Bug 1610616 - Create bugs for expiring probes #181

BenWu · 2020-04-02T19:54:14Z

https://bugzilla.mozilla.org/show_bug.cgi?id=1610616

Here are some example bugs:

with ni?: https://bugzilla-dev.allizom.org/show_bug.cgi?id=1396101
with no accounts: https://bugzilla-dev.allizom.org/show_bug.cgi?id=1396100

This will create a bug in toolkit -> telemetry with the whiteboard probe-expiry-alert and won't close the bug if the probe is no longer expiring

@chutten any additional bug metadata to add that would make triage easier?

fbertsch

This looks good! We just need to ensure that alerts are triaged and heading to the right places.

fbertsch · 2020-04-03T15:50:11Z

probe_scraper/probe_expiry_alert.py

+    }
+
+    found_bugs = search_bugs(version, request_header)
+    print(f"Found {len(found_bugs)} previously created bugs for version {version}")


Why the switch from logging to print? Will these show up in the stackdriver logs?

print will show up in logs since every run right now starts with Unable to parse whitelist ... which is created with print. logging wasn't showing up before

fbertsch · 2020-04-03T15:51:36Z

probe_scraper/probe_expiry_alert.py

-            probes_by_email_by_state[email]["expiring"].append(name)
+def search_bugs(version, request_header):
+    search_query_params = {
+        "summary": BUG_SUMMARY_TEMPLATE.format(version=version, probe=""),


Should we just search for the whiteboard tag? Might make this more resilient.

fbertsch · 2020-04-03T15:55:14Z

probe_scraper/probe_expiry_alert.py

+
+
+def create_bug(probe_name, version, emails, request_header):
+    description_notes = ("No emails associated with the probe have a "


This indicates to me that triaging this list will be more difficult than we intended. I believe many of the emails are not valid bugzilla users.

How about, any email that is not a valid bugzilla email gets an email with a link to the bug. That will hopefully alleviate some of the triaging pain.

fbertsch · 2020-04-03T15:57:09Z

probe_scraper/probe_expiry_alert.py



 if __name__ == "__main__":
    args = parse_args()
-    main(datetime.date.today(), args.dry_run)
+    main(datetime.date.today(), args.dry_run, args.bugzilla_api_key)


It has now occurred to me that we should be using the date sent from Airflow for this check. That way if the Wednesday run fails, we will still file them on Thursday, when that task re-runs.

chutten · 2020-04-03T16:55:22Z

probe_scraper/probe_expiry_alert.py

@@ -41,7 +43,9 @@
 [1] https://wiki.mozilla.org/Release_Management/Calendar


Oooh, could we take the opportunity to instead link to the #telemetry:mozilla.org Matrix channel? Its url is https://chat.mozilla.org/#/room/#telemetry:mozilla.org

chutten · 2020-04-03T16:58:16Z

probe_scraper/probe_expiry_alert.py


-{probes}
+BUG_DESCRIPTION_TEMPLATE = """
+The following Firefox probe will expire in the next major Firefox nightly release: version {version} [1].


Is there going to be one bug per probe? Could we group them for the case where a bunch of devtools probes all expire on the same version? (( the way to group could be by component of the most recent bug in the bug_numbers, by shared prefix (DEVTOOLS_*, FX_*, etc. Scalars and Events have categories to make this easier), or by overlapping notification emails sets ))

This sounds like a great suggestion. We could automate it if the prefix is up to an underscore.

Note: This algorithm is a technical interview question to a tee.

Consider that a gift to you : )

Grouping by component and email set sounds good. Probe names seems a little all over the place so I don't think prefixes will work consistently. For example the last set of expiring probes had:

security.javascriptLoad#javascriptLoad STARTUP_CACHE_REQUESTS WEBEXT_USER_SCRIPT_INJECTION_MS_BY_ADDONID TOUCHBAR_BUTTON_PRESSES QM_REPOSITORIES_INITIALIZATION_TIME_V2 webrtc.sdp.parser_diff startup.profile_selection_reason script.preloader.mainthread_recompile networking.ftp_opened_channels_listings

Oh, we should just be able to group by email, right? What if we create a bug for each email instead of each probe.

That's how email alerts are working now but if there are multiple emails in a set of related probes it would be better to just make one bug and needinfo all the emails.

Okay. In that case let's make a one-off that finds all probes without a valid bugzilla email and we go and require that those teams add a valid bugzilla email. We can perhaps add a check as well that verifies this moving forward. What do you think @ben-wu @chutten?

I checked all the probes active in nightly >= 76 and got this:

Total valid emails: 34 / 66 Probes with valid accounts by type: scalar: 49 / 97 event: 0 / 22 histogram: 78 / 124

About half the probes have valid bugzilla accounts. I'm worried that it would be too prohibitive to require a valid account. Most of the invalid accounts are either email lists or email aliases. I think that sending an email with the bug link to emails that don't have a bugzilla account is sufficient and definitely better than what we currently have. So we're sending an email and creating a bug that will hopefully be triaged. I'm sure @chutten will have more thoughts on this though

Prefixes don't have to work consistently, it'll just add a little pressure for people to organize their histogram names : )

I think creating a bug

per category/prefix (trying to keep the number of bugs filed down)

in the component of the bug_number with the highest id (default to Firefox::General)

with needinfos for the accounts there are

and emails with links to the ones there aren't

is the optimal play for now. At the very least an unprioritized bug in a vaguely-relevant component will get the triage owners to triage it.

chutten · 2020-04-03T16:59:44Z

probe_scraper/probe_expiry_alert.py

+
+    create_params = {
+        "product": "Toolkit",
+        "component": "Telemetry",


We should file a bug in the component that the most recent bug from the bug_numbers list was filed in, or Firefox::General. Toolkit::Telemetry should not contain bugs about probes unless the probes are measuring the Telemetry subsystem itself.

probe_scraper/probe_expiry_alert.py

wlach

Can we somehow include these bugs in the probe expiry emails that I presume will still be sent? I worry that people will still assume they need to file bugs if they see their probes there.

A further improvement which might be worth considering at some point would be to file a metabug per release on which all these bugs would depend, that might be handy for release managers or others to look at.

wlach · 2020-04-08T13:21:53Z

probe_scraper/probe_expiry_alert.py

 BASE_URI = "https://hg.mozilla.org/mozilla-central/raw-file/tip/toolkit/components/telemetry/"
 HISTOGRAMS_FILE = "Histograms.json"
 SCALARS_FILE = "Scalars.yaml"
 EVENTS_FILE = "Events.yaml"

-EMAIL_BODY_FORMAT_STRING = """
-The following Firefox probes have either expired or will expire in the next major Firefox nightly release: version {next_version} [1].
+BUG_WHITEBOARD_TAG = "probe-expiry-alert"


Suggested change

BUG_WHITEBOARD_TAG = "probe-expiry-alert"

BUG_WHITEBOARD_TAG = "[probe-expiry-alert]"

Drive by: I think we generally use square brackets around whiteboard tags which makes them more unique (e.g. probe-expiry-alert would also catch probe-expiry-alerts-on-sundays)

BenWu · 2020-04-08T14:11:51Z

Each email will include a link to the bugs for the probes in the email. But emails will only be sent to emails that aren't associated with a bugzilla account, otherwise it will needinfo the account instead of sending an email.

wlach · 2020-04-08T14:50:55Z

Each email will include a link to the bugs for the probes in the email. But emails will only be sent to emails that aren't associated with a bugzilla account, otherwise it will needinfo the account instead of sending an email.

I think that makes sense, hopefully the whiteboard tag will help us track unaddressed probe expiry over time.

BenWu · 2020-04-09T00:37:11Z

This took longer than expected; I rewrote almost the entire file.

Here are some examples: https://bugzilla-dev.allizom.org/buglist.cgi?status_whiteboard=[probe-expiry-alert]

So now single bug is created for multiple probes that are in the same component and have the same set of emails. The name of bug does is a best effort longest prefix among the probe names which if you look at the examples in the link may result in bugs with the same summary which is not great. A fix for this would be to just use the first probe name or to not group by emails (one bug per component).

Also emails are sent to emails that do not have a bugzilla account but only when bugs are filed, so once per release cycle.

chutten

I like the examples and the behaviour you described. I think this'll help keep the bug volumes down to a reasonable and actionable level.

probe_scraper/probe_expiry_alert.py

fbertsch

This is great. Feels like magic that it ends up in the right component.

fbertsch · 2020-04-29T15:42:27Z

probe_scraper/probe_expiry_alert.py


 What to do about this:
 1. If one, some, or all of the metrics are no longer needed, please remove them from their definitions files (Histograms.json, Scalars.yaml, Events.yaml).
 2. If one, some, or all of the metrics are still required, please submit a Data Collection Review [2] and patch to extend their expiry.

-If you have any problems, please ask for help on the #fx-metrics Slack channel. We'll give you a hand.
+If you have any problems, please ask for help on the #fx-metrics Slack channel or the #telemetry Matrix room at https://chat.mozilla.org/#/room/#telemetry:mozilla.org. We'll give you a hand.


Looking it over, let's also mention that it's easier to extend collection now, with the shorter form.

fbertsch · 2020-04-29T15:51:28Z

probe_scraper/probe_expiry_alert.py

+        values = [v.ljust(longest_value_length) for v in values]
+
+    prefix_length = 0
+    for c in zip(*values):


BenWu requested a review from fbertsch April 2, 2020 19:54

fbertsch reviewed Apr 3, 2020

View reviewed changes

chutten requested changes Apr 3, 2020

View reviewed changes

wlach reviewed Apr 8, 2020

View reviewed changes

BenWu requested review from chutten and fbertsch April 9, 2020 00:37

chutten approved these changes Apr 13, 2020

View reviewed changes

probe_scraper/probe_expiry_alert.py Show resolved Hide resolved

BenWu added 14 commits April 29, 2020 11:36

Create bugs for expiring probes instead of email

19ae22a

Create bugs for expiring probes instead of email

b36692f

oof

64d36d1

Fix update bug, description

958c582

Remove emailer, remove auto resolve bug

d31dd65

Refactor for tests

1480c9a

Test bug creating

3d828bc

Switch to prod bugzilla url

27006be

' -> "

9627821

wip Group by components and emails, reorganize code

bf36cb2

Create bugs for grouped probes

9ae9c77

Put emailer back in for emails with no accounts

a85f2d2

TESTS

557f91d

regex

9db1619

BenWu force-pushed the benwu/bugs-4-expiring-probes branch from 0aa3d64 to 9db1619 Compare April 29, 2020 15:36

fbertsch approved these changes Apr 29, 2020

View reviewed changes

Add data collection renewal short form info

a26175d

BenWu mentioned this pull request Apr 29, 2020

Pass execution date and bugzilla key to probe expiry alert mozilla/telemetry-airflow#961

Merged

BenWu merged commit c84efa6 into master Apr 29, 2020

BenWu deleted the benwu/bugs-4-expiring-probes branch April 29, 2020 16:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug 1610616 - Create bugs for expiring probes #181

Bug 1610616 - Create bugs for expiring probes #181

BenWu commented Apr 2, 2020

fbertsch left a comment

fbertsch Apr 3, 2020

BenWu Apr 3, 2020

fbertsch Apr 3, 2020

fbertsch Apr 3, 2020

fbertsch Apr 3, 2020

chutten Apr 3, 2020

chutten Apr 3, 2020

fbertsch Apr 3, 2020

fbertsch Apr 3, 2020

chutten Apr 3, 2020

BenWu Apr 3, 2020

fbertsch Apr 3, 2020

BenWu Apr 3, 2020

fbertsch Apr 3, 2020

BenWu Apr 6, 2020

chutten Apr 6, 2020

chutten Apr 3, 2020

wlach left a comment

wlach Apr 8, 2020 •

edited

BenWu commented Apr 8, 2020

wlach commented Apr 8, 2020

BenWu commented Apr 9, 2020

chutten left a comment

fbertsch left a comment

fbertsch Apr 29, 2020

fbertsch Apr 29, 2020



		def create_bug(probe_name, version, emails, request_header):
		description_notes = ("No emails associated with the probe have a "

		@@ -41,7 +43,9 @@
		[1] https://wiki.mozilla.org/Release_Management/Calendar

	BUG_WHITEBOARD_TAG = "probe-expiry-alert"
	BUG_WHITEBOARD_TAG = "[probe-expiry-alert]"

Bug 1610616 - Create bugs for expiring probes #181

Bug 1610616 - Create bugs for expiring probes #181

Conversation

BenWu commented Apr 2, 2020

fbertsch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wlach left a comment

Choose a reason for hiding this comment

wlach Apr 8, 2020 • edited

Choose a reason for hiding this comment

BenWu commented Apr 8, 2020

wlach commented Apr 8, 2020

BenWu commented Apr 9, 2020

chutten left a comment

Choose a reason for hiding this comment

fbertsch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wlach Apr 8, 2020 •

edited