500 error on probe-scraper actions

[perrymcmanis144]

I opened a PR against glean and noticed that I had a probe-scraper 500 error. I tried repushing a few minutes later and it still persisted. It appears that some other folks might be experiencing it as well: https://mozilla.slack.com/archives/CEE12R4E8/p1710431615148949?thread_ts=1707921362.947109&cid=CEE12R4E8

Log from the PR: https://github.com/mozilla/glean/actions/runs/8283575535/job/22667074476

[travis79]

Another failure in another repo: https://github.com/mozilla/experimenter/actions/runs/8283756036/job/22668600610?pr=10409

[travis79]

Another: https://github.com/mozilla/blurts-server/actions/runs/8283671119/job/22667400123

[BenWu]

Stack trace from cloud function:

 Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/functions_framework/__init__.py", line 99, in view_func
    return function(request._get_current_object())
  File "/workspace/probe_scraper/glean_push.py", line 47, in main
    updated_paths = runner.main(
  File "/workspace/probe_scraper/runner.py", line 708, in main
    upload_paths += load_glean_metrics(
  File "/workspace/probe_scraper/runner.py", line 609, in load_glean_metrics
    raise ValueError("Found duplicate Glean metrics, check email for details")
ValueError: Found duplicate Glean metrics, check email for details 

[BenWu]

This is what I found in the logs:

Glean has detected duplicated metric identifiers coming from the product 'mozillavpn-backend-cirrus'.
- 'cirrus_events.enrollment' defined more than once in mozillavpn-backend-cirrus, nimbus-cirrus
- 'cirrus_events.enrollment_status' defined more than once in mozillavpn-backend-cirrus, nimbus-cirrus
- 'cirrus_events.instance_name' defined more than once in mozillavpn-backend-cirrus, nimbus-cirrus

I think an email should have been sent too emails don't get sent for the cloud function

[travis79]

I think this may be related: #708

[travis79]

And yes, emails were sent out. I got one along with several nimbus folks. I've actually gotten several emails now, stating the same thing.

[BenWu]

I think the issue is that metrics and pings for mozillavpn-backend-cirrus is still "cached" in the gcs bucket even though #708 removed them. I can create a backup and then try deleting it and it should allow things to run again. Certainly a bug somewhere here but I'll just get things working again

[BenWu]

This should now be fixed after whd deleted the metrics file. I verified that runs are succeeding now.
Some notes:

• The need for some type of monitoring on the cloud function is highlighted here Monitor google cloud function and/or github action #575
• Probe scraper should delete metrics and pings when the files are removed from repositories. Although this is tricky since metrics and pings are only overwritten when a push is triggered from that repo. I'll file an issue to at least document this