ToT is red due to a row of infra failures

[keyonghan]

Lots of tests failed due to infra failure when updating task status:

Example build: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux_android%20devtools_profile_start_test/2056/overview

AuthenticatedClientError:
  URI: https://flutter-dashboard.appspot.com/api/update-task-status
  HTTP Status: 500
  Response body:
Key not found: Commit:flutter/flutter/master/1c2c9421121f345392162e3dde0a784d0cbdb69b
#0      DatastoreDB.lookupValue (package:gcloud/src/db/db.dart:381:9)
<asynchronous suspension>
#1      UpdateTaskStatus._constructCommitKey (package:cocoon_service/src/request_handlers/update_task_status.dart:115:27)
<asynchronous suspension>
#2      UpdateTaskStatus._getTaskFromNamedParams (package:cocoon_service/src/request_handlers/update_task_status.dart:77:35)
<asynchronous suspension>
#3      UpdateTaskStatus.post (package:cocoon_service/src/request_handlers/update_task_status.dart:58:23)
<asynchronous suspension>
#4      RequestHandler.service.<anonymous closure> (package:cocoon_service/src/request_handling/request_handler.dart:52:22)
<asynchronous suspension>
#5      ApiRequestHandler.service.<anonymous closure> (package:cocoon_service/src/request_handling/api_request_handler.dart:151:7)
<asynchronous suspension>
#6      ApiRequestHandler.service (package:cocoon_service/src/request_handling/api_request_handler.dart:150:5)
<asynchronous suspension>
#7      main.<anonymous closure>.<anonymous closure> (file:///app/bin/server.dart:267:9)
<asynchronous suspension>

[keyonghan]

/cc @godofredoc @CaseyHillers This is the failure I mentioned yesterday when updating task status.

[godofredoc]

What is missing to move the test results processing to pub/sub?

[CaseyHillers]

@yusuf-goog was starting something in flutter/cocoon#1574. Once a Cocoon handler has been added, we can update the devicelab test runner to push to pub sub instead of making http requests

[godofredoc]

Sounds like the problem here is that something is creating an entry in the data models that is out of sync with the handlers requests. We need to fix that race condition or avoid depending in the data model entry to exist.

[CaseyHillers]

To recap, this is what happened:

1. flutter/flutter is on the LUCI scheduler, which wasn't impacted by the GitHub outage today
2. Builds triggered and finished before Cocoon received webhook notifications to create the Task entities
3. When the luci builds sent /api/update-task-status they reached the 500 error

This is a duplicate of #78876, which we will mark as being fixed once moved to the cocoon scheduler.

[godofredoc]

After fixing the recipes the test has been passing consistently.

[CaseyHillers]

Any AIs for this being reopened? Since this was caused by the GitHub webhook outage, this will be obsolete once #92300 is landed (which moves the scheduler to be dependent on GitHub webhooks)

[godofredoc]

It was reopened because we were missing the context of what actually fixed it. Thanks for updating!

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.