[coccon] Refactor builds scheduling subscription with pubsub

[keyonghan]

Copied over from https://github.com/flutter/cocoon/pull/1834/files#r869714524

• Create one message per build, this way we can remove the retry logic from here and avoid duplicated builds as a single passed/failed schedule attempt will be associated with the message
• Ensure that messages are not being duplicated - this requires to update the subscription to ensure a single message is delivered. This will prevent the message queue to grow exponentially when there is an outage and prevent multiple builds for the same commit/builder combination.
• Update the configurations to ensure the messages eventually expire, there will be messages that will fail forever until they expire.

[Hirdey-1999]

i can be assigned to the problem

[keyonghan]

This involves interaction with swarming requests in LUCI, and doesn't sound a good fit for first contribution. Just removed the label.

[keyonghan]

Recent related instance: builder Mac mac_host_engine failed to get scheduled in flutter/engine#36624

From backend log:

2022-10-05 12:30:33.325 PDT
Response does not have schedule build: No response
2022-10-05 12:30:33.325 PDT
Non-zero grpc code: No response
2022-10-05 12:30:33.326 PDT
Failed to schedule builds: (Mac mac_host_engine).

This happened in cocoon: https://github.com/flutter/cocoon/blob/main/app_dart/lib/src/request_handlers/scheduler/request_subscription.dart#L88

Refactoring the builds scheduling from batch to one message per build should solve the issue.

[godofredoc]

@keyonghan is this still needed?

[keyonghan]

This has been obsolete. Closing.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.