Several jobs not making progress or being scheduled?

[zanderso]

[ricardoamador]

It appears that the builds in yellow actually did complete but the dashboard was not able to update. I found quite a few errors in the scheduler where we were not able to complete what appears to be the update transaction due to database transaction aborted.

[ricardoamador]

It looks like the transaction here is not running: https://github.com/flutter/cocoon/blob/main/app_dart/lib/src/request_handlers/postsubmit_luci_subscription.dart#L83

[ricardoamador]

Okay this is odd, 71 of the 72 errors are for the same commit in the postsubmit subscription, for userData={commit_key: flutter/flutter/master/d7ed5dc2e212d164fd8733f34a832af10481cd8b, task_key: 6091431327170560}

So this does not explain why the other tasks were hung. There might be more than one issue here. @keyonghan do you think there might be an issue with the scheduler?

[ricardoamador]

Okay there is another issue with warnings where the result of the task is null and attempting to update the datastore is throwing a 400 error:

[keyonghan]

It appears that the builds in yellow actually did complete but the dashboard was not able to update. I found quite a few errors in the scheduler where we were not able to complete what appears to be the update transaction due to database transaction aborted.

We have been hitting Failed to process Instance of 'PushMessage'. (500) The transaction was aborted. frequently in our backend. Wondering if any regression from Datastore.

[ricardoamador]

Yeah it appears that this is happening going back a ways. We are getting un'acked messages which is explainable by the error above. Let me see if I can find out if there are any changes to Datastore recently.