Post submit tasks show In Process though they have finished

[keyonghan]

This happens in different repositories, and is blocking tasks backfilling. Even block trees if any TOT task failed earlier.

Engine:

Framework:

Taking look at one build:
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20Framework%20Smoke%20Tests/14639/overview

We do see three processing entries, with the last one shows correct status update:
Updated datastore from In Progress to Succeeded

Sounds some issue from Datastore side.

[keyonghan]

Recovered Engine tree by manually marking the In Progress as Succeeded (validated that all In Progress tasks have finished with Success status.
Recovered Framework Tree by manually marking the In Progress as New, so the back filler can rerun them.

[keyonghan]

Packages repo are hanging waiting for tasks to be scheduled.

Seems we have a big backlog of unacked messages in pub/sub:

Potential culprit: #135508 (comment)

Reducing the Subscription message retention duration from 4 days to 12 hr.

[vashworth]

I have a framework PR that merged in ~ an hour ago but isn't showing up on the dashboard at all.
#135444

Not sure if it's related or not

[keyonghan]

Immediate issue has been recovered by staling outdated pub/sub messages.
Hanging tasks have been recovered.
The missing commit issue should be resolved by flutter/cocoon#3106.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.