Google Testing did not update PR status despite tests passing

[gmackall]

http://flutter-rob/#/flutter/presubmit?pr=125836

cl/528916263

#125836

I manually set the status in order to resolve.

[gmackall]

made another change and it was stuck again, so overrode once more.

[reidbaker]

Happened again despite presubmit passing https://critique.corp.google.com/cl/529103902
#125002

[reidbaker]

Happened again in
https://critique.corp.google.com/cl/530374404/analysis (no g3 files modified)

#126281

[reidbaker]

https://critique.corp.google.com/cl/530386296/analysis

#126274

[reidbaker]

https://flutter-rob.corp.google.com/#/flutter/presubmit?pr=127689
No diff in G3, g3 green. No frob status.

[XilaiZhang]

from triage: how long did it take for the test results to be propagated back to github? A change was landed yesterday to reduce the cron job latency.

[XilaiZhang]

from triage: have we been seeing this recently after the CL that reduces cron job latency has landed?

[gmackall]

Hey, sorry for the late reply. I haven't seen this recently, unsure if @reidbaker has (he is OOO).

Regarding propagation time, I believe it had been 3+ hours after tests passing when I manually overrode the status. Overriding caused the status to update right away.

[gmackall]

#126959 is an example in which the google testing step was stuck on pending for ~3 weeks (see https://discord.com/channels/608014603317936148/608021351567065092/1135789010892095548).

[gmackall]

Also notable in the current google testing for the same PR above, there is no CL linked (where in others I see that there is a column which shows the CL).

[XilaiZhang]

thanks for the info! Looks like its hitting max number of retries, and therefore the roll was deleted. cl/552900321 has just landed. I will re-trigger google testing on this PR.

I re-marked this PR as untriaged, will go over it in the triage meeting tomorrow.

[XilaiZhang]

landed the fix cl cl/553215015 today.

tested in my local citc client and it works well. currently it seems like the roll worker is not picking up this change made to dart/tools/github, and frob's roll worker has a dependency on dart/tools/github .

looking at the history, the fix cl landed 11am, roll job was set to (the version built at 14:11 pm) at 14:33 pm, and roll workers are deployed at 14:40pm. the test run was triggered at 13:31 pm. It is possible that when I triggered the test run, casey was testing roll job at an older version. will do another test run

[gmackall]

Another example stuck for 18 hours in this PR #131893, at commit 298427b9b5c85cdfadd90e7f62db2fa6cfef4999.

[XilaiZhang]

Google testing is migrating to check run and the issues should have been fixed in previous CLs. feel free to reopen if this happens again

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.