Flutter Gold is not receiving image uploads

[Piinks]

@kjlubick has reported that the Gold dashboard has stopped uploading data.

For example, this test has not seen new uploads in 1 week, since ce8e2bb
--> landed 11/20

A new version of goldctl was rolled to CIPD at the same time: https://chromium-review.googlesource.com/c/infra/infra/+/1924621
--> rolled 11/20
Which included changes to goldctl from https://skia-review.googlesource.com/c/buildbot/+/253698

Other possibly related changes:

• Another goldctl change & roll to cipd (After the one mentioned above)
  • https://skia-review.googlesource.com/c/buildbot/+/256018
  • https://chromium-review.googlesource.com/c/infra/infra/+/1933027
    • --> rolled 11/26
• Open PR to enable tryjobs Pre-Submit Tryjobs for Flutter Gold #44474
  • Noticed yesterday that the golden file changes from Re-Staged: Fixing ContinuousRectangleBorder bottom radius #45304 did not appear for triage
  • They instead showed up in the tryjob results of Pre-Submit Tryjobs for Flutter Gold #44474
  • Was disregarded at the time as the tryjob picking up the change before the other change was properly triaged, may not be the case.
• Open PR to test using Gold for Flutter Web Implement screenshot test for flutter web. #45530
  • does not use tryjobs

[Piinks]

cc/ @goderbauer FYI. The current pre-submit checks do not use goldctl, so changes to golden files won't slip by during this outage.

@kjlubick I will see about getting debug information from image uploads to help identify what is going on.

[Piinks]

https://bugs.chromium.org/p/skia/issues/detail?id=9679 has been filed to set up an alert system to detect this better in the future.

[kjlubick]

I'll check if it's goldctl's bug or not - chrome-gpu is still operating fine, but I don't know if they auto-roll

[kjlubick]

I think I identified the bug - goldctl was passing itself something invalid during imgtest init (or possibly imgtest add). Fix incoming.

As with any good cascading failure, the flutter tests didn't realize the exit code was non-zero, so that would be a helpful follow-up investigation.

[Piinks]

Thanks! 🙂
Our post-submit tests don't throw for a non-zero exit code right now since landing a change or a new test returns a non-zero exit code from the imgtest. Is that right? We use passfail mode, so using the exit code in that way would turn the tree red for every change. Of course, once tryjobs are up and running we'll be able to triage ahead of time so I think that distinguishing between what kind of non-zero exit code it is won't be an issue.

[kjlubick]

Do you run imgtest init before you run your tests or just imgtest add? If imgtest init gives you a non-zero error code, make lots of noise because that means goldctl "herped it's derp" on the inputs.

[Piinks]

Ah! We do. I will add a check for the exit code there in init.

[kjlubick]

goldctl fix is live in CIPD; alert is going live on Gold's side to catch things like this a lot quicker

[Piinks]

Verified this is resolved. :)

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.