Scenario app skia-gold does not work, or does not work as expected

[matanlurey]

STATUS: Known issues are WIP:

• Android scenario app tests with platform views + rotations sometimes display different "gray" color #144657
• Deprecate crop/rotation support for Android plugins, consider Dart alternative #144407
• ExternalTexturesTest still sometimes crashes on rendering the Android image #144365
• Flutter doctor should check using the latest emulator #144232
• [scenario app] golden failure due to single pixel dif #145058

---

While I'm filing this with e: scenario-app it's possible the scope is beyond scenario_app. We are likely "holding it wrong", but it's catastrophic the way it's currently configured - we either turn the tree red because we didn't actually authorize all the digests (example 1), or we straight up miss digests that show incorrect behavior (example 2).

/cc @zanderso @dnfield

---

Example 1: flutter/engine#51061 merged, and turned the tree red.

Only a single image came up in the PR, which was approved.

When it landed post-submit turned the tree red:

Debug information for Gold --------------------------------
stdout: Given image with hash 31dd46a724903b2ba29d64e579a691f1 for test PlatformTextureUiTests_testPlatformViewRotate
Expectation for test: 668bcdeda2f38520c865e639f535b05e (positive)
Expectation for test: 9eb7578aac2db40027790c7f57a2663f (positive)
Expectation for test: 2d6d812504637cb9638d70595733fa0c (positive)
Expectation for test: bbd9733e90b1c0fbde2f7c1689e85f48 (positive)
Expectation for test: fbfc2e60b6b93e5e99659d2674867bb3 (positive)
Expectation for test: af46302575b693eb83c40d2bee719017 (positive)
Expectation for test: 33930e542bfc4c1296f3ad1ad6c49cd2 (positive)
Expectation for test: 3022c9919e2bdef020c73bd1acfd7534 (positive)
Expectation for test: 6bd005faf9dc5939147ef2807c987256 (positive)
Expectation for test: 7d873804f96940f215a7c883d3f8fa4e (positive)
Expectation for test: da554cb3ec3966b1f1bd9c6afb229a45 (positive)
Expectation for test: fd830630715ae54de49c1341e17ad852 (positive)
Expectation for test: 2674732f5e64ea5aafcbd6d7e735bf19 (positive)
Fetching most recent positive digest for trace with ID "331c528375c030eeda1a0d4da269182d".
No recent positive digests for trace with ID "331c528375c030eeda1a0d4da269182d". This probably means that the test was newly added.
Non-exact image comparison using algorithm "fuzzy" against most recent positive digest "".
Untriaged or negative image: https://flutter-engine-gold.skia.org/detail?grouping=name%3DPlatformTextureUiTests_testPlatformViewRotate%26source_type%3Dflutter-engine&digest=31dd46a724903b2ba29d64e579a691f1

stderr: Test: PlatformTextureUiTests_testPlatformViewRotate FAIL

This should ... just never happen?

Example 2: flutter/engine#50730 generated images do not show up.

In flutter/engine#50730, @chinmaygarde fixed/completed a feature for Impeller Vulkan and external textures. Once again, Skia gold did pop up and ask for triage:

... but the images aren't related to the patch. I instead went looking directly in the logs:

... and pulled out the real golden change, which shows a bug not even shown by Skia gold:

[matanlurey]

Trying to debug on CI with flutter/engine#51078.

[chinmaygarde]

Update: As of flutter/engine#50730 (comment), I see the real™ golden changes show up as untriaged digests!

[chinmaygarde]

An instance of change being caught in postsubmit that we know should have been caught in presubmit.

• At 12:42 PM on March 05, we landed a patch with no known untriaged digests in presubmit.
• That patch subsequently broke the tree in postsubmit.
• The failure was in the Android Scenario App Integration Tests.
• This failure showed up as an untriaged digest. The result was accepted as positive at 2:18 PM.
  • Note: The rendering looks incorrect but that is a known issue that is orthogonal to the Gold oddity described here.
• Confusion ensues as to how this escaped presubmit.
• We attempt to pull the images as generated before being submitted to Skia Gold using the instructions here.
• The image pulled is attached below.
• This was same image that was triaged as accepted in post-submit but was not marked as untriaged in presubmit.

[chinmaygarde]

Whoa, additional weirdness. The result that was accepted as positive at 2:18 PM used to show up as a positive (there were 4 images) when searching for all positives by test ExternalTextureTests_testCroppedMediaSurface_bottomLeft. But when I checked it a few mins after writing my comment, the newly positive image is missing from the list. Only three positives are listed, all by @matanlurey more than two weeks ago.

Its as if the new positive was accepted as being a positive (the first link), but not listed in the full list of all positives.

I'm so confused :/

[jonahwilliams]

It looks like the PR flutter/engine#50892 added new golden tests, we triaged them (or @gaaclarke did), autosubmit landed. THen the shard started failing as Skia gold had no images that were even similar: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20Production%20Engine%20Drone/282083/overview

I would have expected that if the test were flakey or something, that we would see diffs - but these had no closest positive/negative at all

[gaaclarke]

Oh interesting, flutter/engine#50892 did add golden tests. Our test for output files verifies that the files were generated. The presubmit check for flutter gold failed though to block the presubmit checks until they were uploaded to gold. Since there was no blockage and no message about new goldens, I didn't triage them as part of landing that PR.

Yet notice that the flutter gold presubmit check is green.

[goderbauer]

This also seems to happen on the framework side: #144978 had some gold failures on the web that were triaged during presubmit. On postsubmit, new failures appeared when running with Impeller.

[matanlurey]

Closing as "completeenough".

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.