pkg/dartdev/test/commands/compile_test is failing on cross-compilation

[alexmarkov]

There are new test failures on Fix some EOL bugs + remove useLineEndingsForPlatform=false from refactor tests.

The tests

pkg/dartdev/test/commands/compile_test RuntimeError (expected Pass)

are failing on configurations

unittest-asserts-release-linux-arm64

Log:

Checking the latest available revision...
Downloading https://storage.googleapis.com/dart-archive/channels/main/raw/latest/VERSION...
Using revision fd33d9eae8ddc741a58d1dee2fe91cb2e63bdcea.
Downloading https://storage.googleapis.com/dart-archive/channels/main/raw/hash/fd33d9eae8ddc741a58d1dee2fe91cb2e63bdcea/sdk/gen_snapshot_linux_arm64_linux_arm...
Failed to download https://storage.googleapis.com/dart-archive/channels/main/raw/hash/fd33d9eae8ddc741a58d1dee2fe91cb2e63bdcea/sdk/gen_snapshot_linux_arm64_linux_arm to /home/chrome-bot/.dart/dartdev/sdk_cache/3.10.0-edge.fd33d9eae8ddc741a58d1dee2fe91cb2e63bdcea/gen_snapshot_linux_arm64_linux_arm.
If the problem persists, try downloading it manually.
SdkCacheException: GET https://storage.googleapis.com/dart-archive/channels/main/raw/hash/fd33d9eae8ddc741a58d1dee2fe91cb2e63bdcea/sdk/gen_snapshot_linux_arm64_linux_arm failed: 404
#0      SdkCache._download (package:dartdev/src/sdk_cache.dart:248:7)
<asynchronous suspension>
#1      SdkCache.ensureArtifact (package:dartdev/src/sdk_cache.dart:207:34)
<asynchronous suspension>
#2      CompileNativeCommand.run (package:dartdev/src/commands/compile.dart:619:29)
<asynchronous suspension>
#3      CommandRunner.runCommand (package:args/command_runner.dart:212:13)
<asynchronous suspension>
#4      DartdevRunner.runCommand (package:dartdev/dartdev.dart:255:18)
<asynchronous suspension>
#5      runDartdev (package:dartdev/dartdev.dart:48:16)
<asynchronous suspension>
#6      main (file:///b/s/w/ir/cache/builder/sdk/pkg/dartdev/bin/dartdev.dart:13:5)
<asynchronous suspension>

full log

Looks unrelated to the commit where it started failing.
Probably related to how we cache Dart SDK artifacts for cross-compilation.
/cc @sstrickl @iinozemtsev

[alexmarkov]

Also failing on other pkg-* bots. The failing tests were recently added in 76ca835.

[iinozemtsev]

Yes, it is not related to the change and related to the test. I think it worth approving the failure for now.

As I've expected, fetching build artifacts from latest in async ci environment is fragile: the latest archive bucket might have missing artifacts, when postsubmit bots are still running.

The test should use locally built artifacts if the goal is to test cross compilation itself, or, if the goal is to test fetching, use some known hash for which we have artifacts, e.g.:

1. allow overriding hash for cross compilation via hidden command line flag / env var / whatever
2. lookup https://dart.googlesource.com/sdk/+/refs/heads/lkgr hash in a test somehow

I'm not sure how easy it is to do the lookup above, and anyway it might be fragile.

Ideally it shouldn't be a pkg test, but instead a special lkgr-triggered bot with a custom recipe, which would download an sdk, cross-compile a binary, trigger a shard on a target arch bot to launch compiled binary and make sure the shard succeeds.

[sstrickl]

Yeah, testing against the downloaded version helped me realize the recent error with the recipes on Windows, but it's not what I really want for the tests either (as they're fragile as mentioned).

Providing a local path for the cross compiling gen_snapshot and the target's dartaotruntime is something we should add anyway for cases where, say, downloading artifacts during build is disallowed. That would fix half the issue in these tests, in that we could provide the path to the locally built cross compiling gen_snapshot, but the other half of the equation, the dartaotruntime built for the target, wouldn't be available...

... unless we copied those over specifically as part of the test environment setup, assuming that the tests only start running after all archs are built, and not only the arch specific to the trybot in question. (I haven't looked into it so I don't know if this is the case.)

Whether or not that can happen, we should definitely add an lkgr-triggered bot that gets the built SDK for both host and target to artifacts from both to do the cross compilation and then sends the result to a bot for the target configuration for running (so again, not depending on the ability to download artifacts), so the result's ability to be successfully executed is checked (and maybe even compare it to a non-cross-compiled version of the same program to see if there are differences).

But it would be nice to have tests that can reliably test the downloadability of the artifacts also, since that's how I caught the recipes error earlier this week :/

[sstrickl]

As a stop-gap for now, I might split out the cross-compilation tests from the compilation tests for pkg/dartdev, just so a failure in the former doesn't mean we miss if there starts being failures in the latter as well.