Update circleci docker image #2041
Conversation
|
It looks like the arm64 Docker container does not build anymore. Some issue in the qemu configure script (logs are sparse). |
rjoleary
force-pushed
the
circleci
branch
2 times, most recently
from
90b2288
to
2176c3a
Compare
rjoleary
added
Awaiting reviewer
rminnich
added
Awaiting author
rjoleary
force-pushed
the
circleci
branch
2 times, most recently
from
4f8b9c1
to
363eb8b
Compare
Maybe we should switch to SATA. |
|
@hugelgupf I think this might be a bug in how we're parsing the output of See: Which seems fine. And then further down: Why only arm64? Is the output of |
Hmmmm, this log output isn't making much sense. There should be a line after this like "--- PASS: TestOpenSignedFile (0.02s)" |
[From the peanut gallery] FWIW, the output of "go test" is not stable from release to release; the Go team does periodically make changes to the output format to improve clarity and readability. In this CL you are moving from Go 1.13 to Go 1.15; in that range there were 27 commits to the Go testing package, including https://go-review.googlesource.com/c/go/+/127120 all of which change test output in some way. From the test log it looks as though maybe you are parsing the output of "go test -test.v" ? |
With that said, I agree that it does seem as though the "Pass: TestOpenSignedFile" is missing. Don't have a good theory there. |
|
I reran the circleci and this seems to be consistent. |
|
I ran the qemu command by hand and here's an excerpt: Still there is no sign of the "--- PASS" for TestOpenSignedFile... |
|
If I only run the tests in the vfile package, it works: |
|
I was able to reproduce the error with testing only two packages in the gotest vm test: It is flaky and fails 9/10 times. When it passes, the logs look normal as expected. On the other hand: Passes 10/10 times. |
|
For the record, it also passes 10/10 times when run on the x86 host outside of qemu: |
|
I improved the logging for the gotest vmtest. Previously, the exit status was ignored and the context was cancelled without enough time to get the stack trace out. When it fails, it is timing out. I tried increasing the timeout to 40s to no avail: It seems to be stuck doing some math crypto stuff. |
|
Oh no, amd64 is failing now too! |
|
Perhaps this has something to do with virtio-rng? |
@thanm we use Go's very own cmd/test2json to parse the output of go test -test.v. The tests are compiled as binaries, shared with the VM, and run in the VM basically as ./testbinary -test.v | test2json -t -p as seen here: I'm checking out the failure. |
|
Running locally with Go 1.15 and my pre-installed QEMU passes. Checking out qemu 6.0.0 now. |
|
I used:
I ran
and I can't repro the issue locally yet. I'll switch to sshing into circleci and checking that out. Maybe it just runs out of memory or nearly runs out of memory and spends a lot of time swapping around. |
|
Something peculiar about running the test locally, however, is that it takes 8 seconds to run, and that time is not attributed to any individual test: On circleci, a timeout after 25 seconds is the issue. |
|
I think it's quite possible that QEMU emulation is just 3x slower on circleci, and that causes the timeout to happen. However, even on Go 1.13, the test takes 8-12 seconds to run. (But let me make sure that it actually uses Go 1.13 to compile the tests -- checking that out.) |
Of course it doesn't, that part is here: Line 63 in dd0caf3 I'll fix that to look for the compiler in GOROOT/bin/go. |
Codecov Report
@@ Coverage Diff @@
## master #2041 +/- ##
==========================================
+ Coverage 54.70% 54.72% +0.01%
==========================================
Files 570 570
Lines 49704 49695 -9
==========================================
+ Hits 27191 27196 +5
+ Misses 21420 21406 -14
Partials 1093 1093
Continue to review full report at Codecov.
|
|
Posted #2076 for that, but it actually didn't change anything. Go 1.15.14 (+ qemu-v6 + kernel-v4.17 + no KVM) with only pkg/{vfile,uzip}: Go 1.13 (+ all the same): If anything, the runtime got better... |
|
Ron wrote us a If it's mostly user time, then it's probably just slow. If it's most sys time, then it's probably blocked on something. |
|
Looks like it's just slow: |
|
I increased the timeout to 100s and ran it a few times: I wonder why is fluctuates so much? Sometimes it is less than 25s, sometimes 10s of seconds more. |
|
Any ideas besides these two options?
|
The big change is the switch to Go 1.15 and QEMU 6.0.0. This change fixes issues arising from this migration. This change also improves the logging of the gotest vmtest to help debug issues with this migration. Signed-off-by: Ryan O'Leary <ryanoleary@google.com>
I went with the first option. |
rjoleary
added
Awaiting reviewer
rminnich
added
automerge
The big change is the switch to Go 1.15 and QEMU 6.0.0.
Signed-off-by: Ryan O'Leary ryanoleary@google.com