Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: LUCI support for plan9 #62025

Open
heschi opened this issue Aug 14, 2023 · 29 comments
Open

x/build: LUCI support for plan9 #62025

heschi opened this issue Aug 14, 2023 · 29 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Plan9
Milestone

Comments

@heschi
Copy link
Contributor

heschi commented Aug 14, 2023

In #61671 (comment), @millerresearch reported that the plan9 builders don't support python3. The LUCI swarming bot requires python3, which means that we aren't going to be able to test plan9 once we finish our migration. We can probably keep the old infrastructure going through the 1.22 cycle, but after that we'll likely have to declare the ports as broken in 1.23.

I have no idea how much work it would be to port python3 to plan9. I imagine that if it were easy to do it'd have been done already...? The LUCI team has long-term plans to port the bot to Go but it won't happen on our timeline. (For the record, even with Python supported, the LUCI project has a number of other programs that we'd really like to have working. Fortunately, those are already written in Go and porting them should be pretty easy. Worst case we may be able to work around them.)

Alternatively, maybe it's possible to run it virtualized somehow, say with the bot running under linux, then booting a qemu VM to do its work? The plan9 builders are already kinda slow, though.

cc @golang/plan9 @golang/release

@heschi heschi added OS-Plan9 Builders x/build issues (builders, bots, dashboards) labels Aug 14, 2023
@gopherbot gopherbot added this to the Unreleased milestone Aug 14, 2023
@heschi heschi added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 14, 2023
@millerresearch
Copy link
Contributor

According to the person who did the python2.7 Plan 9 port, python3 has more dependencies on GCC or Clang (both unavailable on Plan 9), so getting that working would not be trivial.

As for virtualisation, I can only speak for plan9-arm, maybe @0intro can comment about other platforms.

The plan9-arm builders are real hardware (currently a cluster of Raspbery Pi 4 boards); to my knowledge Plan 9 hasn't been ported to qemu for ARM. In any case, I don't think testing Go in a virtual machine is the same as testing on real hardware. The most interesting bugs are usually the ones that depend on timing or caches or other nuances which are likely to be simplified away by emulation.

Do the tasks run by the LUCI bot need to interact with each other (apart from through the shared file system)? If not, it should be feasible to add a linux machine to the plan9-arn cluster to run LUCI bots, which would do a remote execution of tasks on a Plan 9 machine (sharing the bot's file sysem) in place of local execution as a subprocess.

How is testing going to be done for things like android and iOS, which presumably aren't able to run a python3 bot?

@bcmills
Copy link
Contributor

bcmills commented Aug 15, 2023

How is testing going to be done for things like android and iOS, which presumably aren't able to run a python3 bot?

I would expect that the android OS can run a python3 executable directly, although I would also expect that the LUCI bot will run on a Linux host machine and send commands to an emulator to run the tests. (That more closely mimics what a typical Android developer would experience using Go anyway.)

The ios builder currently uses a virtualized environment provided by Corellium, which allows it to exec programs that would normally be disallowed by iOS, presumably including python3.

Adding a Linux machine that sends remote commands to a Plan9 instance sounds like a reasonable architecture to me.

@heschi
Copy link
Contributor Author

heschi commented Aug 15, 2023

Yeah, that sounds viable if you're willing to maintain the extra hop. We'll have to add some kind of interceptor logic to our build script, but that doesn't seem prohibitive.

@0intro
Copy link
Member

0intro commented Oct 17, 2023

I think the easiest way would indeed be to execute the commands in a remote Plan 9 instance, running in either a virtual machine (386 and amd64) or on real hardware (arm).

@heschi Could you indicate where the build scripts are located? I'd like to take a look and see how this could be achieved.

I think the idea would be to execute the commands remotely (think ssh glenda@plan9 ls /), then retrieve the output.

@millerresearch
Copy link
Contributor

Gentle reminder -- we'd like to proceed with moving Plan 9 builders to LUCI, but we're somewhat in the dark. I've found https://chromium.googlesource.com/infra/luci/luci-py -- is this the relevant source or does golang have its own fork? The documentation in that repository refers to "device bots", which is I think what we need, because the Plan 9 tests will run on devices or qemu instances separate from the (linux) platform where the swarming bot runs. It would be helpful to see documentation on setting up a device bot, and -- as @0intro asked a few months ago -- to see examples of how existing build scripts work. I'm guessing android will be the most similar. Also in #63599 a problem launching the swarming bot was reported; we are stalled waiting for a response to that too.

@dmitshur
Copy link
Contributor

dmitshur commented May 17, 2024

Thank you for working on this and your patience. I'd like to try to help move this forward. Apologies for not commenting here sooner.

We don't quite have an ideal existing builder that we can point to as the canonical example of how a builder like this can be implemented. Our work in progress on the Android emulator builder (#61097) and iOS Simulator (#66360) is the closest, in that both of those systems have distinct host and target OSes. An advantage they have is the host OS runs the emulator/simulator locally, whereas here it's likely networking to a Plan 9 system will be needed.

For both iOS and Android builders, we're relying the the approach of having a go_{goos}_{goarch}_exec wrapper in $PATH and setting GOOS + GOARCH to the target OS/arch as well as GOHOSTOS + GOHOSTARCH to the host OS/arch in the environment. (See "If the -exec flag is not given, ..." at https://pkg.go.dev/cmd/go#hdr-Compile_and_run_Go_program.) Provided the go_…_exec wrapper is doing its job as intended, at that point there are no further modifications needed to make a builder work. The builder runs the equivalent of make.bash followed by go tool dist test -json (or go test -json for testing golang.org/x repos) and ends up testing the target OS/arch.

In the case of mobile builders, their go_…_exec scripts invoke appropriate emulators/simulators, but in this case your script would be either SSHing into a Plan 9 machine—or something equivalent like Plan 9 in qemu—in order to execute GOOS=plan9 test binaries. The scripts may also need to copy testdata that tests need. I recently saw @FiloSottile using this approach.

To make progress on all this, I suggest splitting the work into two fairly orthogonal halves:

  1. Get the host machine connected and running. It won't be testing the Plan 9 port yet, but it will make sure everything needed for a functional LUCI builder is in place.
  2. Create a go_plan9_{goarch}_exec script that executes Plan 9 binaries by any means that are convenient. (E.g., such that GOOS=plan9 GOARCH=amd64 go test -short bytes can work in your local testing.)

Once those two pieces are working independently, there's a clear path forward to connect them together to get a LUCI builder that provides initial signal on Plan 9 test execution.

For 1, I think we can track the next steps in any of issues #63599, #63600, or #63601.
For 2, maybe this issue is good to use to track progress.

Thanks again.

@dmitshur dmitshur moved this to In Progress in Go Release May 17, 2024
@0intro
Copy link
Member

0intro commented May 17, 2024

Thank you for the instructions.

The go_$GOOS_$GOARCH_exec script is an interesting approach. The difference, from what I had in mind initially, is that the tests will be cross compiled on Linux, then executed on Plan 9, instead of being built on Plan 9 directly.

I think we should be good for 1. See #63599. The other architectures will follow.

For 2, I've started to experiment the go_$GOOS_$GOARCH_exec script approach, and it looks promising so far.


##### Test execution environment.
$ GOOS=plan9 GOARCH=386 go tool dist test

ssh -p 2222 glenda@localhost /tmp/main
# GOARCH: 386
# CPU: QEMU Virtual CPU version 2.5+
# GOOS: plan9
# OS Version: 2000

##### Testing packages.
ok  	archive/tar	0.297s
ok  	archive/zip	0.250s
ok  	bufio	0.174s
ok  	bytes	0.272s
ok  	cmp	0.145s
ok  	compress/bzip2	0.179s
ssh -p 2222 glenda@localhost /tmp/flate.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestWriterPersistentWriteError (0.00s)
    deflate_test.go:724: ReadFile: open ../../testdata/Isaac.Newton-Opticks.txt: '../../testdata' file does not exist
FAIL
FAIL	compress/flate	0.549s
ok  	compress/gzip	1.068s
ssh -p 2222 glenda@localhost /tmp/lzw.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestWriter (0.00s)
    writer_test.go:30: ../testdata/gettysburg.txt (order=0 litWidth=7): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:30: ../testdata/gettysburg.txt (order=0 litWidth=8): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:30: ../testdata/gettysburg.txt (order=1 litWidth=7): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:30: ../testdata/gettysburg.txt (order=1 litWidth=8): open ../testdata/gettysburg.txt: '../testdata' file does not exist
FAIL
FAIL	compress/lzw	0.139s
ssh -p 2222 glenda@localhost /tmp/zlib.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestWriterBig (0.00s)
    writer_test.go:32: ../testdata/gettysburg.txt (level=-1, dict=""): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:32: ../testdata/gettysburg.txt (level=0, dict=""): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:32: ../testdata/gettysburg.txt (level=-2, dict=""): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:32: ../testdata/gettysburg.txt (level=1, dict=""): open ../testdata/gettysburg.txt: '../testdata' file does not exist
--- FAIL: TestWriterDict (0.00s)
    writer_test.go:32: ../testdata/gettysburg.txt (level=-1, dict="0123456789."): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:32: ../testdata/gettysburg.txt (level=0, dict="0123456789."): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:32: ../testdata/gettysburg.txt (level=-2, dict="0123456789."): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:32: ../testdata/gettysburg.txt (level=1, dict="0123456789."): open ../testdata/gettysburg.txt: '../testdata' file does not exist
--- FAIL: TestWriterReset (0.00s)
    writer_test.go:99: ../testdata/gettysburg.txt (level=0): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:99: ../testdata/gettysburg.txt (level=-1): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:99: ../testdata/gettysburg.txt (level=-2): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:99: ../testdata/gettysburg.txt (level=0): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:99: ../testdata/gettysburg.txt (level=-1): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:99: ../testdata/gettysburg.txt (level=-2): open ../testdata/gettysburg.txt: '../testdata' file does not exist
FAIL
FAIL	compress/zlib	0.143s
ok  	container/heap	0.154s
ok  	container/list	0.142s
ok  	container/ring	0.135s
ok  	context	0.349s
ok  	crypto	0.146s
ok  	crypto/aes	0.152s
ok  	crypto/cipher	1.374s
ok  	crypto/des	0.346s
ok  	crypto/dsa	0.191s
ok  	crypto/ecdh	0.310s
ok  	crypto/ecdsa	0.265s
ok  	crypto/ed25519	0.312s
ok  	crypto/elliptic	0.175s
ok  	crypto/hmac	0.141s
ok  	crypto/internal/alias	0.136s
ok  	crypto/internal/bigmod	0.166s
ok  	crypto/internal/boring	0.142s
ok  	crypto/internal/boring/bcache	0.236s
ok  	crypto/internal/edwards25519	0.460s
ok  	crypto/internal/edwards25519/field	0.170s
ok  	crypto/internal/mlkem768	0.291s
ok  	crypto/internal/nistec	0.759s
ok  	crypto/internal/nistec/fiat	0.135s [no tests to run]
ok  	crypto/md5	0.143s
ok  	crypto/rand	0.148s
ok  	crypto/rc4	0.158s
ok  	crypto/rsa	0.809s
ok  	crypto/sha1	0.138s
ok  	crypto/sha256	0.140s
ok  	crypto/sha512	0.141s
ok  	crypto/subtle	0.259s
ok  	crypto/tls	12.282s
ok  	crypto/x509	1.567s
ok  	database/sql	0.772s
ok  	database/sql/driver	0.136s
ok  	debug/buildinfo	0.153s
ok  	debug/dwarf	0.218s
ok  	debug/elf	0.254s
ok  	debug/gosym	0.159s
ok  	debug/macho	0.162s
ok  	debug/pe	0.222s
ok  	debug/plan9obj	0.145s
ok  	embed	0.148s [no tests to run]
ok  	embed/internal/embedtest	0.184s
ok  	encoding/ascii85	0.140s
ok  	encoding/asn1	0.146s
ok  	encoding/base32	0.154s
ok  	encoding/base64	0.154s
ok  	encoding/binary	0.142s
ok  	encoding/csv	0.161s
ok  	encoding/gob	1.773s
ok  	encoding/hex	0.139s
ok  	encoding/json	0.357s
ok  	encoding/pem	0.463s
ok  	encoding/xml	0.171s
ok  	errors	0.144s
ok  	expvar	0.156s
ok  	flag	0.183s
ok  	fmt	0.199s
ok  	go/ast	0.162s
ssh -p 2222 glenda@localhost /tmp/build.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestLocalDirectory (0.00s)
    build_test.go:145: no buildable Go source files in /tmp
--- FAIL: TestImportCmd (0.00s)
    build_test.go:432: go/build: go list cmd/internal/objfile: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist
        
        
--- FAIL: TestFindImports (0.00s)
    deps_test.go:861: open /usr/local/go/src/go/build: '/usr/local' does not exist
FAIL
FAIL	go/build	1.174s
ok  	go/build/constraint	0.138s
ok  	go/constant	0.144s
ok  	go/doc	0.314s
ok  	go/doc/comment	0.649s
ssh -p 2222 glenda@localhost /tmp/format.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestNode (0.00s)
    format_test.go:43: open format_test.go: 'format_test.go' file does not exist
--- FAIL: TestSource (0.00s)
    format_test.go:101: open format_test.go: 'format_test.go' file does not exist
FAIL
FAIL	go/format	0.144s
ok  	go/importer	0.164s
ok  	go/internal/gccgoimporter	0.196s
ok  	go/internal/gcimporter	0.189s
ssh -p 2222 glenda@localhost /tmp/srcimporter.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestImportedTypes (0.01s)
    srcimporter_test.go:126: go/build: go list flag: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist
        
        
    srcimporter_test.go:126: go/build: go list io: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist
        
        
    srcimporter_test.go:126: go/build: go list io: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist
        
        
    srcimporter_test.go:126: go/build: go list math: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist
        
        
    srcimporter_test.go:126: go/build: go list math: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist
        
        
    srcimporter_test.go:126: go/build: go list math/big: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist
        
        
    srcimporter_test.go:126: go/build: go list golang.org/x/text/unicode/norm: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist
        
        
--- FAIL: TestReimport (0.00s)
    srcimporter_test.go:193: got go/build: go list math: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist
        
        ; want reimport error
--- FAIL: TestIssue20855 (0.00s)
    srcimporter_test.go:204: got unexpected or no error: go/build: go list go/internal/srcimporter/testdata/issue20855: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist
        
--- FAIL: TestIssue24392 (0.00s)
    srcimporter_test.go:220: go/build: go list go/internal/srcimporter/testdata/issue24392: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist
        
        
FAIL
FAIL	go/internal/srcimporter	0.185s
ssh -p 2222 glenda@localhost /tmp/parser.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
panic: open ../printer/nodes.go: '../printer' file does not exist

goroutine 1 [running]:
go/parser.readFile(...)
	/usr/local/go/src/go/parser/performance_test.go:18
go/parser.init()
	/usr/local/go/src/go/parser/performance_test.go:13 +0x329
FAIL	go/parser	0.215s
ssh -p 2222 glenda@localhost /tmp/printer.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestWriteErrors (0.00s)
panic: open printer.go: 'printer.go' file does not exist [recovered]
	panic: open printer.go: 'printer.go' file does not exist

goroutine 21 [running]:
testing.tRunner.func1.2({0x16d240, 0x8717d0})
	/usr/local/go/src/testing/testing.go:1632 +0x282
testing.tRunner.func1()
	/usr/local/go/src/testing/testing.go:1635 +0x3fc
panic({0x16d240, 0x8717d0})
	/usr/local/go/src/runtime/panic.go:759 +0xf8
go/printer.TestWriteErrors(0x898c08)
	/usr/local/go/src/go/printer/printer_test.go:652 +0x231
testing.tRunner(0x898c08, 0x19126c)
	/usr/local/go/src/testing/testing.go:1690 +0x118
created by testing.(*T).Run in goroutine 1
	/usr/local/go/src/testing/testing.go:1743 +0x3d0
FAIL	go/printer	0.223s
ok  	go/scanner	0.140s
ok  	go/token	0.156s
ssh -p 2222 glenda@localhost /tmp/types.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestGenerate (0.00s)
    generate_test.go:59: open /usr/local/go/src/cmd/compile/internal/types2/selection.go: '/usr/local' does not exist
2024/05/17 12:09:33 main:2:8: could not import fmt (can't find import: "fmt": cannot find package "fmt" in any of:
	/usr/local/go/src/fmt (from $GOROOT)
	/usr/glenda/go/src/fmt (from $GOPATH))
FAIL	go/types	0.254s
ok  	go/version	0.139s
ok  	hash	0.141s
ok  	hash/adler32	0.141s
ok  	hash/crc32	0.140s
ok  	hash/crc64	0.140s
ok  	hash/fnv	0.143s
ok  	hash/maphash	0.167s
ok  	html	0.150s
ok  	html/template	0.226s
ok  	image	0.213s
ok  	image/color	0.150s
ssh -p 2222 glenda@localhost /tmp/draw.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestPaletted (0.00s)
    draw_test.go:724: open: open ../testdata/video-001.png: '../testdata' file does not exist
FAIL
FAIL	image/draw	0.184s
ssh -p 2222 glenda@localhost /tmp/gif.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestWriter (0.00s)
    writer_test.go:91: ../testdata/video-001.png open ../testdata/video-001.png: '../testdata' file does not exist
    writer_test.go:91: ../testdata/video-001.gif open ../testdata/video-001.gif: '../testdata' file does not exist
    writer_test.go:91: ../testdata/video-001.interlaced.gif open ../testdata/video-001.interlaced.gif: '../testdata' file does not exist
--- FAIL: TestSubImage (0.00s)
    writer_test.go:121: readImg: open ../testdata/video-001.gif: '../testdata' file does not exist
--- FAIL: TestEncodeAllGo1Dot4 (0.00s)
    writer_test.go:184: ../testdata/video-001.gif open ../testdata/video-001.gif: '../testdata' file does not exist
--- FAIL: TestEncodeAllGo1Dot5 (0.00s)
    writer_test.go:184: ../testdata/video-001.gif open ../testdata/video-001.gif: '../testdata' file does not exist
--- FAIL: TestEncodeAllGo1Dot5GlobalColorModel (0.00s)
    writer_test.go:184: ../testdata/video-001.gif open ../testdata/video-001.gif: '../testdata' file does not exist
--- FAIL: TestEncodeWrappedImage (0.00s)
    writer_test.go:603: readImg: open ../testdata/video-001.gif: '../testdata' file does not exist
FAIL
FAIL	image/gif	0.182s
ssh -p 2222 glenda@localhost /tmp/jpeg.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestDecodeProgressive (0.00s)
    reader_test.go:41: ../testdata/video-001.jpeg: open ../testdata/video-001.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-001.q50.410.jpeg: open ../testdata/video-001.q50.410.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-001.q50.411.jpeg: open ../testdata/video-001.q50.411.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-001.q50.420.jpeg: open ../testdata/video-001.q50.420.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-001.q50.422.jpeg: open ../testdata/video-001.q50.422.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-001.q50.440.jpeg: open ../testdata/video-001.q50.440.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-001.q50.444.jpeg: open ../testdata/video-001.q50.444.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-005.gray.q50.jpeg: open ../testdata/video-005.gray.q50.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-005.gray.q50.2x2.jpeg: open ../testdata/video-005.gray.q50.2x2.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-001.separate.dc.progression.jpeg: open ../testdata/video-001.separate.dc.progression.jpeg: '../testdata' file does not exist
--- FAIL: TestDecodeEOF (0.00s)
    reader_test.go:123: open ../testdata/video-001.jpeg: '../testdata' file does not exist
--- FAIL: TestTruncatedSOSDataDoesntPanic (0.00s)
    reader_test.go:194: open ../testdata/video-005.gray.q50.jpeg: '../testdata' file does not exist
--- FAIL: TestIssue56724 (0.00s)
    reader_test.go:496: open ../testdata/video-001.jpeg: '../testdata' file does not exist
--- FAIL: TestBadRestartMarker (0.00s)
    reader_test.go:510: open ../testdata/video-001.restart2.jpeg: '../testdata' file does not exist
--- FAIL: TestWriter (0.00s)
    writer_test.go:136: ../testdata/video-001.png open ../testdata/video-001.png: '../testdata' file does not exist
    writer_test.go:136: ../testdata/video-001.png open ../testdata/video-001.png: '../testdata' file does not exist
    writer_test.go:136: ../testdata/video-001.png open ../testdata/video-001.png: '../testdata' file does not exist
    writer_test.go:136: ../testdata/video-001.png open ../testdata/video-001.png: '../testdata' file does not exist
    writer_test.go:136: ../testdata/video-001.png open ../testdata/video-001.png: '../testdata' file does not exist
    writer_test.go:136: ../testdata/video-001.png open ../testdata/video-001.png: '../testdata' file does not exist
FAIL
FAIL	image/jpeg	0.197s
ok  	image/png	0.402s
ok  	index/suffixarray	0.235s
ok  	internal/abi	0.154s
ok  	internal/buildcfg	0.140s
ok  	internal/chacha8rand	0.143s
ok  	internal/concurrent	0.166s
ok  	internal/coverage/cformat	0.140s
ok  	internal/coverage/cmerge	0.142s
ok  	internal/coverage/pods	0.147s
ok  	internal/coverage/slicereader	0.151s
ok  	internal/coverage/slicewriter	0.138s
ok  	internal/coverage/test	0.156s
ok  	internal/cpu	0.141s
ok  	internal/dag	0.141s
ok  	internal/diff	0.165s
ok  	internal/fmtsort	0.139s
ok  	internal/fuzz	0.154s
ok  	internal/godebug	0.144s
ok  	internal/godebugs	0.149s
ok  	internal/gover	0.151s
ok  	internal/itoa	0.139s
ok  	internal/platform	0.182s
ok  	internal/poll	0.166s
ok  	internal/profile	0.145s
ok  	internal/reflectlite	0.152s
ok  	internal/runtime/atomic	0.162s
ok  	internal/saferio	0.154s
ok  	internal/singleflight	0.190s
ok  	internal/sysinfo	0.139s
ok  	internal/testenv	0.144s
ssh -p 2222 glenda@localhost /tmp/trace.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestSummarizeGoroutinesTrace (0.00s)
    summary_test.go:269: malformed test v2/testdata/tests/go122-gc-stress.test: bad trace file: failed to read test file for v2/testdata/tests/go122-gc-stress.test: open v2/testdata/tests/go122-gc-stress.test: 'v2' file does not exist
--- FAIL: TestSummarizeGoroutinesRegionsTrace (0.00s)
    summary_test.go:269: malformed test v2/testdata/tests/go122-annotations.test: bad trace file: failed to read test file for v2/testdata/tests/go122-annotations.test: open v2/testdata/tests/go122-annotations.test: 'v2' file does not exist
--- FAIL: TestSummarizeTasksTrace (0.00s)
    summary_test.go:269: malformed test v2/testdata/tests/go122-annotations-stress.test: bad trace file: failed to read test file for v2/testdata/tests/go122-annotations-stress.test: open v2/testdata/tests/go122-annotations-stress.test: 'v2' file does not exist
--- FAIL: TestRelatedGoroutinesV2Trace (0.00s)
    summary_test.go:393: malformed test v2/testdata/tests/go122-gc-stress.test: bad trace file: failed to read test file for v2/testdata/tests/go122-gc-stress.test: open v2/testdata/tests/go122-gc-stress.test: 'v2' file does not exist
FAIL
FAIL	internal/trace	0.223s
ssh -p 2222 glenda@localhost /tmp/trace.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestOldtrace (0.00s)
    oldtrace_test.go:87: didn't see expected test case user_task_region_1_21_good
FAIL
FAIL	internal/trace/v2	0.286s
ok  	internal/trace/v2/internal/oldtrace	0.232s
ssh -p 2222 glenda@localhost /tmp/errors.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestErrorCodeStyle (0.00s)
    codes_test.go:151: open codes.go: 'codes.go' file does not exist
FAIL
FAIL	internal/types/errors	0.159s
ok  	internal/unsafeheader	0.142s
ok  	internal/weak	0.142s
ok  	internal/xcoff	0.194s
ssh -p 2222 glenda@localhost /tmp/zstd.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestAlloc (0.00s)
    zstd_test.go:165: open ../../testdata/Isaac.Newton-Opticks.txt: '../../testdata' file does not exist
FAIL
FAIL	internal/zstd	0.179s
ok  	io	0.317s
ssh -p 2222 glenda@localhost /tmp/fs.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestGlob (0.02s)
    glob_test.go:35: Glob(`glob.go`) = []string(nil) want glob.go
    glob_test.go:35: Glob(`gl?b.go`) = []string(nil) want glob.go
    glob_test.go:35: Glob(`gl\ob.go`) = []string(nil) want glob.go
    glob_test.go:35: Glob(`*`) = []string{"fs.test", "xxx"} want glob.go
    glob_test.go:35: Glob(`*/glob.go`) = []string(nil) want fs/glob.go
FAIL
FAIL	io/fs	0.389s
ssh -p 2222 glenda@localhost /tmp/ioutil.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestReadFile (0.00s)
    ioutil_test.go:36: ReadFile ioutil_test.go: open ioutil_test.go: 'ioutil_test.go' file does not exist
--- FAIL: TestReadOnlyWriteFile (0.00s)
    ioutil_test.go:94: Expected an error when writing to read-only file /tmp/TestReadOnlyWriteFile1731551095/blurp.txt
--- FAIL: TestReadDir (0.00s)
    ioutil_test.go:129: ReadDir ..: io_test.go file not found
FAIL
FAIL	io/ioutil	1.018s
ok  	iter	0.141s
ok  	log	0.157s
ok  	log/slog	0.191s
ok  	log/slog/internal/benchmarks	0.165s
ok  	log/slog/internal/buffer	0.144s
ok  	maps	0.152s
ok  	math	0.156s
ok  	math/big	0.879s
ok  	math/bits	0.151s
ok  	math/cmplx	0.144s
ok  	math/rand	0.212s
ok  	math/rand/v2	0.306s
ok  	mime	0.164s
ok  	mime/multipart	0.703s
ok  	mime/quotedprintable	0.160s
ssh -p 2222 glenda@localhost /tmp/net.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestSendfile (0.01s)
    sendfile_test.go:137: received 0 bytes; expected 567198
    sendfile_test.go:141: retrieved data hash did not match
    sendfile_test.go:145: open ../testdata/Isaac.Newton-Opticks.txt: '../testdata' file does not exist
--- FAIL: TestSendfileParts (0.02s)
    sendfile_test.go:198: unexpected server reply "", want "Produced "
    sendfile_test.go:202: open ../testdata/Isaac.Newton-Opticks.txt: '../testdata' file does not exist
--- FAIL: TestSendfileSeeked (0.02s)
    sendfile_test.go:258: Got 0 bytes; want 10240
    sendfile_test.go:262: open ../testdata/Isaac.Newton-Opticks.txt: '../testdata' file does not exist
--- FAIL: TestSendfileOnWriteTimeoutExceeded (0.02s)
    sendfile_test.go:427: open ../testdata/Isaac.Newton-Opticks.txt: '../testdata' file does not exist
FAIL
FAIL	net	17.559s
ssh -p 2222 glenda@localhost /tmp/http.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestEmptyDirOpenCWD (0.00s)
    fs_test.go:492: open of fs_test.go: open fs_test.go: 'fs_test.go' file does not exist
http.test 33984: warning: process exceeds 100 file descriptors
--- FAIL: TestServeFileFromCWD (0.00s)
    --- FAIL: TestServeFileFromCWD/h1 (0.05s)
        fs_test.go:556: expected 200 OK, got 404 Not Found
    --- FAIL: TestServeFileFromCWD/h2 (0.08s)
        fs_test.go:556: expected 200 OK, got 404 Not Found
FAIL
FAIL	net/http	144.934s
ok  	net/http/cgi	0.523s
ok  	net/http/cookiejar	0.408s
ok  	net/http/fcgi	0.377s
ok  	net/http/httptest	3.828s
ok  	net/http/httptrace	0.154s
ok  	net/http/httputil	14.883s
ok  	net/http/internal	0.261s
ok  	net/http/internal/ascii	0.141s
ok  	net/http/pprof	4.366s
ok  	net/internal/cgotest	0.139s
ok  	net/mail	0.150s
ok  	net/netip	0.174s
ok  	net/rpc	1.715s
ok  	net/rpc/jsonrpc	0.177s
ok  	net/smtp	1.955s
ok  	net/textproto	0.157s
ok  	net/url	0.157s
ssh -p 2222 glenda@localhost /tmp/os.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestRemoveAllButReadOnlyAndPathError (0.00s)
    removeall_test.go:332: RemoveAll succeeded unexpectedly
--- FAIL: TestReadDir (0.00s)
    read_test.go:133: ReadDir .: read_test.go file not found
--- FAIL: TestReadOnlyWriteFile (0.00s)
    read_test.go:96: Expected an error when writing to read-only file /tmp/TestReadOnlyWriteFile1135810301/blurp.txt
--- FAIL: TestReadFile (0.00s)
    read_test.go:37: ReadFile read_test.go: open read_test.go: 'read_test.go' file does not exist
--- FAIL: TestFileReadDir (0.00s)
    --- FAIL: TestFileReadDir/. (0.00s)
        os_test.go:469: could not find dir_unix.go
        os_test.go:469: could not find env.go
        os_test.go:469: could not find error.go
        os_test.go:469: could not find file.go
        os_test.go:469: could not find os_test.go
        os_test.go:469: could not find types.go
        os_test.go:469: could not find stat_darwin.go
        os_test.go:469: could not find stat_linux.go
--- FAIL: TestFileReaddir (0.00s)
    --- FAIL: TestFileReaddir/. (0.00s)
        os_test.go:414: could not find dir_unix.go
        os_test.go:414: could not find env.go
        os_test.go:414: could not find error.go
        os_test.go:414: could not find file.go
        os_test.go:414: could not find os_test.go
        os_test.go:414: could not find types.go
        os_test.go:414: could not find stat_darwin.go
        os_test.go:414: could not find stat_linux.go
--- FAIL: TestFileReaddirnames (0.00s)
    --- FAIL: TestFileReaddirnames/. (0.00s)
        os_test.go:377: could not find dir_unix.go
        os_test.go:377: could not find env.go
        os_test.go:377: could not find error.go
        os_test.go:377: could not find file.go
        os_test.go:377: could not find os_test.go
        os_test.go:377: could not find types.go
        os_test.go:377: could not find stat_darwin.go
        os_test.go:377: could not find stat_linux.go
FAIL
FAIL	os	1.327s
ssh -p 2222 glenda@localhost /tmp/exec.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestCatGoodAndBadFile (0.01s)
    exec_test.go:430: expected test code; got "Error: open exec_test.go: 'exec_test.go' file does not exist\n" (len 61)
FAIL
FAIL	os/exec	0.878s
ok  	os/exec/internal/fdtest	0.151s
ok  	os/signal	0.256s
ok  	os/user	0.146s
ok  	path	0.147s
ssh -p 2222 glenda@localhost /tmp/filepath.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestGlob (0.03s)
    match_test.go:133: Glob(`match.go`) = []string(nil) want match.go
    match_test.go:133: Glob(`mat?h.go`) = []string(nil) want match.go
    match_test.go:133: Glob(`*`) = []string{"filepath.test", "xxx"} want match.go
    match_test.go:133: Glob(`../*/match.go`) = []string(nil) want ../filepath/match.go
--- FAIL: TestNonWindowsGlobEscape (0.00s)
    match_test.go:371: Glob(`\match.go`) = [] want [match.go]
--- FAIL: TestBug3486 (0.00s)
    path_test.go:1735: stat /usr/local/go/src/unicode: '/usr/local' does not exist
FAIL
FAIL	path/filepath	0.217s
ok  	plugin	0.143s
ok  	reflect	0.299s
ok  	regexp	0.422s
ok  	regexp/syntax	0.603s
ssh -p 2222 glenda@localhost /tmp/runtime.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestTracebackSystem (0.04s)
    traceback_system_test.go:114: stderr:
        Callers: [0x3b18ff 0x3b1896 0x3b184b 0x3b6efa 0x3b6ef5 0x3b6ef4 0x3b6ef3 0x3b6ef2 0x82ad1]
        panic: open /usr/local/go/src/runtime/traceback_system_test.go: '/usr/local' does not exist
        
        [...]
FAIL	runtime	8.720s
ok  	runtime/coverage	0.167s
ok  	runtime/debug	0.214s
ok  	runtime/internal/math	0.151s
ok  	runtime/internal/sys	0.144s
ok  	runtime/internal/wasitest	0.161s
ssh -p 2222 glenda@localhost /tmp/metrics.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestDocs (0.00s)
    description_test.go:94: open doc.go: 'doc.go' file does not exist
FAIL
FAIL	runtime/metrics	0.155s
ok  	runtime/pprof	9.515s
ok  	runtime/trace	0.267s
ok  	slices	0.181s
ok  	sort	0.175s
ok  	strconv	0.284s
ok  	strings	0.203s
ssh -p 2222 glenda@localhost /tmp/sync.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: ExampleOnceValues (0.02s)
got:
Reading file once
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
want:
Reading file once
FAIL
FAIL	sync	1.284s
ok  	sync/atomic	1.428s
ok  	syscall	0.141s
ok  	testing	1.020s
ok  	testing/fstest	0.147s
ok  	testing/iotest	0.148s
ok  	testing/quick	0.164s
ok  	testing/slogtest	0.153s
ok  	text/scanner	0.149s
ok  	text/tabwriter	0.143s
ok  	text/template	0.198s
ok  	text/template/parse	0.153s
ssh -p 2222 glenda@localhost /tmp/time.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
panic: cannot load America/Los_Angeles for testing: '../../lib/time' does not exist; you may want to use -tags=timetzdata

goroutine 1 [running]:
time.initTestingZone()
	/usr/local/go/src/time/internal_test.go:21 +0x12f
sync.(*Once).doSlow(0x413e50, 0x294318)
	/usr/local/go/src/sync/once.go:74 +0xa0
sync.(*Once).Do(0x413e50, 0x294318)
	/usr/local/go/src/sync/once.go:65 +0x3e
time.ForceUSPacificForTesting(...)
	/usr/local/go/src/time/export_test.go:18
time.init.0()
	/usr/local/go/src/time/internal_test.go:9 +0x7b
FAIL	time	0.206s
ok  	unicode	0.153s
ok  	unicode/utf16	0.150s
ok  	unicode/utf8	0.150s
ok  	unique	0.187s
ok  	cmd/addr2line	0.170s
ok  	cmd/api	0.235s
ssh -p 2222 glenda@localhost /tmp/asm.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
testdata/386.s:6: #include: open ../../../../runtime/textflag.h: '../../../../runtime' file does not exist
FAIL	cmd/asm/internal/asm	0.247s
ok  	cmd/asm/internal/lex	0.153s
ok  	cmd/cgo/internal/swig	0.172s
ok  	cmd/cgo/internal/test	0.144s
ok  	cmd/cgo/internal/testcshared	0.204s
ok  	cmd/cgo/internal/testerrors	0.184s
ok  	cmd/cgo/internal/testfortran	0.167s
ok  	cmd/cgo/internal/testgodefs	0.165s
ok  	cmd/cgo/internal/testlife	0.163s
ok  	cmd/cgo/internal/testnocgo	0.149s
ok  	cmd/cgo/internal/testplugin	0.295s
ok  	cmd/cgo/internal/testsanitizers	0.202s [no tests to run]
ok  	cmd/cgo/internal/testshared	0.313s
ok  	cmd/cgo/internal/testso	0.175s
ok  	cmd/cgo/internal/teststdio	0.170s
ok  	cmd/cgo/internal/testtls	0.150s
ok  	cmd/compile/internal/abt	0.153s
ok  	cmd/compile/internal/amd64	0.224s
ok  	cmd/compile/internal/base	0.156s
ok  	cmd/compile/internal/compare	0.179s
ok  	cmd/compile/internal/devirtualize	0.186s
ok  	cmd/compile/internal/dwarfgen	0.223s
ok  	cmd/compile/internal/importer	0.221s
ok  	cmd/compile/internal/inline/inlheur	0.211s
ok  	cmd/compile/internal/ir	0.181s
ok  	cmd/compile/internal/liveness	0.222s
ok  	cmd/compile/internal/logopt	0.159s
ok  	cmd/compile/internal/loopvar	0.219s
ok  	cmd/compile/internal/noder	0.231s
ok  	cmd/compile/internal/reflectdata	0.189s [no tests to run]
ok  	cmd/compile/internal/ssa	0.380s
ssh -p 2222 glenda@localhost /tmp/syntax.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestParse (0.00s)
    parser_test.go:30: open parser.go: 'parser.go' file does not exist
--- FAIL: TestVerify (0.00s)
    parser_test.go:34: open parser.go: 'parser.go' file does not exist
FAIL
FAIL	cmd/compile/internal/syntax	0.192s
ok  	cmd/compile/internal/test	0.385s
ok  	cmd/compile/internal/typecheck	0.182s
ok  	cmd/compile/internal/types	0.161s
ssh -p 2222 glenda@localhost /tmp/types2.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
2024/05/17 12:13:54 main:2:8: could not import fmt (can't find import: "fmt": cannot find package "fmt" in any of:
	/usr/local/go/src/fmt (from $GOROOT)
	/usr/glenda/go/src/fmt (from $GOPATH))
FAIL	cmd/compile/internal/types2	0.298s
ok  	cmd/covdata	0.178s
ok  	cmd/cover	0.237s
ok  	cmd/dist	0.151s
ok  	cmd/distpack	0.186s
ssh -p 2222 glenda@localhost /tmp/doc.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
FAIL	cmd/doc	0.204s
ok  	cmd/fix	0.348s
ok  	cmd/go	1.659s
ok  	cmd/go/internal/auth	0.157s
ok  	cmd/go/internal/cache	0.637s
ok  	cmd/go/internal/cfg	0.146s [no tests to run]
ok  	cmd/go/internal/fsys	0.255s
ok  	cmd/go/internal/generate	0.218s
ok  	cmd/go/internal/gover	0.149s
ok  	cmd/go/internal/imports	0.213s
ok  	cmd/go/internal/load	0.191s
ok  	cmd/go/internal/lockedfile	0.327s
ok  	cmd/go/internal/modfetch	0.258s
ok  	cmd/go/internal/modfetch/codehost	0.252s
ok  	cmd/go/internal/modfetch/zip_sum_test	0.199s
ssh -p 2222 glenda@localhost /tmp/modindex.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestIndex (0.00s)
    --- FAIL: TestIndex/crypto (0.00s)
        index_test.go:29: cannot find package in:
            	/usr/local/go/src/crypto
    --- FAIL: TestIndex/encoding (0.00s)
        index_test.go:29: cannot find package in:
            	/usr/local/go/src/encoding
    --- FAIL: TestIndex/unsafe (0.00s)
        index_test.go:29: cannot find package in:
            	/usr/local/go/src/unsafe
    --- FAIL: TestIndex/encoding/json (0.00s)
        index_test.go:29: cannot find package in:
            	/usr/local/go/src/encoding/json
    --- FAIL: TestIndex/runtime (0.00s)
        index_test.go:29: cannot find package in:
            	/usr/local/go/src/runtime
    --- FAIL: TestIndex/net (0.00s)
        index_test.go:29: cannot find package in:
            	/usr/local/go/src/net
    --- FAIL: TestIndex/all (0.00s)
        index_test.go:29: cannot find package in:
            	/usr/local/go/src/crypto
FAIL
FAIL	cmd/go/internal/modindex	0.178s
ok  	cmd/go/internal/modload	0.261s
ok  	cmd/go/internal/mvs	0.164s
ok  	cmd/go/internal/par	0.154s
ok  	cmd/go/internal/str	0.150s
ok  	cmd/go/internal/test	0.240s
ok  	cmd/go/internal/toolchain	0.222s
ok  	cmd/go/internal/vcs	0.197s
ok  	cmd/go/internal/vcweb	0.396s
ssh -p 2222 glenda@localhost /tmp/vcstest.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
2024/05/17 12:14:05 serving /
2024/05/17 12:14:05 serving /
--- FAIL: TestScripts (0.06s)
    vcstest_test.go:168: stat /testdata/vcstest: '/testdata' file does not exist
FAIL
FAIL	cmd/go/internal/vcweb/vcstest	0.350s
ok  	cmd/go/internal/web	0.181s
ok  	cmd/go/internal/work	0.219s
ssh -p 2222 glenda@localhost /tmp/gofmt.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestRewrite (0.03s)
    --- FAIL: TestRewrite/gofmt.go (0.00s)
        gofmt_test.go:62: stat gofmt.go: 'gofmt.go' file does not exist
    --- FAIL: TestRewrite/gofmt_test.go (0.00s)
        gofmt_test.go:62: stat gofmt_test.go: 'gofmt_test.go' file does not exist
FAIL
FAIL	cmd/gofmt	0.258s
ok  	cmd/internal/archive	0.172s
ok  	cmd/internal/bootstrap_test	0.149s
ok  	cmd/internal/buildid	0.271s
ok  	cmd/internal/cov	0.160s
ssh -p 2222 glenda@localhost /tmp/dwarf.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestPutVarAbbrevGenerator (0.00s)
    putvarabbrevgen_test.go:85: open ./dwarf.go: './dwarf.go' file does not exist
FAIL
FAIL	cmd/internal/dwarf	0.153s
ok  	cmd/internal/edit	0.153s
ok  	cmd/internal/goobj	0.147s
ok  	cmd/internal/moddeps	0.155s
ok  	cmd/internal/notsha256	0.521s
ok  	cmd/internal/obj	1.069s
ok  	cmd/internal/obj/loong64	0.148s
ok  	cmd/internal/obj/ppc64	0.157s
ok  	cmd/internal/obj/riscv	0.175s
ok  	cmd/internal/obj/s390x	0.149s
ok  	cmd/internal/obj/x86	0.161s
ok  	cmd/internal/objabi	0.154s
ok  	cmd/internal/pgo	0.175s
ok  	cmd/internal/pkgpath	0.176s
ok  	cmd/internal/pkgpattern	0.152s
ok  	cmd/internal/quoted	0.150s
ok  	cmd/internal/src	0.153s
ok  	cmd/internal/test2json	0.253s
ok  	cmd/link	0.264s
ok  	cmd/link/internal/benchmark	0.152s
ok  	cmd/link/internal/ld	0.263s
ok  	cmd/link/internal/loader	0.159s
ssh -p 2222 glenda@localhost /tmp/nm.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestNonGoExecs (0.04s)
    nm_test.go:80: go tool nm /usr/local/go/src/debug/elf/testdata/gcc-386-freebsd-exec: exit status: 'nm.test 38329: 1'
        open /usr/local/go/src/debug/elf/testdata/gcc-386-freebsd-exec: '/usr/local' does not exist
    nm_test.go:80: go tool nm /usr/local/go/src/debug/elf/testdata/gcc-amd64-linux-exec: exit status: 'nm.test 38332: 1'
        open /usr/local/go/src/debug/elf/testdata/gcc-amd64-linux-exec: '/usr/local' does not exist
    nm_test.go:70: obscuretestdata.DecodeToTempFile(/usr/local/go/src/debug/macho/testdata/gcc-386-darwin-exec.base64): open /usr/local/go/src/debug/macho/testdata/gcc-386-darwin-exec.base64: '/usr/local' does not exist
    nm_test.go:70: obscuretestdata.DecodeToTempFile(/usr/local/go/src/debug/macho/testdata/gcc-amd64-darwin-exec.base64): open /usr/local/go/src/debug/macho/testdata/gcc-amd64-darwin-exec.base64: '/usr/local' does not exist
    nm_test.go:80: go tool nm /usr/local/go/src/debug/pe/testdata/gcc-386-mingw-exec: exit status: 'nm.test 38335: 1'
        open /usr/local/go/src/debug/pe/testdata/gcc-386-mingw-exec: '/usr/local' does not exist
    nm_test.go:80: go tool nm /usr/local/go/src/debug/plan9obj/testdata/amd64-plan9-exec: exit status: 'nm.test 38338: 1'
        open /usr/local/go/src/debug/plan9obj/testdata/amd64-plan9-exec: '/usr/local' does not exist
    nm_test.go:80: go tool nm /usr/local/go/src/debug/plan9obj/testdata/386-plan9-exec: exit status: 'nm.test 38341: 1'
        open /usr/local/go/src/debug/plan9obj/testdata/386-plan9-exec: '/usr/local' does not exist
    nm_test.go:80: go tool nm /usr/local/go/src/internal/xcoff/testdata/gcc-ppc64-aix-dwarf2-exec: exit status: 'nm.test 38344: 1'
        open /usr/local/go/src/internal/xcoff/testdata/gcc-ppc64-aix-dwarf2-exec: '/usr/local' does not exist
FAIL
FAIL	cmd/nm	0.244s
ok  	cmd/objdump	0.206s
ok  	cmd/pack	0.177s
ok  	cmd/pprof	0.255s
ok  	cmd/relnote	0.154s
ok  	cmd/trace	0.223s
ok  	cmd/trace/v2	0.221s
ok  	cmd/vet	0.303s
FAIL

One of the problems I have is that the test data file or directory location is not always deterministic, so we don't really know what should be copied on the remote host to be able to run the test.

@0intro
Copy link
Member

0intro commented May 21, 2024

I've realized that the go_$GOOS_$GOARCH_exec script is executed from each of the test directory, so we can retrieve the current working directory. This solves the test data issue.

Now, most of the tests are passing successfully. There are two remaining issues.

  1. The go/types and cmd/compile/internal/types2 tests expect the Go toolchain binaries to be present on the machine were the tests are executed.
ssh -p 2222 glenda@localhost /tmp/types.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
2024/05/21 16:45:42 main:2:8: could not import fmt (can't find import: "fmt": fork/exec /tmp/gotest/bin/go: '/tmp/gotest/bin/go' does not exist)
FAIL	go/types	0.485s
ssh -p 2222 glenda@localhost /tmp/types2.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
2024/05/21 16:50:43 main:2:8: could not import fmt (can't find import: "fmt": fork/exec /tmp/gotest/bin/go: '/tmp/gotest/bin/go' does not exist)
FAIL	cmd/compile/internal/types2	0.422s
  1. The TestTracebackSystem test expects to access the sources files from the absolute path on the machine on which the Go toolchain has been built.
ssh -p 2222 glenda@localhost /tmp/runtime.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestTracebackSystem (0.04s)
    traceback_system_test.go:114: stderr:
        Callers: [0x3b2bcf 0x3b2b66 0x3b2b1b 0x3b81ca 0x3b81c5 0x3b81c4 0x3b81c3 0x3b81c2 0x83931]
        panic: open /usr/local/go/src/runtime/traceback_system_test.go: '/usr/local' does not exist

@0intro
Copy link
Member

0intro commented May 28, 2024

Hi @dmitshur,

@millerresearch and I finally succeeded to run all tests on a remote Plan 9 machine from Linux, using a go_$GOOS_$GOARCH_exec script.

What is the next step to receive builds on the LUCI bot, for example plan9-386?

##### Test execution environment.
# GOARCH: 386
# CPU: QEMU Virtual CPU version 2.5+
# GOOS: plan9
# OS Version: 2000

##### Testing packages.
ok  	archive/tar	26.857s
ok  	archive/zip	19.933s
ok  	bufio	19.461s
ok  	bytes	20.347s
ok  	cmp	14.258s
ok  	compress/bzip2	11.098s
ok  	compress/flate	21.189s
ok  	compress/gzip	35.042s
ok  	compress/lzw	13.900s
ok  	compress/zlib	17.329s
ok  	container/heap	14.175s
ok  	container/list	14.406s
ok  	container/ring	14.266s
ok  	context	20.261s
ok  	crypto	18.324s
ok  	crypto/aes	13.502s
ok  	crypto/cipher	16.156s
ok  	crypto/des	13.194s
ok  	crypto/dsa	14.896s
ok  	crypto/ecdh	21.032s
ok  	crypto/ecdsa	25.033s
ok  	crypto/ed25519	17.302s
ok  	crypto/elliptic	18.358s
ok  	crypto/hmac	14.029s
ok  	crypto/internal/alias	12.607s
ok  	crypto/internal/bigmod	16.517s
ok  	crypto/internal/boring	12.705s
ok  	crypto/internal/boring/bcache	15.177s
ok  	crypto/internal/edwards25519	17.482s
ok  	crypto/internal/edwards25519/field	14.955s
ok  	crypto/internal/hpke	14.798s
ok  	crypto/internal/mlkem768	17.030s
ok  	crypto/internal/nistec	20.657s
ok  	crypto/internal/nistec/fiat	12.109s [no tests to run]
ok  	crypto/md5	16.397s
ok  	crypto/rand	15.468s
ok  	crypto/rc4	12.910s
ok  	crypto/rsa	23.734s
ok  	crypto/sha1	15.148s
ok  	crypto/sha256	15.467s
ok  	crypto/sha512	14.653s
ok  	crypto/subtle	16.188s
ok  	crypto/tls	57.737s
ok  	crypto/x509	48.663s
ok  	database/sql	31.000s
ok  	database/sql/driver	13.304s
ok  	debug/buildinfo	18.417s
ok  	debug/dwarf	17.190s
ok  	debug/elf	20.040s
ok  	debug/gosym	16.197s
ok  	debug/macho	13.657s
ok  	debug/pe	21.261s
ok  	debug/plan9obj	13.122s
ok  	embed	15.152s [no tests to run]
ok  	embed/internal/embedtest	16.462s
ok  	encoding/ascii85	13.215s
ok  	encoding/asn1	18.056s
ok  	encoding/base32	16.513s
ok  	encoding/base64	15.210s
ok  	encoding/binary	17.202s
ok  	encoding/csv	18.412s
ok  	encoding/gob	28.222s
ok  	encoding/hex	14.567s
ok  	encoding/json	43.914s
ok  	encoding/pem	21.207s
ok  	encoding/xml	22.529s
ok  	errors	15.858s
ok  	expvar	20.351s
ok  	flag	19.004s
ok  	fmt	20.885s
ok  	go/ast	20.500s
ok  	go/build	87.102s
ok  	go/build/constraint	13.869s
ok  	go/constant	17.510s
ok  	go/doc	15.574s
ok  	go/doc/comment	18.992s
ok  	go/format	19.578s
ok  	go/importer	19.980s
ok  	go/internal/gccgoimporter	21.706s
ok  	go/internal/gcimporter	22.117s
ok  	go/internal/srcimporter	181.146s
ok  	go/parser	12.415s
ok  	go/printer	19.353s
ok  	go/scanner	15.300s
ok  	go/token	20.722s
ok  	go/types	56.506s
ok  	go/version	12.694s
ok  	hash	15.305s
ok  	hash/adler32	13.213s
ok  	hash/crc32	15.411s
ok  	hash/crc64	13.802s
ok  	hash/fnv	12.913s
ok  	hash/maphash	13.851s
ok  	html	16.912s
ok  	html/template	36.005s
ok  	image	16.728s
ok  	image/color	13.912s
ok  	image/draw	18.167s
ok  	image/gif	19.915s
ok  	image/jpeg	19.859s
ok  	image/png	21.120s
ok  	index/suffixarray	17.556s
ok  	internal/abi	15.114s
ok  	internal/buildcfg	13.453s
ok  	internal/chacha8rand	12.694s
ok  	internal/concurrent	13.962s
ok  	internal/coverage/cfile	14.778s
ok  	internal/coverage/cformat	14.075s
ok  	internal/coverage/cmerge	12.867s
ok  	internal/coverage/pods	14.925s
ok  	internal/coverage/slicereader	12.454s
ok  	internal/coverage/slicewriter	12.961s
ok  	internal/coverage/test	15.255s
ok  	internal/cpu	13.447s
ok  	internal/dag	13.625s
ok  	internal/diff	13.215s
ok  	internal/fmtsort	13.953s
ok  	internal/fuzz	19.854s
ok  	internal/godebug	14.606s
ok  	internal/godebugs	14.715s
ok  	internal/gover	12.764s
ok  	internal/itoa	12.560s
ok  	internal/platform	17.359s
ok  	internal/poll	14.742s
ok  	internal/profile	13.003s
ok  	internal/reflectlite	16.793s
ok  	internal/runtime/atomic	15.138s
ok  	internal/saferio	14.977s
ok  	internal/singleflight	12.708s
ok  	internal/sysinfo	13.120s
ok  	internal/testenv	14.693s
ok  	internal/trace	22.580s
ok  	internal/trace/internal/oldtrace	16.058s
ok  	internal/types/errors	20.680s
ok  	internal/unsafeheader	12.805s
ok  	internal/weak	14.160s
ok  	internal/xcoff	13.609s
ok  	internal/zstd	19.823s
ok  	io	18.744s
ok  	io/fs	16.100s
ok  	io/ioutil	17.810s
ok  	iter	13.175s
ok  	log	16.713s
ok  	log/slog	31.815s
ok  	log/slog/internal/benchmarks	14.257s
ok  	log/slog/internal/buffer	13.013s
ok  	maps	14.617s
ok  	math	16.266s
ok  	math/big	37.726s
ok  	math/bits	15.721s
ok  	math/cmplx	15.236s
ok  	math/rand	23.625s
ok  	math/rand/v2	24.381s
ok  	mime	16.407s
ok  	mime/multipart	22.123s
ok  	mime/quotedprintable	17.781s
ok  	net	52.135s
ok  	net/http	234.664s
ok  	net/http/cgi	36.081s
ok  	net/http/cookiejar	30.626s
ok  	net/http/fcgi	20.307s
ok  	net/http/httptest	44.136s
ok  	net/http/httptrace	17.389s
ok  	net/http/httputil	53.158s
ok  	net/http/internal	16.070s
ok  	net/http/internal/ascii	12.700s
ok  	net/http/pprof	43.825s
ok  	net/internal/cgotest	12.708s
ok  	net/mail	17.862s
ok  	net/netip	24.957s
ok  	net/rpc	41.945s
ok  	net/rpc/jsonrpc	28.593s
ok  	net/smtp	31.805s
ok  	net/textproto	16.843s
ok  	net/url	21.740s
ok  	os	28.525s
ok  	os/exec	31.105s
ok  	os/exec/internal/fdtest	12.691s
ok  	os/signal	13.719s
ok  	os/user	13.514s
ok  	path	14.604s
ok  	path/filepath	18.894s
ok  	plugin	12.705s
ok  	reflect	39.806s
ok  	regexp	21.934s
ok  	regexp/syntax	17.658s
ok  	runtime	89.929s
ok  	runtime/debug	20.890s
ok  	runtime/internal/math	12.494s
ok  	runtime/internal/sys	12.716s
ok  	runtime/internal/wasitest	15.041s
ok  	runtime/metrics	16.052s
ok  	runtime/pprof	51.264s
ok  	runtime/trace	21.357s
ok  	slices	19.777s
ok  	sort	16.257s
ok  	strconv	18.838s
ok  	strings	20.799s
ok  	sync	22.014s
ok  	sync/atomic	19.580s
ok  	syscall	13.355s
ok  	testing	25.946s
ok  	testing/fstest	16.897s
ok  	testing/iotest	14.854s
ok  	testing/quick	15.453s
ok  	testing/slogtest	17.590s
ok  	text/scanner	15.594s
ok  	text/tabwriter	14.823s
ok  	text/template	27.435s
ok  	text/template/parse	16.385s
ok  	time	57.464s
ok  	unicode	15.071s
ok  	unicode/utf16	13.459s
ok  	unicode/utf8	14.663s
ok  	unique	17.722s
ok  	cmd/addr2line	24.305s
ok  	cmd/api	20.799s
ok  	cmd/asm/internal/asm	20.286s
ok  	cmd/asm/internal/lex	15.089s
ok  	cmd/cgo/internal/swig	13.855s
ok  	cmd/cgo/internal/test	12.590s
ok  	cmd/cgo/internal/testcshared	16.106s
ok  	cmd/cgo/internal/testerrors	15.592s
ok  	cmd/cgo/internal/testfortran	13.631s
ok  	cmd/cgo/internal/testgodefs	14.858s
ok  	cmd/cgo/internal/testlife	12.583s
ok  	cmd/cgo/internal/testnocgo	12.653s
ok  	cmd/cgo/internal/testplugin	14.110s
ok  	cmd/cgo/internal/testsanitizers	12.312s [no tests to run]
ok  	cmd/cgo/internal/testshared	17.599s
ok  	cmd/cgo/internal/testso	15.370s
ok  	cmd/cgo/internal/teststdio	12.510s
ok  	cmd/cgo/internal/testtls	13.061s
ok  	cmd/compile/internal/abt	13.440s
ok  	cmd/compile/internal/amd64	29.045s
ok  	cmd/compile/internal/base	19.046s
ok  	cmd/compile/internal/compare	24.562s
ok  	cmd/compile/internal/devirtualize	25.150s
ok  	cmd/compile/internal/dwarfgen	34.248s
ok  	cmd/compile/internal/importer	29.641s
ok  	cmd/compile/internal/inline/inlheur	28.211s
ok  	cmd/compile/internal/ir	23.560s
ok  	cmd/compile/internal/liveness	27.871s
ok  	cmd/compile/internal/logopt	16.100s
ok  	cmd/compile/internal/loopvar	24.047s
ok  	cmd/compile/internal/noder	30.532s
ok  	cmd/compile/internal/rangefunc	23.316s
ok  	cmd/compile/internal/reflectdata	22.811s [no tests to run]
ok  	cmd/compile/internal/ssa	63.622s
ok  	cmd/compile/internal/syntax	20.536s
ok  	cmd/compile/internal/test	44.259s
ok  	cmd/compile/internal/typecheck	24.457s
ok  	cmd/compile/internal/types	19.342s
ok  	cmd/compile/internal/types2	52.947s
ok  	cmd/covdata	20.150s
ok  	cmd/cover	27.545s
ok  	cmd/dist	16.587s
ok  	cmd/distpack	16.949s
ok  	cmd/doc	36.613s
ok  	cmd/fix	27.678s
ok  	cmd/go	43.206s
ok  	cmd/go/internal/auth	16.043s
ok  	cmd/go/internal/cache	20.150s
ok  	cmd/go/internal/cfg	14.356s [no tests to run]
ok  	cmd/go/internal/fsys	20.581s
ok  	cmd/go/internal/generate	32.038s
ok  	cmd/go/internal/gover	15.568s
ok  	cmd/go/internal/imports	16.262s
ok  	cmd/go/internal/load	26.671s
ok  	cmd/go/internal/lockedfile	17.793s
ok  	cmd/go/internal/modfetch	37.304s
ok  	cmd/go/internal/modfetch/codehost	35.629s
ok  	cmd/go/internal/modfetch/zip_sum_test	26.861s
ok  	cmd/go/internal/modindex	71.032s
ok  	cmd/go/internal/modload	38.336s
ok  	cmd/go/internal/mvs	16.667s
ok  	cmd/go/internal/par	13.364s
ok  	cmd/go/internal/str	14.154s
ok  	cmd/go/internal/test	34.910s
ok  	cmd/go/internal/toolchain	31.300s
ok  	cmd/go/internal/vcs	27.064s
ok  	cmd/go/internal/vcweb	32.042s
ok  	cmd/go/internal/vcweb/vcstest	33.413s
ok  	cmd/go/internal/web	22.206s
ok  	cmd/go/internal/work	33.102s
ok  	cmd/gofmt	18.753s
ok  	cmd/internal/archive	13.542s
ok  	cmd/internal/bootstrap_test	13.520s
ok  	cmd/internal/buildid	16.776s
ok  	cmd/internal/cov	14.618s
ok  	cmd/internal/dwarf	15.905s
ok  	cmd/internal/edit	12.749s
ok  	cmd/internal/goobj	14.402s
ok  	cmd/internal/moddeps	17.277s
ok  	cmd/internal/notsha256	14.388s
ok  	cmd/internal/obj	16.857s
ok  	cmd/internal/obj/loong64	15.035s
ok  	cmd/internal/obj/ppc64	18.101s
ok  	cmd/internal/obj/riscv	16.338s
ok  	cmd/internal/obj/s390x	14.308s
ok  	cmd/internal/obj/x86	19.433s
ok  	cmd/internal/objabi	14.487s
ok  	cmd/internal/pgo	15.714s
ok  	cmd/internal/pkgpath	14.516s
ok  	cmd/internal/pkgpattern	13.804s
ok  	cmd/internal/quoted	12.900s
ok  	cmd/internal/src	13.351s
ok  	cmd/internal/test2json	13.244s
ok  	cmd/link	30.890s
ok  	cmd/link/internal/benchmark	14.212s
ok  	cmd/link/internal/ld	29.405s
ok  	cmd/link/internal/loader	16.953s
ok  	cmd/nm	33.944s
ok  	cmd/objdump	28.518s
ok  	cmd/pack	23.089s
ok  	cmd/pprof	36.629s
ok  	cmd/relnote	18.513s
ok  	cmd/trace	23.768s
ok  	cmd/vet	34.453s

##### os/user with tag osusergo
ok  	os/user	13.522s

##### hash/maphash purego implementation
ok  	hash/maphash	14.188s

##### crypto with tag purego
ok  	crypto	0.072s
ok  	crypto/aes	0.073s
ok  	crypto/cipher	0.073s
ok  	crypto/des	0.072s
ok  	crypto/dsa	0.072s
ok  	crypto/ecdh	0.072s
ok  	crypto/ecdsa	0.072s
ok  	crypto/ed25519	0.072s
ok  	crypto/elliptic	0.075s
ok  	crypto/hmac	0.071s
ok  	crypto/internal/alias	0.073s
ok  	crypto/internal/bigmod	0.072s
ok  	crypto/internal/boring	0.072s
?   	crypto/internal/boring/bbig	[no test files]
ok  	crypto/internal/boring/bcache	0.072s
?   	crypto/internal/boring/sig	[no test files]
ok  	crypto/internal/edwards25519	0.071s
ok  	crypto/internal/edwards25519/field	0.072s
ok  	crypto/internal/hpke	0.073s
ok  	crypto/internal/mlkem768	0.071s
ok  	crypto/internal/nistec	0.072s
ok  	crypto/internal/nistec/fiat	0.072s
?   	crypto/internal/randutil	[no test files]
ok  	crypto/md5	0.073s
ok  	crypto/rand	0.071s
ok  	crypto/rc4	0.182s
ok  	crypto/rsa	0.072s
ok  	crypto/sha1	0.072s
ok  	crypto/sha256	0.072s
ok  	crypto/sha512	0.078s
ok  	crypto/subtle	0.072s
ok  	crypto/tls	0.073s
ok  	crypto/x509	0.072s
?   	crypto/x509/pkix	[no test files]

##### GOEXPERIMENT=rangefunc
ok  	iter	13.064s
ok  	slices	19.868s
ok  	maps	14.614s

##### Testing without libgcc.
ok  	net	52.157s
ok  	os/user	15.028s

##### sync -cpu=10
ok  	sync	22.105s

##### GOMAXPROCS=2 runtime -cpu=1 -quick
ok  	runtime	64.892s

##### GOMAXPROCS=2 runtime -cpu=2 -quick
ok  	runtime	64.991s

##### GOMAXPROCS=2 runtime -cpu=4 -quick
ok  	runtime	91.586s

ALL TESTS PASSED

@dmitshur
Copy link
Contributor

dmitshur commented May 28, 2024

That's great to hear, thanks!

One idea I wanted to briefly mention about testdata: I wonder if could work well to rely on //go:embed testdata directives in _test.go files to embed testdata directory content into test binaries. It can be constrained to Plan 9 needs in _plan9_test.go files, or applied more generally if it's a pattern we're okay with using more broadly.

For next steps, using the current plan9-386 LUCI builder as the example, we'll need to update the builder definition at https://ci.chromium.org/ui/p/golang/builders/luci.golang.ci/gotip-plan9-386 to match your builder. If you follow the "Machine Pool" link there now, it filters by cipd_platform:plan9-386 pool:luci.golang.shared-workers (you'll need to login to view the page content) which currently matches no bots. Updating cipd_platform to linux-amd64 (since your builder is a Linux AMD64 host) matches the plan9-386 builder and more, see here. We'll need to think the dimensions to use long term, but maybe narrowing down by bot id can work initially:

https://chromium-swarm.appspot.com/botlist?f=cipd_platform%3Alinux-amd64&f=pool%3Aluci.golang.shared-workers&f=id%3Aplan9-386

Sent CL 588756.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/588756 mentions this issue: main.star: use id dimension to select Plan 9 builders

gopherbot pushed a commit to golang/build that referenced this issue May 28, 2024
The current Plan 9 builders are expected to be Linux AMD64 hosts in the
shared-workers pool that will provide functional go_plan9_{goarch}_exec
scripts in order to test the Plan 9 Go ports.

For a lack of a better alternative available now, rely on their unique
bot id dimension to tell them apart from other Linux AMD64 hosts in the
shared-workers pool.

For golang/go#62025.

Change-Id: Ib53de20dd4a56dd222b281abc8b7dbbeeb18a0ab
Reviewed-on: https://go-review.googlesource.com/c/build/+/588756
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Reviewed-by: David du Colombier <0intro@gmail.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
@dmitshur
Copy link
Contributor

@0intro https://chromium-swarm.appspot.com/task?id=69d9f835adb76f10 was a recent build after the aforementioned CL:

15819 2024-05-28 21:41:44.550 E: Failed to run run_isolated
Traceback (most recent call last):
  File "/home/swarming/.swarming/swarming_bot.1.zip/__main__.py", line 324, in main
    return fn(args)
           ^^^^^^^^
  File "/home/swarming/.swarming/swarming_bot.1.zip/__main__.py", line 142, in CMDrun_isolated
    import run_isolated
  File "/home/swarming/.swarming/swarming_bot.1.zip/client/run_isolated.py", line 53, in <module>
    import distutils
ModuleNotFoundError: No module named 'distutils'

@millerresearch
Copy link
Contributor

One idea I wanted to briefly mention about testdata: I wonder if could work well to rely on //go:embed testdata directives in _test.go files to embed testdata directory content into test binaries. It can be constrained to Plan 9 needs in _plan9_test.go files, or applied more generally if it's a pattern we're okay with using more broadly.

Our plan is to have one filesystem shared via 9p between the swarming-bot proxy and the plan9 test [virtual] machine, so the source and test files will be visible with the same names (presumably under /home/swarming/.swarming/c). Will these mean it's not necessary to embed testdata?

@millerresearch
Copy link
Contributor

A plan9-arm builder is now running. Would you like to try sending it a task?

This builder has python 3.11 so I think the missing distutils problem will not occur here. It seems python 3.12 has removed the distutils module from the standard library?

@millerresearch
Copy link
Contributor

The first task seems to have got as far as building go for linux-amd64 and plan9-arm, then failed in the "update prebuilt go / cas archive" stage:

[E2024-05-29T15:05:21.028203+01:00 29195 0 annotate.go:273] original error: retry budget exhausted (6 attempts): context deadline exceeded

goroutine 1:
#0 go.chromium.org/luci/client/cmd/cas/casimpl/archive.go:169 - casimpl.(*archiveRun).doArchive()
  reason: failed to call UploadIfMissing

#1 go.chromium.org/luci/client/cmd/cas/casimpl/archive.go:221 - casimpl.(*archiveRun).Run()
#2 github.com/maruel/subcommands@v1.1.1/subcommands.go:395 - subcommands.Run()
#3 cas/main.go:95 - main.main()
#4 runtime/proc.go:271 - runtime.main()
#5 runtime/asm_amd64.s:1695 - runtime.goexit()
cas: failed to call UploadIfMissing: retry budget exhausted (6 attempts): context deadline exceeded

@0intro
Copy link
Member

0intro commented May 29, 2024

I've just installed the Python distutils package.

One idea I wanted to briefly mention about testdata: I wonder if could work well to rely on //go:embed testdata directives in _test.go files to embed testdata directory content into test binaries. It can be constrained to Plan 9 needs in _plan9_test.go files, or applied more generally if it's a pattern we're okay with using more broadly.

Our plan is to have one filesystem shared via 9p between the swarming-bot proxy and the plan9 test [virtual] machine, so the source and test files will be visible with the same names (presumably under /home/swarming/.swarming/c). Will these mean it's not necessary to embed testdata?

That said, if we had the possibility to run Go tests without access to the full sources tree, it would be great.

@dmitshur
Copy link
Contributor

dmitshur commented May 29, 2024

Sharing a filesystem seems good if it works well. Embedding is just an alternative path. It would make the test binaries larger and may require them to write out the embedded data to a temporary directory, so the main advantage is that it doesn't require copying files via a separate mechanism, and that it might make it easier to avoid copying unnecessary files.

That "update prebuilt go / cas archive" build step ran for 6.2 minutes, which is quite long to upload a pre-built Go toolchain. Is the uplink on that machine expected to be slow? Let's also wait and see if the 6.2 minute time keeps happening more than just once.

@millerresearch
Copy link
Contributor

That "update prebuilt go / cas archive" build step ran for 6.2 minutes, which is quite long to upload a pre-built Go toolchain. Is the uplink on that machine expected to be slow?

The link is nominally 20Mbit/sec upstream.

@dmitshur
Copy link
Contributor

dmitshur commented May 30, 2024

In b8746572283039962513, the 'upload prebuilt go' step took 54 seconds. A pre-built toolchain is around 150 MB compressed, so that seems to line up with 20 Mbit/s.

The build ran for 57 minutes. 27020 tests passed, 74 failed. It's great to see this progress. It seems we should expect these builders to be slow—please feel free add them here so that they get increased timeouts.

@millerresearch
Copy link
Contributor

Today's tests have been failing because I restarted the bot and accidentally didn't have go_plan9_arm_exec in the PATH, so it was trying to run arm test binaries on amd64. Fixed now.

With the old build infrastructure we could use x/build/cmd/retrybuilds to re-run a failed test. Does LUCI have a way for us to do that?

@dmitshur
Copy link
Contributor

dmitshur commented May 30, 2024

We haven't updated retrybuilds for LUCI, but may do so in the future. So far it hasn't been a high priority because it's generally okay to wait for future commits to come in and be built. In LUCI UI, on the Infra tab there is a "Retry Build" action available (for authorized users) that acts similarly, though it causes a new build to be scheduled without making the old one unavailable.

@0intro
Copy link
Member

0intro commented Jun 4, 2024

Is there a way with the Swarming Bot to execute a script before or after running a task?
It could be helpful, for example, to reboot the remote Plan 9 VM after running tests.

@0intro
Copy link
Member

0intro commented Jun 17, 2024

To answer my own question, there is actually a way to execute a script after running a task.

When the SWARMING_NEVER_REBOOT environment variable is not set, the Swarming Bot executes both /sbin/shutdown and /sbin/reboot after running a task. Since these files are just symbolic links to systemd nowadays, it's easy to replace them by custom scripts.

In my case, I've deleted /sbin/reboot and replaced /sbin/shutdown by the following script, which reboots the remote Plan 9 machine:

#!/bin/bash
timeout --preserve-status 10 ssh -i /home/swarming/.ssh/id_rsa_plan9 glenda@go-plan9-386-luci fshalt -r >/dev/null

The remaining issue is that when the SWARMING_NEVER_REBOOT environment variable is not set, the Swarming Bot attempts to reboot the machine aggressively after some uptime. I worked around this issue by creating a /.docker_env file, which fakes a Docker environment and fools the Swarming Bot.

@millerresearch
Copy link
Contributor

In LUCI UI, on the Infra tab there is a "Retry Build" action available (for authorized users) that acts similarly, though it causes a new build to be scheduled without making the old one unavailable.

Is there a way for us to become authorized to retry builds for the plan9-* builders? It would be easier for us to debug the new infrastructure we're having to develop because of LUCI, if we could arrange to receive tasks in a controlled way instead of having long and unpredictable waits until the coordinator decides to send one.

At this moment I have a plan9-arm swarming bot running, I see a long list of tasks for it showing state NO_RESOURCE, but nothing is being sent.

@dmitshur
Copy link
Contributor

@0intro Sorry it took a while before I had a chance to get back to this, but I'm glad you were able to make progress. There is open source documentation for the Swarming Bot at https://chromium.googlesource.com/infra/luci/luci-py.git/+/main/appengine/swarming/doc/Bot.md. Its Hooks section mentions on_before_task and on_after_task callbacks, which sound like what you were looking for. I hope this helps.

@millerresearch Yes, I agree you should be able to trigger these builds yourself. may-start-trybots is the relevant access tier (see the current LUCI configuration that uses it). Please follow the process there to request that access.

@millerresearch
Copy link
Contributor

Now that we're making more progress with tests, I observe that LUCI is sending a larger set of tests to the Plan 9 builders than the old coordinator was doing. For example:

  • the x/benchmarks repo tests weren't sent by the coordinator but are being sent by LUCI; they fail to compile because of a Unix syscall dependency
  • subrepo tests like x/tools are being sent for several go releases, where the coordinator just sent them for gotip
  • the main repo test is including the $GOROOT/test tests, which adds about an hour to the execution time on plan9-arm

Were these changes intended, or just an unexpected side effect of switching to LUCI?

@dmitshur
Copy link
Contributor

This is a consequence of the simplification of the build policy during the migration to LUCI. With coordinator, there were many custom policies applied to builders individually, including what you mention:

https://cs.opensource.google/go/x/build/+/master:dashboard/builders.go;l=2701-2702;drc=e41b08676471cd283c5de077647f9177f7e8ac88

The LUCI configuration tried to generalize a higher level classification at a repository level, instead of doing it per builder. See:

https://cs.opensource.google/go/x/build/+/luci-config:main.star;l=636-675;drc=67d27da0c4496ee84405c96517dfdaaf4e960cfc

Because the x/benchmarks repo is considered a "library" category, it's tested on a wider set of platforms, which includes Plan 9. Also see the commit message of CL 515355 for a bit more motivation.

If the current strategy doesn't work well for testing the Plan 9 OS, I think it's reasonable for us to consider adjusting the LUCI configuration accordingly. For tests in the main repo, the change can also be done inside cmd/dist rather than in LUCI configuration if the goal is to apply test skips to all.bash as well. It'll probably work better to file a separate issue to discuss those details.

@0intro
Copy link
Member

0intro commented Jun 26, 2024

By setting GO_TEST_SHARDS=0, you should be able to skip $GOROOT/test tests.

@millerresearch
Copy link
Contributor

millerresearch commented Jul 9, 2024

All tests have now passed on plan9-arm. A few are spectacularly slow (probably build cache is confused by cross-compiling of packages on the swarming machine).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Plan9
Projects
Status: In Progress
Development

No branches or pull requests

6 participants