x/build: linux-mips64 builder not reporting success or failure

[bradfitz]

(Continued from #19930 (comment))

The linux-mips64 builder keeps working, but no success or failure shows up on build.golang.org, which means the builder is disappearing or something.

I see make.bash finishes in 13 minutes, and now I'm watching it do the snapshot copy to Google Cloud Storage. There's a comment on that code that says:

func (st *buildStatus) writeSnapshot(bc *buildlet.Client) (err error) {
        sp := st.createSpan("write_snapshot_to_gcs")
        defer func() { sp.done(err) }()
        // This should happen in 15 seconds or so, but I saw timeouts                                                                                                          
        // a couple times at 1 minute. Some buildlets might be far                                                                                                             
        // away on the network, so be more lenient. The timeout mostly                                                                                                         
        // is here to prevent infinite hangs.                                                                                                                                  
        ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
        defer cancel()

There's a 5 minute timeout. It's possible those arm5 machines can't gzip quickly, and/or their network is slow.

Yup, confirmed:

Error: writeSnapshot: net/http: request canceled

Maybe we don't need to snapshot those builds to Google Cloud Storage. That means we can't shard tests, but we're not doing that currently anyway.

/cc @vstefanovic @adams-sarah @cybrcodr

[gopherbot]

CL https://golang.org/cl/40505 mentions this issue.

[bradfitz]

mips64 has an "ok" on the dashboard now.

[vstefanovic]

@bradfitz, thanks!

(mips and mips64 builders are slow partly because they run on the same machine, they are slowing down each other.)