Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: add openbsd/arm builder #25498

Open
FiloSottile opened this issue May 22, 2018 · 19 comments
Open

x/build: add openbsd/arm builder #25498

FiloSottile opened this issue May 22, 2018 · 19 comments

Comments

@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented May 22, 2018

@4a6f656c used to maintain one but it doesn't seem to be running anymore.

Cloud arm32 is hard to find, so this might end up being a Raspberry Pi on my desk.

@4a6f656c
Copy link
Contributor

@4a6f656c 4a6f656c commented Dec 11, 2018

I've resurrected the openbsd/arm builder and should be able to continue running it for the foreseeable future (or until another option is found).

@bcmills
Copy link
Member

@bcmills bcmills commented May 6, 2019

I see that the builder is running, but it seems not to have enough memory to complete the tests reliably. Basically all of the runs are failing with fatal error: runtime: out of memory.

Is it possible to add more RAM to the builder, or perhaps configure it with some swap to deal with bigger tests?

Examples:
https://build.golang.org/log/17984e5b3a059c1882855a2c8774fdb7d8ecfad6
https://build.golang.org/log/5233126dc3a18b78a2fee33288f378485eeb66ac
https://build.golang.org/log/ce51b2eebed1b20dc6be028224d613fda8bcc216

@4a6f656c
Copy link
Contributor

@4a6f656c 4a6f656c commented Jun 30, 2019

@bcmills - the platform is limited to 1GB of data size and this was a regression from Go 1.12, which was fixed via issue #27739.

@bcmills
Copy link
Member

@bcmills bcmills commented Aug 30, 2019

@4a6f656c, please disable the openbsd-arm builder until it can be converted to use cmd/buildlet.

The old-style builders are no longer supported (#21191 (comment)), and we can't easily fix configuration issues with them (such as #32838).

@4a6f656c
Copy link
Contributor

@4a6f656c 4a6f656c commented Jan 4, 2020

@bcmills @dmitshur - a replacement openbsd-arm builder (host-openbsd-arm-joelsing) is running with cmd/buildlet, however is repeatedly failing in the coordinator (server side) with a writeSnapshot: failed to write snapshot to GCS timeout... I presume that my somewhat limited upstream bandwidth, combined with being a long way away (network wise), means that the time limit is insufficient to complete the upload. Could you please investigate?

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 6, 2020

@4a6f656c
Copy link
Contributor

@4a6f656c 4a6f656c commented Jan 31, 2020

@bcmills @toothrot @cagedmantis - I've turned the buildlet off since it has just been spinning its wheels for the last month. Please let me know when I should try again.

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Apr 13, 2020

@4a6f656c Thanks for working on this and sorry about the delay. A few questions to better understand the writeSnapshot issue that's blocking you from proceeding.

Have you considered the SkipSnapshot configuration option for this builder? Do you know what the effect of using it on your builder would be?

The writeSnapshot timeout was increased from 1 minute to 5 minutes 3 years ago in CL 37468, with a comment noting that most operations should happen in well under a minute. Do you have information about the upstream network speed of your builder, so we can estimate if increasing that timeout slightly would be sufficient, or if we need to approach this differently.

@4a6f656c
Copy link
Contributor

@4a6f656c 4a6f656c commented Apr 14, 2020

@dmitshur - ah, I was not aware of SkipSnapshot - that does in fact look like it would solve the issue, not sure if sharded tests are an issue here (for openbsd/arm and openbsd/arm64 they would seem unlikely, not sure for linux/riscv64). The upstream network is ~1Mbps, the bigger issue is likely the RTT from .au to .us.

I'll raise a CL...

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 14, 2020

Change https://golang.org/cl/228238 mentions this issue: dashboard: enable SkipSnapshot for openbsd-arm-jsing builder

gopherbot pushed a commit to golang/build that referenced this issue Apr 14, 2020
The builder network is too slow/distant to upload snapshots successfully.

Updates golang/go#25498

Change-Id: Ida86eb1f84e34c9810227c98370cef980529796f
Reviewed-on: https://go-review.googlesource.com/c/build/+/228238
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@4a6f656c
Copy link
Contributor

@4a6f656c 4a6f656c commented Apr 15, 2020

@dmitshur - could you please let me know when that change is deployed and I'll try restarting the openbsd/arm builder.

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Apr 15, 2020

@4a6f656c Sure. It's deployed now.

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 16, 2020

Change https://golang.org/cl/228597 mentions this issue: dashboard: reduce the amount of work for openbsd/arm

@4a6f656c
Copy link
Contributor

@4a6f656c 4a6f656c commented Apr 16, 2020

@dmitshur - thanks, we're now up and running, although I've just mailed another CL to reduce the amount of work the builder is trying to do.

gopherbot pushed a commit to golang/build that referenced this issue Apr 16, 2020
Given the performance of this hardware, reduce the amount of work that
it is trying to do.

Updates golang/go#25498

Change-Id: I943ab2b01bd10c5217c097464d3904b0f4de2d0a
Reviewed-on: https://go-review.googlesource.com/c/build/+/228597
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@4a6f656c
Copy link
Contributor

@4a6f656c 4a6f656c commented Apr 20, 2020

@dmitshur - any chance of getting the last change deployed (openbsd/arm currently has 873 builds queued)?

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Apr 20, 2020

@4a6f656c Yes, it will be deployed at the nearest opportunity, likely today.

@4a6f656c
Copy link
Contributor

@4a6f656c 4a6f656c commented Apr 20, 2020

@dmitshur - thanks!

@andybons
Copy link
Member

@andybons andybons commented May 27, 2020

@4a6f656c the openbsd-arm-jsing builder has been failing builds for a bit now. Can you take a look?

@4a6f656c
Copy link
Contributor

@4a6f656c 4a6f656c commented May 30, 2020

@andybons the builder was upgraded to OpenBSD 6.7 - builds resumed functioning after https://go-review.googlesource.com/c/go/+/234381 was landed. There are still a handful of intermittent failures which I need to look at - most of these are likely related to the fact that it is a slow non-MP platform.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.