Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: update NetBSD builders to NetBSD 8-beta #20852

Closed
bradfitz opened this issue Jun 29, 2017 · 77 comments

Comments

Projects
None yet
6 participants
@bradfitz
Copy link
Member

commented Jun 29, 2017

In https://go-review.googlesource.com/c/47036/, Christos Zoulas mentions that there are known bugs in the NetBSD 7.1 kqueue code and we should switch our builders to NetBSD 8-beta.

Also mentioned in #20836 (comment)

This is a tracking bug to update our NetBSD builders past 7.1

I've updated https://golang.org/wiki/MinimumRequirements too.

/cc @krytarowski @bsiegert @zoulasc @adams-sarah @andybons

@bradfitz bradfitz added the OS-NetBSD label Jun 29, 2017

@gopherbot gopherbot added this to the Unreleased milestone Jun 29, 2017

@gopherbot gopherbot added the Builders label Jun 29, 2017

@bradfitz bradfitz modified the milestones: Go1.10, Unreleased Jun 29, 2017

@zoulasc

This comment has been minimized.

Copy link
Contributor

commented Jul 9, 2017

This should also be fixed in the 7.1 branch now, builds available from:
https://releng.netbsd.org/builds/netbsd-7-1/201707090640Z/

@bsiegert

This comment has been minimized.

Copy link
Contributor

commented Jul 11, 2017

I would be more comfortable with a netbsd-8 system than with 7.1. http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=52034 is still pending, and it means that significant disk activity is likely to crash the VM, or make it hang.

@gopherbot

This comment has been minimized.

Copy link

commented Jul 14, 2017

CL https://golang.org/cl/48091 mentions this issue.

gopherbot pushed a commit to golang/build that referenced this issue Jul 14, 2017

x/build: upgrade netbsd-amd64 builder to netbsd-8
There have been a number of fixes for issues found with the Go test
suite in kqueue, signal handling and vio drivers.

Updates golang/go#20852

Change-Id: I95861220990c05f3e89a4a40893e17758d29ebe0
Reviewed-on: https://go-review.googlesource.com/48091
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Jul 14, 2017

CL https://golang.org/cl/48552 mentions this issue.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Jul 14, 2017

@bsiegert, this doesn't boot on GCE as far as I can tell:

$ gomote create netbsd-amd64-8branch
Error running create: failed to create buildlet: 500 Internal Server Error: buildlet didn't come up at http://10.240.0.21 in 5m0s

Please let me know once you've actually run it on GCE.

Until that time, Go will continue to not run NetBSD builders.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Jul 14, 2017

I didn't mean to close this. Reopened.

@zoulasc

This comment has been minimized.

Copy link
Contributor

commented Jul 15, 2017

Are there any log messages on how -8 fails to boot on GCE, or are there any GCE hosts available for us to try?

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Jul 15, 2017

I didn't have time to capture the serial console. I had assumed @bsiegert tested this.

You can create small GCE VM instances yourself for free: https://cloud.google.com/free/

@bsiegert

This comment has been minimized.

Copy link
Contributor

commented Jul 15, 2017

The relevant bit from the serial console:

+ curl -H 'Metadata-Flavor: Google' 'http://metadata.google.internal/computeMetadata/v1/instance/attributes/buildle
t-binary-url'
Shared object "libssl.so.10" not found
+ curl -o /buildlet
Shared object "libssl.so.10" not found
+ echo 'giving up'
+ command echo 'giving up'
giving up
+ sleep 10
+ halt -p
Jul 15 19:11:28 netbsd8-builder-20170715 halt: halted by root

The binary packages are for NetBSD-7, and I suppose they are linked against base OpenSSL from NetBSD-7 :( Sorry for not catching this before sending out the CL.

@zoulasc

This comment has been minimized.

Copy link
Contributor

commented Jul 15, 2017

Yes, it is unfortunate that there are no pre-built packages for -8 yet. I will work on that. Do you need anything else from me?

@bsiegert

This comment has been minimized.

Copy link
Contributor

commented Jul 17, 2017

I'll try unpacking a /lib and /usr/lib from NetBSD-7 to fix this.

@zoulasc

This comment has been minimized.

Copy link
Contributor

commented Jul 17, 2017

the better way is to make the symlinks to the missing shared libraries to point to the new ones...

@bsiegert

This comment has been minimized.

Copy link
Contributor

commented Jul 18, 2017

@zoulasc That sounds unsafe if there was a major bump in between.

@gopherbot

This comment has been minimized.

Copy link

commented Jul 18, 2017

CL https://golang.org/cl/49670 mentions this issue.

@zoulasc

This comment has been minimized.

Copy link
Contributor

commented Jul 19, 2017

it is not an issue, people were just being conservative with libcrypto. OTOH if you replace libraries with older versions every binary on the system will break. But no worries, I'll have packages built soon.

@gopherbot

This comment has been minimized.

Copy link

commented Aug 3, 2017

Change https://golang.org/cl/53010 mentions this issue: x/build: use NetBSD-8 packages for netbsd-amd64 builder

gopherbot pushed a commit to golang/build that referenced this issue Aug 4, 2017

x/build: use NetBSD-8 packages for netbsd-amd64 builder
This should fix the latest set of errors that bradfitz@ was seeing.

Updates golang/go#20852

Change-Id: Iad7d6a479c5546395de907731643c1d817fe625d
Reviewed-on: https://go-review.googlesource.com/53010
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Aug 4, 2017

@bsiegert, it still doesn't work.

Logs from the new x/build/cmd/debugnewvm tool running debugnewvm --host=host-netbsd-8branch --run-build=netbsd-amd64-8branch:

https://gist.github.com/bradfitz/3a6ba42ed3f9d7a47338e2172cfab9bb

@bsiegert

This comment has been minimized.

Copy link
Contributor

commented Aug 4, 2017

Thanks for testing, and thanks for making me aware of debugnewvm.

It appears that this time, there has been a regression around setting routes. There is a default route via .0.1 but the host route to .0.1 is missing for some reason.

@bsiegert

This comment has been minimized.

Copy link
Contributor

commented Sep 5, 2017

It took me a long time to find out, but it is a small bug in dhcpcd, the DHCP client. I will get this fix into NetBSD-8, then try again.

@bsiegert

This comment has been minimized.

Copy link
Contributor

commented Sep 5, 2017

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Dec 1, 2017

@bsiegert, I just rebuilt the 386 image. It boots saying NetBSD 8.0_BETA (GENERIC.201711301510Z) which is newer than the 201711280120Z I see above. But still no luck:

virtio0: Virtio SCSI Device (rev. 0x00)
virtio0: no matching child driver; not configured

I'll try again tomorrow.

@krytarowski

This comment has been minimized.

Copy link
Contributor

commented Dec 1, 2017

https://releng.netbsd.org/cgi-bin/req-8.cgi just monitor 413. Martin who committed the fix cannot pullup it and we need another person from releng to test and merge the patch.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Dec 1, 2017

@krytarowski, thanks for the info & link!

@gopherbot

This comment has been minimized.

Copy link

commented Dec 1, 2017

Change https://golang.org/cl/81356 mentions this issue: dashboard, cmd/buildlet, env/netbsd: enable netbsd ssh, re-disable netbsd-386

gopherbot pushed a commit to golang/build that referenced this issue Dec 1, 2017

dashboard, cmd/buildlet, env/netbsd: enable netbsd ssh, re-disable ne…
…tbsd-386

Fixes golang/go#22872
Updates golang/go#20852

Change-Id: I2b3454cb59a434cb12b10bb93cb4dd21d5187281
Reviewed-on: https://go-review.googlesource.com/81356
Reviewed-by: Andrew Bonventre <andybons@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Dec 4, 2017

Change https://golang.org/cl/81735 mentions this issue: dashboard: re-enable NetBSD/amd64 builder

gopherbot pushed a commit to golang/build that referenced this issue Dec 4, 2017

dashboard: re-enable NetBSD/amd64 builder
Updates golang/go#22968
Updates golang/go#20852

Change-Id: Ida2bc7213d418735e897d0bae8e5fba9ae18fd6c
Reviewed-on: https://go-review.googlesource.com/81735
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Dec 5, 2017

@bsiegert, the NetBSD/386 VM now boots on GCE, but I can't find any NetBSD mirror with binary packages for i386, which means we have no curl and we can't download the buildlet.

The builder script fails on these lines,

https://github.com/golang/build/blob/master/env/netbsd-amd64/mkvm.py#L63

But doesn't stop building the VM image on error.

localhost# PS1='anita-root-shell-pr''ompt-1512499919.93# '
PS1='anita-root-shell-pr''ompt-1512499919.93# '
anita-root-shell-prompt-1512499919.93# env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All/ pkg_add bash curl
env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All/ pkg_add bash curl
pkg_add: Can't process http://ftp.netbsd.org:80/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All//bash*: Not Found
pkg_add: no pkg found for 'bash', sorry.
pkg_add: Can't process http://ftp.netbsd.org:80/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All//curl*: Not Found
pkg_add: no pkg found for 'curl', sorry.
pkg_add: 2 package additions failed
anita-root-shell-prompt-1512499919.93# echo exit_status=$?=
echo exit_status=$?=
exit_status=1=
anita-root-shell-prompt-1512499919.93# exec /bin/sh
exec /bin/sh
localhost# PS1='anita-root-shell-pr''ompt-1512499922.04# '
PS1='anita-root-shell-pr''ompt-1512499922.04# '
anita-root-shell-prompt-1512499922.04# env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All/ pkg_add git-base
env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All/ pkg_add git-base
pkg_add: Can't process http://ftp.netbsd.org:80/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q2/All//git-base*: Not Found
pkg_add: no pkg found for 'git-base', sorry.
pkg_add: 1 package addition failed

Does somebody build i386 packages?

Or do we need to build from source? (That's what I did on my test NetBSD/i386 VM, but it was super tedious and involved checking out all of the pkgsrc cvs repo.... I'd rather not do that all in qemu)

@krytarowski

This comment has been minimized.

Copy link
Contributor

commented Dec 5, 2017

We need to rebuild the packages for NetBSD-8/i386.

I've noted execution of "race" tests for FreeBSD, Linux, Windows - is this ThreadSanitizer (TSan) related? If so, we could add a NetBSD/amd64 target.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Dec 5, 2017

@krytarowski, who rebuilds packages? When can that be done by?

As for race detector support, that's unrelated to this issue. There is at least #19273 open for more platform support. Or you could file a new bug. But it would likely require upstream TSAN support. It won't happen for Go 1.10.

@krytarowski

This comment has been minimized.

Copy link
Contributor

commented Dec 5, 2017

We, have already passed the requests internally to responsible developers.If I recall correctly, Manuel Bouyer was generating them.. however -current branches (like -8) are special cases so they will be generated on demand.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Dec 5, 2017

Oh, you already ported TSAN (https://blog.netbsd.org/tnf/entry/one_year_checkpoint_and_thread). Feel free to file a new bug for tracking Go support.

@zoulasc

This comment has been minimized.

Copy link
Contributor

commented Dec 5, 2017

I am trying to build i386 packages for NetBSD-8.0...

@gopherbot

This comment has been minimized.

Copy link

commented Dec 5, 2017

Change https://golang.org/cl/82018 mentions this issue: dashboard: re-disable netbsd/386 builder until binary packages are built

gopherbot pushed a commit to golang/build that referenced this issue Dec 6, 2017

dashboard: re-disable netbsd/386 builder until binary packages are built
Also update the bootstrap toolchain (verified in VMware VM locally).

Updates golang/go#20852

Change-Id: Iac875419ed8dc51fb9db09d8948067f24c601e17
Reviewed-on: https://go-review.googlesource.com/82018
Reviewed-by: Andrew Bonventre <andybons@golang.org>
@zoulasc

This comment has been minimized.

Copy link
Contributor

commented Dec 6, 2017

There should be packages now for 8.0 + i386. Please let me know if I missed any dependencies.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Dec 6, 2017

@zoulasc, I don't seen mozilla-rootcerts here:

http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q3/All/

The complete set we try to install is:

    "env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/%s/%s/All/ pkg_add bash curl" % (arch, release),
    "env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/%s/%s/All/ pkg_add git-base" % (arch, release),
    "env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/%s/%s/All/ pkg_add mozilla-rootcerts go14" % (arch, release),
    # Interactive debugging tools for users using gomote ssh:
    "env PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/%s/%s/All/ pkg_add emacs25-nox11 vim screen" % (arch, release),

But emacs, vim, screen, go14, and even git-base should be optional. They're more nice-to-have for debugging. But without mozilla-rootcerts, curl & Go might have issues fetching resources.

@zoulasc

This comment has been minimized.

Copy link
Contributor

commented Dec 6, 2017

stay tuned :-)

@zoulasc

This comment has been minimized.

Copy link
Contributor

commented Dec 6, 2017

you should have everything but emacs... which should come later.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Dec 6, 2017

@zoulasc, thanks. I'm kicking off a build of the image.

@zoulasc

This comment has been minimized.

Copy link
Contributor

commented Dec 6, 2017

thank you for all your patience :-) I am building git now.

@gopherbot

This comment has been minimized.

Copy link

commented Dec 6, 2017

Change https://golang.org/cl/82019 mentions this issue: dashboard, env/netbsd-386: update and re-enable netbsd/386 builder

gopherbot pushed a commit to golang/build that referenced this issue Dec 6, 2017

dashboard, env/netbsd-386: update and re-enable netbsd/386 builder
The necessary binary packages are now available at:
http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/i386/8.0_2017Q3/All/

Updates golang/go#20852

Change-Id: I20660a29f9e4044899de75d1b632c34619fca19b
Reviewed-on: https://go-review.googlesource.com/82019
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Dec 6, 2017

Okay, deployed. I have it configured to only use 1 VM at a time so it won't use our self-imposed 500 VM quota all at once to catch up, but it'll catch up slowly on its own. I ran one test build manually and watched it get through the start, so I see networking on GCE now works, and bash & curl work, the buildlet starts (I needed to rebuild it for the sysctl fix), and make.bash starts running. I assume it'll be fine. Famous last words. Off to sleep...

Thanks again, @zoulasc and @bsiegert and @krytarowski!

@krytarowski

This comment has been minimized.

Copy link
Contributor

commented Dec 6, 2017

Thanks for your effort! The next release should ship with TSan, MSan and ARM CI machines on NetBSD!

@andybons

This comment has been minimized.

Copy link
Member

commented Dec 6, 2017

Really impressed with all the collaboration on this effort. Nice work, everyone! 🎉

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Dec 6, 2017

And looks good this morning. Still some flakes here & there, but much better.

I'm going to close this bug. We can open new ones for specific issues that come up.

@bradfitz bradfitz closed this Dec 6, 2017

@golang golang locked and limited conversation to collaborators Dec 6, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.