Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
x/build: find new cloud provider for Solaris, Illumos builders? #15581
Now that we have SmartOS builders on Joyent in a custom image using the buildlet, let's use the Joyent API and dynamically create the containers as needed. This should be done by creating a new pool type in x/build/cmd/coordinator, similar to the GCE, reverse, and Kubernetes pool types.
This will both be cheaper (run zero when we need zero), but also let us scale from 0 to dozens as needed and let us do sharded builds and let SmartOS be a trybot. (currently we just run 2 containers all the time)
I see lots of joyent stuff at https://godoc.org/?q=joyent
I was just about to file this dup bug, forgetting I'd already filed it, so I'll copy the text I was about to post:
Currently the Joyent builders (GOOS=solaris, but really GOOS=illumos once #20603 happens) are statically created and the instances sit there idle most of the time, long polling the build coordinator for work. And when there is a burst of work, we can't process a burst, because we only have N instances.
That is, they use the buildlet's "reverse" mode, where the buildlets connect to farmer.golang.org and register themselves, rather than being dynamically created.
We currently have three implementations of the coordinator's BuildletPool interface,
It's kinda a waste that we're paying for N static Joyent instances just to run in reverse mode, since Joyent can already quickly spin up containers.
We should implement a JoyentBuildletPool implementations of BuildletPool and implement the Joyent API.
Of course, if we could run illumos or OmniOS on GCE that would be more ideal from a less-code-to-write angle, but I don't think they run there yet.
I do see references to EC2 AMIs for illumos and OmniOS, so maybe writing an EC2BuidlletPool implementation of the BuildletPool interface is a better use of our time and could be used for other OSes that don't run on GCE's KVM.
In any case, the static reverse builder situation is not ideal.
The future of OmniOS is uncertain: https://lists.omniti.com/pipermail/omnios-discuss/2017-April/008699.html
I just ran OmniOS-CE (the community edition) at home (omniosce-r151026u.iso, 7th of May, 2018) on KVM/QEMU and it works fine and passes all.bash.
It supports running under virtio-net but not virtio-scsi (that driver exists somewhere for ilumos, but it's not merged? or not in omnios-ce?). It does, however, support virtio-blk. But GCE doesn't support virtio-blk.
So we can't run OmniOS directly on GCE.
But because GCE now supports nested virtualization, we could do something slightly gross or lovely:
I think that's our best bet for Solaris scalable, trybots at this point. It's slightly tedious, but it stays within the GCP ecosystem we're already mostly using and where we have tons of quota, and the network is super fast, not leaving a building.
…ting This adds a linux-amd64 COS builder that should be just like our existing linux-amd64 COS builder except that it's using a forked image that has the VMX license bit enabled for nested virtualization. (GCE appears to be using the license mechanism as some sort of opt-in mechanism for features that aren't yet GA; might go away?) Once this is in, it won't do any new builds as regular+trybot builders are disabled. But it means I can then use gomote + debugnewvm to work on preparing the other four image types. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ic55f17eea17908dba7f58618d8cd162a2ed9b015 Reviewed-on: https://go-review.googlesource.com/c/162959 Reviewed-by: Dmitri Shuralyov <email@example.com>
The COS image I'd forked from earlier didn't have CONFIG_KVM or CONFIG_KVM_INTEL enabled in its kernel, so even though I'd enabled the VMX license bit for the VM, the kernel was unable to use it. Now I've instead rebuilt the ChromiumOS "lakitu" board with a modified kernel config: https://cloud.google.com/container-optimized-os/docs/how-to/building-from-open-source More docs later. Still tinkering. Nothing uses this yet. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Id2839066e67d9ddda939d96c5f4287af3267a769 Reviewed-on: https://go-review.googlesource.com/c/163057 Reviewed-by: Dmitri Shuralyov <firstname.lastname@example.org>
…d OS + vmx This adds scripts to create a new builder host image that acts like Container-Optimized OS (has docker, runs konlet on startup) but with a Debian 9 kernel + userspace that permits KVM for nested virtualization. Updates golang/go#15581 (solaris) Updates golang/go#23060 (dragonfly) Updates golang/go#30262 (riscv) Updates golang/go#30267 (fuchsia) Updates golang/go#23824 (android) Change-Id: Ib1d3a250556703856083c222be2a70c4e8d91884 Reviewed-on: https://go-review.googlesource.com/c/163301 Reviewed-by: Dmitri Shuralyov <email@example.com>
Joyent.com is shutting down their public cloud, so we no longer have our GOOS=solaris or GOOS=illumos builders there. Maybe somebody will find a new place to run them. Or maybe the ports will be abandoned. We'll see. Updates golang/go#15581 Change-Id: I0590227ce61b6b298b6aa4554e5e3bc9e4c464b5 Reviewed-on: https://go-review.googlesource.com/c/build/+/200219 Reviewed-by: Dmitri Shuralyov <firstname.lastname@example.org>
The Virtio SCSI support is still a WIP, but I'm circling back around to look at it. The other critical issue we had with GCE was this bug in the GCE hypervisor itself -- but I received notification that it's been fixed in the last week, so I'm going to try it out!
In the interim, I've seen people asking for some kind of key for a builder on the mailing list. If I can provide a zone similar to the one that was provided by Joyent, is that something I can get configured as a stop gap for this week?
Do you mean this one?
I'll have a look!
On a Linux machine, I ran:
I've made this available inside the zone:
I also built a
I was able to use
So I think this is all good to go, with the addition to the dashboard in the CL? I didn't put in a health check entry because it seems like that's just for infrastructure that's currently managed by the Go team.
Please let me know what to do next!
While the work to make illumos a first class GCE guest is completed, use this interim zone provided by an illumos community member to run illumos builds. Updates golang/go#15581 Change-Id: I1784847e5407894d01ce0aadf489b38d7e5c1924 Reviewed-on: https://go-review.googlesource.com/c/build/+/201597 Reviewed-by: Brad Fitzpatrick <email@example.com> Run-TryBot: Brad Fitzpatrick <firstname.lastname@example.org> TryBot-Result: Gobot Gobot <email@example.com>