x/build: find new cloud provider for Solaris, Illumos builders?

[bradfitz]

Now that we have SmartOS builders on Joyent in a custom image using the buildlet, let's use the Joyent API and dynamically create the containers as needed. This should be done by creating a new pool type in x/build/cmd/coordinator, similar to the GCE, reverse, and Kubernetes pool types.

This will both be cheaper (run zero when we need zero), but also let us scale from 0 to dozens as needed and let us do sharded builds and let SmartOS be a trybot. (currently we just run 2 containers all the time)

I see lots of joyent stuff at https://godoc.org/?q=joyent

/cc @davecheney @4ad

[bradfitz]

Old bug: #9515

[bradfitz]

I was just about to file this dup bug, forgetting I'd already filed it, so I'll copy the text I was about to post:

---

Currently the Joyent builders (GOOS=solaris, but really GOOS=illumos once #20603 happens) are statically created and the instances sit there idle most of the time, long polling the build coordinator for work. And when there is a burst of work, we can't process a burst, because we only have N instances.

That is, they use the buildlet's "reverse" mode, where the buildlets connect to farmer.golang.org and register themselves, rather than being dynamically created.

We currently have three implementations of the coordinator's BuildletPool interface,

• dynamically create GCE VMs
• dynamically create GKE containers
• "reverse" (dedicated machines connected to the coordinator)

It's kinda a waste that we're paying for N static Joyent instances just to run in reverse mode, since Joyent can already quickly spin up containers.

We should implement a JoyentBuildletPool implementations of BuildletPool and implement the Joyent API.

Of course, if we could run illumos or OmniOS on GCE that would be more ideal from a less-code-to-write angle, but I don't think they run there yet.

I do see references to EC2 AMIs for illumos and OmniOS, so maybe writing an EC2BuidlletPool implementation of the BuildletPool interface is a better use of our time and could be used for other OSes that don't run on GCE's KVM.

In any case, the static reverse builder situation is not ideal.

/cc @adams-sarah @cybrcodr

[4ad]

I do see references to EC2 AMIs for illumos and OmniOS

The future of OmniOS is uncertain: https://lists.omniti.com/pipermail/omnios-discuss/2017-April/008699.html

[bradfitz]

I just ran OmniOS-CE (the community edition) at home (omniosce-r151026u.iso, 7th of May, 2018) on KVM/QEMU and it works fine and passes all.bash.

It supports running under virtio-net but not virtio-scsi (that driver exists somewhere for ilumos, but it's not merged? or not in omnios-ce?). It does, however, support virtio-blk. But GCE doesn't support virtio-blk.

So we can't run OmniOS directly on GCE.

But because GCE now supports nested virtualization, we could do something slightly gross or lovely:

• boot Linux on GCE that then runs KVM/QEMU to run OmniOS-CE using virtio-blk+virtio-net.

I think that's our best bet for Solaris scalable, trybots at this point. It's slightly tedious, but it stays within the GCP ecosystem we're already mostly using and where we have tons of quota, and the network is super fast, not leaving a building.

/cc @dmitshur

[gopherbot]

Change https://golang.org/cl/162959 mentions this issue: dashboard, buildlet: add a disabled builder with nested virt, for testing