x/build/cmd/coordinator: give buildlets access to suspend themselves for N seconds

[bradfitz]

Watching the debugging in #35482 (including people running out to buy laptops), I realize we could probably provide some help in the build system.

GCE VMs support suspend:

https://cloud.google.com/sdk/gcloud/reference/alpha/compute/instances/suspend

gcloud alpha compute instances suspend is used to suspend a Google Compute Engine virtual machine. Suspending a VM is the equivalent of sleep or standby mode: the guest receives an ACPI S3 suspend signal, after which all VM state is saved to temporary storage

So we could have the coordinator do that suspend on behalf of buildlets, followed by a resume in the {buildlet/user/test}-requested duration.

I figure we'd pass an environment variable to tests containing a URL containing a secret build-specific secret, and tests could hit that URL with a seconds parameter to say how long they'd like to be suspended.

We'd need to also suspend the buildlet health checking on the coordinator side so they don't fail health checks and get killed.

This would permit writing unit tests that test program time vs real time, at least in longtest mode.

I assume this would be useful, @aclements, @ianlancetaylor?

/cc @golang/osp-team