Taskcluster: use Windows instances is multiple AWS regions

[SimonSapin]

The CI queue is slow right now because Windows tasks for the PR being tested keep being killed and restarted.

https://tools.taskcluster.net/groups/UowzfhVIR7q_1bl9hagSeQ/tasks/HA3OnWLmS-yP0wDfJiZyLA/runs/3/logs/public%2Flogs%2Flive.log

[taskcluster:error] AWS has issued a spot termination - need to abort task

The above is the fourth run of this particular task.

https://tools.taskcluster.net/aws-provisioner/servo-win2016/errors shows many errors with code Server.SpotInstanceTermination and InsufficientInstanceCapacity.

People on #taskcluster on IRC suggest that this problem is made worse by our worker type configuration at https://tools.taskcluster.net/aws-provisioner/servo-win2016/edit only specifying the us-west-2 region. With our usage spread over multiple AWS regions, we would be less subject to low availability in a given region.

However, AMIs are region-local and we manage the AMI for Windows CI ourselves, and it’d be nice to not increase the number of steps needed every time we want to update it.

Possible steps to fix this:

• In etc/taskcluster/windows/build-ami.py, after the AMI is built:
  • Use taskcluster api awsProvisioner workerType servo-win2016-staging to get the current definition of the servo-win2016-staging worker type
  • Assert the region and update the ImageId for the new AMI
  • Pipe the modified JSON into taskcluster api awsProvisioner updateWorkerType servo-win2016-staging
• (While we’re touching this file, also save the AMI’s admin password through taskcluster api secrets set so that manual copy-paste into 1Password is not needed.)
• Make another script that queue a task for the servo-win2016-staging worker type.
• Make a third script that obtains the ImageId from the servo-win2016-staging definition, copies the AMI into multiple regions, and updates servo-win2016 to use them.

[SimonSapin]

Docs for aws-provisioner are gone from https://docs.taskcluster.net/ because it is “deprecated”, even though its replacement won’t be ready for months :/

In the meantime, I was pointed at the source for the exact set of API and their schema:

• https://github.com/taskcluster/aws-provisioner/blob/c1a05150f2369069c6db5ab1470b5a3a7fa7be80/lib/api-v1.js#L277
• https://github.com/taskcluster/aws-provisioner/blob/master/schemas/create-worker-type-request.yml