Migrate windows runners to non-ephemeral instances

[jeanschmidt]

🚀 The feature, motivation and pitch

Due to increased pressure over our windows runners, and the elevated cost of instantiating and bringing down those instances, we want to migrate instances from ephemeral to not ephemeral.

Possible impacts are related to breakages in or misbehaves on CI jobs that puts the runners in a bad state. Other possible impacts are related to exhaustion of resources, especially disk space, but memory might be a contender, as CI trash piles up on those instances.

As a somewhat middle of the road approach to this, currently nonephemeral instances are stochastically rotated as older instances get higher priority to be terminated when demand is lower.

Instances definition can be found here: pytorch/test-infra#4072

• ✅ migrate windows.4xlarge to windows.4xlarge.nonephemeral instances under pytorch/pytorch (Migrate jobs from windows.4xlarge windows.4xlarge.nonephemeral instances #100377)
• 📣 migrate windows.8xlarge.nvidia.gpu to windows.8xlarge.nvidia.gpu.nonephemeral instances under pytorch/pytorch (Migrate jobs from windows.8xlarge.nvidia.gpu to nonephemeral #104404)
• ⏳ submit PRs to all repositories under pytorch/ organization to migrate windows.4xlarge to windows.4xlarge.nonephemeral
• ⏳ submit PRs to all repositories under pytorch/ organization to migrate windows.8xlarge.nvidia.gpu to windows.8xlarge.nvidia.gpu.nonephemeral
• ⏳ terminate the existence of windows.4xlarge and windows.8xlarge.nvidia.gpu
• ⏳ evaluate and start the work related to the adoption of windows.g5.4xlarge.nvidia.gpu to replace windows.8xlarge.nvidia.gpu.nonephemeral in other repositories and use cases (proposed by @huydhn)

The reasoning for this phased approach is to reduce the scope of possible contenders to investigate in case of misbehave of particular CI jobs.

Alternatives

No response

Additional context

No response

cc @seemethere @malfet @pytorch/pytorch-dev-infra

[malfet]

Is there a tool (if not I think it's important to write one) that would allow admin to retire runner if it becomes corrupted

[jeanschmidt]

this can be accomplish by AWS UI, manually terminating it. But it is possible to provide a more friendly tooling for this if we believe this is relevant.

Do you have ideas on how we can implement it in a way that is both safe and simple from a user perspective?

[malfet]

@jeanschmidt terminating instance from AWS UI is fine, but it would be nice to have some sort of catchup script that deregisters terminated runners from GitHub (that runs say once a day or something)

[jeanschmidt]

Thanks for the feedback, I have a task to do that since january, but kept pushing back as raised the priority of other ones. Will focus on this one as soon as the time permits.