Make aarch64 tests complete consistently

[workingjubilee]

Seriously it's annoying that they constantly have to be restarted because it means the test doesn't complete. Then GH only allows kicking it off again after all jobs have finished. This was okay at first but at some point it has started to happen way more than bearable.

Options, I think, are:

• Improve test runtimes significantly
• Stop using spot instances
• Find a way to have the instance hibernate, then rerun it?
• Find a way to have GitHub rerun the test if it was interrupted

[BradyBonnette]

There is a 5th way possibly:

Figure out if plrust can be cross-compiled in an x86 environment, then run it in docker with qemu or something, which would be hosted in Github's environment instead of something we set up in The Cloud™️ .

There are probably a lot of things that could be done to the aarch64 CI infrastructure to make it better, but I just assembled the "quickest thing possible that could work". Also being mindful of costs and whatnot.

[workingjubilee]

ime QEMU takes even more time to execute.

[BradyBonnette]

ime QEMU takes even more time to execute.

Yeah that's the fear.

Trying to match Github 1:1 on infrastructure implementation details is a bit tricky. I could use normal on-demand instances (as opposed to spot instances) that could "spring to life" when they need, but then I'd probably have to incorporate something with AWS Lambda to get that to work properly (i.e. adds more machinery). I could also consider going down the route of using something like ECS to handle the workload as well if this keeps becoming an issue or if we have so many CI runs that everything sits in a queue forever.

Maybe the short term remedy for now is to increase the number of spot instances available? Right now it's set to 2 available at any given time (simply because commits happened so infrequently), but there's a ramp-up and ramp-down time when the spot instances are terminated and the new ones are instantiated to take their place.

[eeeebbbbrrrr]

aarch64 tests have been running just fine for as long as I can recall now. CI is super slow, but that's a different problem.