Live upgrade and integration tests

[brson]

It's critical that self-updates do not ever break. Right now I'm very unconfident that we can guarantee this.

To rectify this I intend to do several things:

• Modify the deployment scripts to create archives for the release artifacts.
• Create a new crate rustup-live-test that just contains tests
  • Its test scenarios work by setting RUSTUP_DIST_ROOT into the live archives to simulate updates over time
  • The dist root is configurable so that it can be run against dev-static.rlo, static.rlo, or the file system
• Create a new crate rustup-build-archives that rebuilds the archives. Use this to backfill the live archives and for local testing.
• Create two git branches master-live-test, stable-live-test, with new CI configs
  • These just run the new tests. We'll use these to test the dev deployment before the stable deployment.

cc @alexcrichton @aturon @Diggsey

This is what I intend to do next. I think before doing any further production deployments.

[Diggsey]

Sounds good, although it wouldn't necessarily catch problems due to state, or side effects of performing several upgrades. (For example, upgrading A -> B inadvertently breaks the rustup configuration in a way not detectable on B, then upgrading B -> C fails due to that error, despite upgrades A -> C and B -> C both working.

[alexcrichton]

@brson you're thinking of gating releases of rustup on this as well?

[brson]

@alexcrichton Yes, but it will be a manual process. We'll do the dev deployment, wait for the CDN to synchronize, run the live test suite against dev-static.rlo. Then do the production deployment (there will be no opportunity to run the tests against static.rlo before it's deployed).

On that note, I want to change the way deployment works so that the stable deployment is just copying from dev-static. Way less opportunity for unexpected failures that way.

[brson]

@Diggsey Oh, I think we can be quite exhaustive with the upgrade scenarios. We can generate tests that upgrade A->B->C->D, A->C->D, A->D, etc. performing various operations to manipulate state along the way.

[alexcrichton]

I guess I'm just thinking that it's nice having the deployment process be so easy today (just push to a branch), and if it gets significantly more complicated than that it may be more difficult to make releases. That being said you'll probably be doing most of the releases so it may be a moot point!

[brson]

@alexcrichton I think I can set this up so deployment only has two steps: build and deploy in dev, then run a script to copy the dev artifacts to prod. The only sacrifice I'll have to make in the fidelity of the tests is that they will need to go directly to the s3 server instead of cloudfront.