Should download and install concurrently

[nagisa]

Currently rustup is scarily sequential. It could easily download more stuff while installing the previously downloaded stuff.

[brson]

I don't know if it would be profitable to do multiple network requests in parallel, or multiple file ops in parallel, but it would definitely be profitable to be doing networking while doing file I/O.

Today rustup does installation in two phases: first it acquires all the resources off the network; then it installs them. It does this to eliminate the uncertainty of the network failing during installation.

The actual file I/O though in rustup does have a transactional system that is supposed to be able to roll back, and it definitely does in the test suite, and I've seen it do rollbacks live. I am not super-confident that it is bulletproof, though we could try interleaving downloading and installation and see how it goes.

Adding parallelism here would make status messages nondeterministic and more confusing.

There are definitely opportunities for improvement here, though I'm not sure I'm ready to pull the trigger yet without thinking about the constraints more. If somebody wanted to give it a shot I'd be happy to review.

[nagisa]

I don't know if it would be profitable to do multiple network requests in parallel

I think the reason I proposed this in the first place is that rustc’s CDN is terribly bad with not yet cached stuff. Rereading original report, I guess it wasn’t the reason, but it is still worth doing it IMO.

For reference, on my 100Mbps connection, downloading the tarball for x86_64-unknown-linux-gnu or x86_64-pc-windows-msvc is pretty fast (~8-11MB/s), but downloading artefacts for something like powerpc64-unknown-linux-gnu will run at some abysmal sub-1MB/s (yes, it is THAT terrible). Having 3 parallel network streams (rustc/libstd/cargo) downloading at 3MB/s combined would cut the installation time by 3. Using HTTP range requests could improve the download times even further.

As for output status/messages, maybe something like indicatif could work?