Update submodules in parallel

[Zoxc]

No description provided.

[Zoxc]

This seems to save a couple of seconds.

r? @alexcrichton

[Zoxc]

I changed to code to get tarballs for more submodules. I've seen the time taken to check out submodules go as low as 13 seconds on Travis, it used to take around 90 seconds before. This is probably more helpful for AppVeyor since it takes longer to check out submodules.

This may result in submodules incorrectly assuming that the rustc's repo commit hash is their own. We should check that that doesn't happen.

[alexcrichton]

r? @aidanhs

[aidanhs]

This may result in submodules incorrectly assuming that the rustc's repo commit hash is their own. We should check that that doesn't happen.

Regarding this, if you touch .git inside each of the submodules, it appears that confuses git sufficiently that it will error (hopefully allowing us to catch this kind of error when it happens):

/tmp/tmp.ZwUF7zEhcg $ git status
fatal: Not a git repository (or any of the parent directories): .git
/tmp/tmp.ZwUF7zEhcg $ git init
Initialised empty Git repository in /tmp/tmp.ZwUF7zEhcg/.git/
/tmp/tmp.ZwUF7zEhcg $ git status
On branch master

Initial commit

nothing to commit (create/copy files and use "git add" to track)
/tmp/tmp.ZwUF7zEhcg $ mkdir a
/tmp/tmp.ZwUF7zEhcg $ cd a
/tmp/tmp.ZwUF7zEhcg/a $ git status
On branch master

Initial commit

nothing to commit (create/copy files and use "git add" to track)
/tmp/tmp.ZwUF7zEhcg/a $ touch .git
/tmp/tmp.ZwUF7zEhcg/a $ git status
fatal: Invalid gitfile format: .git
/tmp/tmp.ZwUF7zEhcg/a $ echo $?
128

One example that springs to mind - cargo --version prints a commitish. I assume that means it's one of the ones that needs cloning.

[aidanhs]

There are two parts here:

1. cloning in parallel (-j to submodule init)
2. defaulting to download from a .zip

I love 1! That flag must've been added recently, I don't have it on my version - since this only runs in CI, it's probably fine to require it (we can see if the appveyor git version plays ball).

I understand the motivation of 2 and the results certainly speak for themselves, but I feel a little cautious about an 'on-by-default' approach. Taking an arbitrary recent build log from travis, we can get most of the win just by including a couple more repos to be downloaded as a .zip (book, rust-by-example). Combined with 1, I suspect it'll bring us down to a similar speed but with less additional risk.

[00:00:05] Cloning into '/home/travis/rustsrc/src/src/dlmalloc'...
[00:00:05] Cloning into '/home/travis/rustsrc/src/src/doc/book'...
[00:00:09] Cloning into '/home/travis/rustsrc/src/src/doc/nomicon'...
[00:00:10] Cloning into '/home/travis/rustsrc/src/src/doc/reference'...
[00:00:11] Cloning into '/home/travis/rustsrc/src/src/doc/rust-by-example'...
[00:00:43] Cloning into '/home/travis/rustsrc/src/src/jemalloc'...
[00:00:44] Cloning into '/home/travis/rustsrc/src/src/libcompiler_builtins'...
[00:00:45] Cloning into '/home/travis/rustsrc/src/src/liblibc'...
[00:00:47] Cloning into '/home/travis/rustsrc/src/src/stdsimd'...
[00:00:48] Cloning into '/home/travis/rustsrc/src/src/tools/cargo'...
[00:00:50] Cloning into '/home/travis/rustsrc/src/src/tools/clippy'...
[00:00:52] Cloning into '/home/travis/rustsrc/src/src/tools/lld'...
[00:00:54] Cloning into '/home/travis/rustsrc/src/src/tools/miri'...
[00:00:55] Cloning into '/home/travis/rustsrc/src/src/tools/rls'...
[00:00:56] Cloning into '/home/travis/rustsrc/src/src/tools/rust-installer'...
[00:00:56] Cloning into '/home/travis/rustsrc/src/src/tools/rustfmt'...

[Zoxc]

I've changed it to use git submodules by default.

[Zoxc]

Could we use git submodules only for dist build? This would ensure that artifacts shipped to users don't have the wrong commit hash.

[Zoxc]

I ran this on AppVeyor. I got the following timings (best 3 of 4):

Tarball by default: 36s 42s 44s
Git submodule by default: 64s 77s 80s

[aidanhs]

I'm ok with 30s for the increased peace of mind :)

@bors r+

[bors]

📌 Commit 1578b1e has been approved by aidanhs

[aidanhs]

(thanks for the great work by the way, the slowness of cloning has long been an annoyance of mine)