Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reducing the memory/bandwidth footprint of submodules with ./x.py? #63978

Open
gendx opened this issue Aug 28, 2019 · 1 comment
Open

Reducing the memory/bandwidth footprint of submodules with ./x.py? #63978

gendx opened this issue Aug 28, 2019 · 1 comment

Comments

@gendx
Copy link

@gendx gendx commented Aug 28, 2019

I recently tried to compile rust by using the ./x.py script. I noticed that the first thing that happens is a full clone of all the git submodules (see https://github.com/rust-lang/rust/blob/master/src/bootstrap/bootstrap.py#L679-L687).

Cloning the whole history of all the submodules is generally of little interest to compile the current version of rust, but it has a non-negligible memory footprint. Today, the largest submodules are:

  • src/llvm-project with 696.10 MiB of history
  • src/doc/rust-by-example with 136.06 MiB of history

Downloading so much data can be detrimental for several reasons:

  • This increases the latency of the build, especially for people with a slow downloading bandwidth.
  • The remote git server takes some time to compress the objects before sending them, also increasing the latency for everyone.
  • The history takes space on disk.

I've tried to tweak the lines in the above bootstrap.py script to limit the depth of the submodule update. Unfortunately, using --depth=1 doesn't work, because the submodule commit referenced here may not be the latest in the submodule repo, and apparently git cannot clone at a specific commit.

However, I found that --depth=20 (and --depth=1 for llvm-project) was working on today's state. Overall, this reduced the overhead of submodule updates quite significantly.

Of course, this isn't bullet-proof:

  • Until git supports cloning a repo at a specific commit, the heuristic depth of 20 may not be enough in the future.
  • I only tested this on a fresh clone of rust-lang/rust. Updating from an existing clone may not work.

Nonetheless, would it make sense to add an option (CLI flag to ./x.py for example) that allows to manually reduce the cloning depth?

@BO41

This comment has been minimized.

Copy link
Contributor

@BO41 BO41 commented Aug 28, 2019

apparently git cannot clone at a specific commit
Until git supports cloning a repo at a specific commit

AFAIK git is able to do this, but github isn't allowing it.

Probably needs a different solution 😞

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.