Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cargo eating all RAM during aarch64 build under Docker on x86-64 #10583

Open
nazar-pc opened this issue Apr 20, 2022 · 9 comments
Open

Cargo eating all RAM during aarch64 build under Docker on x86-64 #10583

nazar-pc opened this issue Apr 20, 2022 · 9 comments
Labels
A-git Area: anything dealing with git C-bug Category: bug Performance Gotta go fast! S-blocked-external Status: ❌ blocked on something out of the direct control of the Cargo project, e.g., upstream fix

Comments

@nazar-pc
Copy link

nazar-pc commented Apr 20, 2022

Problem

I'm trying to build an application, this time to build aarch64 container image on x86-64 machine.

My machine is a beefy 5900X with 128G of RAM, but my system runs out of memory (Cargo eats it all) when cross-compiling aarch64 container.

Steps

  1. Clone this branch: https://github.com/nazar-pc/subspace/tree/cargo-memory-bug-aarch64
  2. Have Docker installed on Linux x86-64 machine
  3. Run docker buildx build --platform linux/arm64 -t test -f Dockerfile-farmer .
  4. Observe system freezing/crashing because of running out of memory

Possible Solution(s)

No response

Notes

x86-64 build on the same machine works fine with ~20G of system memory.

Not entirely sure if this is Cargo's fault or QEMU or something else, but I used similar QEMU-based setups before and never seen anything remotely like this.

Version

cargo 1.60.0-nightly (c082648 2022-02-08)
release: 1.60.0-nightly
commit-hash: c082648646cbb2be266df9ecbcdc253058158d68
commit-date: 2022-02-08
host: aarch64-unknown-linux-gnu
libgit2: 1.3.0 (sys:0.13.23 vendored)
libcurl: 7.80.0-DEV (sys:0.4.51+curl-7.80.0 vendored ssl:OpenSSL/1.1.1m)
os: Linux [64-bit]

UPD: Tried the most recent version, still the same issue:

cargo 1.62.0-nightly (dba5baf 2022-04-13)
release: 1.62.0-nightly
commit-hash: dba5baf4345858c591517b24801902a062c399f8
commit-date: 2022-04-13
host: aarch64-unknown-linux-gnu
libgit2: 1.4.2 (sys:0.14.2 vendored)
libcurl: 7.80.0-DEV (sys:0.4.51+curl-7.80.0 vendored ssl:OpenSSL/1.1.1m)
os: Linux [64-bit]
@nazar-pc nazar-pc added the C-bug Category: bug label Apr 20, 2022
@epage
Copy link
Contributor

epage commented Apr 20, 2022

#8405 is one potential solution for this.

@nazar-pc
Copy link
Author

Hm, I feel like this is something different because I only observe this issue in above scenario with QEMU-based emulation. Let me know if I can add some instrumentation to help debugging this (though reproduction steps should be reliable too as everything is containerized).

@nazar-pc
Copy link
Author

I'm fairly certain that memory consumption happens during repo cloning (dependencies from git), before doing any compilation

@weihanglo
Copy link
Member

If it is related to git clone, perhaps set the config net.git-fetch-with-cli = true to see whether libgit2 is the suspect.

@nazar-pc
Copy link
Author

Thank you @weihanglo, it helped!

@Flawm
Copy link

Flawm commented Aug 31, 2022

I'm seeing the exact same behavior as OP, however setting the net git fetch flag didn't seem to reduce the memory consumption

In my case, I was able to get around it by binding my cargo caches into the container:

    -v $HOME/.cargo/git:/usr/local/cargo/git \
    -v $HOME/.cargo/registry:/usr/local/cargo/registry \

I suppose the docker-compose.yml equivalent would work as well, but in some CI scenarios this may not be possible

@weihanglo weihanglo removed the A-external-dependencies Area: dependencies on things outside of cargo label Oct 11, 2022
MichaIng added a commit to MichaIng/DietPi that referenced this issue Oct 16, 2022
- CI | Another attempt to fix vaultwarden builds on emulated aarch64: rust-lang/cargo#10583
MichaIng added a commit to MichaIng/DietPi that referenced this issue Oct 16, 2022
- GMediaRender | Updated to version 0.0.9 and aligned service name with Debian and upstream service and executable name. The update can be applied via reinstall: dietpi-software reinstall 163
- CI | Fix vaultwarden builds on emulated aarch64: rust-lang/cargo#10583
- Other minor updates to new software packages
mtekman pushed a commit to mtekman/DietPi that referenced this issue Oct 31, 2022
- GMediaRender | Updated to version 0.0.9 and aligned service name with Debian and upstream service and executable name. The update can be applied via reinstall: dietpi-software reinstall 163
- CI | Fix vaultwarden builds on emulated aarch64: rust-lang/cargo#10583
- Other minor updates to new software packages
alexpdp7 added a commit to alexpdp7/ragent that referenced this issue Nov 10, 2022
alexpdp7 added a commit to alexpdp7/ragent that referenced this issue Nov 10, 2022
alexpdp7 added a commit to alexpdp7/ragent that referenced this issue Nov 10, 2022
* Workaround rust-lang/cargo#10583

* Bump hyper from 0.14.20 to 0.14.22

Bumps [hyper](https://github.com/hyperium/hyper) from 0.14.20 to 0.14.22.
- [Release notes](https://github.com/hyperium/hyper/releases)
- [Changelog](https://github.com/hyperium/hyper/blob/v0.14.22/CHANGELOG.md)
- [Commits](hyperium/hyper@v0.14.20...v0.14.22)

---
updated-dependencies:
- dependency-name: hyper
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump clap from 4.0.18 to 4.0.22

Bumps [clap](https://github.com/clap-rs/clap) from 4.0.18 to 4.0.22.
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](clap-rs/clap@v4.0.18...v4.0.22)

---
updated-dependencies:
- dependency-name: clap
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Run cargo update

* (cargo-release) version 1.2.8

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
TeoZosa added a commit to TeoZosa/structlog-sentry-logger that referenced this issue Dec 18, 2022
Qqwy added a commit to snappy-cobra/WiiDevEnv that referenced this issue Dec 19, 2022
Fix found here: rust-lang/cargo#10583
This problem seems to exist on aarch64 building for amd64 and vice-versa.
tk-nguyen added a commit to tk-nguyen/speedmetrics that referenced this issue Jan 1, 2023
MatthieuCoder added a commit to discordnova/Nova that referenced this issue Jan 16, 2023
@msrd0
Copy link

msrd0 commented Mar 5, 2023

I just ran into this issue as well trying to execute cargo fetch on an alpine aarch64 guest system running on an x86_64 archlinux host. It never made it past cloning the crates.io index.

@ThomasWaldmann
Copy link

Maybe related: pyca/cryptography#8640

wofferl added a commit to wofferl/proxmox-backup-arm64 that referenced this issue Mar 31, 2023
stefano-garzarella added a commit to stefano-garzarella/rust-vmm-container that referenced this issue Apr 7, 2023
libgit2 can consume a lot of memory when cross-compiling for
arm64. As suggested here [1], let's use the git executable
to prevent this issue.

[1] rust-lang/cargo#10583 (comment)

Fixes: rust-vmm#79
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
andreeaflorescu pushed a commit to rust-vmm/rust-vmm-container that referenced this issue Apr 11, 2023
libgit2 can consume a lot of memory when cross-compiling for
arm64. As suggested here [1], let's use the git executable
to prevent this issue.

[1] rust-lang/cargo#10583 (comment)

Fixes: #79
Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
@weihanglo weihanglo added S-blocked-external Status: ❌ blocked on something out of the direct control of the Cargo project, e.g., upstream fix Performance Gotta go fast! labels Apr 21, 2023
bors bot pushed a commit to sigp/lighthouse that referenced this issue Oct 19, 2023
…ing cross-compiling (#4828)

## Issue Addressed
#4827 

## Proposed Changes

This PR introduces a new build-arg to the Lighthouse Dockerfile: `CARGO_USE_GIT_CLI`. This arg will be passed into the `CARGO_NET_GIT_FETCH_WITH_CLI` [environment variable](https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli), which instructs `cargo` to use the git CLI during `fetch` operations instead of the git library. Doing so works around [a bug](rust-lang/cargo#10583) with the git library that causes it to go OOM during `fetch` operations on `arm64` platforms.

The default value is `false` so this doesn't affect Lighthouse builds or the CI pipeline. Running a build with `--build-arg CARGO_USE_GIT_CLI=true` will activate it, which is necessary to cross-compile the `arm64` binary when not using `cross` (i.e., when building via the Dockerfile instead of natively if you don't have a rust environment ready to go).

Special thanks to @michaelsproul for helping me repro the initial problem.

Co-authored-by: Michael Sproul <micsproul@gmail.com>
ThomasBrinker pushed a commit to Elektrobit/rupdate that referenced this issue Mar 4, 2024
Cargo uses libgit2, which causes issues while building for aarch64 targets:
rust-lang/cargo#10583

As this is one of many errors caused by libgit2, it is adviced to use the local git binary:
https://docs.shipyard.rs/configuration/git-fetch-with-cli.html
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-git Area: anything dealing with git C-bug Category: bug Performance Gotta go fast! S-blocked-external Status: ❌ blocked on something out of the direct control of the Cargo project, e.g., upstream fix
Projects
None yet
Development

No branches or pull requests

7 participants