Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter out cargo dependencies for unsupported platforms #5475

Open
vashirov opened this issue Oct 5, 2022 · 1 comment
Open

Filter out cargo dependencies for unsupported platforms #5475

vashirov opened this issue Oct 5, 2022 · 1 comment
Labels
needs triage The issue will be triaged during scrum

Comments

@vashirov
Copy link
Member

vashirov commented Oct 5, 2022

Issue Description
cargo vendor downloads all dependencies including for Windows. For example, our vendor directory includes winapi crate and libraries required by ansi_term:

$ du -hs vendor/winapi* 
7.6M    vendor/winapi
52M     vendor/winapi-i686-pc-windows-gnu
54M     vendor/winapi-x86_64-pc-windows-gnu

cargo vendor doesn't currently support filtering out platform-specific dependencies: rust-lang/cargo#7058
But there is a cargo-vendor-filterer project:
rust-lang/cargo#7058 (comment)
https://github.com/coreos/cargo-vendor-filterer

It should shave about 5MB from the size of the compressed released tarball:

$ du 389-ds-base-latest.tar.bz2 
14M     389-ds-base-latest.tar.bz2

vs.

$ du 389-ds-base-latest_filtered.tar.bz2
8.6M    389-ds-base-latest_filtered.tar.bz2
@vashirov vashirov added the needs triage The issue will be triaged during scrum label Oct 5, 2022
@vashirov
Copy link
Member Author

vashirov commented Oct 5, 2022

One issue I found is with Cargo.lock file:

SLAPD_HEADER_DIR=/home/vashirov/src/ds/rpmbuild/BUILD/389-ds-base-2.3.0.202210051402gitcf4c82c21/ \
        cargo rustc --locked --offline --manifest-path=./src/librnsslapd/Cargo.toml \
        --release --verbose -- -C debuginfo=2 
    Blocking waiting for file lock on package cache
error: the lock file /home/vashirov/src/ds/rpmbuild/BUILD/389-ds-base-2.3.0.202210051402gitcf4c82c21/src/Cargo.lock needs to be updated but --locked was passed to prevent this
If you want to try to generate the lock file without accessing the network, remove the --locked flag and use --offline instead.
make[1]: *** [Makefile:12983: /home/vashirov/src/ds/rpmbuild/BUILD/389-ds-base-2.3.0.202210051402gitcf4c82c21/rs/rnsslapd/release/librnsslapd.a] Error 101
make[1]: *** Waiting for unfinished jobs....
error: the lock file /home/vashirov/src/ds/rpmbuild/BUILD/389-ds-base-2.3.0.202210051402gitcf4c82c21/src/Cargo.lock needs to be updated but --locked was passed to prevent this
If you want to try to generate the lock file without accessing the network, remove the --locked flag and use --offline instead.
make[1]: *** [Makefile:12972: /home/vashirov/src/ds/rpmbuild/BUILD/389-ds-base-2.3.0.202210051402gitcf4c82c21/rs/rslapd/release/librslapd.a] Error 101
make[1]: Leaving directory '/home/vashirov/src/ds/rpmbuild/BUILD/389-ds-base-2.3.0.202210051402gitcf4c82c21'
error: Bad exit status from /var/tmp/rpm-tmp.MYjwuT (%build)

If I remove --locked from the build options, it builds fine. Or I can modify the Cargo.lock file to remove the dependencies that were filtered out. But next time someone runs cargo update or cargo vendor without cargo vendor-filterer, it will overwrite Cargo.lock file to again include those dependencies.

Another issue is missing support for --manifest-path: coreos/cargo-vendor-filterer#31
It's not critical and can be worked around.

So until cargo vendor supports filtering natively, we might run into issues like these.

Here's the diff if someone wants to play with this:

diff --git a/rpm.mk b/rpm.mk                                                                                                                       
index ed72ee567..c3b40c7ed 100644
--- a/rpm.mk
+++ b/rpm.mk
@@ -38,9 +38,9 @@ update-cargo-dependencies:
  
 download-cargo-dependencies:
        cargo update --manifest-path=./src/Cargo.toml
-       cargo vendor --manifest-path=./src/Cargo.toml
-       cargo fetch --manifest-path=./src/Cargo.toml
-       tar -czf vendor.tar.gz vendor
+       cd src; cargo vendor-filterer --platform=x86_64-unknown-linux-gnu --prefix=vendor --format=tar.gz
+       mv src/vendor.tar.gz .
+       tar -xf vendor.tar.gz
  
 bundle-rust:
        python3 rpm/bundle-rust-downstream.py ./src/Cargo.lock $(DS_SPECFILE) ./vendor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs triage The issue will be triaged during scrum
Projects
None yet
Development

No branches or pull requests

1 participant