Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Post job cleanup errors with "argument list too long" in repository with many Cargo.toml/Cargo.lock files #179

Closed
max-heller opened this issue Nov 24, 2023 · 2 comments · Fixed by #180
Closed

Post job cleanup errors with "argument list too long" in repository with many Cargo.toml/Cargo.lock files #179

max-heller opened this issue Nov 24, 2023 · 2 comments · Fixed by #180

Comments

@max-heller
Copy link
Contributor

max-heller commented Nov 24, 2023 •
edited

Problem

I'm using rust-cache in a repository with a large number of Cargo.toml/Cargo.lock files due to including cargo and the rust book as submodules (see e.g. this subdirectory):

Cargo.toml and Cargo.lock files 
Lockfiles considered:
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/.cargo/config.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/benches/benchsuite/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/benches/capture/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/crates/cargo-platform/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/crates/cargo-test-macro/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/crates/cargo-test-support/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/crates/cargo-test-support/containers/apache/bar/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/crates/cargo-test-support/containers/sshd/bar/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/crates/cargo-util/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/crates/crates-io/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/crates/home/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/crates/mdman/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/crates/resolver-tests/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/crates/semver-check/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/crates/xtask-build-man/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/crates/xtask-bump-check/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/crates/xtask-stale-label/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/credential/cargo-credential-1password/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/credential/cargo-credential-libsecret/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/credential/cargo-credential-macos-keychain/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/credential/cargo-credential-wincred/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/credential/cargo-credential/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/tests/testsuite/cargo_add/add_basic/out/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/cargo/tests/testsuite/cargo_add/add_multiple/out/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-03/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-05/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-05/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-06/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-06/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-07/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-07/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-09/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-09/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-10/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-10/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-11/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-11/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-12/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-12/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-13/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-13/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-14/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-14/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-15/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-15/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-16/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-16/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-17/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-17/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-18/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-18/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-19/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-19/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-20/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-20/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-21/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-21/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-22/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-22/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-23/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-23/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-24/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-24/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-25/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/listing-20-25/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/no-listing-01-define-threadpool-struct/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/no-listing-01-define-threadpool-struct/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/no-listing-02-impl-threadpool-new/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/no-listing-02-impl-threadpool-new/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/no-listing-03-define-execute/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/no-listing-03-define-execute/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/no-listing-04-update-worker-definition/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/no-listing-04-update-worker-definition/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/no-listing-05-fix-worker-new/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/no-listing-05-fix-worker-new/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/no-listing-06-fix-threadpool-drop/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/no-listing-06-fix-threadpool-drop/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/no-listing-07-final-code/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/listings/ch20-web-server/no-listing-07-final-code/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-book/rust-toolchain
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-reference/style-check/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rust-reference/style-check/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rustc-dev-guide/ci/date-check/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/books/rustc-dev-guide/ci/date-check/Cargo.toml
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/Cargo.lock
    - /home/runner/work/mdbook-pandoc/mdbook-pandoc/Cargo.toml

These lockfiles aren't relevant to the compilation/caching of my project, since I only build the mdbook contained in each repository.

The number of lockfiles breaks the post-run cleanup step (it works fine without submodules cloned):

Post job cleanup.
Error: An error occurred trying to start process '/home/runner/runners/2.311.0/externals/node16/bin/node' with working directory '/home/runner/work/mdbook-pandoc/mdbook-pandoc'. Argument list too long

Possible fixes

  • Only consider lockfiles for workspace members instead of globbing for all lockfiles within a workspace. This would require parsing the workspace manifests to determine member crates
  • Keep globbing for all lockfiles but pass them to the cleanup script in a way that doesn't result in the argument list too long error
@Swatinem
Copy link
Owner

Thanks for the feedback. We already parse some of the toml files. So might as well also do that for the workspace definition.
I don’t have too much time to dedicate to maintenance right now though.

@max-heller max-heller mentioned this issue Nov 25, 2023
Merged
@max-heller
Copy link
Contributor Author

We already parse some of the toml files. So might as well also do that for the workspace definition.

Turns out there was already logic for calling out to cargo metadata, so all that was required was adding the --no-deps flag to determine workspace members. Opened #180

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Only key by Cargo.toml and Cargo.lock files of workspace members max-heller/rust-cache
2 participants
@Swatinem @max-heller