cargo build --dependencies-only

[nagisa]

There should be an option to only build dependencies.

[KalitaAlexey]

@nagisa,
Why do you want it?

[nagisa]

I do not remember exactly why, but I do remember that I ended just running rustc manually.

[KalitaAlexey]

@posborne, @mcarton, @devyn,
You reacted with thumbs up.
Why do you want it?

[mcarton]

Sometimes you add a bunch of dependencies to your project, know it will take a while to compile next time you cargo build, but want your computer to do that as you start coding so the next cargo build is actually fast.
But I guess I got here searching for a cargo doc --dependencies-only, which allows you to get the doc of your dependencies while your project does not compile because you'd need the doc to know how exactly to fix that compilation error you've had for a half hour 😄

[gregwebs]

As described in #3615 this is useful with build to setup a cache of all dependencies.

[alexcrichton]

@gregwebs out of curiosity do you want to cache compiled dependencies or just downloaded dependencies? Caching compiled dependencies isn't implemented today (but would be with a command such as this) but downloading dependencies is available via cargo fetch.

[gregwebs]

Generally, as with my caching use case, the dependencies change infrequently and it makes sense to cache the compilation of them.

The Haskell tool stack went through all this and they seemed to generally decided to merge things into a single command where possible. For fetch they did end up with something kinda confusing though: build --dry-run --prefetch. For build --dependencies-only mentioned here they do have the same: build --only-dependencies

[alexcrichton]

@gregwebs ok thanks for the info!

[KalitaAlexey]

@alexcrichton,
It looks like I should continue my work on the PR.
Will Cargo's team accept it?

[alexcrichton]

@KalitaAlexey I personally wouldn't be convinced just yet, but it'd be good to canvas opinions from others on @rust-lang/tools as well

[KalitaAlexey]

@alexcrichton,
Anyway I have no time right now)

[nrc]

I don't see much of a use case - you can just do cargo build and ignore the output for the last crate. If you really need to do this (for efficiency) then there is API you can use.

[gregwebs]

What's the API?

[nrc]

Implement an Executor. That lets you intercept every call to rustc and you can do nothing if it is the last crate.

[gregwebs]

I wasn't able to find any information about an Executor for cargo. Do you have any links to documentation?

[nrc]

Docs are a little thin, but start here:

cargo/src/cargo/ops/cargo_rustc/mod.rs

Lines 62 to 64 in 609371f

	/// A glorified callback for executing calls to rustc. Rather than calling rustc
	/// directly, we'll use an Executor, giving clients an opportunity to intercept
	/// the build calls.

You can look at the RLS for an example of how to use them: https://github.com/rust-lang-nursery/rls/blob/master/src/build.rs#L288

[shepmaster]

A question of Stack Overflow wanted this feature. In that case, the OP wanted to build the dependencies for a Docker layer.

A similar situation exists for the playground, where I compile all the crates once. In my case, I just put in a dummy lib.rs / main.rs. All the dependencies are built, and the real code is added in the future.

[alexcrichton]

@shepmaster unfortunately the proposed solution wouldn't satisfy that question because a Cargo.toml won't parse without associated files in src (e.g. src/lib.rs, etc). So that question would still require "dummy files", in which case it wouldn't specifically be serviced by this change.

[lolgesten]

I ended up here because I also am thinking about the Docker case. To do a good docker build I want to:

COPY Cargo.toml Cargo.lock /mything

RUN cargo build-deps --release  # creates a layer that is cached

COPY src /mything/src

RUN cargo build --release       # only rebuild this when src files changes

This means the dependencies would be cached between docker builds as long as Cargo.toml and Cargo.lock doesn't change.

I understand src/lib.rs src/main.rs are needed to do a good build, but maybe build-deps simply builds all the deps.

[ghost]

The dockerfile template in shepmaster's linked stackoverflow post above SOLVES this problem

I came to this thread because I also wanted the docker image to be cached after building the dependencies. After later resolving this issue, I posted something explaining docker caching, and was informed that the answer was already linked in the stackoverflow post. I made this mistake, someone else made this mistake, it's time to clarify.

RUN cd / && \
    cargo new playground
WORKDIR /playground                      # a new project has a src/main.rs file

ADD Cargo.toml /playground/Cargo.toml 
RUN cargo build                          # DEPENDENCIES ARE BUILD and CACHED
RUN cargo build --release
RUN rm src/*.rs                          # delete dummy src files

# here you add your project src to the docker image

After building, changing only the source and rebuilding starts from the cached image with dependencies already built.

[lolgesten]

someone needs to relax...

[lolgesten]

Also @karlfish what you're proposing is not actually working. If using FROM rust:1.20.0.

1. cargo new playground fails because it wants USER env variable to be set.
2. RUN cargo build does not build dependencies for release, but for debug. why do you need that?

[lolgesten]

Here's a better version.

FROM rust:1.20.0

WORKDIR /usr/src

# Create blank project
RUN USER=root cargo new umar

# We want dependencies cached, so copy those first.
COPY Cargo.toml Cargo.lock /usr/src/umar/

WORKDIR /usr/src/umar

# This is a dummy build to get the dependencies cached.
RUN cargo build --release

# Now copy in the rest of the sources
COPY src /usr/src/umar/src/

# This is the actual build.
RUN cargo build --release \
    && mv target/release/umar /bin \
    && rm -rf /usr/src/umar

WORKDIR /

EXPOSE 3000

CMD ["/bin/umar"]

[shepmaster]

You can always review the complete Dockerfile for the playground.

[maelvls]

Hi!
What is the current state of the --deps-only idea? (mainly for dockerization)

[AdrienneCohea]

I agree that it would be really cool to have a --deps-only option so that we could cache our filesystem layers better in Docker.

I haven't tried replicating this yet, but it looks very promising. This is in glibc and not musl, by the way. My main priority is to get to a build that doesn't take 3-5 minutes ever time, not a 5 MB alpine-based image.

[trezm]

I've been lurking on this thread for quite a while, and might have a helpful tidbit for those looking for a docker workaround. I did a scan and couldn't see anywhere this has been suggested, but the following seems to work for caching deps for me in a docker build:

# Create a fake main file
RUN echo "fn main() {println!(\"if you see this, the build broke\")}" > src/main.rs
# copy over your manifests
COPY ./Cargo.toml ./Cargo.toml

# this build step will cache your dependencies
RUN cargo build --release

COPY ./src ./src
COPY build.rs build.rs

RUN cargo build --release

RUN cp target/release/pepsi /usr/local/cargo/bin/pepsi

Pepsi is just the codename for a project of mine, and build.rs is essentially a single println in a main function. For some reason the build.rs is necessary in order to do an actual cache bust for the binary.

Anyway, not looking to comment either way, just thought I'd leave this here for docker folks looking for a quick workaround!

[camerondavison]

@trezm I have been playing with the same docker commands but ran into some strange thing where it is not rebuilding the binary. Does it have to do with timestamps of the main.rs file? Why did the build rs fix the problem?

[trezm]

@camerondavison do be 100% honest, I'm really not sure why the build.rs affects the result! I came across it as a solution by accident. I had a project that I have a dockerfile for that uses tonic, so I needed a build.rs to build proto files.

I noticed that I could super simplify the Dockerfile for that project, but when I copied the same Dockerfile to a new project (sans build.rs) it no longer worked because of what you mentioned. I added the build.rs file back with a single dummy println and it seemed to work again!

Really not sure why this would change anything, maybe something to do with the presence of a build.rs file forcing cargo to look at the actual timestamp on all the files? This is me wildly speculating, but I encourage you to try adding a dummy build.rs and see if you get the same results!

[camerondavison]

@trezm adding a build.rs file with

fn main() {}

does fix the problem. so crazy. I am going to keep trying to figure out why.

EDIT: found a solution
because the file that is created is newer than the source code rust thinks that the build artifacts that get created are newer than the source code. I was able to fix this problem by touching the build artifacts for just my source code before doing the next build. not sure if that is better than using the build.rs or not. this is the dockerfile I ended up with

FROM rust:1.44 as build

# app
ENV app=my-app

# dependencies
WORKDIR /tmp/${app}
COPY Cargo.toml Cargo.lock ./

# compile dependencies
RUN set -x\
 && mkdir -p src\
 && echo "fn main() {println!(\"broken\")}" > src/main.rs\
 && cargo build --release

# copy source and rebuild
COPY src/ src/
RUN set -x\
 && find target/release -type f -name "$(echo "${app}" | tr '-' '_')*" -exec touch -t 200001010000 {} +\
 && cargo build --release

[vn971]

@camerondavison adding to your work-around, you can replace pre-last line with: find target/release/ -type f -executable -maxdepth 1 -delete

[anitnilay20]

hey @trezm @camerondavison doesn't work if there are multiple workspaces. 😢

[trezm]

Bummer! Does @vn971's solution work? I don't claim to have any substantial knowledge about cargo, I was just hopefully helping with a workaround :)

[vn971]

@anitnilay20 just copy the needed Cargo.* files only, and otherwise repeat the steps? Should work?

[anitnilay20]

@trezm Solution does work for single workspace cargo projects but it doesn't work for multiple workspaces..

[vn971]

Let's better raise a StackOverflow question or something. This ticket is to add support for the flag. I think multiple workspaces work if done correctly, in the right order.

[anitnilay20]

@vn971 it worked.. issue was bad Dockerfile. anyway thanx for your help.

[dessalines]

Does anyone have a working version that can cache deps for workspaces too?

[anitnilay20]

@dessalines You can check this https://github.com/sha-el/Vouch/blob/master/Dockerfile

[dessalines]

@anitnilay20 Thank you so much, that did it!

It does re-fetch the workspace deps when anything changes in there, but any changes in the main project don't re-fetch the workspace deps!

[jorgecarleitao]

I have been dealing with the issue over a year now, both in libs, bins, and workspaces, with projects with and without examples and benches, mostly in docker images. Here are my 2 cents:

The underlying consequence is the too high compilation time of slowly-varying code in the context of CI/CD. Specifically, there is a number of rust projects whose majority of their CI/CD time is spent compiling non-changing code, which can be cached (e.g. check logs of arrow/rust or its issues within docker/CI).

The core issue is that, in the context of containerization (e.g. docker), engines have no way of knowing that the resulting artifacts in target/*/deps are independent of src/ after RUN cargo build. As such, they must invalidate all artifacts in target/*/deps when src/ changes (e.g. via a COPY . . before RUN). This is the reason why cargo build is not sufficient: we need a command to

build the stuff in target/*/deps using the dependants of target/*/deps

Cargo supports this locally by not recompiling dependencies that are already compiled. However, it does this because it knows that target/*/deps is independent of src/; a generic technology like containerization has no way of knowing this.

In theory, solving this issue requires two different API endpoints:

• a command to get the list of dependants (which we have been assuming to be only Cargo.toml, src/[lib.rs/main.rs] and optionally build.rs)
• a command to build target/*/deps

IMO this is issue a sub-part of one of the top 5 requested improvements to rust - compilation time: slow compilation is not only addressed by producing better plans for the LLVM, but also providing the tools to efficiently cache slowly varying code, such as external dependencies.

Finally, why the current status quo and "replacing lib.rs/main.rs for a dummy" is not enough:

non-workspace projects

This approach works well for simple projects, but does not work for projects with examples or benches. Essentially, given a Cargo.toml with a [[bench]]/[[example]], cargo build [--lib] [--release] requires the respective file benches/examples to be there. This implies that we also need to copy the benches and examples (which causes a recompilation on every change of example), or parse the Cargo.toml and touch a file for every [[bench]]/[[example]] on it (high maintenance).

workspace projects

When there are inter-workspace dependencies, say w1 <- w2, we can't compile w2's dependencies because it requires compiling w1 dependencies, like @shepmaster mentioned.

One current "solution" for this is to repeat the step for non-workspace projects on each workspace including examples/benches (cumbersome).

In the end, a command like cargo build --dependencies-only would collect the necessary files required to build targets/*/deps based on Cargo.tomls and would execute the download/compile to achieve that final state, in the same way that the hacks aforementioned currently achieve with different levels of "hackiness".

[Dushistov]

@jorgecarleitao

The underlying consequence is the too high compilation time of slowly-varying code in the context of CI/CD

Why you can not use sccache for that?
It also automatically handle changes of versions of dependencies, compiler version changes and so on.

[jorgecarleitao]

@jorgecarleitao

The underlying consequence is the too high compilation time of slowly-varying code in the context of CI/CD

Why you can not use sccache for that?
It also automatically handle changes of versions of dependencies, compiler version changes and so on.

We have:

This is complex, and doesn't seem to be helping IRL, so I'm closing this

[technimad]

@jorgecarleitao
Why you can not use sccache for that?
It also automatically handle changes of versions of dependencies, compiler version changes and so on.

In our case we don’t go for sccache because:

• it adds yet another tool to our stack, which comes with a maintenance and knowledge cost.
• it isn’t supported natively by our cloud CI (Bitbucket)

Our product consists of multiple components written in different languages. We support this with a small team, so we stick to native tooling and defaults as much as possible.

[parisholley]

for those who have the benefit of a buildkit based docker CI process... this is how i've been able to achieve < 1 minute test builds and < 2 minute release builds

For Testing:

# define dependencies for temporary build
COPY src/Cargo.toml src/Cargo.toml
COPY test/Cargo.toml test/Cargo.toml
COPY Cargo.lock Cargo.lock
COPY Cargo.toml Cargo.toml

# dummy files so we can compile and build depenencies
RUN echo "fn main(){}" > src/lib.rs
RUN echo "fn main(){}" > test/lib.rs

# cache dependency compilation
RUN --mount=type=cache,target=/root/.cargo/registry/ RUSTFLAGS="-C target-feature=-crt-static" cargo test --package test

# remove dummy files and compilation cache (not dependency cache)
RUN rm -rf src test
RUN rm -rf target/debug/**/libsrc*
RUN rm -rf target/debug/**/libtest*

# copy actual src over that tends to change more often than dependencies
COPY build/cdn build/cdn
COPY test test
COPY src src

# run another build using our actual source code
RUN RUSTFLAGS="-C target-feature=-crt-static" cargo test --package test

For release/deployment

# dummy files so we can compile and build depenencies
RUN mkdir src && echo "fn main(){}" > src/lib.rs
RUN echo "[workspace]" > Cargo.toml && echo "members = [\"src\"] " >> Cargo.toml

# define dependencies for temporary build
COPY src/Cargo.toml src/Cargo.toml
COPY Cargo.lock src/Cargo.lock

# cache dependency compilation
RUN --mount=type=cache,target=/root/.cargo/registry/ RUSTFLAGS="-C target-feature=-crt-static"  cargo build --release --lib

# remove dummy files and compilation cache (not dependency cache)
RUN rm -rf src
RUN rm -rf target/release/**/libsrc*

# copy actual src over that tends to change more often than dependencies
COPY build/cdn build/cdn
COPY src src

# run another build using our actual source code
RUN RUSTFLAGS="-C target-feature=-crt-static" cargo build --release

FROM release as bin1

WORKDIR /rust

COPY src/cfg src/cfg
COPY --from=release /rust/target/release/bin1 bin1

ENTRYPOINT ["/rust/bin1"]

FROM release as bin2

WORKDIR /rust

COPY src/cfg src/cfg
COPY --from=release /rust/target/release/bin2 bin2

ENTRYPOINT ["/rust/bin2"]

[gilescope]

Having lost more of my life than I care to mention trying to get non-trivial incremental compilation working in something docker-like, I have come to the conclusion that only by removing mtime from the picture can we achieve stable caching. I think we need #6529 to be solved before there's any point in having a command that builds just the dependencies (for which we have reasonable workarounds). Currently there's no way to work around the mtime dependency.

[dessalines]

I figured out how to get this also working with cargo workspaces, using romac's fork of cargo-build-deps.

This example has my_app, and two workspaces: utils and db.

FROM rust:nightly as rust

# Cache deps
WORKDIR /app
RUN sudo chown -R rust:rust .
RUN USER=root cargo new myapp

# Install cache-deps
RUN cargo install --git https://github.com/romac/cargo-build-deps.git

WORKDIR /app/myapp
RUN mkdir -p db/src/ utils/src/

# Copy the Cargo tomls
COPY myapp/Cargo.toml myapp/Cargo.lock ./
COPY myapp/db/Cargo.toml ./db/
COPY myapp/utils/Cargo.toml ./utils/

# Cache the deps
RUN cargo build-deps

# Copy the src folders
COPY myapp/src ./src/
COPY myapp/db/src ./db/src/
COPY myapp/utils/src/ ./utils/src/

# Build for debug
RUN cargo build

[uselessscat]

In python there is a file called dependencies.txt, So in your dockerfile you do something like this:

WORKDIR /my-stuff 
COPY dev.dependencies.txt dependencies.txt ./

# in dependencies i declared fastapi that install uvicorn
RUN  pip install -r dev.dependencies.txt -r dependencies.txt

COPY . .

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080", "--reload"]

So you dont have to download all dependencies everytime you changes the code. With Rust Im unable to generate compilation cache and build without creating a dummy project. It would be great something like:

WORKDIR /my-stuff

COPY Cargo.toml Cargo.lock ./
# this runs cargo fetch, cargo install, and finally cargo build
RUN cargo cache build

COPY . .
RUN cargo build

CMD ["cargo", "watch", "-x", "run"]

Then cargo cache can contain clean and build subcommands (or generate, or something that describes the action of building the cache)
Look at #5120

[vojtechkral]

@uselessscat This is why I created #8362 , I believe that's more or less what you mean...

[LukeMathWalker]

As mentioned in #8362, I believe cargo-chef is a MVP of what this could look like from an API perspective.
A sample Dockerfile looks like this:

FROM rust as planner
WORKDIR app
RUN cargo install cargo-chef 
COPY . .
RUN cargo chef prepare  --recipe-path recipe.json

FROM rust as cacher
WORKDIR app
RUN cargo install cargo-chef
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json

FROM rust as builder
WORKDIR app
COPY . .
COPY --from=cacher /app/target target
RUN cargo build --release --bin app

FROM rust as runtime
WORKDIR app
COPY --from=builder /app/target/release/app /usr/local/bin
ENTRYPOINT ["./usr/local/bin/app"]

It does not require Buildkit.
I still believe this functionality should somehow find its way into cargo, but hopefully we can explore the space a bit with cargo-chef outside of it to understand the tradeoffs and actual usage patterns.

[dessalines]

@LukeMathWalker this is much cleaner to have the recipe planning inside docker as well... you should make this the default example in the cargo-chef readme.

[LukeMathWalker]

I thought I did, but it turns I had not pushed the commit to remote 😅 Updated! @dessalines