Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cargo build --dependencies-only #2644

Open
nagisa opened this issue May 4, 2016 · 241 comments
Open

cargo build --dependencies-only #2644

nagisa opened this issue May 4, 2016 · 241 comments

Comments

@nagisa
Copy link

@nagisa nagisa commented May 4, 2016

There should be an option to only build dependencies.

@KalitaAlexey
Copy link
Contributor

@KalitaAlexey KalitaAlexey commented Jan 17, 2017

@nagisa,
Why do you want it?

@nagisa
Copy link
Author

@nagisa nagisa commented Jan 17, 2017

I do not remember exactly why, but I do remember that I ended just running rustc manually.

@KalitaAlexey
Copy link
Contributor

@KalitaAlexey KalitaAlexey commented Jan 17, 2017

@posborne, @mcarton, @devyn,
You reacted with thumbs up.
Why do you want it?

@mcarton
Copy link
Member

@mcarton mcarton commented Jan 17, 2017

Sometimes you add a bunch of dependencies to your project, know it will take a while to compile next time you cargo build, but want your computer to do that as you start coding so the next cargo build is actually fast.
But I guess I got here searching for a cargo doc --dependencies-only, which allows you to get the doc of your dependencies while your project does not compile because you'd need the doc to know how exactly to fix that compilation error you've had for a half hour 😄

@gregwebs
Copy link

@gregwebs gregwebs commented Jan 30, 2017

As described in #3615 this is useful with build to setup a cache of all dependencies.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Jan 31, 2017

@gregwebs out of curiosity do you want to cache compiled dependencies or just downloaded dependencies? Caching compiled dependencies isn't implemented today (but would be with a command such as this) but downloading dependencies is available via cargo fetch.

@gregwebs
Copy link

@gregwebs gregwebs commented Jan 31, 2017

Generally, as with my caching use case, the dependencies change infrequently and it makes sense to cache the compilation of them.

The Haskell tool stack went through all this and they seemed to generally decided to merge things into a single command where possible. For fetch they did end up with something kinda confusing though: build --dry-run --prefetch. For build --dependencies-only mentioned here they do have the same: build --only-dependencies

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Jan 31, 2017

@gregwebs ok thanks for the info!

@KalitaAlexey
Copy link
Contributor

@KalitaAlexey KalitaAlexey commented Jan 31, 2017

@alexcrichton,
It looks like I should continue my work on the PR.
Will Cargo's team accept it?

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Feb 2, 2017

@KalitaAlexey I personally wouldn't be convinced just yet, but it'd be good to canvas opinions from others on @rust-lang/tools as well

@KalitaAlexey
Copy link
Contributor

@KalitaAlexey KalitaAlexey commented Feb 2, 2017

@alexcrichton,
Anyway I have no time right now)

@nrc
Copy link
Member

@nrc nrc commented Feb 2, 2017

I don't see much of a use case - you can just do cargo build and ignore the output for the last crate. If you really need to do this (for efficiency) then there is API you can use.

@gregwebs
Copy link

@gregwebs gregwebs commented Feb 4, 2017

What's the API?

@nrc
Copy link
Member

@nrc nrc commented Feb 6, 2017

Implement an Executor. That lets you intercept every call to rustc and you can do nothing if it is the last crate.

@gregwebs
Copy link

@gregwebs gregwebs commented Feb 6, 2017

I wasn't able to find any information about an Executor for cargo. Do you have any links to documentation?

@nrc
Copy link
Member

@nrc nrc commented Feb 6, 2017

Docs are a little thin, but start here:

/// A glorified callback for executing calls to rustc. Rather than calling rustc
/// directly, we'll use an Executor, giving clients an opportunity to intercept
/// the build calls.

You can look at the RLS for an example of how to use them: https://github.com/rust-lang-nursery/rls/blob/master/src/build.rs#L288

@shepmaster
Copy link
Member

@shepmaster shepmaster commented Feb 10, 2017

A question of Stack Overflow wanted this feature. In that case, the OP wanted to build the dependencies for a Docker layer.

A similar situation exists for the playground, where I compile all the crates once. In my case, I just put in a dummy lib.rs / main.rs. All the dependencies are built, and the real code is added in the future.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Feb 10, 2017

@shepmaster unfortunately the proposed solution wouldn't satisfy that question because a Cargo.toml won't parse without associated files in src (e.g. src/lib.rs, etc). So that question would still require "dummy files", in which case it wouldn't specifically be serviced by this change.

@lolgesten
Copy link

@lolgesten lolgesten commented Oct 9, 2017

I ended up here because I also am thinking about the Docker case. To do a good docker build I want to:

COPY Cargo.toml Cargo.lock /mything

RUN cargo build-deps --release  # creates a layer that is cached

COPY src /mything/src

RUN cargo build --release       # only rebuild this when src files changes

This means the dependencies would be cached between docker builds as long as Cargo.toml and Cargo.lock doesn't change.

I understand src/lib.rs src/main.rs are needed to do a good build, but maybe build-deps simply builds all the deps.

@ghost
Copy link

@ghost ghost commented Oct 9, 2017

The dockerfile template in shepmaster's linked stackoverflow post above SOLVES this problem

I came to this thread because I also wanted the docker image to be cached after building the dependencies. After later resolving this issue, I posted something explaining docker caching, and was informed that the answer was already linked in the stackoverflow post. I made this mistake, someone else made this mistake, it's time to clarify.

RUN cd / && \
    cargo new playground
WORKDIR /playground                      # a new project has a src/main.rs file

ADD Cargo.toml /playground/Cargo.toml 
RUN cargo build                          # DEPENDENCIES ARE BUILD and CACHED
RUN cargo build --release
RUN rm src/*.rs                          # delete dummy src files

# here you add your project src to the docker image

After building, changing only the source and rebuilding starts from the cached image with dependencies already built.

@lolgesten
Copy link

@lolgesten lolgesten commented Oct 9, 2017

someone needs to relax...

@lolgesten
Copy link

@lolgesten lolgesten commented Oct 9, 2017

Also @karlfish what you're proposing is not actually working. If using FROM rust:1.20.0.

  1. cargo new playground fails because it wants USER env variable to be set.
  2. RUN cargo build does not build dependencies for release, but for debug. why do you need that?
@lolgesten
Copy link

@lolgesten lolgesten commented Oct 9, 2017

Here's a better version.

FROM rust:1.20.0

WORKDIR /usr/src

# Create blank project
RUN USER=root cargo new umar

# We want dependencies cached, so copy those first.
COPY Cargo.toml Cargo.lock /usr/src/umar/

WORKDIR /usr/src/umar

# This is a dummy build to get the dependencies cached.
RUN cargo build --release

# Now copy in the rest of the sources
COPY src /usr/src/umar/src/

# This is the actual build.
RUN cargo build --release \
    && mv target/release/umar /bin \
    && rm -rf /usr/src/umar

WORKDIR /

EXPOSE 3000

CMD ["/bin/umar"]
@shepmaster
Copy link
Member

@shepmaster shepmaster commented Oct 9, 2017

You can always review the complete Dockerfile for the playground.

@maelvls
Copy link

@maelvls maelvls commented Nov 10, 2017

Hi!
What is the current state of the --deps-only idea? (mainly for dockerization)

@AdrienneCohea
Copy link

@AdrienneCohea AdrienneCohea commented Feb 3, 2018

I agree that it would be really cool to have a --deps-only option so that we could cache our filesystem layers better in Docker.

I haven't tried replicating this yet, but it looks very promising. This is in glibc and not musl, by the way. My main priority is to get to a build that doesn't take 3-5 minutes ever time, not a 5 MB alpine-based image.

@trezm
Copy link

@trezm trezm commented Jun 13, 2020

@camerondavison do be 100% honest, I'm really not sure why the build.rs affects the result! I came across it as a solution by accident. I had a project that I have a dockerfile for that uses tonic, so I needed a build.rs to build proto files.

I noticed that I could super simplify the Dockerfile for that project, but when I copied the same Dockerfile to a new project (sans build.rs) it no longer worked because of what you mentioned. I added the build.rs file back with a single dummy println and it seemed to work again!

Really not sure why this would change anything, maybe something to do with the presence of a build.rs file forcing cargo to look at the actual timestamp on all the files? This is me wildly speculating, but I encourage you to try adding a dummy build.rs and see if you get the same results!

@camerondavison
Copy link

@camerondavison camerondavison commented Jun 13, 2020

@trezm adding a build.rs file with

fn main() {}

does fix the problem. so crazy. I am going to keep trying to figure out why.

EDIT: found a solution
because the file that is created is newer than the source code rust thinks that the build artifacts that get created are newer than the source code. I was able to fix this problem by touching the build artifacts for just my source code before doing the next build. not sure if that is better than using the build.rs or not. this is the dockerfile I ended up with

FROM rust:1.44 as build

# app
ENV app=my-app

# dependencies
WORKDIR /tmp/${app}
COPY Cargo.toml Cargo.lock ./

# compile dependencies
RUN set -x\
 && mkdir -p src\
 && echo "fn main() {println!(\"broken\")}" > src/main.rs\
 && cargo build --release

# copy source and rebuild
COPY src/ src/
RUN set -x\
 && find target/release -type f -name "$(echo "${app}" | tr '-' '_')*" -exec touch -t 200001010000 {} +\
 && cargo build --release
@vn971
Copy link

@vn971 vn971 commented Jun 29, 2020

@camerondavison adding to your work-around, you can replace pre-last line with: find target/release/ -type f -executable -maxdepth 1 -delete

@anitnilay20
Copy link

@anitnilay20 anitnilay20 commented Jul 1, 2020

hey @trezm @camerondavison doesn't work if there are multiple workspaces. 😢

@trezm
Copy link

@trezm trezm commented Jul 1, 2020

Bummer! Does @vn971's solution work? I don't claim to have any substantial knowledge about cargo, I was just hopefully helping with a workaround :)

@vn971
Copy link

@vn971 vn971 commented Jul 1, 2020

@anitnilay20 just copy the needed Cargo.* files only, and otherwise repeat the steps? Should work?

@anitnilay20
Copy link

@anitnilay20 anitnilay20 commented Jul 1, 2020

@trezm Solution does work for single workspace cargo projects but it doesn't work for multiple workspaces..

@vn971
Copy link

@vn971 vn971 commented Jul 1, 2020

Let's better raise a StackOverflow question or something. This ticket is to add support for the flag. I think multiple workspaces work if done correctly, in the right order.

@anitnilay20
Copy link

@anitnilay20 anitnilay20 commented Jul 1, 2020

@vn971 it worked.. issue was bad Dockerfile. anyway thanx for your help.

@dessalines
Copy link

@dessalines dessalines commented Jul 11, 2020

Does anyone have a working version that can cache deps for workspaces too?

@dessalines
Copy link

@dessalines dessalines commented Jul 12, 2020

@anitnilay20 Thank you so much, that did it!

It does re-fetch the workspace deps when anything changes in there, but any changes in the main project don't re-fetch the workspace deps!

@jorgecarleitao
Copy link

@jorgecarleitao jorgecarleitao commented Jul 19, 2020

I have been dealing with the issue over a year now, both in libs, bins, and workspaces, with projects with and without examples and benches, mostly in docker images. Here are my 2 cents:

The underlying consequence is the too high compilation time of slowly-varying code in the context of CI/CD. Specifically, there is a number of rust projects whose majority of their CI/CD time is spent compiling non-changing code, which can be cached (e.g. check logs of arrow/rust or its issues within docker/CI).

The core issue is that, in the context of containerization (e.g. docker), engines have no way of knowing that the resulting artifacts in target/*/deps are independent of src/ after RUN cargo build. As such, they must invalidate all artifacts in target/*/deps when src/ changes (e.g. via a COPY . . before RUN). This is the reason why cargo build is not sufficient: we need a command to

build the stuff in target/*/deps using the dependants of target/*/deps

Cargo supports this locally by not recompiling dependencies that are already compiled. However, it does this because it knows that target/*/deps is independent of src/; a generic technology like containerization has no way of knowing this.

In theory, solving this issue requires two different API endpoints:

  • a command to get the list of dependants (which we have been assuming to be only Cargo.toml, src/[lib.rs/main.rs] and optionally build.rs)
  • a command to build target/*/deps

IMO this is issue a sub-part of one of the top 5 requested improvements to rust - compilation time: slow compilation is not only addressed by producing better plans for the LLVM, but also providing the tools to efficiently cache slowly varying code, such as external dependencies.

Finally, why the current status quo and "replacing lib.rs/main.rs for a dummy" is not enough:

non-workspace projects

This approach works well for simple projects, but does not work for projects with examples or benches. Essentially, given a Cargo.toml with a [[bench]]/[[example]], cargo build [--lib] [--release] requires the respective file benches/examples to be there. This implies that we also need to copy the benches and examples (which causes a recompilation on every change of example), or parse the Cargo.toml and touch a file for every [[bench]]/[[example]] on it (high maintenance).

workspace projects

When there are inter-workspace dependencies, say w1 <- w2, we can't compile w2's dependencies because it requires compiling w1 dependencies, like @shepmaster mentioned.

One current "solution" for this is to repeat the step for non-workspace projects on each workspace including examples/benches (cumbersome).

In the end, a command like cargo build --dependencies-only would collect the necessary files required to build targets/*/deps based on Cargo.tomls and would execute the download/compile to achieve that final state, in the same way that the hacks aforementioned currently achieve with different levels of "hackiness".

@Dushistov
Copy link

@Dushistov Dushistov commented Jul 19, 2020

@jorgecarleitao

The underlying consequence is the too high compilation time of slowly-varying code in the context of CI/CD

Why you can not use sccache for that?
It also automatically handle changes of versions of dependencies, compiler version changes and so on.

@jorgecarleitao
Copy link

@jorgecarleitao jorgecarleitao commented Jul 19, 2020

@jorgecarleitao

The underlying consequence is the too high compilation time of slowly-varying code in the context of CI/CD

Why you can not use sccache for that?
It also automatically handle changes of versions of dependencies, compiler version changes and so on.

We have:

This is complex, and doesn't seem to be helping IRL, so I'm closing this

@technimad
Copy link

@technimad technimad commented Jul 19, 2020

@jorgecarleitao
Why you can not use sccache for that?
It also automatically handle changes of versions of dependencies, compiler version changes and so on.

In our case we don’t go for sccache because:

  • it adds yet another tool to our stack, which comes with a maintenance and knowledge cost.
  • it isn’t supported natively by our cloud CI (Bitbucket)

Our product consists of multiple components written in different languages. We support this with a small team, so we stick to native tooling and defaults as much as possible.

@parisholley
Copy link

@parisholley parisholley commented Jul 19, 2020

for those who have the benefit of a buildkit based docker CI process... this is how i've been able to achieve < 1 minute test builds and < 2 minute release builds

For Testing:

# define dependencies for temporary build
COPY src/Cargo.toml src/Cargo.toml
COPY test/Cargo.toml test/Cargo.toml
COPY Cargo.lock Cargo.lock
COPY Cargo.toml Cargo.toml

# dummy files so we can compile and build depenencies
RUN echo "fn main(){}" > src/lib.rs
RUN echo "fn main(){}" > test/lib.rs

# cache dependency compilation
RUN --mount=type=cache,target=/root/.cargo/registry/ RUSTFLAGS="-C target-feature=-crt-static" cargo test --package test

# remove dummy files and compilation cache (not dependency cache)
RUN rm -rf src test
RUN rm -rf target/debug/**/libsrc*
RUN rm -rf target/debug/**/libtest*

# copy actual src over that tends to change more often than dependencies
COPY build/cdn build/cdn
COPY test test
COPY src src

# run another build using our actual source code
RUN RUSTFLAGS="-C target-feature=-crt-static" cargo test --package test

For release/deployment

# dummy files so we can compile and build depenencies
RUN mkdir src && echo "fn main(){}" > src/lib.rs
RUN echo "[workspace]" > Cargo.toml && echo "members = [\"src\"] " >> Cargo.toml

# define dependencies for temporary build
COPY src/Cargo.toml src/Cargo.toml
COPY Cargo.lock src/Cargo.lock

# cache dependency compilation
RUN --mount=type=cache,target=/root/.cargo/registry/ RUSTFLAGS="-C target-feature=-crt-static"  cargo build --release --lib

# remove dummy files and compilation cache (not dependency cache)
RUN rm -rf src
RUN rm -rf target/release/**/libsrc*

# copy actual src over that tends to change more often than dependencies
COPY build/cdn build/cdn
COPY src src

# run another build using our actual source code
RUN RUSTFLAGS="-C target-feature=-crt-static" cargo build --release

FROM release as bin1

WORKDIR /rust

COPY src/cfg src/cfg
COPY --from=release /rust/target/release/bin1 bin1

ENTRYPOINT ["/rust/bin1"]

FROM release as bin2

WORKDIR /rust

COPY src/cfg src/cfg
COPY --from=release /rust/target/release/bin2 bin2

ENTRYPOINT ["/rust/bin2"]
kszucs added a commit to apache/arrow that referenced this issue Jul 30, 2020
This allow us to build the crate without having to re-build nor download all its dependencies at the expense of a larger (build) image.

Here I am extending the ideas of a non-workspace rust project to a workspace rust project. See [here](rust-lang/cargo#2644) for a _long_ discussion on the issue. Essentially, it is not easy to do this atm with cargo.

IMO I think that this is worth it, but I need your help to judge it; it incurs some maintenance (duplication of configurations).

Closes #7799 from jorgecarleitao/docker_caching

Lead-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Co-authored-by: Jorge C. Leitao <jorgecarleitao@gmail.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
@gilescope
Copy link
Contributor

@gilescope gilescope commented Jul 31, 2020

Having lost more of my life than I care to mention trying to get non-trivial incremental compilation working in something docker-like, I have come to the conclusion that only by removing mtime from the picture can we achieve stable caching. I think we need #6529 to be solved before there's any point in having a command that builds just the dependencies (for which we have reasonable workarounds). Currently there's no way to work around the mtime dependency.

@dessalines
Copy link

@dessalines dessalines commented Sep 2, 2020

I figured out how to get this also working with cargo workspaces, using romac's fork of cargo-build-deps.

This example has my_app, and two workspaces: utils and db.

FROM rust:nightly as rust

# Cache deps
WORKDIR /app
RUN sudo chown -R rust:rust .
RUN USER=root cargo new myapp

# Install cache-deps
RUN cargo install --git https://github.com/romac/cargo-build-deps.git

WORKDIR /app/myapp
RUN mkdir -p db/src/ utils/src/

# Copy the Cargo tomls
COPY myapp/Cargo.toml myapp/Cargo.lock ./
COPY myapp/db/Cargo.toml ./db/
COPY myapp/utils/Cargo.toml ./utils/

# Cache the deps
RUN cargo build-deps

# Copy the src folders
COPY myapp/src ./src/
COPY myapp/db/src ./db/src/
COPY myapp/utils/src/ ./utils/src/

# Build for debug
RUN cargo build
@uselessscat
Copy link

@uselessscat uselessscat commented Oct 5, 2020

In python there is a file called dependencies.txt, So in your dockerfile you do something like this:

WORKDIR /my-stuff 
COPY dev.dependencies.txt dependencies.txt ./

# in dependencies i declared fastapi that install uvicorn
RUN  pip install -r dev.dependencies.txt -r dependencies.txt

COPY . .

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080", "--reload"]

So you dont have to download all dependencies everytime you changes the code. With Rust Im unable to generate compilation cache and build without creating a dummy project. It would be great something like:

WORKDIR /my-stuff

COPY Cargo.toml Cargo.lock ./
# this runs cargo fetch, cargo install, and finally cargo build
RUN cargo cache build

COPY . .
RUN cargo build

CMD ["cargo", "watch", "-x", "run"]

Then cargo cache can contain clean and build subcommands (or generate, or something that describes the action of building the cache)
Look at #5120

@vojtechkral
Copy link
Contributor

@vojtechkral vojtechkral commented Oct 5, 2020

@uselessscat This is why I created #8362 , I believe that's more or less what you mean...

@LukeMathWalker
Copy link

@LukeMathWalker LukeMathWalker commented Oct 25, 2020

As mentioned in #8362, I believe cargo-chef is a MVP of what this could look like from an API perspective.
A sample Dockerfile looks like this:

FROM rust as planner
WORKDIR app
RUN cargo install cargo-chef 
COPY . .
RUN cargo chef prepare  --recipe-path recipe.json

FROM rust as cacher
WORKDIR app
RUN cargo install cargo-chef
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json

FROM rust as builder
WORKDIR app
COPY . .
COPY --from=cacher /app/target target
RUN cargo build --release --bin app

FROM rust as runtime
WORKDIR app
COPY --from=builder /app/target/release/app /usr/local/bin
ENTRYPOINT ["./usr/local/bin/app"]

It does not require Buildkit.
I still believe this functionality should somehow find its way into cargo, but hopefully we can explore the space a bit with cargo-chef outside of it to understand the tradeoffs and actual usage patterns.

@dessalines
Copy link

@dessalines dessalines commented Oct 25, 2020

@LukeMathWalker this is much cleaner to have the recipe planning inside docker as well... you should make this the default example in the cargo-chef readme.

@LukeMathWalker
Copy link

@LukeMathWalker LukeMathWalker commented Oct 25, 2020

I thought I did, but it turns I had not pushed the commit to remote 😅 Updated! @dessalines

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet