New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cargo build --dependencies-only #2644

Open
nagisa opened this Issue May 4, 2016 · 59 comments

Comments

Projects
None yet
@nagisa

nagisa commented May 4, 2016

There should be an option to only build dependencies.

@KalitaAlexey

This comment has been minimized.

Contributor

KalitaAlexey commented Jan 17, 2017

@nagisa,
Why do you want it?

@nagisa

This comment has been minimized.

nagisa commented Jan 17, 2017

I do not remember exactly why, but I do remember that I ended just running rustc manually.

@KalitaAlexey

This comment has been minimized.

Contributor

KalitaAlexey commented Jan 17, 2017

@posborne, @mcarton, @devyn,
You reacted with thumbs up.
Why do you want it?

@mcarton

This comment has been minimized.

Contributor

mcarton commented Jan 17, 2017

Sometimes you add a bunch of dependencies to your project, know it will take a while to compile next time you cargo build, but want your computer to do that as you start coding so the next cargo build is actually fast.
But I guess I got here searching for a cargo doc --dependencies-only, which allows you to get the doc of your dependencies while your project does not compile because you'd need the doc to know how exactly to fix that compilation error you've had for a half hour 😄

@gregwebs

This comment has been minimized.

gregwebs commented Jan 30, 2017

As described in #3615 this is useful with build to setup a cache of all dependencies.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jan 31, 2017

@gregwebs out of curiosity do you want to cache compiled dependencies or just downloaded dependencies? Caching compiled dependencies isn't implemented today (but would be with a command such as this) but downloading dependencies is available via cargo fetch.

@gregwebs

This comment has been minimized.

gregwebs commented Jan 31, 2017

Generally, as with my caching use case, the dependencies change infrequently and it makes sense to cache the compilation of them.

The Haskell tool stack went through all this and they seemed to generally decided to merge things into a single command where possible. For fetch they did end up with something kinda confusing though: build --dry-run --prefetch. For build --dependencies-only mentioned here they do have the same: build --only-dependencies

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jan 31, 2017

@gregwebs ok thanks for the info!

@KalitaAlexey

This comment has been minimized.

Contributor

KalitaAlexey commented Jan 31, 2017

@alexcrichton,
It looks like I should continue my work on the PR.
Will Cargo's team accept it?

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Feb 2, 2017

@KalitaAlexey I personally wouldn't be convinced just yet, but it'd be good to canvas opinions from others on @rust-lang/tools as well

@KalitaAlexey

This comment has been minimized.

Contributor

KalitaAlexey commented Feb 2, 2017

@alexcrichton,
Anyway I have no time right now)

@nrc

This comment has been minimized.

Member

nrc commented Feb 2, 2017

I don't see much of a use case - you can just do cargo build and ignore the output for the last crate. If you really need to do this (for efficiency) then there is API you can use.

@gregwebs

This comment has been minimized.

gregwebs commented Feb 4, 2017

What's the API?

@nrc

This comment has been minimized.

Member

nrc commented Feb 6, 2017

Implement an Executor. That lets you intercept every call to rustc and you can do nothing if it is the last crate.

@gregwebs

This comment has been minimized.

gregwebs commented Feb 6, 2017

I wasn't able to find any information about an Executor for cargo. Do you have any links to documentation?

@nrc

This comment has been minimized.

Member

nrc commented Feb 6, 2017

Docs are a little thin, but start here:

/// A glorified callback for executing calls to rustc. Rather than calling rustc
/// directly, we'll use an Executor, giving clients an opportunity to intercept
/// the build calls.

You can look at the RLS for an example of how to use them: https://github.com/rust-lang-nursery/rls/blob/master/src/build.rs#L288

@shepmaster

This comment has been minimized.

Member

shepmaster commented Feb 10, 2017

A question of Stack Overflow wanted this feature. In that case, the OP wanted to build the dependencies for a Docker layer.

A similar situation exists for the playground, where I compile all the crates once. In my case, I just put in a dummy lib.rs / main.rs. All the dependencies are built, and the real code is added in the future.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Feb 10, 2017

@shepmaster unfortunately the proposed solution wouldn't satisfy that question because a Cargo.toml won't parse without associated files in src (e.g. src/lib.rs, etc). So that question would still require "dummy files", in which case it wouldn't specifically be serviced by this change.

@lolgesten

This comment has been minimized.

lolgesten commented Oct 9, 2017

I ended up here because I also am thinking about the Docker case. To do a good docker build I want to:

COPY Cargo.toml Cargo.lock /mything

RUN cargo build-deps --release  # creates a layer that is cached

COPY src /mything/src

RUN cargo build --release       # only rebuild this when src files changes

This means the dependencies would be cached between docker builds as long as Cargo.toml and Cargo.lock doesn't change.

I understand src/lib.rs src/main.rs are needed to do a good build, but maybe build-deps simply builds all the deps.

@ghost

This comment has been minimized.

ghost commented Oct 9, 2017

The dockerfile template in shepmaster's linked stackoverflow post above SOLVES this problem

I came to this thread because I also wanted the docker image to be cached after building the dependencies. After later resolving this issue, I posted something explaining docker caching, and was informed that the answer was already linked in the stackoverflow post. I made this mistake, someone else made this mistake, it's time to clarify.

RUN cd / && \
    cargo new playground
WORKDIR /playground                      # a new project has a src/main.rs file

ADD Cargo.toml /playground/Cargo.toml 
RUN cargo build                          # DEPENDENCIES ARE BUILD and CACHED
RUN cargo build --release
RUN rm src/*.rs                          # delete dummy src files

# here you add your project src to the docker image

After building, changing only the source and rebuilding starts from the cached image with dependencies already built.

@lolgesten

This comment has been minimized.

lolgesten commented Oct 9, 2017

someone needs to relax...

@lolgesten

This comment has been minimized.

lolgesten commented Oct 9, 2017

Also @karlfish what you're proposing is not actually working. If using FROM rust:1.20.0.

  1. cargo new playground fails because it wants USER env variable to be set.
  2. RUN cargo build does not build dependencies for release, but for debug. why do you need that?
@lolgesten

This comment has been minimized.

lolgesten commented Oct 9, 2017

Here's a better version.

FROM rust:1.20.0

WORKDIR /usr/src

# Create blank project
RUN USER=root cargo new umar

# We want dependencies cached, so copy those first.
COPY Cargo.toml Cargo.lock /usr/src/umar/

WORKDIR /usr/src/umar

# This is a dummy build to get the dependencies cached.
RUN cargo build --release

# Now copy in the rest of the sources
COPY src /usr/src/umar/src/

# This is the actual build.
RUN cargo build --release \
    && mv target/release/umar /bin \
    && rm -rf /usr/src/umar

WORKDIR /

EXPOSE 3000

CMD ["/bin/umar"]
@shepmaster

This comment has been minimized.

Member

shepmaster commented Oct 9, 2017

@maelvalais

This comment has been minimized.

maelvalais commented Nov 10, 2017

Hi!
What is the current state of the --deps-only idea? (mainly for dockerization)

@AdrienneCohea

This comment has been minimized.

AdrienneCohea commented Feb 3, 2018

I agree that it would be really cool to have a --deps-only option so that we could cache our filesystem layers better in Docker.

I haven't tried replicating this yet, but it looks very promising. This is in glibc and not musl, by the way. My main priority is to get to a build that doesn't take 3-5 minutes ever time, not a 5 MB alpine-based image.

@lolgesten

This comment has been minimized.

lolgesten commented Feb 16, 2018

What impact does a build.rs have on your case?

My (several) build.rs are for invoking bindgen around c-libs. Some are simply: bindgen + include!(concat!(env!("OUT_DIR"), "/bindings.rs")); in the lib.rs, and some lib.rs add a bit more to the bindings

I guess my expectation of a cargo build --deps only in a workspace scenario would be that, as a minimum, it would build any non project local deps of those submodules together with the non project local deps of my main lib (like libc, which is often in my submodules).

I only mention build.rs because the solution of doing empty projects, build, and then copying in the source becomes more messy when you also have a bespoke build with git submodules of various c sources built via a build.rs.

Maybe a cargo build --deps-only would do build only dependencies that are not local to the project. I.e. any dep that in a Cargo.toml (root level or submodule) that references the central cargo repo, own repos, or git sources.

@gregwebs

This comment has been minimized.

gregwebs commented Feb 16, 2018

I think it would be useful to decide whether to accept or reject this as a desirable feature (even though there is no implementation yet) and clearly state that at the top of the issue (or a new issue).

@chris13524

This comment has been minimized.

chris13524 commented Feb 25, 2018

I'd also like to have this feature. I'm also doing a Docker cache thing.

@shepmaster 's Dockerfile isn't working in my case (cross compiling stuff).

@cmac4603

This comment has been minimized.

cmac4603 commented Mar 4, 2018

I am very sad this got closed, this would be great for docker builds, and way less hacky than current solutions. I feel docker builds in their own right are also a good enough reason alone

@mathroc

This comment has been minimized.

mathroc commented Mar 4, 2018

@cmac4603 this has not been closed

@cmac4603

This comment has been minimized.

cmac4603 commented Mar 4, 2018

Oh my goodness! I'm so glad! Yup, totally misread, thanks @mathroc

I would love to see this for docker builds then, particularly local dev builds involving docker-compose. Generally for final test/prod builds, I use a musl builder from scratch, but for development, would be a very nice-to-have in the toolkit

@axos88

This comment has been minimized.

axos88 commented Mar 18, 2018

Related: #5198

@briansmith

This comment has been minimized.

briansmith commented Mar 21, 2018

downloading dependencies is available via cargo fetch

cargo fetch downloads too much in general without having something like #5216 to allow the fetch to be constrained to a target.

@golddranks

This comment has been minimized.

Contributor

golddranks commented Jul 27, 2018

I had a need for this feature today for pre-building a docker image cache. The trick to cargo init a fresh project and reusing only one's Cargo.toml almost worked, but I have a build.rs script, so I had to make a stub build.rs file too. It would be easier to have a "standard" way of downloading and building the deps that doesn't depend on the particulars of your build and handles your corner cases correctly.

@RReverser

This comment has been minimized.

Contributor

RReverser commented Jul 30, 2018

but I have a build.rs script, so I had to make a stub build.rs file too

@golddranks Why did you need it though? If the reason is that you have build = "build.rs" line in Cargo.toml, then just remove that line as it's not required nowadays and build.rs is automatically assumed to be a build script if found.

@golddranks

This comment has been minimized.

Contributor

golddranks commented Jul 31, 2018

Ah, didn't know that. Thanks.

@ArtemGr

This comment has been minimized.

ArtemGr commented Aug 9, 2018

Yet another reason to have this is that some CIs (AppVeyour in particular) have a limit on how much information is logged from the build. With cargo build --dependencies-only one can build dependencies normally, then switch to cargo build -vv for the source code that's being the primary target of the CI build.

@wycats

This comment has been minimized.

Contributor

wycats commented Aug 10, 2018

I think we should probably consider the docker use-case in more detail. I have a hunch that there's more to the use-case than meets the eye.

I think that the CI use-case is a more interesting rationale for this feature, but I wonder if it argues instead for more control over the verbosity options (--crate-verbosity "my-crate=vv" or something along those lines), which I could imagine being useful for other purposes ("I really want to look more closely at what's going on when I build clap")

@JeanMertz

This comment has been minimized.

JeanMertz commented Aug 17, 2018

I came here looking for the same use-case of wanting to properly layer my Docker images to decrease build times.

I ended up with the following solution, in case this helps anyone else:

FROM ekidd/rust-musl-builder AS builder

RUN sudo chown -R rust:rust /home/rust

RUN mkdir src && touch src/lib.rs
COPY Cargo.lock .
COPY Cargo.toml .
RUN cargo build --release

ADD . .
RUN cargo build --release

FROM scratch
COPY --from=builder \
  /home/rust/src/target/x86_64-unknown-linux-musl/release/my-app \
  /my-app
ENTRYPOINT ["/my-app"]

Here's the result of the first run, vs a second run with only source code changes, no dependency changes:

collapsed wall of text
❯ time docker build -t hello .; and docker run hello

Sending build context to Docker daemon  225.3kB
Step 1/10 : FROM ekidd/rust-musl-builder AS builder
 ---> 792e654be291
Step 2/10 : RUN mkdir src/ && touch src/lib.rs
 ---> Running in 2208ab6053f4
Removing intermediate container 2208ab6053f4
 ---> 572926d2738a
Step 3/10 : COPY Cargo.lock .
 ---> e8eb753e4881
Step 4/10 : COPY Cargo.toml .
 ---> 43f3d04fe0e8
Step 5/10 : RUN cargo build --release
 ---> Running in 374ba4a8a864
    Updating registry `https://github.com/rust-lang/crates.io-index`
    <SNIP>
    Finished release [optimized] target(s) in 43.65s
Removing intermediate container 374ba4a8a864
 ---> 843e36ec3ce7
Step 6/10 : ADD . .
 ---> c66cc9b7f4dc
Step 7/10 : RUN cargo build --release
 ---> Running in bcd95a11ff47
   Compiling hello-world v0.1.0 (file:///home/rust/src)
    Finished release [optimized] target(s) in 0.40s
Removing intermediate container bcd95a11ff47
 ---> 1789bdcc6ed3
Step 8/10 : FROM scratch
 ---> 
Step 9/10 : COPY --from=builder   /home/rust/src/target/x86_64-unknown-linux-musl/release/hello-world   /hello-world
 ---> Using cache
 ---> e8501d0c8738
Step 10/10 : ENTRYPOINT ["/hello-world"]
 ---> Using cache
 ---> fc179245d85b
Successfully built fc179245d85b
Successfully tagged hello:latest
       53.53 real         2.17 user         0.80 sys
hello world
❯ time docker build -t hello .; and docker run hello

Sending build context to Docker daemon  225.3kB
Step 1/10 : FROM ekidd/rust-musl-builder AS builder
 ---> 792e654be291
Step 2/10 : RUN mkdir src/ && touch src/lib.rs
 ---> Using cache
 ---> 572926d2738a
Step 3/10 : COPY Cargo.lock .
 ---> Using cache
 ---> e8eb753e4881
Step 4/10 : COPY Cargo.toml .
 ---> Using cache
 ---> 43f3d04fe0e8
Step 5/10 : RUN cargo build --release
 ---> Using cache
 ---> 843e36ec3ce7
Step 6/10 : ADD . .
 ---> a9698b8f6a88
Step 7/10 : RUN cargo build --release
 ---> Running in 980a6d70a441
   Compiling hello-world v0.1.0 (file:///home/rust/src)
    Finished release [optimized] target(s) in 0.35s
Removing intermediate container 980a6d70a441
 ---> 1661e048b501
Step 8/10 : FROM scratch
 ---> 
Step 9/10 : COPY --from=builder   /home/rust/src/target/x86_64-unknown-linux-musl/release/hello-world   /hello-world
 ---> d8b93e6a4ff9
Step 10/10 : ENTRYPOINT ["/hello-world"]
 ---> Running in e26f084ed361
Removing intermediate container e26f084ed361
 ---> 6684afc6b641
Successfully built 6684afc6b641
Successfully tagged hello:latest
        7.43 real         2.26 user         1.00 sys
hello universe!
@gilescope

This comment has been minimized.

Contributor

gilescope commented Sep 8, 2018

Same docker use case here. We can do the above, but anyone reading it would wonder what's going on. It would be much clearer to have an -dependencies-only flag.

Is the question whether there are sufficient use cases (Docker, performance timing, any others?)? (I'm assuming the cargo implementation would be fairly simple.).

To be fair docker is a pretty big use case...

@emilk

This comment has been minimized.

emilk commented Sep 10, 2018

Same docker use case here. I would love this feature.

@anuraags

This comment has been minimized.

anuraags commented Sep 28, 2018

Same docker use case here. I don't want to have to pull and compile all the dependencies in my image everytime I change the source code.

@gilescope

This comment has been minimized.

Contributor

gilescope commented Oct 3, 2018

I tried @JeanMertz approach but when you've got a workspace with a few crates in it, it's not so pretty. docker's COPY doesn't seem to like recursive globs, COPY **/Cargo.toml . for example doesn't seem to work for me.

That said, I'm not sure there's a nice way of doing this, because even if we said build deps only, if we copied in all our source to the docker image and the source had changed then it would be a cache miss anyway. I think the best we can ever do with docker is to generalise @JeanMertz 's approach.

@dwijnand

This comment has been minimized.

Member

dwijnand commented Oct 3, 2018

Note that there is also https://crates.io/crates/cargo-build-deps.

@dwijnand dwijnand referenced this issue Oct 10, 2018

Open

Cargo team: planning areas #5

0 of 5 tasks complete
@azban

This comment has been minimized.

azban commented Nov 7, 2018

@nrc i tried the 'Executor' path to no avail (https://github.com/azban/cargo-build-deps/blob/executor/src/main.rs). i had to hack around fingerprint updates because there is no way to be lenient here. but i finally got stuck on ignoring custom build scripts, because they don't use the executor and rather call commands directly here.

i ended up hacking around build_plan, which is messy because seemingly the only way to get it, is from stdout, and i was making assumptions about how to tell if something is an external dependency or not. (https://github.com/azban/cargo-build-deps/blob/build-plan/src/main.rs)

@JeanMertz @gilescope @emilk @anuraags , i ended up 'generalizing' the approach above in order to support more complex crates (workspaces mostly). mostly involves a separate build image which you only pass what's required to build (Cargo.toml, Cargo.lock, etc) into the docker context.. and then inherit from that image in a later build. you can check out an example here: paritytech/substrate@a12e304

@denzp

This comment has been minimized.

denzp commented Nov 8, 2018

Unfortunately, in many usecases approach based on Cargo.toml and Cargo.lock can't really work, because you need to update at least current crate version. And therefore Docker will invalidate cache immediately, even if dependencies versions are still the same.

I was recently experimenting with BuildKit approach: every rustc or build script invocation is done in own build stage (something like this). The image builder can manage "cache invalidation" and building concurrency on its own then.

I can call it a successful experiment, but I decided to go even deeper - I'm currently trying to build a BuildKit frontend for Cargo crates. This will help to achieve a mind-blowing

docker build -t my-tag -f Cargo.toml .

Apart from these experiments, you can use an "almost" production-ready caching approach today! You need to have Docker 18.06+, DOCKER_BUILDKIT=1 environment variable, and a special Dockerfile syntax:

# syntax=docker/dockerfile-upstream:experimental

FROM rustlang/rust:nightly as builder

# Copy crate sources and workspace lockfile
WORKDIR /rust-src
COPY Cargo.lock /rust-src/Cargo.lock
COPY cargo-container-tools /rust-src

# Build with mounted cache
RUN --mount=type=cache,target=/rust-src/target \
    --mount=type=cache,target=/usr/local/cargo/git \
    --mount=type=cache,target=/usr/local/cargo/registry \
    ["cargo", "build", "--release"]

# Copy binaries into normal layers
RUN --mount=type=cache,target=/rust-src/target \
    ["cp", "/rust-src/target/release/cargo-buildscript", "/usr/local/bin/cargo-buildscript"]
RUN --mount=type=cache,target=/rust-src/target \
    ["cp", "/rust-src/target/release/cargo-test-runner", "/usr/local/bin/cargo-test-runner"]
RUN --mount=type=cache,target=/rust-src/target \
    ["cp", "/rust-src/target/release/cargo-ldd", "/usr/local/bin/cargo-ldd"]

# Copy the binaries into final stage
FROM debian:stable-slim
COPY --from=builder /usr/local/bin/cargo-buildscript /usr/local/bin/cargo-buildscript
COPY --from=builder /usr/local/bin/cargo-test-runner /usr/local/bin/cargo-test-runner
COPY --from=builder /usr/local/bin/cargo-ldd /usr/local/bin/cargo-ldd

# syntax=... defines a Dockerfile builder frontend image and caching is done with --mount flags of RUN instruction.

Update: corrected Dockerfile stage names.
Update 2: changed nightly Cargo registry cache path.

@vladikoff

This comment has been minimized.

vladikoff commented Nov 8, 2018

After trying out cargo-build-deps, I noticed that it runs cargo update and seems slower than latest cargo build in Rust nightly. I would say it probably makes sense to just bake this functionality into cargo main for Docker and caching purposes.

@azban

This comment has been minimized.

azban commented Nov 8, 2018

awesome @denzp .. i hadn't checked out buildkit, and it looks like good timing as it's no longer experimental on the docker-ce released today :). i'm going to favor this over cargo-build-deps.

@J-Zeitler

This comment has been minimized.

J-Zeitler commented Nov 15, 2018

@denzp That looks promising. Do you think it would be possible to support both cargo build and cargo check with this? Currently I'm doing something like this mess (based on the suggestions above + rust-aws-lambda):

ADD $SRC/Cargo.toml $SRC/Cargo.lock ./
RUN mkdir src && touch src/lib.rs
ARG checkonly
RUN if [ "x$checkonly" = "x" ] ; then cargo build --target $BUILD_TARGET --release ; else cargo check ; fi
ADD $SRC/src src
RUN if [ "x$checkonly" = "x" ] ; then cargo build --target $BUILD_TARGET --release ; else cargo check && echo "return early after check, this is not an error" && false ; fi

which seems to cache both branches (with/without checkonly supplied as build-arg) separately

@denzp

This comment has been minimized.

denzp commented Nov 15, 2018

@J-Zeitler in your example, change of checkonly build argument alters the whole command. And therefore Docker decides to drop the cached result layer (containing target dir with build artifacts).

With RUN --mount it shouldn't be a problem anymore, because regardless of the cached layer presence, target dir with Cargo cache will be preserved.

Looks like you can simplify the Dockerfile and use only the second command. But is there a reason to run cargo check during image building stage?

@J-Zeitler

This comment has been minimized.

J-Zeitler commented Nov 15, 2018

@denzp Cool, will try to find some time to test it.

The reason is that I'm on a windows machine and building for an "amazonlinux" host. I guess I could run cargo check locally. Will cargo check always produce the same output regardless of --target?

(also, I noticed that I actually don't even use --target in the check command so it might not do what I want anyways)

eidt: also, yes, with a proper cache I could probably remove the second command. What I do now (as the others above) is using docker layers as cache and ADDing $SRC/src src will evict the cached layers below. That's the reason people poke in the fake src/lib.rs etc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment