Avoid rebuilding a project when cwd changes #4788

alexcrichton · 2017-12-07T00:33:36Z

This commit is targeted at solving a use case which typically comes up during CI
builds -- the target directory is cached between builds but the cwd of the
build changes over time. For example the following scenario can happen:

A project is compiled at /projects/a.
The target directory is cached.
A new build is started in /projects/b.
The previous target directory is restored to /projects/b.
The build start, and Cargo rebuilds everything.

The last piece of behavior is indeed unfortunate! Cargo's internal hashing
currently isn't that resilient to changing cwd and this PR aims to help improve
the situation!

The first point of too-much-hashing came up with Target::src_path. Each
Target was hashed and stored for all compilations, and the src_path field
was an absolute path on the filesystem to the file that needed to be compiled.
This path then changed over time when cwd changed, but otherwise everything else
remained the same!

This commit updates the handling of the src_path field to simply ignore it
when hashing. Instead the path we actually pass to rustc is later calculated and
then passed to the fingerprint calculation.

The next problem this fixes is that the dep info files were augmented after
creation to have the cwd of the compiler at the time to find the files at a
later date. This, unfortunately, would cause issues if the cwd itself changed.
Instead the cwd is now left out of dep-info files (they're no longer augmented)
and instead the cwd is recalculated when parsing the dep info later.

The final problem that this commit fixes is actually an existing issue in Cargo
today. Right now you can actually execute cargo build from anywhere in a
project and Cargo will execute the build. Unfortunately though the argument to
rustc was actually different depending on what directory you were in (the
compiler was invoked with a path relative to cwd). This path ends up being used
for metadata like debuginfo which means that different directories would cause
different artifacts to be created, but Cargo wouldn't rerun the compiler!

To fix this issue the matter of cwd is now entirely excluded from compilation
command lines. Instead rustc is unconditionally invoked with a relative path
if the path is underneath the workspace root, and otherwise it's invoked as an
absolute path (in which case the cwd doesn't matter).

Once all these fixes were added up it means that now we can have projects where
if you move the entire directory Cargo won't rebuild the original source!

Note that this may be a bit of a breaking change, however. This means that the
paths in error messages for cargo will no longer be unconditionally relative to
the current working directory, but rather relative to the root of the workspace
itself. Unfortunately this is moreso of a feature right now rather than a bug,
so it may be one that we just have to stomach.

Closes #3273

rust-highfive · 2017-12-07T00:33:40Z

r? @matklad

(rust_highfive has picked a reviewer for you, use r? to override)

matklad

The implementation looks good to me, and I think "purge CWD, use paths relative to a fixed position" is the right approach, but I am not sure that "workspace root heuristic" is a good enough implementation. It's certainly a heuristic because some workspace members may be outside of the workspace, and some non-members may be inside workspace. Not sure if we need 100% correct implementation here though.

matklad · 2017-12-07T08:55:01Z

src/cargo/ops/cargo_rustc/fingerprint.rs

-    if fs_try!(f.read_until(0, &mut cwd)) == 0 {
-        return Ok(None)
-    }
-    let cwd = util::bytes2path(&cwd[..cwd.len()-1])?;


Am I correct that new versions of Cargo will fail to parse old depinfo, which is actually OK, because it'll cause only a rebuild, and not a build failure?

Hm I'm not actually sure what will happen but the hashes are all changing so I don't think newer cargo will use the same files from older cargo.

matklad · 2017-12-07T08:58:35Z

src/cargo/ops/cargo_rustc/mod.rs

+    let ws_root = cx.ws.root();
+    let src = unit.target.src_path();
+    assert!(src.is_absolute());
+    match src.strip_prefix(ws_root) {


I am not too happy about this solution, because there's no guarantee that workspace members reside below workspace root (see also #4787 (comment) :) ).

I think the ideal behavior here is to use all explicit worksapce members which lay outside of the root package as anchors for paths as well. Not sure if we need to support this right from the start though.

matklad · 2017-12-07T09:05:42Z

src/cargo/core/manifest.rs

+    fn hash<H: Hasher>(&self, _: &mut H) {
+        // ...
+    }
+}


A slightly more elaborte design here would be to introduce a notion of explicitly relative path here and track path relative to workspace root and to other workspace members outside the root. Sort of like

struct PathRoots { // a number of anchors, such as current workspace dir, a particular workspace member, users's home directory, // CARGO_HOME, etc. This structure is "global" for cargo and is stored in Workspace/config. roots: Vec::<PathBuf>, } struct RelativePath { root_idx: usize // index in the global PathRoots rel_path: PathBuf } impl RelativePath { fn to_path(&self, roots: &PathRoots) -> PathBuf { roots[self.root_idx].join(&self.rel_path) } }

Not sure that we need this, but this can cope with workspaces which are not entirely under a workspace root. (Or perhaps it was a mistake to allow non-subdirectory members?).

alexcrichton · 2017-12-07T15:34:12Z

@matklad thanks for taking a look! It's true yeah that workspace members outside the workspace root do pose a problem. That being said I wasn't really sure how we'd handle them at all :(

I originally had a much more complicated patch which entirely refactored the src_path field into a "root" and a "extra" like you mentioned (only hashing the extra, not the root) but that ended up not working for a whole number of reasons.

So thinking about it... Right now the workspace root comes up in a few locations:

When we invoke rustc we give it a relative or non-relative path. This is actually pretty important for debuginfo and backtraces and such (so everything doesn't show up as src/lib.rs).
When we parse and manage the dep-info files

I think it's a pretty reasonable assumption to assume all rust files for a crate are within the same folder, so I think we could pretty easily adapt the dep-info files to only list files relative to the package root rather than the workspace root (basically do some serious postprocessing).

For invocations of rustc though I'm not sure what we could do. I think (especially due to the existing bug in Cargo) we need to pick a root path to have and to me the workspace root seems the most obvious, but it's true that it then means you have to rename the entire workspace at once and if you have a crate outside of a workspace you basically just can't rename it. (or we could figure out a scheme where we pass things like ../foo/src/lib.rs to rustc, aka figuring out relative paths with ..)

What do you think? Maybe we should fix up dep-info parsing to work without using ws.root() and leave the "what path do we provide to rustc" for later?

matklad · 2017-12-07T20:46:26Z

Hm, the part about rustc is interesting from the IDE point of view: we need to linkify stack traces, so it's important that stacktraces use absolute paths or relative paths with a known root. Semantically, the proper solution seems to root everything at workspace, and use paths exactly as the user specified them.

So, if you write members = ["../foo", "/home/workspace/bar"], ../foo and /home/workspace/bar end up in the compiled binary.

I think it's a pretty reasonable assumption to assume all rust files for a crate are within the same folder, so I think we could pretty easily adapt the dep-info files to only list files relative to the package root rather than the workspace root (basically do some serious postprocessing).

Hm, depinfo is produced by the compiler, so in theory solving the problem with rustc should automatically solve the depinfo problem?

How hard is it to implement the .. approach?

matklad · 2017-12-07T20:53:07Z

I think it's a pretty reasonable assumption to assume all rust files for a crate are within the same folder, so I think we could pretty easily adapt the dep-info files to only list files relative to the package root rather than the workspace root (basically do some serious postprocessing).

This assumption is also pretty important for IDEs, because we need to know which stuff to index, so +1 to making it an official and documented requirement :)

Though, I can imagine someone doing weird stuff like include_str("/etc/passwd"). We should probably not flat out break for such cases, but weird stuff like not doing a rebuild looks fine to me

This commit is targeted at solving a use case which typically comes up during CI builds -- the `target` directory is cached between builds but the cwd of the build changes over time. For example the following scenario can happen: 1. A project is compiled at `/projects/a`. 2. The `target` directory is cached. 3. A new build is started in `/projects/b`. 4. The previous `target` directory is restored to `/projects/b`. 5. The build start, and Cargo rebuilds everything. The last piece of behavior is indeed unfortunate! Cargo's internal hashing currently isn't that resilient to changing cwd and this PR aims to help improve the situation! The first point of too-much-hashing came up with `Target::src_path`. Each `Target` was hashed and stored for all compilations, and the `src_path` field was an absolute path on the filesystem to the file that needed to be compiled. This path then changed over time when cwd changed, but otherwise everything else remained the same! This commit updates the handling of the `src_path` field to simply ignore it when hashing. Instead the path we actually pass to rustc is later calculated and then passed to the fingerprint calculation. The next problem this fixes is that the dep info files were augmented after creation to have the cwd of the compiler at the time to find the files at a later date. This, unfortunately, would cause issues if the cwd itself changed. Instead the cwd is now left out of dep-info files (they're no longer augmented) and instead the cwd is recalculated when parsing the dep info later. The final problem that this commit fixes is actually an existing issue in Cargo today. Right now you can actually execute `cargo build` from anywhere in a project and Cargo will execute the build. Unfortunately though the argument to rustc was actually different depending on what directory you were in (the compiler was invoked with a path relative to cwd). This path ends up being used for metadata like debuginfo which means that different directories would cause different artifacts to be created, but Cargo wouldn't rerun the compiler! To fix this issue the matter of cwd is now entirely excluded from compilation command lines. Instead rustc is unconditionally invoked with a relative path *if* the path is underneath the workspace root, and otherwise it's invoked as an absolute path (in which case the cwd doesn't matter). Once all these fixes were added up it means that now we can have projects where if you move the entire directory Cargo won't rebuild the original source! Note that this may be a bit of a breaking change, however. This means that the paths in error messages for cargo will no longer be unconditionally relative to the current working directory, but rather relative to the root of the workspace itself. Unfortunately this is moreso of a feature right now rather than a bug, so it may be one that we just have to stomach.

alexcrichton · 2017-12-08T15:57:18Z

@matklad ok I've amended the first commit and pushed up another one. This should reduce the reliance on ws.root() except in one location. Everything dealing with fingerprints/dep-info doesn't deal with ws.root() at all and should now be improved to be resilient to renames.

The one usage of ws.root() is purely about how we actually invoke rustc itself. Let's say we have a project like:

foo/rust/Cargo.toml // <- workspace root
foo/rust/bar/Cargo.toml
foo/third-party/more-rust // <- workspace member

With this PR as-is if you rename the rust directory to a different name (like rust2) then Cargo should avoid rebuilding everything. Where this PR as-is falls down, however, is if you rename the foo directory (let's say that's the overarching git repo for the whole project). If that happens then Cargo will rebuild everything because the more-rust crate is compiled as rustc /path/to/foo/third-party/... and renaming /path/to/foo would cause the output artifact to change.

I believe the fix for this would be to change how Cargo invokes rustc, instead invoking rustc ../third-party/more-rust/... which makes it possible at all for Cargo to avoid recompilation when the foo directory is renamed.

I also believe that such a fix would purely involve local modification to this logic. I think though that this may be rare enough that we may want to punt on this for now?

Does that makes sense? Curious what you think!

matklad · 2017-12-11T12:36:30Z

src/cargo/ops/cargo_rustc/mod.rs

+    assert!(src.is_absolute());
+    match src.strip_prefix(ws_root) {
+        Ok(path) => (path.to_path_buf(), Some(ws_root.to_path_buf())),
+        Err(_) => (src.to_path_buf(), None),


Returning None here means that rustc will use pretty-much arbitrary cwd, is that right? Maybe it's better to lock this down to ws_root even in this case, just to get a bit more determinism? Than we can return just PathBuf here and avoid that comment about hashing only .0.

Certainly plausible! I was originally thinking that this wouldn't matter but I think with plugins nowadays it could matter for sure. I'll change.

That being said though I don't think we can avoid hashing the second field of the return value because ws_root changes over time (if the whole dir is renamed) but we don't want that to cause recompiles.

That being said though I don't think we can avoid hashing the second field of the return value because ws_root changes over time (if the whole dir is renamed) but we don't want that to cause recompiles.

The idea is that if we lock down cwd, then we don't need to return a pair here, we can return just the path.

matklad · 2017-12-11T12:45:30Z

src/cargo/ops/cargo_rustc/fingerprint.rs

+/// when it was invoked.
+///
+/// The serialized Cargo format will contain a list of files, all of which are
+/// relative if they're under `root`. or absolute if they're elsewehre.


Hm, weren't depinfo files, produced by Cargo, used by some other tools? #3557

So, this change will break such clients :(

Oh sure!

I think though (at least the intention) that these files are just those in .fingerprint, so the ones like target/debug/foo.d should still be present unmangled from rustc itself.

matklad · 2017-12-11T12:51:23Z

Yeah, so I think anchoring to the package root in most places is the correct thing to do, but I still have a couple of question (left inline as well):

Looks like dep-info files have external clients, so we can't just change the format as we see fit?
Let's always lock down rustc cwd to some value?
Implementation wise, I don't really get thrilled by the NonHashedPathBuf. If it's simple enough to just use relative paths for targets, I would probably gone that way.

alexcrichton · 2017-12-12T04:29:24Z

For the last point about NonHashedPathBuf I actually did try to go down that road initially (actually required a huge amount of refactoring in toml/targets.rs but it ended up not panning out unfortunately. When we construct a Target I don't think we know enough about the workspace to store workspace-relative paths is sort of the tl;dr at this point

matklad · 2017-12-12T08:58:01Z

I don't think we know enough about the workspace to store workspace-relative paths is sort of the tl;dr at this point

Targets should be package relative, and not workspace relative. In theory, we do have necessary information:

either the target has an explicitly declared path = /foo/bar/baz.rs in Cargo.toml, in which case we should use this path as is (it can be either relative or absolute, and we should not change that).
or this is an implicit target, which we search by checking ./src/lib.rs, etc. In such case, we should be able to get the relative path out.

But I totally understand that it might be hard to actually implement.

So yeah, I think it might worth it to look into package relative paths for targets and locking down cwd for rustc, but, otherwise, r+ at will, this looks great! :)

This commit alters the format of the dependency info that Cargo keeps track of for each crate. In order to be more resilient against directory renames and such Cargo will now postprocess the compiler's dep-info output and serialize into its own format. This format is intended to primarily list relative paths *to the root of the relevant package* rather than absolute or relative to some other location. If paths aren't actually relative to the package root they're still stored as absolute, but there's not much we can do about that!

alexcrichton · 2017-12-12T19:00:30Z

Oh sure yeah targets are package relative but we still don't want to hash the src_path field in Target (no matter what we do). That field actually has no bearing on the output artifact as it's never factored in. What actually matters, the path to rustc, is the one we hash.

It was after I ended up realizing that when I jettisoned the ~500 lines of changes to get the refactoring done as it ended up not being used in the end :(

In any case, thought I already did it, but just pushed up the changes for cwd from the package in non-workspace situations

matklad · 2017-12-12T20:00:05Z

@bors r+

bors · 2017-12-12T20:00:07Z

📌 Commit f688e9c has been approved by matklad

bors · 2017-12-12T20:00:16Z

⌛ Testing commit f688e9c with merge 4005bc4...

Avoid rebuilding a project when cwd changes This commit is targeted at solving a use case which typically comes up during CI builds -- the `target` directory is cached between builds but the cwd of the build changes over time. For example the following scenario can happen: 1. A project is compiled at `/projects/a`. 2. The `target` directory is cached. 3. A new build is started in `/projects/b`. 4. The previous `target` directory is restored to `/projects/b`. 5. The build start, and Cargo rebuilds everything. The last piece of behavior is indeed unfortunate! Cargo's internal hashing currently isn't that resilient to changing cwd and this PR aims to help improve the situation! The first point of too-much-hashing came up with `Target::src_path`. Each `Target` was hashed and stored for all compilations, and the `src_path` field was an absolute path on the filesystem to the file that needed to be compiled. This path then changed over time when cwd changed, but otherwise everything else remained the same! This commit updates the handling of the `src_path` field to simply ignore it when hashing. Instead the path we actually pass to rustc is later calculated and then passed to the fingerprint calculation. The next problem this fixes is that the dep info files were augmented after creation to have the cwd of the compiler at the time to find the files at a later date. This, unfortunately, would cause issues if the cwd itself changed. Instead the cwd is now left out of dep-info files (they're no longer augmented) and instead the cwd is recalculated when parsing the dep info later. The final problem that this commit fixes is actually an existing issue in Cargo today. Right now you can actually execute `cargo build` from anywhere in a project and Cargo will execute the build. Unfortunately though the argument to rustc was actually different depending on what directory you were in (the compiler was invoked with a path relative to cwd). This path ends up being used for metadata like debuginfo which means that different directories would cause different artifacts to be created, but Cargo wouldn't rerun the compiler! To fix this issue the matter of cwd is now entirely excluded from compilation command lines. Instead rustc is unconditionally invoked with a relative path *if* the path is underneath the workspace root, and otherwise it's invoked as an absolute path (in which case the cwd doesn't matter). Once all these fixes were added up it means that now we can have projects where if you move the entire directory Cargo won't rebuild the original source! Note that this may be a bit of a breaking change, however. This means that the paths in error messages for cargo will no longer be unconditionally relative to the current working directory, but rather relative to the root of the workspace itself. Unfortunately this is moreso of a feature right now rather than a bug, so it may be one that we just have to stomach. Closes #3273

bors · 2017-12-12T20:25:24Z

☀️ Test successful - status-appveyor, status-travis
Approved by: matklad
Pushing 4005bc4 to master...

This commit fixes flycheck#1397. By the recent change of cargo(rust-lang/cargo#4788), flycheck can't detect the error file. So I changed to use 'workspace_root' in 'cargo metadata's outputs, as working directory. If the user's cargo is a bit old and there's no 'workspace_root', flycheck-rust-manifest-directory search a directory with 'Cargo.toml', as ever.

Due to a recent change to cargo [1], in a workspace setting (multiple crates) filenames are now relative to the workspace root, whereas previously they were relative to the crate. This commit uses `cargo metadata` to figure out the workspace root, if it exists. If it doesn't, it fallbacks on the manifest directory. [1]: rust-lang/cargo#4788

rust-highfive assigned matklad Dec 7, 2017

matklad reviewed Dec 7, 2017

View reviewed changes

alexcrichton force-pushed the rename-works branch from fc7c39a to dd0b41b Compare December 8, 2017 15:52

matklad reviewed Dec 11, 2017

View reviewed changes

alexcrichton force-pushed the rename-works branch from dd0b41b to 4493400 Compare December 12, 2017 04:28

alexcrichton force-pushed the rename-works branch from 4493400 to f688e9c Compare December 12, 2017 19:00

bors merged commit f688e9c into rust-lang:master Dec 12, 2017

bors mentioned this pull request Dec 12, 2017

Start migration to the failure crate #4795

Merged

nrc mentioned this pull request Dec 18, 2017

Update Cargo and its dependencies rust-lang/rust#46630

Merged

alexcrichton deleted the rename-works branch December 18, 2017 19:47

This was referenced Dec 26, 2017

can't find target file on nightlies >= 2017-12-24 japaric/xargo#186

Closed

Relative path use in RUSTC_WRAPPER has broken #4865

Closed

robert-w-gries mentioned this pull request Jan 1, 2018

Allow users to use latest nightly compiler without manually setting environment variable phil-opp/blog_os#381

Merged

phil-opp mentioned this pull request Jan 9, 2018

xargo build fails after using stale dependencies built against different version of 'core' japaric/xargo#189

Open

Zoxc mentioned this pull request Jan 11, 2018

x.py build changed how paths are printed => hyperlinks (in e.g. emacs) do not work rust-lang/rust#47355

Closed

This was referenced Jan 11, 2018

How to detect workspace root? #4933

Closed

Add workspace root to metadata command. #4940

Merged

alexcrichton mentioned this pull request Jan 15, 2018

Regression in pwd of cargo doc rust-lang/rust#47434

Closed

ehuss mentioned this pull request Jan 15, 2018

It infers the wrong file path from json error msg rust-lang/rust-enhanced#234

Closed

alexcrichton mentioned this pull request Jan 16, 2018

Regression in the behavior of file!() when using workspaces rust-lang/rust#47462

Closed

davemilter mentioned this pull request Feb 1, 2018

warning/errors reporting breaks at nightly #4998

Closed

alexcrichton mentioned this pull request Feb 1, 2018

Add a compiler option for a 'pseudo-cwd' for rustc errors rust-lang/rust#47939

Closed

fmdkdd mentioned this pull request Feb 3, 2018

Checking doesn't work for workspace projects. flycheck/flycheck-rust#61

Closed

imp mentioned this pull request Feb 15, 2018

Workspace support may need some love after cargo#4788 rust-lang/atom-ide-rust#54

Closed

kngwyu mentioned this pull request Mar 7, 2018

Changed flycheck-rust-manifest-directory to use 'cargo metadata' flycheck/flycheck#1422

Merged

blueyed mentioned this pull request Sep 9, 2018

Neomake - workspaces are broken rust-lang/rust.vim#259

Closed

ehuss mentioned this pull request Jun 22, 2019

Utilize dep-info for all dependencies #7034

Closed

alexcrichton mentioned this pull request Jul 15, 2019

Fix some issues with absolute paths in dep-info files. #7137

Merged

bjorn3 mentioned this pull request Dec 5, 2020

Cargo runs rustc from the workspace root #8947

Open

ehuss added this to the 1.24.0 milestone Feb 6, 2022

grothesque mentioned this pull request Mar 21, 2024

Cargo should print appropriate relative paths when being run from a non-root folder #9887

Open

bjorn3 mentioned this pull request Aug 6, 2024

Rust-analyzer + rustc repo: build failures in the standard library are not shown in the editor any more rust-lang/rust#128726

Open

pnkfelix mentioned this pull request Aug 9, 2024

Opt-in flag for absolute paths in diagnostics rust-lang/compiler-team#770

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid rebuilding a project when cwd changes #4788

Avoid rebuilding a project when cwd changes #4788

alexcrichton commented Dec 7, 2017 •

edited

Loading

rust-highfive commented Dec 7, 2017

matklad left a comment

matklad Dec 7, 2017

alexcrichton Dec 7, 2017

matklad Dec 7, 2017

matklad Dec 7, 2017

alexcrichton commented Dec 7, 2017

matklad commented Dec 7, 2017

matklad commented Dec 7, 2017

alexcrichton commented Dec 8, 2017

matklad Dec 11, 2017

alexcrichton Dec 12, 2017

matklad Dec 12, 2017

matklad Dec 11, 2017

alexcrichton Dec 12, 2017

matklad commented Dec 11, 2017

alexcrichton commented Dec 12, 2017

matklad commented Dec 12, 2017

alexcrichton commented Dec 12, 2017

matklad commented Dec 12, 2017

bors commented Dec 12, 2017

bors commented Dec 12, 2017

bors commented Dec 12, 2017

Avoid rebuilding a project when cwd changes #4788

Avoid rebuilding a project when cwd changes #4788

Conversation

alexcrichton commented Dec 7, 2017 • edited Loading

rust-highfive commented Dec 7, 2017

matklad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Dec 7, 2017

matklad commented Dec 7, 2017

matklad commented Dec 7, 2017

alexcrichton commented Dec 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matklad commented Dec 11, 2017

alexcrichton commented Dec 12, 2017

matklad commented Dec 12, 2017

alexcrichton commented Dec 12, 2017

matklad commented Dec 12, 2017

bors commented Dec 12, 2017

bors commented Dec 12, 2017

bors commented Dec 12, 2017

alexcrichton commented Dec 7, 2017 •

edited

Loading