Implement a file-path remapping feature in support of debuginfo and reproducible builds #41508

michaelwoerister · 2017-04-24T17:15:52Z

This PR adds the -Zremap-path-prefix-from/-Zremap-path-prefix-to commandline option pair and is a more general implementation of #41419. As opposed to the previous attempt, this implementation should enable reproducible builds regardless of the working directory of the compiler.

This implementation of the feature is more general in the sense that the re-mapping will affect all paths the compiler emits, including the ones in error messages.

r? @alexcrichton

nagisa · 2017-04-24T20:27:19Z

src/librustc/session/config.rs

+        FilePathMapping::new(
+            self.debugging_opts.remap_path_prefix_from.iter().zip(
+                self.debugging_opts.remap_path_prefix_to.iter()
+            ).map(|(src, dst)| (src.clone(), dst.clone())).collect()


This seems like a potentially confusing CLUI to me. At this point I would simply make -Z remap-path take two arguments… obviously not something our argument parser can do.

Yeah, I know, it's not perfect. There's been quite some discussion about the CLUI in the initial issue (#38322). This seemed to be an acceptable compromise. It's an unstable feature at the moment. If our parser had some improvements before this is stabilized, we could certainly revisit this.

Given that the values are paths, and most file systems do not like colon ':' in path names, can't a single argument be bifurcated on a colon? Example: -Z remap-path="original_path_from:preferred_path_to".

alexcrichton · 2017-04-24T20:54:53Z

Looks great to me! Was it intentional to remove the same-number-arguments validation? (checking that the same number of -to as -from was supplied)

michaelwoerister · 2017-04-24T21:10:08Z

Was it intentional to remove the same-number-arguments validation?

Good catch! I'll add it back in tomorrow.

michaelwoerister · 2017-04-25T14:22:44Z

@rust-lang/docs, is there a place where a not-yet-stable feature like this can be documented?

nagisa · 2017-04-25T14:30:35Z

Unstable book would be the place except it doesn't support unstable flags due to overzealous tidy lint. On Apr 25, 2017 5:23 PM, "Michael Woerister" <notifications@github.com> wrote: @rust-lang/docs <https://github.com/orgs/rust-lang/teams/docs>, is there a place where a not-yet-stable feature like this can be documented? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#41508 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AApc0i66l7IKfBrC2LZELtMi5Kcib0nGks5rzgHMgaJpZM4NGdXh> .

frewsxcv · 2017-04-25T15:11:28Z

@rust-lang/docs, is there a place where a not-yet-stable feature like this can be documented?

Yep! Just add a new file to this directory and link to it in the 'compiler flags' section of the summary file.

frewsxcv · 2017-04-25T15:12:17Z

An example of an existing entry:

https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/linker-flavor.html

michaelwoerister · 2017-04-26T10:47:54Z

OK, I added some documentation to the unstable book and made the codegen test check more things.
r? @alexcrichton

bors · 2017-04-26T12:36:09Z

☔ The latest upstream changes (presumably #41504) made this pull request unmergeable. Please resolve the merge conflicts.

…eproducible builds.

alexcrichton

just a few nits, otherwise r=me

alexcrichton · 2017-04-26T13:59:54Z

src/doc/unstable-book/src/SUMMARY.md

@@ -2,6 +2,7 @@

 - [Compiler flags](compiler-flags.md)
    - [linker_flavor](compiler-flags/linker-flavor.md)
+    - [linker_flavor](compiler-flags/remap-path-prefix.md)


s/linker_flavor/remap_path_prefix/

alexcrichton · 2017-04-26T14:00:05Z

src/doc/unstable-book/src/compiler-flags/remap-path-prefix.md

@@ -0,0 +1,37 @@
+# `remap-path-prefix`
+
+The tracking issue for this feature is: None


Mind opening a tracking issue for this?

michaelwoerister · 2017-04-26T14:14:56Z

@bors r=alexcrichton

Nits fixed, I think.

bors · 2017-04-26T14:14:58Z

📌 Commit ab9691d has been approved by alexcrichton

jsgf · 2017-04-26T16:38:51Z

Thanks for turning this around @michaelwoerister! I hope to try this out soon.

BTW, assuming this gets stabilized in more or less the current form, what would the stable command-line options look like?

jsgf · 2017-04-26T17:15:34Z

src/doc/unstable-book/src/compiler-flags/remap-path-prefix.md

+```text
+rustc -Zremap-path-prefix-from="/home/foo/my-project/src" -Zremap-path-prefix-to="/sources/my-project"
+```
+


It's unclear from this how the remapping is actually operating. It should also answer:

Is it delimited by path element boundaries, or is it operating on literal strings?

Is there any normalization performed before/after doing the transformation? (Redundant / elimination?)

Are the input paths absolute, relative, or whatever was provided to the compiler?

In this example, if the input path were /home/foo/my-project/src-orig, would this remap to /sources/my-project-orig? Should there be a trailing / to prevent this if its undesirable?

Good points. I'll provide some more documentation in a follow-up PR. The implementation as it is does as little as possible: (1) Take each string as it is provided to the compiler, (2) don't do any normalization, (3) do a simple string-level prefix replacement.

jsgf · 2017-04-26T17:22:31Z

src/doc/unstable-book/src/compiler-flags/remap-path-prefix.md

+
+When the options are given multiple times, the nth `-from` will be matched up
+with the nth `-to` and they can appear anywhere on the commandline. Mappings
+specified later on the line will take precedence over earlier ones.


What does

Mappings specified later on the line will take precedence over earlier ones.

actually mean? Do you mean that if the -Zremap-path-prefix-from is identical it is replaced? Or does it more generally apply, such that a more specific path overrides a more general one?

For example, what's the mapping for:

rustc -Zremap-path-prefix-from=my-project/src -Zremap-path-prefix-to=/sources/my-project \ -Zremap-path-prefix-from=my-project/src/vendored/otherproject -Zremap-path-prefix-to=/sources/otherproject

when applied to my-project/src/vendored/otherproject/src/lib.rs?

In other words, does the ordering of the -from options matter when multiple mappings could apply? Or is there some other way to disambiguate multiple matching mappings?

Is it guaranteed that at most a single mapping can be applied, or is there some notion of repeated application of mappings until nothing matches?

I wonder if defining the matching as:

later remap-path-prefix-from replaces a previous identical one

mappings are matched from longest from prefix to shortest

would be better.

I think this would make it deterministically always select more specific mappings over more general ones, rather than relying purely on ordering.

That would differ from the current behaviour in the example a/b -> y, a -> x. I'm not sure which is better, but without a concrete argument that the new one is better I'd suggest to stick with the current behaviour which is simple to describe and implement. The other behaviour would require a trie-like data structure or other more complex thing.

I'm trying to standardise this behaviour across multiple compilers and GCC is already doing something very similar based on the ordering of command-line flags (and I have a patch pending to make it exactly match the behaviour being proposed in this PR). Changing this in the suggested way would make the spec for this standard even more complex. :(

In real-world cases I don't think the issue will crop up, in rustc or anywhere else - it's unlikely that a child process inheriting a prefix-map from a parent process would want to add a less-specific mapping for a higher-level directory.

The implementation is very simple: Walk all from/to pairs as provided on the commandline, starting at the last, and just stop at the first match.

In my opinion, relying only on ordering is a good approach, since the rules are clear and without surprises.

The current behaviour allows you to specify useless mappings (by putting more general before more specific in the reverse ordering), which might be inadvertent or unexpected.

I don't think it would require very fancy structures; I think you could implement it pretty simply using BTreeMap<String, String> instead of Vec<String, String>. It would be nice to have an algorithm that can have more efficient implementations (ie, not linear search), to cope with the case where there are lots of mappings (but that's definitely not a requirement for the first implementation).

@michaelwoerister

In my opinion, relying only on ordering is a good approach, since the rules are clear and without surprises

I think that's fine - I think the wording could do with clarification (ie, I managed to misinterpret what it meant until I looked at the source).

Noted. I'll provide an update in a subsequent PR and make sure to ping the usual suspects for review.

Using the ordering of flags for the lookup allows easy overriding of previous flags without needing to have a way to remove items from an accumulated list of flags (ex: presuming a variable in some higher level build env is accumulating RUST_FLAGS, using ordering means there is no need to parse all the accumulated flags to figure out which ones to remove when adding a higher priority remapping).

Well, that might be useful to have anyway, since an identity mapping is semantically different from no mapping at all.

codyps · 2017-04-26T17:28:53Z

As this remaps all paths, will this also resolve #40552 ? ie: does it affect the expansion of file!()?

jsgf · 2017-04-26T17:34:14Z

@jmesmon - I believe that's the intent.

michaelwoerister · 2017-04-27T10:43:34Z

@TimNN Thanks for the tip! But the problem is already caused by the arguments passed to the compiler, I think.

michaelwoerister · 2017-04-27T10:49:55Z

But the problem is already caused by the arguments passed to the compiler, I think.

More specifically, the test case tells the compiler to replace something Unix-specific here:

-Zremap-path-prefix-from={{src-base}}/remap_path_prefix/auxiliary

But on Windows the compiler will see the equivalent of

{{src-base}}\remap_path_prefix\auxiliary
            ^                 ^

😭

alexcrichton · 2017-04-27T14:14:18Z

@michaelwoerister I'd also consider it ok to ignore the test on windows, it seems like we may not benefit much from the extra coverage there

michaelwoerister · 2017-04-28T11:34:27Z

@bors r=alexcrichton

OK, since we are only doing string replacement there should be nothing platform-specific in there atm, so I
disabled the test on Windows. Thanks, @alexcrichton!

bors · 2017-04-28T11:34:28Z

📌 Commit 8ea050d has been approved by alexcrichton

bors · 2017-04-28T11:34:36Z

⌛ Testing commit 8ea050d with merge 9339127...

bors · 2017-04-28T12:06:15Z

💔 Test failed - status-travis

TimNN · 2017-04-28T12:09:35Z

@bors retry

Spurious failures in sccache #40240

bors · 2017-04-28T12:09:44Z

⌛ Testing commit 8ea050d with merge 2971d49...

@alexcrichton

…xcrichton Implement a file-path remapping feature in support of debuginfo and reproducible builds This PR adds the `-Zremap-path-prefix-from`/`-Zremap-path-prefix-to` commandline option pair and is a more general implementation of #41419. As opposed to the previous attempt, this implementation should enable reproducible builds regardless of the working directory of the compiler. This implementation of the feature is more general in the sense that the re-mapping will affect *all* paths the compiler emits, including the ones in error messages. r? @alexcrichton

bors · 2017-04-28T14:45:39Z

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 2971d49 to master...

rust-lang/rust#41508

Fix `--remap-path-prefix` not correctly remapping `rust-src` component paths and unify handling of path mapping with virtualized paths This PR fixes rust-lang#73167 ("Binaries end up containing path to the rust-src component despite `--remap-path-prefix`") by preventing real local filesystem paths from reaching compilation output if the path is supposed to be remapped. `RealFileName::Named` introduced in rust-lang#72767 is now renamed as `LocalPath`, because this variant wraps a (most likely) valid local filesystem path. `RealFileName::Devirtualized` is renamed as `Remapped` to be used for remapped path from a real path via `--remap-path-prefix` argument, as well as real path inferred from a virtualized (during compiler bootstrapping) `/rustc/...` path. The `local_path` field is now an `Option<PathBuf>`, as it will be set to `None` before serialisation, so it never reaches any build output. Attempting to serialise a non-`None` `local_path` will cause an assertion faliure. When a path is remapped, a `RealFileName::Remapped` variant is created. The original path is preserved in `local_path` field and the remapped path is saved in `virtual_name` field. Previously, the `local_path` is directly modified which goes against its purpose of "suitable for reading from the file system on the local host". `rustc_span::SourceFile`'s fields `unmapped_path` (introduced by rust-lang#44940) and `name_was_remapped` (introduced by rust-lang#41508 when `--remap-path-prefix` feature originally added) are removed, as these two pieces of information can be inferred from the `name` field: if it's anything other than a `FileName::Real(_)`, or if it is a `FileName::Real(RealFileName::LocalPath(_))`, then clearly `name_was_remapped` would've been false and `unmapped_path` would've been `None`. If it is a `FileName::Real(RealFileName::Remapped{local_path, virtual_name})`, then `name_was_remapped` would've been true and `unmapped_path` would've been `Some(local_path)`. cc `@eddyb` who implemented `/rustc/...` path devirtualisation

rust-highfive assigned alexcrichton Apr 24, 2017

This was referenced Apr 24, 2017

Implement debuginfo path remapping #41419

Closed

Implement debuginfo path remapping #38348

Closed

nagisa reviewed Apr 24, 2017

View reviewed changes

arielb1 added the relnotes Marks issues that should be documented in the release notes of the next release. label Apr 25, 2017

michaelwoerister force-pushed the generic-path-remapping branch from 5b20acc to 7be6f22 Compare April 26, 2017 13:41

michaelwoerister added 3 commits April 26, 2017 15:44

Implement a file-path remapping feature in support of debuginfo and r…

39ffea3

…eproducible builds.

remap-path-prefix: Validate number of commandline arguments passed.

eb6308f

Make codegen test for remap-path-prefix more thorough.

5c62ce4

michaelwoerister force-pushed the generic-path-remapping branch from 7be6f22 to 4dbd8a3 Compare April 26, 2017 13:44

alexcrichton reviewed Apr 26, 2017

View reviewed changes

michaelwoerister mentioned this pull request Apr 26, 2017

Tracking issue for Path Prefix Remapping #41555

Closed

Add some documentation for -Zremap-path-prefix to the unstable book

ab9691d

michaelwoerister force-pushed the generic-path-remapping branch from 4dbd8a3 to ab9691d Compare April 26, 2017 14:13

aidanhs added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Apr 26, 2017

jsgf reviewed Apr 26, 2017

View reviewed changes

alexcrichton added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Apr 27, 2017

Disable path remapping test on Windows.

8ea050d

bors merged commit 8ea050d into rust-lang:master Apr 28, 2017

bors mentioned this pull request Apr 28, 2017

Build instruction profiler runtime as part of compiler-rt #38608

Closed

kevinmehall added a commit to kevinmehall/rust-peg that referenced this pull request Apr 29, 2017

rust: Fix peg-syntax-ext for filemap API change

de96c18

rust-lang/rust#41508

infinity0 mentioned this pull request May 13, 2017

Bit-for-bit deterministic / reproducible builds #34902

Closed

dwrensha mentioned this pull request May 16, 2017

update for latest nightly rustc serde-deprecated/aster#134

Closed

Mark-Simulacrum mentioned this pull request Jun 6, 2017

Rust 1.19 regression, bugsnag 0.1.3 #42457

Closed

cbeuw mentioned this pull request Apr 3, 2021

Fix --remap-path-prefix not correctly remapping rust-src component paths and unify handling of path mapping with virtualized paths #83813

Merged

jsgf mentioned this pull request Aug 19, 2021

--remap-path-prefix no longer works for compiler messages #87745

Closed

ehuss mentioned this pull request May 9, 2022

Docs say --remap-path-prefix treats paths as strings during matching but compiler does not #96870

Open

cassaundra mentioned this pull request Jun 14, 2023

Frequent use of line 0 in debug info reduces effectiveness of Cachegrind #65487

Open

		@@ -0,0 +1,37 @@
		# `remap-path-prefix`

		The tracking issue for this feature is: None

Implement a file-path remapping feature in support of debuginfo and reproducible builds #41508

Implement a file-path remapping feature in support of debuginfo and reproducible builds #41508

Conversation

michaelwoerister commented Apr 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Apr 24, 2017

michaelwoerister commented Apr 24, 2017

michaelwoerister commented Apr 25, 2017

nagisa commented Apr 25, 2017 via email

frewsxcv commented Apr 25, 2017

frewsxcv commented Apr 25, 2017

michaelwoerister commented Apr 26, 2017

bors commented Apr 26, 2017

alexcrichton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelwoerister commented Apr 26, 2017

bors commented Apr 26, 2017

jsgf commented Apr 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsgf Apr 26, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codyps Apr 26, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codyps commented Apr 26, 2017

jsgf commented Apr 26, 2017

michaelwoerister commented Apr 27, 2017

michaelwoerister commented Apr 27, 2017

alexcrichton commented Apr 27, 2017

michaelwoerister commented Apr 28, 2017

bors commented Apr 28, 2017

bors commented Apr 28, 2017

bors commented Apr 28, 2017

TimNN commented Apr 28, 2017

bors commented Apr 28, 2017

bors commented Apr 28, 2017

jsgf Apr 26, 2017 •

edited

codyps Apr 26, 2017 •

edited