[WIP] Content hash support. (See also cargo changes) #75594

gilescope · 2020-08-16T16:08:02Z

This PR adds a makefile comment (on the following line as can't be the same line in makefiles) with the file size and a hash of the file in the deps/*.d files.

The main PR is on cargo's repo: rust-lang/cargo#8623

This is to enable cargo to be able to use file sizes and content hashes as well as mtimes to both detect rapid changes that are currently missed and also to prevent compile cascades where mtimes have been altered but no actual change to the content occured (E.g. docker layers typically don't store precise times or copying a target dir from one place to another).

It seems that the hashes are already being calculated in the compiler so this is almost zero cost to rustc - the changes are:

Inside .rlib, lib.rmeta is renamed to libcrate_name-svh.rmeta
For .rmeta files we add the svh to a known location in the uncompressed header.
For dylibs we add one additional symbol containing the svh.

(For outanding points please see the cargo PR. For binary file tracking we build on #63012 )

rust-highfive · 2020-08-16T16:08:05Z

r? @ecstatic-morse

(rust_highfive has picked a reviewer for you, use r? to override)

gilescope · 2020-08-16T20:45:54Z

That looks a legit test failure. Let me look into that.

src/librustc_interface/passes.rs

gilescope · 2020-08-18T07:12:05Z

Hmm, so that test failure wasn't legit and went away. Meanwhile spotted we already use a SourceFileHashAlgorithm for hashing source files for putting in debug info.... maybe there's no need for an additional hashing of the source - that would be v. cute.

bjorn3 · 2020-08-18T10:18:50Z

Compiled dependencies and probably the output files also need to be hashed.

gilescope · 2020-08-20T07:43:33Z

Updated it so that we use the precomputed hashes - this makes this change effectively zero cost now as all the source was being hashed elsewhere. That's much better than I was exepcting (and given that maybe there's no need to put it behind a -Z rustc flag?).

That's a really weird compiler error - I don't have a mental model how that could happen. Locally it's an Rc but the CI thinks it's an Arc. It's like the CI isn't actually compiling the branch of code I'm on, but is merging the patch onto master and building that? (Maybe the process has changed a bit? - I've taken a fair few months off from submissions)

gilescope · 2020-08-20T07:46:14Z

How should the hashes [u8] should be turned into utf8? Seems there's pleanty of options to turn bin to text - for now I've dodge that and just used the debug formatting of [u8] to turn it to a string but suggestions welcome!

bjorn3 · 2020-08-20T08:13:41Z

Locally it's an Rc but the CI thinks it's an Arc.

There is a type in rustc_data_structures that is an Rc when parallel rustc is disabled and an Arc when enabled.

bors · 2020-08-30T17:59:13Z

☔ The latest upstream changes (presumably #74862) made this pull request unmergeable. Please resolve the merge conflicts.

gilescope · 2020-10-25T06:09:40Z

@ehuss do my changes to base.rs look vaguely in the right direction? Once I can get one symbol appearing in the output that I can find I'll be unstuck. But I'm really clueless about this codegen side at the moment - I've not touched this part of the compiler at all before. Any hints are greatfully appreciated and may shave months of headbanging off this PR...

gilescope · 2020-10-25T06:50:39Z

Well this have been a voyage of discovery, I made a hi.rs that printed 'Hi'. I can see the section coming out in the llvm IR code and I can see it coming out in the generated assembly code if I do:

rustc --emit=llvm-bc,llvm-ir

I can see it in hi.s machine assembler that's produced by llc ./hi.bc and if I do as hi.s then I get an a.out and when I vi inside that I can see the section (but the a.out doesn't run so I can't be incanting everything in the same way that cargo/rustc/llvm is).

So I guess it's all working fine but some optimisation layer is removing it (linker maybe?). I will try and see if I can hint to the linker not to remove it...

compiler/rustc_codegen_llvm/src/base.rs

bjorn3 · 2020-10-25T08:22:00Z

So I guess it's all working fine but some optimisation layer is removing it (linker maybe?). I will try and see if I can hint to the linker not to remove it...

This for executables? You may need to do the same as for the gdb debug scripts section. (reference it from the main shim using a volatile load)

rust/compiler/rustc_codegen_ssa/src/base.rs

Line 429 in 17cc9b6

bx.insert_reference_to_gdb_debug_scripts_section_global();

gilescope · 2020-10-28T10:47:35Z

Seem to get the symbol coming out for rmeta and executables now. Just trying to add the symbol to the dylib (it's not coming out in the .ll so I've got a way to go there...).

compiler/rustc_codegen_llvm/src/base.rs

bjorn3 · 2020-10-28T15:29:52Z

The cdylib doesn't contain the svh symbol, as the metadata module is omitted for cdylibs.

compiler/rustc_codegen_ssa/src/back/link.rs

compiler/rustc_metadata/src/rmeta/mod.rs

compiler/rustc_codegen_ssa/src/back/write.rs

bjorn3 · 2020-11-01T15:12:44Z

compiler/rustc_interface/src/passes.rs

            .collect();

        if let Some(ref backend) = sess.opts.debugging_opts.codegen_backend {
-            files.push(backend.to_string());
+            files.push((backend.to_string(), None)); // TO DO: is this something we need to hash?


Yes. It is a dylib, so it contains an SVH. You could add a parameter with the result of codegen_backend.metadata_loader() to this function to read it, but that looks like a bit of a hack.

compiler/rustc_metadata/src/rmeta/encoder.rs

compiler/rustc_interface/src/passes.rs

compiler/rustc_span/src/lib.rs

denser encoding format Co-authored-by: bjorn3 <bjorn3@users.noreply.github.com>

bjorn3 · 2020-11-14T10:56:30Z

Did you also add the corresponding decode call?

gilescope · 2020-11-14T16:00:19Z

I see where I was being tripped up - the encode was converting an array to a slice before writing it so there's a length in there as well messing up my offsets. I should have guessed that might happen sooner...

bors · 2020-11-15T15:45:26Z

☔ The latest upstream changes (presumably #79070) made this pull request unmergeable. Please resolve the merge conflicts.

Note that reviewers usually do not review pull requests until merge conflicts are resolved! Once you resolve the conflicts, you should change the labels applied by bors to indicate that your PR is ready for review. Post this as a comment to change the labels:

@rustbot modify labels: +S-waiting-on-review -S-waiting-on-author

bors · 2020-12-16T15:43:45Z

☔ The latest upstream changes (presumably #77117) made this pull request unmergeable. Please resolve the merge conflicts.

Note that reviewers usually do not review pull requests until merge conflicts are resolved! Once you resolve the conflicts, you should change the labels applied by bors to indicate that your PR is ready for review. Post this as a comment to change the labels:

@rustbot modify labels: +S-waiting-on-review -S-waiting-on-author

Dylan-DPC-zz · 2021-02-28T00:12:46Z

@gilescope any updates on this?

gilescope · 2021-03-24T08:24:49Z

This PR is good, but blocked waiting on the cargo changes. The cargo PR needs another round of work to nail the build.rs interactions.

JohnCSimon · 2022-11-27T01:43:43Z

@gilescope
Ping from triage: I'm closing this due to inactivity, Please reopen when you are ready to continue with this.
Note: if you do please open the PR BEFORE you push to it, else you won't be able to reopen - this is a quirk of github.
Thanks for your contribution.

@rustbot label: +S-inactive

rust-highfive assigned ecstatic-morse Aug 16, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 16, 2020

gilescope mentioned this pull request Aug 16, 2020

[WIP] mtime+content tracking rust-lang/cargo#8623

Closed

12 tasks

bjorn3 reviewed Aug 16, 2020

View reviewed changes

src/librustc_interface/passes.rs Outdated Show resolved Hide resolved

bjorn3 reviewed Aug 17, 2020

View reviewed changes

src/librustc_interface/passes.rs Outdated Show resolved Hide resolved

gilescope mentioned this pull request Aug 30, 2020

Add SVH based symbol to dylibs #73917

Open

gilescope force-pushed the endmtime branch from 46ad019 to eeea611 Compare September 5, 2020 11:49

gilescope changed the title ~~WIP: Content hash support. (See also cargo changes)~~ [WIP] Content hash support. (See also cargo changes) Sep 20, 2020

crlf0710 marked this pull request as draft October 8, 2020 11:46

crlf0710 added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 8, 2020

bjorn3 reviewed Oct 25, 2020

View reviewed changes

compiler/rustc_codegen_llvm/src/base.rs Outdated Show resolved Hide resolved

bjorn3 reviewed Oct 28, 2020

View reviewed changes

compiler/rustc_codegen_llvm/src/base.rs Outdated Show resolved Hide resolved

gilescope force-pushed the endmtime branch from d6ccfed to 5a65dfa Compare October 28, 2020 16:09

gilescope mentioned this pull request Oct 29, 2020

Rustc does not warn when .so/.rlib files are not in sync #10786

Closed

bjorn3 reviewed Oct 29, 2020

View reviewed changes

compiler/rustc_codegen_ssa/src/back/link.rs Outdated Show resolved Hide resolved

gilescope force-pushed the endmtime branch from 092d2d5 to fad2de6 Compare October 29, 2020 10:23

gilescope added 2 commits November 1, 2020 14:14

undoing cargo fmt

efb32e5

Reducing diff

4a1b9e0

bjorn3 requested changes Nov 1, 2020

View reviewed changes

gilescope and others added 3 commits November 1, 2020 18:09

tidy

e979534

Apply suggestions from code review

21ad026

denser encoding format Co-authored-by: bjorn3 <bjorn3@users.noreply.github.com>

Work on non-osx platforms

ced4bfb

gilescope added 5 commits November 14, 2020 17:55

svh moved before version and metadata version bumped

b3452a4

EXEs can't be dependend on so no need currently to add an svh in them.

aaccda2

Merge branch 'master' into endmtime

dd8d335

Fixup following introduction of sha256

e4c08f3

cargo fmt

a9c9e6b

gilescope added 4 commits November 15, 2020 20:47

consistent ordering

5021fd7

Merge branch 'master' into endmtime

56e671c

renamed fn

1b3c8c0

./x.py fmt

6848ea6

ehuss mentioned this pull request Jan 6, 2021

Warnings missing on second compilation on OSX #80751

Closed

gilescope mentioned this pull request Jun 3, 2021

(Option to) Fingerprint by file contents instead of mtime rust-lang/cargo#6529

Open

Dylan-DPC-zz added S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 26, 2021

ecstatic-morse removed their assignment Feb 25, 2022

jonhoo mentioned this pull request Jun 13, 2022

Tracking issue for -Z binary-dep-depinfo #63012

Open

2 tasks

JohnCSimon closed this Nov 27, 2022

rustbot added the S-inactive Status: Inactive and waiting on the author. This is often applied to closed PRs. label Nov 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Content hash support. (See also cargo changes) #75594

[WIP] Content hash support. (See also cargo changes) #75594

gilescope commented Aug 16, 2020 •

edited

Loading

rust-highfive commented Aug 16, 2020

gilescope commented Aug 16, 2020

gilescope commented Aug 18, 2020

bjorn3 commented Aug 18, 2020

gilescope commented Aug 20, 2020 •

edited

Loading

gilescope commented Aug 20, 2020

bjorn3 commented Aug 20, 2020

bors commented Aug 30, 2020

gilescope commented Oct 25, 2020

gilescope commented Oct 25, 2020 •

edited

Loading

bjorn3 commented Oct 25, 2020

gilescope commented Oct 28, 2020

bjorn3 commented Oct 28, 2020

bjorn3 Nov 1, 2020

bjorn3 commented Nov 14, 2020

gilescope commented Nov 14, 2020

bors commented Nov 15, 2020

bors commented Dec 16, 2020

Dylan-DPC-zz commented Feb 28, 2021

gilescope commented Mar 24, 2021

JohnCSimon commented Nov 27, 2022

[WIP] Content hash support. (See also cargo changes) #75594

[WIP] Content hash support. (See also cargo changes) #75594

Conversation

gilescope commented Aug 16, 2020 • edited Loading

rust-highfive commented Aug 16, 2020

gilescope commented Aug 16, 2020

gilescope commented Aug 18, 2020

bjorn3 commented Aug 18, 2020

gilescope commented Aug 20, 2020 • edited Loading

gilescope commented Aug 20, 2020

bjorn3 commented Aug 20, 2020

bors commented Aug 30, 2020

gilescope commented Oct 25, 2020

gilescope commented Oct 25, 2020 • edited Loading

bjorn3 commented Oct 25, 2020

gilescope commented Oct 28, 2020

bjorn3 commented Oct 28, 2020

bjorn3 Nov 1, 2020

Choose a reason for hiding this comment

bjorn3 commented Nov 14, 2020

gilescope commented Nov 14, 2020

bors commented Nov 15, 2020

bors commented Dec 16, 2020

Dylan-DPC-zz commented Feb 28, 2021

gilescope commented Mar 24, 2021

JohnCSimon commented Nov 27, 2022

gilescope commented Aug 16, 2020 •

edited

Loading

gilescope commented Aug 20, 2020 •

edited

Loading

gilescope commented Oct 25, 2020 •

edited

Loading