Cargo should not consider the build script as stale with every modification #1162

tomaka · 2015-01-13T19:12:03Z

When a package has a build script and you modify any file in your project, everything gets recompiled.

I'm currently working on a library named glium which is starting be have a moderate size and which uses build scripts only for code generation.

Modifying the code of an example or a test and running "cargo build" recompiles everything, which takes 3-4 minutes on my machine. Due to this, it's very annoying to write tests or examples.

Suggestion

Build scripts should output cargo:depends=/path/to/file just like they output rustc-flags. Only modifications in the files which they depend upon, plus the build script itself and its modules, trigger a recompilation.

The text was updated successfully, but these errors were encountered:

alexcrichton · 2015-01-14T02:06:29Z

This seems like a pretty great use of the metadata output to me, I like the idea!

I would add one caveat that if cargo:depends is not printed, then the behavior is the same as it is today. That means that if you go out of your way to print dependencies then you get better incremental times, but if you don't do anything we at least won't do anything wrong!

tomaka · 2015-01-14T07:26:58Z

What if a build script has no dependency? Should it print cargo:depends= with nothing behind, or should it print cargo:no-deps?

alexcrichton · 2015-01-14T07:37:17Z

I'd be down with either, but the latter seems like a better idea

marcusklaas · 2015-07-07T18:22:08Z

Is this something that could be done by someone with little knowledge of Cargo's internals?

I'd like to work on this, but it would be nice to have a sense of the complexity involved first..

alexcrichton · 2015-07-07T23:46:59Z

@marcusklaas yeah I think this should be relatively easy. You'd want to add parsing code for this key being printed out first. Next you'd want to serialize it out to a custom location (probably something in the relevant fingerprint directory).

You'd then update the build script fingerprint routine to take this file into account.

marcusklaas · 2015-07-08T20:59:27Z

Thanks for the pointers. I'll give it a shot!

The script crate currently has a build script, and Cargo will consider all files in the script crate as inputs to the build script as it otherwise doesn't know [what the input files are][cargo-1162]. This means that if any file in the source tree of the script crate changes (or is created) then Cargo will think it needs to re-run the build script and rebuild the crate. [cargo-1162]: rust-lang/cargo#1162 The build script of the script crate is invoking python, and consequently Python is generating some bytecode files in the source tree. On the second build of Servo, Cargo will see these new files, think that something has changed, and will re-run the build script of the script crate. This change passes the `-B` flag to python to avoid generating these bytecode files, which should avoid tampering with the source tree and appease Cargo by ensuring that it doesn't get rebuilt. --- As a helpful tip to if this comes up again, this was discovered by using the changes in rust-lang/cargo@c447e9d plus the change in rust-lang/cargo#2044. Once `RUST_LOG` was set to `cargo::ops::cargo_rustc::fingerprint=info`, the second run of `./mach build` printed out: ``` precalculated components have changed: 1444364448.000000000s (/build/servo/components/script/dom/bindings/codegen/parser/WebIDL.pyc) != 1444364235.000000000s (/build/servo/components/script/document_loader.rs) ``` Which should help easily diagnose these kinds of problems in the future!

Fix spurious rebuilds of the script crate The script crate currently has a build script, and Cargo will consider all files in the script crate as inputs to the build script as it otherwise doesn't know [what the input files are][cargo-1162]. This means that if any file in the source tree of the script crate changes (or is created) then Cargo will think it needs to re-run the build script and rebuild the crate. [cargo-1162]: rust-lang/cargo#1162 The build script of the script crate is invoking python, and consequently Python is generating some bytecode files in the source tree. On the second build of Servo, Cargo will see these new files, think that something has changed, and will re-run the build script of the script crate. This change passes the `-B` flag to python to avoid generating these bytecode files, which should avoid tampering with the source tree and appease Cargo by ensuring that it doesn't get rebuilt. --- As a helpful tip to if this comes up again, this was discovered by using the changes in rust-lang/cargo@c447e9d plus the change in rust-lang/cargo#2044. Once `RUST_LOG` was set to `cargo::ops::cargo_rustc::fingerprint=info`, the second run of `./mach build` printed out: ``` precalculated components have changed: 1444364448.000000000s (/build/servo/components/script/dom/bindings/codegen/parser/WebIDL.pyc) != 1444364235.000000000s (/build/servo/components/script/document_loader.rs) ``` Which should help easily diagnose these kinds of problems in the future!  [<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/servo/servo/7936)

Currently Cargo is quite conservative in how it determines whether a build script should be run. The heuristic used is "did any file in the project directory change", but this is almost always guaranteed to be too coarse grained in situations like: * If the build script takes a long time to run it's advantageous to run it as few times as possible. Being able to inform Cargo about precisely when a build script should be run should provide more robust support here. * Build scripts may not always have all of their dependencies in-tree or in the crate root. Sometimes a dependency could be elsewhere in a repository and scripts need a method of informing Cargo about this (as currently these compiles don't happen then they should). This commit adds this support in build scripts via a new `rerun-if-changed` directive which can be printed to standard output (using the standard Cargo metadata format). The value for this key is a path relative to the crate root, and Cargo will only look at these paths when determining whether to rerun the build script. Any other file changes will not trigger the build script to be rerun. Future extensions could possibly include the usage of glob patterns in build script paths like the `include` and `exclude` features of `Cargo.toml`, but these should be backwards compatible to add in the future. Closes rust-lang#1162

Currently Cargo is quite conservative in how it determines whether a build script should be run. The heuristic used is "did any file in the project directory change", but this is almost always guaranteed to be too coarse grained in situations like: * If the build script takes a long time to run it's advantageous to run it as few times as possible. Being able to inform Cargo about precisely when a build script should be run should provide more robust support here. * Build scripts may not always have all of their dependencies in-tree or in the crate root. Sometimes a dependency could be elsewhere in a repository and scripts need a method of informing Cargo about this (as currently these compiles don't happen then they should). This commit adds this support in build scripts via a new `rerun-if-changed` directive which can be printed to standard output (using the standard Cargo metadata format). The value for this key is a path relative to the crate root, and Cargo will only look at these paths when determining whether to rerun the build script. Any other file changes will not trigger the build script to be rerun. Currently the printed paths may either be a file or a directory, and a directory is deeply traversed. The heuristic for trigger a rerun is detecting whether any input file has been modified since the build script was last run (determined by looking at the modification time of the output file of the build script). This current implementation means that if you depend on a directory and then delete a file within it the build script won't be rerun, but this is already the case and can perhaps be patched up later. Future extensions could possibly include the usage of glob patterns in build script paths like the `include` and `exclude` features of `Cargo.toml`, but these should be backwards compatible to add in the future. Closes #1162

Currently Cargo is quite conservative in how it determines whether a build script should be run. The heuristic used is "did any file in the project directory change", but this is almost always guaranteed to be too coarse grained in situations like: * If the build script takes a long time to run it's advantageous to run it as few times as possible. Being able to inform Cargo about precisely when a build script should be run should provide more robust support here. * Build scripts may not always have all of their dependencies in-tree or in the crate root. Sometimes a dependency could be elsewhere in a repository and scripts need a method of informing Cargo about this (as currently these compiles don't happen then they should). This commit adds this support in build scripts via a new `rerun-if-changed` directive which can be printed to standard output (using the standard Cargo metadata format). The value for this key is a path relative to the crate root, and Cargo will only look at these paths when determining whether to rerun the build script. Any other file changes will not trigger the build script to be rerun. Future extensions could possibly include the usage of glob patterns in build script paths like the `include` and `exclude` features of `Cargo.toml`, but these should be backwards compatible to add in the future. Closes rust-lang#1162

…alexcrichton:script-rebuild); r=jdm The script crate currently has a build script, and Cargo will consider all files in the script crate as inputs to the build script as it otherwise doesn't know [what the input files are][cargo-1162]. This means that if any file in the source tree of the script crate changes (or is created) then Cargo will think it needs to re-run the build script and rebuild the crate. [cargo-1162]: rust-lang/cargo#1162 The build script of the script crate is invoking python, and consequently Python is generating some bytecode files in the source tree. On the second build of Servo, Cargo will see these new files, think that something has changed, and will re-run the build script of the script crate. This change passes the `-B` flag to python to avoid generating these bytecode files, which should avoid tampering with the source tree and appease Cargo by ensuring that it doesn't get rebuilt. --- As a helpful tip to if this comes up again, this was discovered by using the changes in rust-lang/cargo@c447e9d plus the change in rust-lang/cargo#2044. Once `RUST_LOG` was set to `cargo::ops::cargo_rustc::fingerprint=info`, the second run of `./mach build` printed out: ``` precalculated components have changed: 1444364448.000000000s (/build/servo/components/script/dom/bindings/codegen/parser/WebIDL.pyc) != 1444364235.000000000s (/build/servo/components/script/document_loader.rs) ``` Which should help easily diagnose these kinds of problems in the future! Source-Repo: https://github.com/servo/servo Source-Revision: bc5444067f7614ab7e3c83d8e131800d0db817b2

…alexcrichton:script-rebuild); r=jdm The script crate currently has a build script, and Cargo will consider all files in the script crate as inputs to the build script as it otherwise doesn't know [what the input files are][cargo-1162]. This means that if any file in the source tree of the script crate changes (or is created) then Cargo will think it needs to re-run the build script and rebuild the crate. [cargo-1162]: rust-lang/cargo#1162 The build script of the script crate is invoking python, and consequently Python is generating some bytecode files in the source tree. On the second build of Servo, Cargo will see these new files, think that something has changed, and will re-run the build script of the script crate. This change passes the `-B` flag to python to avoid generating these bytecode files, which should avoid tampering with the source tree and appease Cargo by ensuring that it doesn't get rebuilt. --- As a helpful tip to if this comes up again, this was discovered by using the changes in rust-lang/cargoc447e9d plus the change in rust-lang/cargo#2044. Once `RUST_LOG` was set to `cargo::ops::cargo_rustc::fingerprint=info`, the second run of `./mach build` printed out: ``` precalculated components have changed: 1444364448.000000000s (/build/servo/components/script/dom/bindings/codegen/parser/WebIDL.pyc) != 1444364235.000000000s (/build/servo/components/script/document_loader.rs) ``` Which should help easily diagnose these kinds of problems in the future! Source-Repo: https://github.com/servo/servo Source-Revision: bc5444067f7614ab7e3c83d8e131800d0db817b2 UltraBlame original commit: 2ce25f1cba4b757bbbd0a09e9b0dc22a187eefa7

Upgrade to mdBook v0.3.7 This bumps the requirement from Rust v1.34.0 to v1.35.0 for building docs. AFAICT CI is using nightlies so that should be fine, but I thought I'd mention it in case someone thinks this impacts contributors in any way. Other than that, there are a few changes that might impact some users in a visible way, like automatic dark theme support for those who picked that perference in their browser, possible color changes to the scrollbar and to the font size, change in the spacing in the sidebar entries, and many more changes and fixes that won't be too immediately impactful but very good all around. I checked changes from transitive dependency bumps as well, AFAICT there is nothing that *should* impact the final rendering. **tl;dr:** Nothing will explode. Probably. For completeness, my raw notes of outtakes as I was reviewing the change logs: ``` [cosmetic] v0.3.2 https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#mdbook-032 Added automatic dark-theme detection based on the CSS prefers-color-scheme feature. This may be enabled by setting output.html.preferred-dark-theme to your preferred dark theme. #1037 rust-lang/mdBook#1037 v0.3.3 https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#mdbook-033 Improvements to the automatic dark theme selection. #1069 rust-lang/mdBook#1069 v0.3.7 https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#mdbook-037 Fixed theme selector focus. #1170 rust-lang/mdBook#1170 * https://developer.mozilla.org/en-US/docs/Web/CSS/@media/prefers-color-scheme * users who picked the dark color scheme in their browser will see the cargo doc in dark. [cosmetic] v0.3.2 https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#mdbook-032 Use standard scrollbar-color CSS along with webkit extension #816 rust-lang/mdBook#816 * https://developer.mozilla.org/en-US/docs/Web/CSS/scrollbar-color * scroll bar color might change i guess. [helpful] v0.3.2 https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#mdbook-032 Updated highlight.js for syntax highlighting updates (primarily to add async/await to Rust highlighting). #1041 rust-lang/mdBook#1041 * not sure cargo doc has many code examples with async/await, but there we go. [warning] v0.3.2 https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#mdbook-032 Raised minimum supported rust version to 1.35. #1003 rust-lang/mdBook#1003 * from 1.34.0. [cosmetic] v0.3.4 https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#mdbook-034 Switch to relative rem font sizes from px. #894 rust-lang/mdBook#894 * will impact some displays, but px is already an abstract thing so maybe not that big of an impact. [warning] v0.3.5 https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#mdbook-035 Updated pulldown-cmark to 0.6.1, fixing several issues. #1021 rust-lang/mdBook#1021 * from 0.5, breaking changes. * parsing only -- the team had to do multiple changes but nothing seems like it would impact final rendering [cosmetic] v0.3.6 https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#mdbook-036 Adjusted spacing of sidebar entries. #1137 rust-lang/mdBook#1137 [warning] v0.3.6 https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#mdbook-036 Handlebars updated to 3.0. #1130 rust-lang/mdBook#1130 * from 2.0 * https://github.com/sunng87/handlebars-rust/blob/master/CHANGELOG.md * strictly maintenance and perf AFAICS, no changes to final rendering. [cosmetic] v0.3.6 https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#mdbook-036 Adjusted the menu bar animation to not immediately obscure the top content. #989 rust-lang/mdBook#989 * personal fave. [cosmetic] v0.3.7 https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#mdbook-037 Code spans in headers are no longer highlighted as code. #1162 * users will see some headers change, probably. [fixes] + ~13 fixes impacting rendering in less immediately visible ways. rest should have no impact on end-user experience. ```

alexcrichton mentioned this issue Jan 14, 2015

Should include! be recommended? #824

Closed

daboross mentioned this issue Mar 15, 2015

Option for whether lib.rs should be rebuilt after build.rs is rerun #1422

Closed

This was referenced Jun 22, 2015

Rustfmt always recompiles rust-lang/rustfmt#110

Closed

Prevent build.rs from forcing unnecessary recompiles rust-lang/rustfmt#111

Closed

crumblingstatue mentioned this issue Jun 29, 2015

It should be possible to tell cargo when to rerun the build script #1769

Closed

alexcrichton mentioned this issue Oct 9, 2015

Fix spurious rebuilds of the script crate servo/servo#7936

Merged

alexcrichton mentioned this issue Oct 19, 2015

Add support for test build scripts #1581

Closed

larsbergstrom mentioned this issue Nov 7, 2015

Tracking issue for Rust high-pri or blocking issues for Servo & Gecko servo/servo#2853

Closed

1 task

alexcrichton mentioned this issue Nov 10, 2015

Build Script Support - input dirty checking #2133

Closed

alexcrichton mentioned this issue Nov 11, 2015

Allow build scripts to specify dependencies #2134

Merged

bors closed this as completed in #2134 Dec 4, 2015

BigBadE mentioned this issue Nov 26, 2022

Building recompiles bootloader every build, resulting in extreme build times rust-osdev/bootloader#286

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cargo should not consider the build script as stale with every modification #1162

Cargo should not consider the build script as stale with every modification #1162

tomaka commented Jan 13, 2015

alexcrichton commented Jan 14, 2015

tomaka commented Jan 14, 2015

alexcrichton commented Jan 14, 2015

marcusklaas commented Jul 7, 2015

alexcrichton commented Jul 7, 2015

marcusklaas commented Jul 8, 2015

Cargo should not consider the build script as stale with every modification #1162

Cargo should not consider the build script as stale with every modification #1162

Comments

tomaka commented Jan 13, 2015

Suggestion

alexcrichton commented Jan 14, 2015

tomaka commented Jan 14, 2015

alexcrichton commented Jan 14, 2015

marcusklaas commented Jul 7, 2015

alexcrichton commented Jul 7, 2015

marcusklaas commented Jul 8, 2015