Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cargo should not consider the build script as stale with every modification #1162

Closed
tomaka opened this issue Jan 13, 2015 · 6 comments

Comments

@tomaka
Copy link
Contributor

commented Jan 13, 2015

When a package has a build script and you modify any file in your project, everything gets recompiled.

I'm currently working on a library named glium which is starting be have a moderate size and which uses build scripts only for code generation.

Modifying the code of an example or a test and running "cargo build" recompiles everything, which takes 3-4 minutes on my machine. Due to this, it's very annoying to write tests or examples.

Suggestion

Build scripts should output cargo:depends=/path/to/file just like they output rustc-flags. Only modifications in the files which they depend upon, plus the build script itself and its modules, trigger a recompilation.

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Jan 14, 2015

This seems like a pretty great use of the metadata output to me, I like the idea!

I would add one caveat that if cargo:depends is not printed, then the behavior is the same as it is today. That means that if you go out of your way to print dependencies then you get better incremental times, but if you don't do anything we at least won't do anything wrong!

@tomaka

This comment has been minimized.

Copy link
Contributor Author

commented Jan 14, 2015

What if a build script has no dependency? Should it print cargo:depends= with nothing behind, or should it print cargo:no-deps?

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Jan 14, 2015

I'd be down with either, but the latter seems like a better idea

@marcusklaas

This comment has been minimized.

Copy link

commented Jul 7, 2015

Is this something that could be done by someone with little knowledge of Cargo's internals?

I'd like to work on this, but it would be nice to have a sense of the complexity involved first..

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Jul 7, 2015

@marcusklaas yeah I think this should be relatively easy. You'd want to add parsing code for this key being printed out first. Next you'd want to serialize it out to a custom location (probably something in the relevant fingerprint directory).

You'd then update the build script fingerprint routine to take this file into account.

@marcusklaas

This comment has been minimized.

Copy link

commented Jul 8, 2015

Thanks for the pointers. I'll give it a shot!

alexcrichton added a commit to alexcrichton/servo that referenced this issue Oct 9, 2015

Fix spurious rebuilds of the script crate
The script crate currently has a build script, and Cargo will consider all files
in the script crate as inputs to the build script as it otherwise doesn't know
[what the input files are][cargo-1162]. This means that if any file in the
source tree of the script crate changes (or is created) then Cargo will think it
needs to re-run the build script and rebuild the crate.

[cargo-1162]: rust-lang/cargo#1162

The build script of the script crate is invoking python, and consequently Python
is generating some bytecode files in the source tree. On the second build of
Servo, Cargo will see these new files, think that something has changed, and
will re-run the build script of the script crate.

This change passes the `-B` flag to python to avoid generating these bytecode
files, which should avoid tampering with the source tree and appease Cargo by
ensuring that it doesn't get rebuilt.

---

As a helpful tip to if this comes up again, this was discovered by using the
changes in rust-lang/cargo@c447e9d plus the change in rust-lang/cargo#2044. Once
`RUST_LOG` was set to `cargo::ops::cargo_rustc::fingerprint=info`, the second
run of `./mach build` printed out:

```
precalculated components have changed:
  1444364448.000000000s (/build/servo/components/script/dom/bindings/codegen/parser/WebIDL.pyc) !=
  1444364235.000000000s (/build/servo/components/script/document_loader.rs)
```

Which should help easily diagnose these kinds of problems in the future!

bors-servo pushed a commit to servo/servo that referenced this issue Oct 9, 2015

bors-servo
Auto merge of #7936 - alexcrichton:script-rebuild, r=glennw
Fix spurious rebuilds of the script crate

The script crate currently has a build script, and Cargo will consider all files
in the script crate as inputs to the build script as it otherwise doesn't know
[what the input files are][cargo-1162]. This means that if any file in the
source tree of the script crate changes (or is created) then Cargo will think it
needs to re-run the build script and rebuild the crate.

[cargo-1162]: rust-lang/cargo#1162

The build script of the script crate is invoking python, and consequently Python
is generating some bytecode files in the source tree. On the second build of
Servo, Cargo will see these new files, think that something has changed, and
will re-run the build script of the script crate.

This change passes the `-B` flag to python to avoid generating these bytecode
files, which should avoid tampering with the source tree and appease Cargo by
ensuring that it doesn't get rebuilt.

---

As a helpful tip to if this comes up again, this was discovered by using the
changes in rust-lang/cargo@c447e9d plus the change in rust-lang/cargo#2044. Once
`RUST_LOG` was set to `cargo::ops::cargo_rustc::fingerprint=info`, the second
run of `./mach build` printed out:

```
precalculated components have changed:
  1444364448.000000000s (/build/servo/components/script/dom/bindings/codegen/parser/WebIDL.pyc) !=
  1444364235.000000000s (/build/servo/components/script/document_loader.rs)
```

Which should help easily diagnose these kinds of problems in the future!

<!-- Reviewable:start -->
[<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/servo/servo/7936)
<!-- Reviewable:end -->

bors-servo pushed a commit to servo/servo that referenced this issue Oct 9, 2015

bors-servo
Auto merge of #7936 - alexcrichton:script-rebuild, r=jdm
Fix spurious rebuilds of the script crate

The script crate currently has a build script, and Cargo will consider all files
in the script crate as inputs to the build script as it otherwise doesn't know
[what the input files are][cargo-1162]. This means that if any file in the
source tree of the script crate changes (or is created) then Cargo will think it
needs to re-run the build script and rebuild the crate.

[cargo-1162]: rust-lang/cargo#1162

The build script of the script crate is invoking python, and consequently Python
is generating some bytecode files in the source tree. On the second build of
Servo, Cargo will see these new files, think that something has changed, and
will re-run the build script of the script crate.

This change passes the `-B` flag to python to avoid generating these bytecode
files, which should avoid tampering with the source tree and appease Cargo by
ensuring that it doesn't get rebuilt.

---

As a helpful tip to if this comes up again, this was discovered by using the
changes in rust-lang/cargo@c447e9d plus the change in rust-lang/cargo#2044. Once
`RUST_LOG` was set to `cargo::ops::cargo_rustc::fingerprint=info`, the second
run of `./mach build` printed out:

```
precalculated components have changed:
  1444364448.000000000s (/build/servo/components/script/dom/bindings/codegen/parser/WebIDL.pyc) !=
  1444364235.000000000s (/build/servo/components/script/document_loader.rs)
```

Which should help easily diagnose these kinds of problems in the future!

<!-- Reviewable:start -->
[<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/servo/servo/7936)
<!-- Reviewable:end -->

bors-servo pushed a commit to servo/servo that referenced this issue Oct 9, 2015

bors-servo
Auto merge of #7936 - alexcrichton:script-rebuild, r=jdm
Fix spurious rebuilds of the script crate

The script crate currently has a build script, and Cargo will consider all files
in the script crate as inputs to the build script as it otherwise doesn't know
[what the input files are][cargo-1162]. This means that if any file in the
source tree of the script crate changes (or is created) then Cargo will think it
needs to re-run the build script and rebuild the crate.

[cargo-1162]: rust-lang/cargo#1162

The build script of the script crate is invoking python, and consequently Python
is generating some bytecode files in the source tree. On the second build of
Servo, Cargo will see these new files, think that something has changed, and
will re-run the build script of the script crate.

This change passes the `-B` flag to python to avoid generating these bytecode
files, which should avoid tampering with the source tree and appease Cargo by
ensuring that it doesn't get rebuilt.

---

As a helpful tip to if this comes up again, this was discovered by using the
changes in rust-lang/cargo@c447e9d plus the change in rust-lang/cargo#2044. Once
`RUST_LOG` was set to `cargo::ops::cargo_rustc::fingerprint=info`, the second
run of `./mach build` printed out:

```
precalculated components have changed:
  1444364448.000000000s (/build/servo/components/script/dom/bindings/codegen/parser/WebIDL.pyc) !=
  1444364235.000000000s (/build/servo/components/script/document_loader.rs)
```

Which should help easily diagnose these kinds of problems in the future!

<!-- Reviewable:start -->
[<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/servo/servo/7936)
<!-- Reviewable:end -->

alexcrichton added a commit to alexcrichton/cargo that referenced this issue Nov 11, 2015

Allow build scripts to specify dependencies
Currently Cargo is quite conservative in how it determines whether a build
script should be run. The heuristic used is "did any file in the project
directory change", but this is almost always guaranteed to be too coarse
grained in situations like:

* If the build script takes a long time to run it's advantageous to run it as
  few times as possible. Being able to inform Cargo about precisely when a build
  script should be run should provide more robust support here.
* Build scripts may not always have all of their dependencies in-tree or in the
  crate root. Sometimes a dependency could be elsewhere in a repository and
  scripts need a method of informing Cargo about this (as currently these
  compiles don't happen then they should).

This commit adds this support in build scripts via a new `rerun-if-changed`
directive which can be printed to standard output (using the standard Cargo
metadata format). The value for this key is a path relative to the crate root,
and Cargo will only look at these paths when determining whether to rerun the
build script. Any other file changes will not trigger the build script to be
rerun.

Currently the printed paths may either be a file or a directory, and a directory
is deeply traversed. The heuristic for trigger a rerun is detecting whether any
input file has been modified since the build script was last run (determined by
looking at the modification time of the output file of the build script). This
current implementation means that if you depend on a directory and then delete a
file within it the build script won't be rerun, but this is already the case and
can perhaps be patched up later.

Future extensions could possibly include the usage of glob patterns in build
script paths like the `include` and `exclude` features of `Cargo.toml`, but
these should be backwards compatible to add in the future.

Closes rust-lang#1162

alexcrichton added a commit to alexcrichton/cargo that referenced this issue Nov 11, 2015

Allow build scripts to specify dependencies
Currently Cargo is quite conservative in how it determines whether a build
script should be run. The heuristic used is "did any file in the project
directory change", but this is almost always guaranteed to be too coarse
grained in situations like:

* If the build script takes a long time to run it's advantageous to run it as
  few times as possible. Being able to inform Cargo about precisely when a build
  script should be run should provide more robust support here.
* Build scripts may not always have all of their dependencies in-tree or in the
  crate root. Sometimes a dependency could be elsewhere in a repository and
  scripts need a method of informing Cargo about this (as currently these
  compiles don't happen then they should).

This commit adds this support in build scripts via a new `rerun-if-changed`
directive which can be printed to standard output (using the standard Cargo
metadata format). The value for this key is a path relative to the crate root,
and Cargo will only look at these paths when determining whether to rerun the
build script. Any other file changes will not trigger the build script to be
rerun.

Currently the printed paths may either be a file or a directory, and a directory
is deeply traversed. The heuristic for trigger a rerun is detecting whether any
input file has been modified since the build script was last run (determined by
looking at the modification time of the output file of the build script). This
current implementation means that if you depend on a directory and then delete a
file within it the build script won't be rerun, but this is already the case and
can perhaps be patched up later.

Future extensions could possibly include the usage of glob patterns in build
script paths like the `include` and `exclude` features of `Cargo.toml`, but
these should be backwards compatible to add in the future.

Closes rust-lang#1162

alexcrichton added a commit to alexcrichton/cargo that referenced this issue Nov 11, 2015

Allow build scripts to specify dependencies
Currently Cargo is quite conservative in how it determines whether a build
script should be run. The heuristic used is "did any file in the project
directory change", but this is almost always guaranteed to be too coarse
grained in situations like:

* If the build script takes a long time to run it's advantageous to run it as
  few times as possible. Being able to inform Cargo about precisely when a build
  script should be run should provide more robust support here.
* Build scripts may not always have all of their dependencies in-tree or in the
  crate root. Sometimes a dependency could be elsewhere in a repository and
  scripts need a method of informing Cargo about this (as currently these
  compiles don't happen then they should).

This commit adds this support in build scripts via a new `rerun-if-changed`
directive which can be printed to standard output (using the standard Cargo
metadata format). The value for this key is a path relative to the crate root,
and Cargo will only look at these paths when determining whether to rerun the
build script. Any other file changes will not trigger the build script to be
rerun.

Future extensions could possibly include the usage of glob patterns in build
script paths like the `include` and `exclude` features of `Cargo.toml`, but
these should be backwards compatible to add in the future.

Closes rust-lang#1162

bors added a commit that referenced this issue Dec 4, 2015

Auto merge of #2134 - alexcrichton:build-script-input, r=brson
Currently Cargo is quite conservative in how it determines whether a build
script should be run. The heuristic used is "did any file in the project
directory change", but this is almost always guaranteed to be too coarse
grained in situations like:

* If the build script takes a long time to run it's advantageous to run it as
  few times as possible. Being able to inform Cargo about precisely when a build
  script should be run should provide more robust support here.
* Build scripts may not always have all of their dependencies in-tree or in the
  crate root. Sometimes a dependency could be elsewhere in a repository and
  scripts need a method of informing Cargo about this (as currently these
  compiles don't happen then they should).

This commit adds this support in build scripts via a new `rerun-if-changed`
directive which can be printed to standard output (using the standard Cargo
metadata format). The value for this key is a path relative to the crate root,
and Cargo will only look at these paths when determining whether to rerun the
build script. Any other file changes will not trigger the build script to be
rerun.

Currently the printed paths may either be a file or a directory, and a directory
is deeply traversed. The heuristic for trigger a rerun is detecting whether any
input file has been modified since the build script was last run (determined by
looking at the modification time of the output file of the build script). This
current implementation means that if you depend on a directory and then delete a
file within it the build script won't be rerun, but this is already the case and
can perhaps be patched up later.

Future extensions could possibly include the usage of glob patterns in build
script paths like the `include` and `exclude` features of `Cargo.toml`, but
these should be backwards compatible to add in the future.

Closes #1162

@bors bors closed this in #2134 Dec 4, 2015

psyomn added a commit to psyomn/cargo that referenced this issue Dec 8, 2015

Allow build scripts to specify dependencies
Currently Cargo is quite conservative in how it determines whether a build
script should be run. The heuristic used is "did any file in the project
directory change", but this is almost always guaranteed to be too coarse
grained in situations like:

* If the build script takes a long time to run it's advantageous to run it as
  few times as possible. Being able to inform Cargo about precisely when a build
  script should be run should provide more robust support here.
* Build scripts may not always have all of their dependencies in-tree or in the
  crate root. Sometimes a dependency could be elsewhere in a repository and
  scripts need a method of informing Cargo about this (as currently these
  compiles don't happen then they should).

This commit adds this support in build scripts via a new `rerun-if-changed`
directive which can be printed to standard output (using the standard Cargo
metadata format). The value for this key is a path relative to the crate root,
and Cargo will only look at these paths when determining whether to rerun the
build script. Any other file changes will not trigger the build script to be
rerun.

Future extensions could possibly include the usage of glob patterns in build
script paths like the `include` and `exclude` features of `Cargo.toml`, but
these should be backwards compatible to add in the future.

Closes rust-lang#1162

JanLikar added a commit to JanLikar/cargo that referenced this issue Jan 26, 2016

Allow build scripts to specify dependencies
Currently Cargo is quite conservative in how it determines whether a build
script should be run. The heuristic used is "did any file in the project
directory change", but this is almost always guaranteed to be too coarse
grained in situations like:

* If the build script takes a long time to run it's advantageous to run it as
  few times as possible. Being able to inform Cargo about precisely when a build
  script should be run should provide more robust support here.
* Build scripts may not always have all of their dependencies in-tree or in the
  crate root. Sometimes a dependency could be elsewhere in a repository and
  scripts need a method of informing Cargo about this (as currently these
  compiles don't happen then they should).

This commit adds this support in build scripts via a new `rerun-if-changed`
directive which can be printed to standard output (using the standard Cargo
metadata format). The value for this key is a path relative to the crate root,
and Cargo will only look at these paths when determining whether to rerun the
build script. Any other file changes will not trigger the build script to be
rerun.

Future extensions could possibly include the usage of glob patterns in build
script paths like the `include` and `exclude` features of `Cargo.toml`, but
these should be backwards compatible to add in the future.

Closes rust-lang#1162

jrmuizel pushed a commit to jrmuizel/gecko-cinnabar that referenced this issue Jun 12, 2017

servo: Merge #7936 - Fix spurious rebuilds of the script crate (from …
…alexcrichton:script-rebuild); r=jdm

The script crate currently has a build script, and Cargo will consider all files
in the script crate as inputs to the build script as it otherwise doesn't know
[what the input files are][cargo-1162]. This means that if any file in the
source tree of the script crate changes (or is created) then Cargo will think it
needs to re-run the build script and rebuild the crate.

[cargo-1162]: rust-lang/cargo#1162

The build script of the script crate is invoking python, and consequently Python
is generating some bytecode files in the source tree. On the second build of
Servo, Cargo will see these new files, think that something has changed, and
will re-run the build script of the script crate.

This change passes the `-B` flag to python to avoid generating these bytecode
files, which should avoid tampering with the source tree and appease Cargo by
ensuring that it doesn't get rebuilt.

---

As a helpful tip to if this comes up again, this was discovered by using the
changes in rust-lang/cargo@c447e9d plus the change in rust-lang/cargo#2044. Once
`RUST_LOG` was set to `cargo::ops::cargo_rustc::fingerprint=info`, the second
run of `./mach build` printed out:

```
precalculated components have changed:
  1444364448.000000000s (/build/servo/components/script/dom/bindings/codegen/parser/WebIDL.pyc) !=
  1444364235.000000000s (/build/servo/components/script/document_loader.rs)
```

Which should help easily diagnose these kinds of problems in the future!

Source-Repo: https://github.com/servo/servo
Source-Revision: bc5444067f7614ab7e3c83d8e131800d0db817b2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.