-
Notifications
You must be signed in to change notification settings - Fork 12.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tracking Issue for consolidating command invocations in bootstrap #126819
Labels
C-tracking-issue
Category: A tracking issue for an RFC or an unstable feature.
T-bootstrap
Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Comments
matthiaskrgr
added a commit
to matthiaskrgr/rust
that referenced
this issue
Jun 29, 2024
…=onur-ozkan Bootstrap command refactoring: port more `Command` usages to `BootstrapCmd` (step 2) This PR moves more of bootstrap to use `BooststrapCmd`, and also refactors the struct to allow it to serve as a proper command wrapper. Tracking issue: rust-lang#126819 Best reviewed commit-by-commit, I have been adding some helper impls along the way to ease the migration, and then later I remove some of them since they were no longer needed. r? `@onur-ozkan`
rust-timer
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jun 29, 2024
Rollup merge of rust-lang#126822 - Kobzol:bootstrap-cmd-refactor-2, r=onur-ozkan Bootstrap command refactoring: port more `Command` usages to `BootstrapCmd` (step 2) This PR moves more of bootstrap to use `BooststrapCmd`, and also refactors the struct to allow it to serve as a proper command wrapper. Tracking issue: rust-lang#126819 Best reviewed commit-by-commit, I have been adding some helper impls along the way to ease the migration, and then later I remove some of them since they were no longer needed. r? `@onur-ozkan`
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jul 4, 2024
…nur-ozkan Bootstrap command refactoring: consolidate output modes (step 3) This PR is a continuation to rust-lang#126731. It consolidates the output modes of bootstrap (`Print` vs `CaptureAll` vs `CaptureStdout`) and simplifies the logic around error printing (now a command error is always printed if the failure is not ignored). It also ports even more usages of `Command` to `BootstrapCommand`, most notably the git helpers and many usages of the `output` function. The last commit was added because the third commit made two variants of the `Tool` enum unused (no idea why, but it seems to have been a false positive that they were used before). It can be reviewed now, but I would wait with merging until at least a few days after rust-lang#126731, just to catch any potential issues from that PR before we move further. As a next step, I want to clean up the API of the command a little bit to make usage easier (currently it's a bit verbose), and then continue with the rest of the tasks from the tracking issue. As always, best reviewed commit by commit. Tracking issue: rust-lang#126819 r? `@onur-ozkan`
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jul 4, 2024
…try> Bootstrap command refactoring: consolidate output modes (step 3) This PR is a continuation to rust-lang#126731. It consolidates the output modes of bootstrap (`Print` vs `CaptureAll` vs `CaptureStdout`) and simplifies the logic around error printing (now a command error is always printed if the failure is not ignored). It also ports even more usages of `Command` to `BootstrapCommand`, most notably the git helpers and many usages of the `output` function. The last commit was added because the third commit made two variants of the `Tool` enum unused (no idea why, but it seems to have been a false positive that they were used before). It can be reviewed now, but I would wait with merging until at least a few days after rust-lang#126731, just to catch any potential issues from that PR before we move further. As a next step, I want to clean up the API of the command a little bit to make usage easier (currently it's a bit verbose), and then continue with the rest of the tasks from the tracking issue. As always, best reviewed commit by commit. Tracking issue: rust-lang#126819 r? `@onur-ozkan` try-job: aarch64-apple
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jul 4, 2024
…nur-ozkan Bootstrap command refactoring: consolidate output modes (step 3) This PR is a continuation to rust-lang#126731. It consolidates the output modes of bootstrap (`Print` vs `CaptureAll` vs `CaptureStdout`) and simplifies the logic around error printing (now a command error is always printed if the failure is not ignored). It also ports even more usages of `Command` to `BootstrapCommand`, most notably the git helpers and many usages of the `output` function. The last commit was added because the third commit made two variants of the `Tool` enum unused (no idea why, but it seems to have been a false positive that they were used before). It can be reviewed now, but I would wait with merging until at least a few days after rust-lang#126731, just to catch any potential issues from that PR before we move further. As a next step, I want to clean up the API of the command a little bit to make usage easier (currently it's a bit verbose), and then continue with the rest of the tasks from the tracking issue. As always, best reviewed commit by commit. Tracking issue: rust-lang#126819 r? `@onur-ozkan` try-job: aarch64-apple
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jul 7, 2024
…nur-ozkan Bootstrap command refactoring: quality-of-life improvements (step 4) Continuation of rust-lang#127120. This PR simply introduce two new functions (`BootstrapCommand:run` and `command`) that make it a bit easier to use commands in bootstrap. It also adds several `#[must_use]` annotations. This shouldn't (hopefully) have any effect on behavior. Especially the first commit IMO makes any code that runs commands more readable, and allows using the API in a fluent way, without needing to jump back and forth between the command and the `Build(er)`. Tracking issue: rust-lang#126819 r? `@onur-ozkan`
github-actions bot
pushed a commit
to rust-lang/miri
that referenced
this issue
Jul 9, 2024
Bootstrap command refactoring: quality-of-life improvements (step 4) Continuation of rust-lang/rust#127120. This PR simply introduce two new functions (`BootstrapCommand:run` and `command`) that make it a bit easier to use commands in bootstrap. It also adds several `#[must_use]` annotations. This shouldn't (hopefully) have any effect on behavior. Especially the first commit IMO makes any code that runs commands more readable, and allows using the API in a fluent way, without needing to jump back and forth between the command and the `Build(er)`. Tracking issue: rust-lang/rust#126819 r? `@onur-ozkan`
lnicola
pushed a commit
to lnicola/rust-analyzer
that referenced
this issue
Jul 11, 2024
Bootstrap command refactoring: quality-of-life improvements (step 4) Continuation of rust-lang/rust#127120. This PR simply introduce two new functions (`BootstrapCommand:run` and `command`) that make it a bit easier to use commands in bootstrap. It also adds several `#[must_use]` annotations. This shouldn't (hopefully) have any effect on behavior. Especially the first commit IMO makes any code that runs commands more readable, and allows using the API in a fluent way, without needing to jump back and forth between the command and the `Build(er)`. Tracking issue: rust-lang/rust#126819 r? `@onur-ozkan`
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jul 13, 2024
…nur-ozkan Bootstrap command refactoring: improve debuggability (step 5) Continuation of rust-lang#127321. This PR improves the debuggability of command execution, by improving the output logged when a command fails (it now includes the exact location where the command was created and where it was executed), and also by adding a "drop bomb", which will panic if a command was created, but not executed (which is probably a bug). Here is how the output of a failed command looks like: ``` Command "git" "foo" "[bar]" (failure_mode=Exit, stdout_mode=Capture, stderr_mode=Capture) did not execute successfully. Expected success, got exit status: 1 Created at: src/core/build_steps/compile.rs:1699:9 Executed at: src/core/build_steps/compile.rs:1699:58 STDOUT ---- STDERR ---- git: 'foo' is not a git command. See 'git --help'. ``` And this is what is printed if you forget to execute a command: ``` thread 'main' panicked at /projects/personal/rust/rust/src/tools/build_helper/src/drop_bomb/mod.rs:42:13: command constructed at `src/core/build_steps/compile.rs:1699:9` was dropped without being executed: `git` ``` Best reviewed commit by commit. Tracking issue: rust-lang#126819 r? `@onur-ozkan`
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jul 13, 2024
…nur-ozkan Bootstrap command refactoring: improve debuggability (step 5) Continuation of rust-lang#127321. This PR improves the debuggability of command execution, by improving the output logged when a command fails (it now includes the exact location where the command was created and where it was executed), and also by adding a "drop bomb", which will panic if a command was created, but not executed (which is probably a bug). Here is how the output of a failed command looks like: ``` Command "git" "foo" "[bar]" (failure_mode=Exit, stdout_mode=Capture, stderr_mode=Capture) did not execute successfully. Expected success, got exit status: 1 Created at: src/core/build_steps/compile.rs:1699:9 Executed at: src/core/build_steps/compile.rs:1699:58 STDOUT ---- STDERR ---- git: 'foo' is not a git command. See 'git --help'. ``` And this is what is printed if you forget to execute a command: ``` thread 'main' panicked at /projects/personal/rust/rust/src/tools/build_helper/src/drop_bomb/mod.rs:42:13: command constructed at `src/core/build_steps/compile.rs:1699:9` was dropped without being executed: `git` ``` Best reviewed commit by commit. Tracking issue: rust-lang#126819 r? `@onur-ozkan`
tgross35
added a commit
to tgross35/rust
that referenced
this issue
Jul 16, 2024
…=onur-ozkan Bootstrap command refactoring: port remaining commands with access to `Build` (step 6) Continuation of rust-lang#127450. This PR ports commands in bootstrap that can easily get access to `Build(er)` to `BootstrapCommand`. After this PR, everything that can access `Build(er)` should be using the new API. Statistics of `bootstrap` code (ignoring `src/bin/<shims>`) after this PR: ``` 7 usages of `Command::new` 69 usages of `command()` (new API) - out of that: 16 usages of `as_command_mut()` (new API, but accesses the inner command) ``` Tracking issue: rust-lang#126819 r? `@onur-ozkan`
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jul 16, 2024
…nur-ozkan Bootstrap command refactoring: port remaining commands with access to `Build` (step 6) Continuation of rust-lang#127450. This PR ports commands in bootstrap that can easily get access to `Build(er)` to `BootstrapCommand`. After this PR, everything that can access `Build(er)` should be using the new API. Statistics of `bootstrap` code (ignoring `src/bin/<shims>`) after this PR: ``` 7 usages of `Command::new` 69 usages of `command()` (new API) - out of that: 16 usages of `as_command_mut()` (new API, but accesses the inner command) ``` Tracking issue: rust-lang#126819 r? `@onur-ozkan`
github-actions bot
pushed a commit
to rust-lang/miri
that referenced
this issue
Jul 20, 2024
Bootstrap command refactoring: port remaining commands with access to `Build` (step 6) Continuation of rust-lang/rust#127450. This PR ports commands in bootstrap that can easily get access to `Build(er)` to `BootstrapCommand`. After this PR, everything that can access `Build(er)` should be using the new API. Statistics of `bootstrap` code (ignoring `src/bin/<shims>`) after this PR: ``` 7 usages of `Command::new` 69 usages of `command()` (new API) - out of that: 16 usages of `as_command_mut()` (new API, but accesses the inner command) ``` Tracking issue: rust-lang/rust#126819 r? `@onur-ozkan`
tgross35
added a commit
to tgross35/rust
that referenced
this issue
Jul 20, 2024
…=onur-ozkan Bootstrap command refactoring: make command output API more bulletproof (step 7) Continuation of rust-lang#127680. This PR modifies the API of running commands to make it more explicit when a command is expected to produce programmatically handled output. Now if you call just `run`, you cannot access the stdout/stderr by accident, because it will not be returned to the caller. This API change might be seen as overkill, let me know what do you think. In any case, I'd like to land the second commit, to make it harder to accidentally read stdout/stderr of commands that did not capture output (now you'd get an empty string as a result, but you should probably get a panic instead, if you try to read uncaptured stdout/stderr). Tracking issue: rust-lang#126819 r? `@onur-ozkan`
tgross35
added a commit
to tgross35/rust
that referenced
this issue
Jul 20, 2024
…=onur-ozkan Bootstrap command refactoring: make command output API more bulletproof (step 7) Continuation of rust-lang#127680. This PR modifies the API of running commands to make it more explicit when a command is expected to produce programmatically handled output. Now if you call just `run`, you cannot access the stdout/stderr by accident, because it will not be returned to the caller. This API change might be seen as overkill, let me know what do you think. In any case, I'd like to land the second commit, to make it harder to accidentally read stdout/stderr of commands that did not capture output (now you'd get an empty string as a result, but you should probably get a panic instead, if you try to read uncaptured stdout/stderr). Tracking issue: rust-lang#126819 r? ``@onur-ozkan``
matthiaskrgr
added a commit
to matthiaskrgr/rust
that referenced
this issue
Jul 20, 2024
…=onur-ozkan Bootstrap command refactoring: make command output API more bulletproof (step 7) Continuation of rust-lang#127680. This PR modifies the API of running commands to make it more explicit when a command is expected to produce programmatically handled output. Now if you call just `run`, you cannot access the stdout/stderr by accident, because it will not be returned to the caller. This API change might be seen as overkill, let me know what do you think. In any case, I'd like to land the second commit, to make it harder to accidentally read stdout/stderr of commands that did not capture output (now you'd get an empty string as a result, but you should probably get a panic instead, if you try to read uncaptured stdout/stderr). Tracking issue: rust-lang#126819 r? ```@onur-ozkan```
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jul 23, 2024
…nur-ozkan Bootstrap command refactoring: make command output API more bulletproof (step 7) Continuation of rust-lang#127680. This PR modifies the API of running commands to make it more explicit when a command is expected to produce programmatically handled output. Now if you call just `run`, you cannot access the stdout/stderr by accident, because it will not be returned to the caller. This API change might be seen as overkill, let me know what do you think. In any case, I'd like to land the second commit, to make it harder to accidentally read stdout/stderr of commands that did not capture output (now you'd get an empty string as a result, but you should probably get a panic instead, if you try to read uncaptured stdout/stderr). Tracking issue: rust-lang#126819 r? `@onur-ozkan`
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jul 24, 2024
…nur-ozkan Bootstrap command refactoring: make command output API more bulletproof (step 7) Continuation of rust-lang#127680. This PR modifies the API of running commands to make it more explicit when a command is expected to produce programmatically handled output. Now if you call just `run`, you cannot access the stdout/stderr by accident, because it will not be returned to the caller. This API change might be seen as overkill, let me know what do you think. In any case, I'd like to land the second commit, to make it harder to accidentally read stdout/stderr of commands that did not capture output (now you'd get an empty string as a result, but you should probably get a panic instead, if you try to read uncaptured stdout/stderr). Tracking issue: rust-lang#126819 r? `@onur-ozkan`
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jul 25, 2024
…nur-ozkan Bootstrap command refactoring: make command output API more bulletproof (step 7) Continuation of rust-lang#127680. This PR modifies the API of running commands to make it more explicit when a command is expected to produce programmatically handled output. Now if you call just `run`, you cannot access the stdout/stderr by accident, because it will not be returned to the caller. This API change might be seen as overkill, let me know what do you think. In any case, I'd like to land the second commit, to make it harder to accidentally read stdout/stderr of commands that did not capture output (now you'd get an empty string as a result, but you should probably get a panic instead, if you try to read uncaptured stdout/stderr). Tracking issue: rust-lang#126819 r? `@onur-ozkan`
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jul 25, 2024
…nur-ozkan Bootstrap command refactoring: make command output API more bulletproof (step 7) Continuation of rust-lang#127680. This PR modifies the API of running commands to make it more explicit when a command is expected to produce programmatically handled output. Now if you call just `run`, you cannot access the stdout/stderr by accident, because it will not be returned to the caller. This API change might be seen as overkill, let me know what do you think. In any case, I'd like to land the second commit, to make it harder to accidentally read stdout/stderr of commands that did not capture output (now you'd get an empty string as a result, but you should probably get a panic instead, if you try to read uncaptured stdout/stderr). Tracking issue: rust-lang#126819 r? `@onur-ozkan`
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jul 25, 2024
…try> Bootstrap command refactoring: make command output API more bulletproof (step 7) Continuation of rust-lang#127680. This PR modifies the API of running commands to make it more explicit when a command is expected to produce programmatically handled output. Now if you call just `run`, you cannot access the stdout/stderr by accident, because it will not be returned to the caller. This API change might be seen as overkill, let me know what do you think. In any case, I'd like to land the second commit, to make it harder to accidentally read stdout/stderr of commands that did not capture output (now you'd get an empty string as a result, but you should probably get a panic instead, if you try to read uncaptured stdout/stderr). Tracking issue: rust-lang#126819 r? `@onur-ozkan` try-job: x86_64-msvc
bors
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jul 28, 2024
…nur-ozkan Bootstrap command refactoring: make command output API more bulletproof (step 7) Continuation of rust-lang#127680. This PR modifies the API of running commands to make it more explicit when a command is expected to produce programmatically handled output. Now if you call just `run`, you cannot access the stdout/stderr by accident, because it will not be returned to the caller. This API change might be seen as overkill, let me know what do you think. In any case, I'd like to land the second commit, to make it harder to accidentally read stdout/stderr of commands that did not capture output (now you'd get an empty string as a result, but you should probably get a panic instead, if you try to read uncaptured stdout/stderr). Tracking issue: rust-lang#126819 r? `@onur-ozkan` try-job: x86_64-msvc
github-actions bot
pushed a commit
to rust-lang/miri
that referenced
this issue
Jul 30, 2024
Bootstrap command refactoring: make command output API more bulletproof (step 7) Continuation of rust-lang/rust#127680. This PR modifies the API of running commands to make it more explicit when a command is expected to produce programmatically handled output. Now if you call just `run`, you cannot access the stdout/stderr by accident, because it will not be returned to the caller. This API change might be seen as overkill, let me know what do you think. In any case, I'd like to land the second commit, to make it harder to accidentally read stdout/stderr of commands that did not capture output (now you'd get an empty string as a result, but you should probably get a panic instead, if you try to read uncaptured stdout/stderr). Tracking issue: rust-lang/rust#126819 r? `@onur-ozkan` try-job: x86_64-msvc
matthiaskrgr
added a commit
to matthiaskrgr/rust
that referenced
this issue
Aug 12, 2024
Slightly refactor `Flags` in bootstrap The next step for rust-lang#126819 is to track commands executed inside `Config::parse`. This is quite challenging, because (tracked) command execution needs to access some state that is stored inside `Config`, which creates a sort of a chicken-and-egg problem. I would like to first untangle `Config::parse` a little bit, which is what this PR starts with. Tracking issue: rust-lang#126819 r? `@onur-ozkan`
matthiaskrgr
added a commit
to matthiaskrgr/rust
that referenced
this issue
Aug 12, 2024
Slightly refactor `Flags` in bootstrap The next step for rust-lang#126819 is to track commands executed inside `Config::parse`. This is quite challenging, because (tracked) command execution needs to access some state that is stored inside `Config`, which creates a sort of a chicken-and-egg problem. I would like to first untangle `Config::parse` a little bit, which is what this PR starts with. Tracking issue: rust-lang#126819 r? `@onur-ozkan`
GuillaumeGomez
added a commit
to GuillaumeGomez/rust
that referenced
this issue
Aug 12, 2024
Slightly refactor `Flags` in bootstrap The next step for rust-lang#126819 is to track commands executed inside `Config::parse`. This is quite challenging, because (tracked) command execution needs to access some state that is stored inside `Config`, which creates a sort of a chicken-and-egg problem. I would like to first untangle `Config::parse` a little bit, which is what this PR starts with. Tracking issue: rust-lang#126819 r? ``@onur-ozkan``
GuillaumeGomez
added a commit
to GuillaumeGomez/rust
that referenced
this issue
Aug 12, 2024
Slightly refactor `Flags` in bootstrap The next step for rust-lang#126819 is to track commands executed inside `Config::parse`. This is quite challenging, because (tracked) command execution needs to access some state that is stored inside `Config`, which creates a sort of a chicken-and-egg problem. I would like to first untangle `Config::parse` a little bit, which is what this PR starts with. Tracking issue: rust-lang#126819 r? `@onur-ozkan`
rust-timer
added a commit
to rust-lang-ci/rust
that referenced
this issue
Aug 12, 2024
Rollup merge of rust-lang#128878 - Kobzol:refactor-flags, r=onur-ozkan Slightly refactor `Flags` in bootstrap The next step for rust-lang#126819 is to track commands executed inside `Config::parse`. This is quite challenging, because (tracked) command execution needs to access some state that is stored inside `Config`, which creates a sort of a chicken-and-egg problem. I would like to first untangle `Config::parse` a little bit, which is what this PR starts with. Tracking issue: rust-lang#126819 r? `@onur-ozkan`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
C-tracking-issue
Category: A tracking issue for an RFC or an unstable feature.
T-bootstrap
Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
This is a tracking issue for monitoring the work of improving how we deal with command invocations in bootstrap (proposed here).
Currently, bootstrap executes commands on many places, often in arbitrary ways. This makes it harder to debug, profile and test the command invocations executed by it. The goal of this refactoring is to make sure that all external command invocations use the same API (a custom command wrapper) and that they go through a centralized location.
Intermediate steps:
&Build(er)
toBootstrapCommand
BootstrapCommand
(step 1) #126731Command
usages toBootstrapCmd
(step 2) #126822BootstrapCmd
so that it storesCommand
instead of&mut Command
and remove all calls toBootstrapCmd::from
by changingCommand::new
toBootstrapCmd::new
Command
usages toBootstrapCmd
(step 2) #126822#[must_use]
#[track_caller]
to command execution functions to make debugging easier.BootstrapCmd
that can access Builder to use the correct output and failure modes. This will include passing Builder to additional places.Build
(step 6) #127680Config
.Flags
in bootstrap #128878BootstrapCommand
.Builder
,Build
,Config
and command context. There are some fields and logic used during command execution that are distributed amongstBuilder/Build/Config
, so it will require some refactoring to make it work if the execution will happen on a separate place (in the command context).Config::parse
) by introducing a new command context that will be passed to these places, and then stored inBuilder
. Move certain fields (such asfail_fast
) fromBuilder
to the command context.Step
debug hierarchical output.metrics.json
).The text was updated successfully, but these errors were encountered: