cargo_test: remove test roots after success #9701
Conversation
The test directories under `.../cit/` can take up a lot of space, and they're not useful to keep around for successful tests. The test guard now removes each "root" directory, using `panicking` as a proxy for failure to instead leave it for inspection.
|
(rust-highfive has picked a reviewer for you, use r? to override) |
rust-highfive
added
the
S-waiting-on-review
|
For context, I was looking around in our distro builds of Rust to see what takes up so much space, and Cargo's test |
|
As you can see from the test failure, I don't think this can land as it doesn't work on Windows. There's more information at #9585 and #7042. It would be great to figure out how to deal with Windows, but I'm uncertain how. FWIW, I very frequently run commands inside test directories as I am doing development, even after success. Perhaps this could be an option? Or maybe the distro scripts can just delete it after the tests run? EDIT: Would also need to be careful if this increased CI time significantly. |
I'll see if I can manage that. It does mean there will still be that 12GB spike in disk usage throughout the build, but combined with other transient cleanup I'm adding, it may be good enough. (Ideally, I'd just provision our builders to always have plenty of total space, and just let all cleanup happen afterward, but we don't always have that flexibility across all targets.) |
|
12 seems a lot larger than I would expect. Now you've got me curious where all that space is being wasted. On my linux system, it is 7GB, and on macOS it is 2.2GB. That seems like a crazy difference. Hmm. One test has a hard-link to cargo itself which is 200MB (!). Test binaries are almost 6MB each, whereas on macos they are 1.5MB (with debuginfo)! Why are linux binaries so much heavier? |
|
Oh I thought the macos/linux differences would be "unpacked" debuginfo? On Linux we duplicate libstd's debuginfo everywhere but on macOS it's never duplicated since it's left to all reside in libstd itself |
|
Yeah, I think split debuginfo explains macOS, and then I suspect our respective Linux differences may be due to building |
|
Closing since this has become a bit stale, and this can't be accepted as-is due to the Windows issue. I suggest if possible to delete the |
Clear disk space on CI. Cargo's testsuite uses a considerable amount of disk space. On windows-gnu, the target directory can get over 12GB, and there is only 13GB free. We're starting to run out of disk space, so this is a stop-gap that clears out the test data before running the smoke test which uses a fair bit of space itself. We will probably need to think about addressing #9701 somehow, otherwise we will start running out of space as we add more tests. See the linked issues in #9701 (comment) for additional context.
Clean more aggressively in CI The Windows x86_64 gnu CI job is running dangerously low on disk space. This PR adds another step to delete test output more aggressively. The test output with x86_64-pc-windows-gnu is nearly 9.5GB. The benchmark step is adding about 1GB of space (unfortunately it is rebuilding cargo, which may be hard to avoid without a workspace). Eventually we should probably look at figuring out how to reduce the amount of disk space used by the testsuite. Perhaps something like #9701 (see comments there). Or, making aggressive changes to the tests themselves. Many tests can probably be changed to use `cargo check` instead of `cargo build` (or maybe even `cargo tree`). We can default to not generating debuginfo. Or perhaps there are other changes to put the tests on a diet.
Clean more aggressively in CI The Windows x86_64 gnu CI job is running dangerously low on disk space. This PR adds another step to delete test output more aggressively. The test output with x86_64-pc-windows-gnu is nearly 9.5GB. The benchmark step is adding about 1GB of space (unfortunately it is rebuilding cargo, which may be hard to avoid without a workspace). Eventually we should probably look at figuring out how to reduce the amount of disk space used by the testsuite. Perhaps something like rust-lang#9701 (see comments there). Or, making aggressive changes to the tests themselves. Many tests can probably be changed to use `cargo check` instead of `cargo build` (or maybe even `cargo tree`). We can default to not generating debuginfo. Or perhaps there are other changes to put the tests on a diet.
The test directories under
.../cit/can take up a lot of space, andthey're not useful to keep around for successful tests. The test guard
now removes each "root" directory, using
panickingas a proxy forfailure to instead leave it for inspection.