-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Port from bespoke assertions of snapbox #14039
Labels
A-testing-cargo-itself
Area: cargo's tests
C-enhancement
Category: enhancement
S-accepted
Status: Issue or feature is accepted, and has a team member available to help mentor or review
Comments
This comment was marked as resolved.
This comment was marked as resolved.
#14041 highlights a blocker for |
This comment was marked as resolved.
This comment was marked as resolved.
bors
added a commit
that referenced
this issue
Jun 12, 2024
test: migrate build_script_env to snapbox ### What does this PR try to resolve? part of #14039.
bors
added a commit
that referenced
this issue
Jun 13, 2024
test: migrate features_are_quoted to snapbox ### What does this PR try to resolve? Part of #14039. Migrate `tests/testsuite/shell_quoting.rs` to snapbox. ### How should we test and review this PR? N/A ### Additional information N/A
This comment was marked as resolved.
This comment was marked as resolved.
bors
added a commit
that referenced
this issue
Jun 13, 2024
Migrate a few test files to snapbox This migrates the following files to `snapbox` - `artifact_dep` - Has a few `does_not_contain` - `artifact_dir` - `bad_config` - `bad_manifest_path` - Does not use `str!` for all tests - `bench` Note: This also adds auto-redactions for: - `[HOST_TARGET]` - `[ALT_TARGET]` - Only added if cross-compilation is allowed for the target - `[AVG_ELAPSED]` - For `bench` output - `[JITTER]` - For `bench` output Part of #14039
2 tasks
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
dieterplex
added a commit
to dieterplex/cargo
that referenced
this issue
Jun 15, 2024
dieterplex
added a commit
to dieterplex/cargo
that referenced
this issue
Jun 15, 2024
dieterplex
added a commit
to dieterplex/cargo
that referenced
this issue
Jun 15, 2024
bors
added a commit
that referenced
this issue
Jun 15, 2024
test: Migrate tests/testsuite/co*.rs to snapbox Migrating files: - tests/testsuite/collisions.rs - `with_stderr_does_not_contain` in test `collision_doc_host_target_feature_split` - tests/testsuite/concurrent.rs - tests/testsuite/config.rs - tests/testsuite/config_cli.rs - tests/testsuite/config_include.rs - tests/testsuite/corrupt_git.rs Testing with command `SNAPSHOTS=overwrite cargo test collisions::` or so. Part of #14039
This comment was marked as resolved.
This comment was marked as resolved.
bors
added a commit
that referenced
this issue
Jul 4, 2024
test: Migrate network tests to snapbox Part of #14039.
This comment was marked as resolved.
This comment was marked as resolved.
bors
added a commit
that referenced
this issue
Jul 4, 2024
test: migrate install* and inheritable_workspace_fields to snapbox ### What does this PR try to resolve? Part of #14039. Migrate following to snapbox: - `tests/testsuite/inheritable_workspace_fields.rs` - `tests/testsuite/install.rs` - `tests/testsuite/install_upgrade.rs`
bors
added a commit
that referenced
this issue
Jul 5, 2024
test: migrate cfg and check to snapbox ### What does this PR try to resolve? Part of #14039. Migrate following to snapbox: - `tests/testsuite/cfg.rs` - `tests/testsuite/check.rs`
bors
added a commit
that referenced
this issue
Jul 5, 2024
test: migrate build_plan and build_script to snapbox ### What does this PR try to resolve? Part of #14039. Migrate following to snapbox: - `tests/testsuite/build_plan.rs` - `tests/testsuite/build_script.rs`
bors
added a commit
that referenced
this issue
Jul 5, 2024
test: Migrate jobserver to snapbox Part of #14039.
bors
added a commit
that referenced
this issue
Jul 5, 2024
test: migrate patch, pkgid, proc_macro and progress to snapbox ### What does this PR try to resolve? Part of #14039. Migrate following to snapbox: - `tests/testsuite/patch.rs` - `tests/testsuite/pkgid.rs` - `tests/testsuite/proc_macro.rs` - `tests/testsuite/progress.rs`
bors
added a commit
that referenced
this issue
Jul 6, 2024
test: migrate serveral files to snapbox ### What does this PR try to resolve? Part of #14039. Migrate following to snapbox: - `tests/testsuite/read_manifest.rs` - `tests/testsuite/rustdoc_extern_html.rs` - `tests/testsuite/tool_paths.rs` - `tests/testsuite/warn_on_failure.rs`
bors
added a commit
that referenced
this issue
Jul 7, 2024
test: migrate metabuild, metadata and net_config to snapbox ### What does this PR try to resolve? Part of #14039. Migrate following to snapbox: - `tests/testsuite/metabuild.rs` - `tests/testsuite/metadata.rs` - `tests/testsuite/net_config.rs`
bors
added a commit
that referenced
this issue
Jul 8, 2024
test: migrate member_errors, multitarget and new to snapbox ### What does this PR try to resolve? Part of #14039. Migrate following to snapbox: - `tests/testsuite/member_errors.rs` - `tests/testsuite/multitarget.rs` - `tests/testsuite/new.rs`
bors
added a commit
that referenced
this issue
Jul 8, 2024
test: migrate member_errors, multitarget and new to snapbox ### What does this PR try to resolve? Part of #14039. Migrate following to snapbox: - `tests/testsuite/member_errors.rs` - `tests/testsuite/multitarget.rs` - `tests/testsuite/new.rs`
bors
added a commit
that referenced
this issue
Jul 8, 2024
test: migrate generate_lockfile and glob_targets to snapbox ### What does this PR try to resolve? Part of #14039. Migrate following to snapbox: - `tests/testsuite/generate_lockfile.rs` - `tests/testsuite/glob_targets.rs`
bors
added a commit
that referenced
this issue
Jul 9, 2024
test: migrate member_errors, multitarget and new to snapbox ### What does this PR try to resolve? Part of #14039. Migrate following to snapbox: - `tests/testsuite/member_errors.rs` - `tests/testsuite/multitarget.rs` - `tests/testsuite/new.rs`
bors
added a commit
that referenced
this issue
Jul 9, 2024
test: Migrate lto tests to snapbox Part of #14039.
bors
added a commit
that referenced
this issue
Jul 10, 2024
test: migrate test to snapbox ### What does this PR try to resolve? Part of #14039. Migrate following to snapbox: - `tests/testsuite/test.rs`
bors
added a commit
that referenced
this issue
Jul 10, 2024
test: Migrate lto tests to snapbox Part of #14039.
bors
added a commit
that referenced
this issue
Jul 12, 2024
test: migrate check_cfg to snapbox ### What does this PR try to resolve? Part of #14039. Migrate following to snapbox: - `tests/testsuite/check_cfg.rs` --- This commit allow deprecation for most of the tests currently in this file. This is because the tests either use `does_not_contain` or perform multiple `contains` assertion for single tool with macro `x!`. Here are some potential options to further eliminate the deprecation: 1. Remove the `x!` macro and replace the it with `str![]` for both Windows and non-Windows platforms. 2. Modify the `x!` macro to only generate the `--check-cfg` part for different platforms and manually compose it with `format!`.
bors
added a commit
that referenced
this issue
Jul 12, 2024
test: migrate build-std/main to snapbox ### What does this PR try to resolve? Part of #14039. Migrate following to snapbox: - `tests/build-std/main.rs`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
A-testing-cargo-itself
Area: cargo's tests
C-enhancement
Category: enhancement
S-accepted
Status: Issue or feature is accepted, and has a team member available to help mentor or review
Traditionally,
cargo-test-support
has had bespoke assertions, requiring hand implemented diff algorithms. As there is less of a community around this, the knowledge is more specialized, its less likely to be documented, and we are on our own for feature development.In #13980 and #14031, we introduced the use of snapbox as replacement for our own assertions and need to work to migrate to them.
Porting Instructions
1. Select a file to port
The files should have
#![allow(deprecated)]
at the top, e.g.Please check the comments on this issue for anyone to have claimed the file and then post that you claim the file
2. Remove
#![allow(deprecated)]
to identify what work is neededIf nothing, congrats, that was easy!
3. Resolve basic deprecations
Replace
Execs::with_stdout("...")
withReplace
Execs::with_stderr("...")
with(prelude is only needed for some steps)
Commit these changes
Run
SNAPSHOTS=overwrite cargo test && cargo check --test testsuite
. In rare cases,SNAPSHOTS=overwrite
may make bad edits. Feel free to create a reproduction case and create an issue. They are usually obvious how to fix, so don't worry.Diff the snapshots. Is there anything machine or run specific in them that previously was elided out with
[..]
?[..]
Once its working, amend your commit with the snapshots
4. Resolve non-literal deprecations
Like Step 3, but not just
with_stdout("...")
but with variables orformat!
.Evaluate whether a literal could be used
format!
for, then keep itSo this can end up with either
with_std*_data(expression)
with_std*_data(str![])
like in Step 35. Repeat with "straight forward" deprecations
with_stdout_unordered(expected)
->with_stdout_data(expected.unordered())
with_stderr_unordered(expected)
->with_stderr_data(expected.unordered())
with_json(expected)
->with_stdout_data(expected.json_lines())
orwith_stdout_data(expected.json())
.unordered()
6. Contains / Does not Contains deprecations
A judgement needs to be made for how to handle these.
A
contains
can be modeled by surrounding a entry with...
(text) or"...",
(json lines). The question is whether we should still do this or use a different method.contains
with an equality check, use multi-line globs (...
) for rustc errors that weren't previously matched to minimize tying Cargo's tests to the exact output of rustccontains
can be merged into a singleeq
with..
interspersed (or one...
and.unordered()
). For example, see 3054936unordered
, likely string literals, rather thanstr![]
should be used, as these can't correctly be updatedA
does_not_contain
/line_without
cannot be modeled at this time. The challenge with these is that they are brittle and you can't tell when they are no longer testing for what you think because the output changed. We should evaluate what to do with these on a case-by-case basis.When in doubt, feel free to put
#[allow(deprecated)]
on a statement and move on. We can come back to these later.The text was updated successfully, but these errors were encountered: