add recursive settings and clean up test whitelist #4142

bradleypmartin · 2020-04-27T04:44:55Z

Close (issue) #4113.

Within the cfg_if directory, there are some nested files that can be helped out with a recursive config setting and then removed from the test whitelist in rustfmt-core/rustfmt-lib/src/test/mod.rs.

In addition to the cfg_if files on the whitelist, there are quite a few others that don't seem to be needed there.

The files that remain appear to be whitelisted for reasons outside the scope of the Issue linked above. For example, issue-3253/paths nests some files whose tests fail despite having a // rustfmt-recursive: true config setting put into the lib.rs file in the issue-3253 root folders (source/target).

bradleypmartin · 2020-04-27T04:54:08Z

rustfmt-core/rustfmt-lib/src/test/mod.rs

@@ -21,21 +21,10 @@ const DIFF_CONTEXT_SIZE: usize = 3;

 // A list of files on which we want to skip testing.
 const SKIP_FILE_WHITE_LIST: &[&str] = &[
+    "issue-3253/paths",


Placing a remaining whitelisted file up in lexicographical order after 'cleaning house'

bradleypmartin · 2020-04-27T04:55:59Z

rustfmt-core/rustfmt-lib/src/test/mod.rs

-    "cfg_mod/dir",
-    "cfg_mod/bar.rs",
-    "cfg_mod/foo.rs",
-    "cfg_mod/wasm32.rs",


Between adding the recursive config option to the cfg_if/lib.rs files and possibly some changes that have occurred outside the scope of this PR, a good many of these whitelisted file strings don't seem necessary at this time.

We need to keep these files in the list to make sure that recursive is working correctly. The test must pass even when these files are skipped.

calebcartwright · 2020-04-27T15:37:22Z

Thanks for the PR @bradleypmartin! Will try to take a look at this later today

topecongiro · 2020-04-28T02:55:44Z

rustfmt-core/rustfmt-lib/src/test/mod.rs

-    "cfg_mod/dir",
-    "cfg_mod/bar.rs",
-    "cfg_mod/foo.rs",
-    "cfg_mod/wasm32.rs",


We need to keep these files in the list to make sure that recursive is working correctly. The test must pass even when these files are skipped.

calebcartwright · 2020-04-28T03:29:49Z

rustfmt-core/rustfmt-lib/tests/source/cfg_if/lib.rs

@@ -10,6 +10,7 @@
 //! * `mips64`: [`is_mips64_feature_detected`]
 //! * `powerpc`: [`is_powerpc_feature_detected`]
 //! * `powerpc64`: [`is_powerpc64_feature_detected`]
+// rustfmt-recursive: true


I think the system/idempotence checks need the inline config options to be at the top of the files as well

Thanks, @calebcartwright ! Even if the config options can 'survive' being down below, I also like how having them at the top matches with their location in other files already. I've gone ahead and moved them up there.

I'll wait on topecongiro's feedback on the particular files we can take off the whitelist, then make those changes as well (feel free to weigh in there too if you've got strong opinions).

bradleypmartin · 2020-04-28T03:37:19Z

Hey @topecongiro , thanks for the feedback! To clarify: it was pretty evident that the following three files were being intentionally ignored to test recursive behavior, so I left those in:

"configs/recursive/disabled/foo.rs",
"configs/recursive/enabled/foo.rs",
"mods-relative-path/mod_b.rs",

...as for the others, though (the cfg_mod files highlighted in your change request, for instance), are there any specifically that we want kept in the whitelist?

It seems like we can probably at least leave out the two cfg_if files, but maybe I'm thinking about that incorrectly.

Just let me know and I'll fix it up!

calebcartwright · 2020-04-28T04:30:06Z

...as for the others, though (the cfg_mod files highlighted in your change request, for instance), are there any specifically that we want kept in the whitelist?

I'll wait on topecongiro's feedback on the particular files we can take off the whitelist, then make those changes as well (feel free to weigh in there too if you've got strong opinions).

I don't expect we'll need to remove any files from the skip file list.

The main integration test suite for rustfmt (usually referenced as the system test suite) works by checking every file individually under the rustfmt-core/rustfmt-lib/tests/source directory, and comparing the formatting result of each file with it's corresponding counterpart in rustfmt-core/rustfmt-lib/tests/target.

A few of these tests (including the cfg_mod ones @topecongiro noted) are also validating other functionality of rustfmt, such as config options or module resolution. We keep those imported files on the skip list so that rustfmt can only get to those files (and thus format them) by following the imports.

If we were to remove those files from the skiplist then that test would no longer necessarily validate the mod behavior, since rustfmt would have formatted each of those other files individually.

bradleypmartin · 2020-04-28T15:00:21Z

Once again, I appreciate the patient help and context while I sync up here!

For the moment I'm reverting the whitelist and I'll take a closer look this evening. I think something I find challenging is that - at first glance - it's difficult to parse exactly what the purpose of some of these files' whitelisting is...especially if existing unit/integration tests pass regardless of whether the files are present in the whitelist or not.

...it's like, for someone looking at the SKIP_FILE list for the first time, the intuitive/'first-order' understanding is that not skipping the files (having them absent from the list) would result in 'more rigorous' testing (i.e. not being skipped).

But with both of your feedback and thinking about it some more, it's certainly clear that having a file in an 'ignore/skip' list might in fact convey exactly the testing logic/behavior that you're after...it's just (totally in my opinion) more counterintuitive and takes more time to wrap my head around the context.

Thanks again for the help!

calebcartwright · 2020-04-29T00:13:07Z

No worries, and thanks for the feedback! We want rustfmt to be as easy as possible for folks to contribute to, so it's great to hear about your experience.

especially if existing unit/integration tests pass regardless of whether the files are present in the whitelist or not.

The gist of #4113 is essentially that some integration tests were no longer running at all, and the cause of that was due to a misconfiguration in those tests (the absence of the recursive setting). We want to fix those by correcting the configuration used for those tests, and not by removing files from the skip list.

There's a pretty comprehensive (and ever-growing) testing suite within rustfmt, but I think it's fair to say we could take another look through the Contributor and testing documentation to make sure things are up to date and clear especially to include some things we know are missing (#4071 (comment)).

IMHO that's something best tackled in a separate PR though

calebcartwright · 2020-04-29T03:11:01Z

it's difficult to parse exactly what the purpose of some of these files' whitelisting is...especially if existing unit/integration tests pass regardless of whether the files are present in the whitelist or not.

...it's like, for someone looking at the SKIP_FILE list for the first time, the intuitive/'first-order' understanding is that not skipping the files (having them absent from the list) would result in 'more rigorous' testing (i.e. not being skipped).

Just to follow up on these points, I think there may still be some confusion on what that skip list is used for.

rustfmt only formats the files provided as args, unless the --recursive flag is included, in which case rustfmt will also format the submods that are imported by those input files. The test runner here is basically invoking rustfmt against each file under source (i.e. rustfmt tests/source/foo.rs) and then comparing the formatted output of each file to the expected output under target (tests/target/foo.rs)

The skip list in many cases is being used so that we don't run rustfmt directly against those individual files like all of the others, so that we can validate those particular files get formatted because they were imported elsewhere and resolved correctly.

For example with cfg_mod, the tests will format all the files within that directory, but the real purpose of these test files is that we want to validate that rustfmt correctly resolved and formatted those other files based on their path starting from mod.rs.

We want the test to run this, and only this, for the cfg_mod tests:

rustfmt --recursive rustfmt-core/rustfmt-lib/tests/source/cfg_mod/mod.rs

And then validate that all the other files (foo.rs, bar.rs, dir/dir1/dir2/wasm32.rs, dir/dir1/dir3/wasm32.rs, etc.) were also formatted as expected.

If we remove these entries from the skip list, then the test runner will end up running the equivalent of rustfmt for each file:

rustfmt rustfmt-core/rustfmt-lib/tests/source/cfg_mod/mod.rs
rustfmt rustfmt-core/rustfmt-lib/tests/source/cfg_mod/bar.rs
rustfmt rustfmt-core/rustfmt-lib/tests/source/cfg_mod/foo.rs
rustfmt rustfmt-core/rustfmt-lib/tests/source/cfg_mod/somewhere.rs
rustfmt rustfmt-core/rustfmt-lib/tests/source/cfg_mod/dir/dir1/dir2/wasm32.rs
rustfmt rustfmt-core/rustfmt-lib/tests/source/cfg_mod/dir/dir1/dir2/wasm32.rs
etc.

The end result of the formatting out is the same in both cases, but in the latter case we wouldn't have a way of knowing whether the mod path resolution was done correctly. By adding those other files to the skip list, then the test will fail if the path resolution does ever break.

We have tests like these to validate core behavior of rustfmt and to prevent any regressions. Internally rustfmt uses a pinned version of rustc parser to parse the input files, and it's especially important when we upgrade the rustc parser version that rustfmt is still working correctly with those upstream parser changes.

The impetus for #4113 was actually such recent rustc upgrade that broke the path resolution within rustfmt. We caught it manually through some other testing, but realized our automated testing for those scenarios (like cfg_mod) wasn't actually running.

Anyway, probably way more than you wanted to know 😄 Most of the time the skip list isn't something rustfmt contributors have to deal with, but I hope that helps explain the context.

bradleypmartin · 2020-04-30T03:05:04Z

Hey, no worries on giving more information than I need - anything and everything's appreciated! :)

Got kind of busy at work, but this extra info will be immensely helpful when I get some bandwidth free...hopefully at the weekend if not before. Thanks once again!

bradleypmartin · 2020-05-25T18:43:37Z

@calebcartwright Thanks for adding that label and sorry about the wait! Looking at this in earnest right now. Hope you're all enjoying a relaxing Memorial Day weekend (for those in the US)!

bradleypmartin · 2020-05-25T20:06:07Z

Also saw that recent builds were kinda problematic via rust-lang/rust#72232; I'll work from a less recent commit until that gets resolved.

EDIT: Sandboxing my changes over at my forked rustfmt: bradleypmartin#1.

calebcartwright · 2020-05-25T22:44:28Z

Also saw that recent builds were kinda problematic via rust-lang/rust#72232; I'll work from a less recent commit until that gets resolved.

At this point the upstream issue is only impacting the availability of rustfmt via rustup, but you should be able to rebase your changes on master and compile using a recent nightly

Servo has used this since forever, and it'd be useful to be able to use rustfmt stable there so that we can use the same rustfmt version in both Firefox and Servo. Feel free to close this if there's any reason it shouldn't be done. Closes rust-lang#3368

Previously, non-trivial type aliases in extern blocks were dropped by rustfmt because only the type alias name would be passed to a rewritter. This commit fixes that by passing all type information (generics, bounds, and assignments) to a type alias rewritter, and consolidates `rewrite_type_alias` and `rewrite_associated_type` as one function. Closes rust-lang#4159

* Preserve comments in empty statements Closes rust-lang#4018 * fixup! Preserve comments in empty statements

…ving a struct's commas (rust-lang#4201)

…om/bradleypmartin/rustfmt into bradleypmartin-test-recursive-mode

topecongiro · 2020-05-26T10:36:44Z

Thanks!

bradleypmartin · 2020-05-26T14:58:37Z

Hey @topecongiro @calebcartwright , thanks a lot again for the patience and help with this one, and I'll try and do a better job of understanding the problem space/scope before I submit my next OSS PR.

Have a great day!

calebcartwright · 2020-05-27T01:41:47Z

No worries @bradleypmartin, thank you for your interest and contribution!

jeremystucki · 2020-05-31T21:22:36Z

Hey, I think you forgot to close the associated issue

add recursive mode and clean up whitelist where we can

60ed5e9

bradleypmartin commented Apr 27, 2020

View reviewed changes

topecongiro suggested changes Apr 28, 2020

View reviewed changes

calebcartwright reviewed Apr 28, 2020

View reviewed changes

move recursive setting to top of lib.rs files

3f40a9e

rustfmt-core/rustfmt-lib/src/test/mod.rs : revert skipped files

009421c

calebcartwright mentioned this pull request May 1, 2020

Add recursive config setting to appropriate tests #4113

Closed

calebcartwright added the pr-waiting-on-author label May 21, 2020

Merge branch 'master' into bradleypmartin-test-recursive-mode

fa3395d

bradleypmartin mentioned this pull request May 25, 2020

[WIP] Testing out forked changes for PR 4142 at rustfmt upstream bradleypmartin/rustfmt#1

Open

bradleypmartin and others added 9 commits May 25, 2020 17:26

add recursive mode and clean up whitelist where we can

ddd5c6e

move recursive setting to top of lib.rs files

278a4f1

rustfmt-core/rustfmt-lib/src/test/mod.rs : revert skipped files

f0cce5b

Update diff.rs (rust-lang#4150)

010c678

docs: clarify config opts availability [skip ci] (rust-lang#4155)

759aaf7

update to rustc 656

58c6c79

Remove disable_all_formatting (rust-lang#4147)

ce8493a

Stabilize binop_separator.

445f71e

Servo has used this since forever, and it'd be useful to be able to use rustfmt stable there so that we can use the same rustfmt version in both Firefox and Servo. Feel free to close this if there's any reason it shouldn't be done. Closes rust-lang#3368

topecongiro and others added 22 commits May 25, 2020 17:27

Fix test

d9f7064

Audit lib.rs

69f6b76

Cargo fmt

b68f06b

Fix merge conflicts

991ab17

Preserve comments in empty statements (rust-lang#4180)

fffc7ff

* Preserve comments in empty statements Closes rust-lang#4018 * fixup! Preserve comments in empty statements

Add test cases for fixed issues (rust-lang#4189)

037c064

Do not break empty blocks in match arm patterns (rust-lang#4187)

25d5f9f

Add cargo-make support (rust-lang#4191)

0e3d5f0

Parse comma-separated branches in macro definitions (rust-lang#4173)

b56a370

Fix module resolution in inner modules with paths (rust-lang#4194)

24fa639

Update Contributing.md (rust-lang#4195)

5b62710

Align string type for edition (rust-lang#4197)

20a52b7

Merge configs from parent directories (rust-lang#4179)

b042b37

Stabilize ignore config option (rust-lang#4139)

cbf44c8

Upload annotate-snippets to 0.8 (rust-lang#4161)

4861ef4

Improve error message when module resolution failed (rust-lang#4198)

35121ea

Fix CFG_RELEASE value (rust-lang#4202)

5936675

rust-lang#4099: trailing_comma + struct_field_align_threshold -> remo…

0a45e91

…ving a struct's commas (rust-lang#4201)

Fix makefile (rust-lang#4203)

b272c77

Cargo update

256c024

Release v2.0.0-rc.2

77a7f95

Merge branch 'bradleypmartin-test-recursive-mode' of https://github.c…

0b8341f

…om/bradleypmartin/rustfmt into bradleypmartin-test-recursive-mode

topecongiro merged commit 68fc133 into rust-lang:master May 26, 2020

bradleypmartin deleted the bradleypmartin-test-recursive-mode branch May 26, 2020 14:58

karyon added backport-triage and removed pr-waiting-on-author labels Oct 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add recursive settings and clean up test whitelist #4142

add recursive settings and clean up test whitelist #4142

bradleypmartin commented Apr 27, 2020

bradleypmartin Apr 27, 2020

bradleypmartin Apr 27, 2020

topecongiro Apr 28, 2020

calebcartwright commented Apr 27, 2020

topecongiro Apr 28, 2020

calebcartwright Apr 28, 2020

bradleypmartin Apr 28, 2020

bradleypmartin commented Apr 28, 2020

calebcartwright commented Apr 28, 2020

bradleypmartin commented Apr 28, 2020

calebcartwright commented Apr 29, 2020

calebcartwright commented Apr 29, 2020 •

edited

Loading

bradleypmartin commented Apr 30, 2020

bradleypmartin commented May 25, 2020

bradleypmartin commented May 25, 2020 •

edited

Loading

calebcartwright commented May 25, 2020

topecongiro commented May 26, 2020

bradleypmartin commented May 26, 2020

calebcartwright commented May 27, 2020

jeremystucki commented May 31, 2020

add recursive settings and clean up test whitelist #4142

add recursive settings and clean up test whitelist #4142

Conversation

bradleypmartin commented Apr 27, 2020

bradleypmartin Apr 27, 2020

Choose a reason for hiding this comment

bradleypmartin Apr 27, 2020

Choose a reason for hiding this comment

topecongiro Apr 28, 2020

Choose a reason for hiding this comment

calebcartwright commented Apr 27, 2020

topecongiro Apr 28, 2020

Choose a reason for hiding this comment

calebcartwright Apr 28, 2020

Choose a reason for hiding this comment

bradleypmartin Apr 28, 2020

Choose a reason for hiding this comment

bradleypmartin commented Apr 28, 2020

calebcartwright commented Apr 28, 2020

bradleypmartin commented Apr 28, 2020

calebcartwright commented Apr 29, 2020

calebcartwright commented Apr 29, 2020 • edited Loading

bradleypmartin commented Apr 30, 2020

bradleypmartin commented May 25, 2020

bradleypmartin commented May 25, 2020 • edited Loading

calebcartwright commented May 25, 2020

topecongiro commented May 26, 2020

bradleypmartin commented May 26, 2020

calebcartwright commented May 27, 2020

jeremystucki commented May 31, 2020

calebcartwright commented Apr 29, 2020 •

edited

Loading

bradleypmartin commented May 25, 2020 •

edited

Loading