Add clippy and fix commands to x.py #56595

ljedrz · 2018-12-07T14:48:22Z

Since they are kind of similar in nature, I have used the same approach as for cargo check. At least some of the boilerplate could probably be shared, but I'd prefer to gather some feedback before I decide to merge them more aggressively.

This works reasonably well for clippy; with -A clippy::all and some extra #![feature(rustc_private)]s almost the whole codebase can be processed. There are some concerns, though:

unlike check, in order to be able to traverse all the crates, some of them need to be marked with the #![feature(rustc_private)] attribute
-W clippy::all breaks on any error. Is there a way to produce errors but not have them break the progress?
I'm not sure how to redirect the errors in a way that would show colors; for now I was able to de-jsonize and print them (something not needed for check)

cargo fix is much more stubborn; it refuses to acknowledge crates like core and std, so it doesn't progress much at all.

Since this is a bit more tricky than I have envisioned, I need some guidance:

is this the right approach or am I doing something very wrong ^^?
why are the extra rustc_private features necessary? I was hoping for the same treatment as check
are changes in clippy and cargo fix needed e.g. in order to produce errors in the same manner as check or did I miss something?
do we need this level of file granularity (e.g. for futureproofing) or can check, clippy and fix files be condensed?

Hopes-to-fix: #53896

Cc @alexcrichton, @zackmdavis

rust-highfive · 2018-12-07T14:48:33Z

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

bors · 2018-12-08T01:54:49Z

☔ The latest upstream changes (presumably #56258) made this pull request unmergeable. Please resolve the merge conflicts.

nikomatsakis · 2018-12-10T11:22:49Z

r? @alexcrichton

bootstrap: fix edition A byproduct of work on rust-lang#56595; done with `cargo fix --edition`.

alexcrichton · 2018-12-14T22:17:41Z

Thanks for the PR! I'm somewhat hesitant to merge this in though because it seems like it's presumably duplicating a lot of code already there for other aspects of rustbuild. Rustbuild already suffers from a good deal of duplication, so I think I'd personally prefer if we could try to explore strategies of deduplication before merging these in

mark-i-m · 2018-12-14T23:25:57Z

I would kind of like to merge this actually. It would be a big step towards moving the compiler to edition 2018.

ljedrz · 2018-12-15T05:58:38Z

@alexcrichton there is definitely room for improvement in terms of deduplication, that's one of the reasons for the WIP; I just wanted to make sure the general approach was fine before squeezing the code a bit more.

I'll rebase and deduplicate the code shortly.

ljedrz · 2018-12-15T08:41:32Z

@alexcrichton I updated the commit - the new commands are now integrated into check.rs.

bootstrap: fix edition A byproduct of work on #56595; done with `cargo fix --edition`.

bors · 2018-12-16T02:24:26Z

☔ The latest upstream changes (presumably #56600) made this pull request unmergeable. Please resolve the merge conflicts.

ljedrz · 2018-12-16T07:22:00Z

Rebased.

sinkuu · 2018-12-16T13:13:00Z

src/bootstrap/check.rs

@@ -23,6 +23,22 @@ pub struct Std {
    pub target: Interned<String>,
 }

+fn args(kind: Kind) -> Vec<String> {
+    match kind {
+        Kind::Clippy => vec!["--".to_owned(), "-W".to_owned(), "clippy::all".to_owned()],


-W clippy::all breaks on any error. Is there a way to produce errors but not have them break the progress?

--cap-lints warn should work.

I managed to pass it into clippy arguments, but it doesn't seem to be enough 🤔.

sinkuu · 2018-12-16T13:52:17Z

unlike check, in order to be able to traverse all the crates, some of them need to be marked with the #![feature(rustc_private)] attribute

You can make Builder::cargo to add RUSTFLAGS="-Zforce-unstable-if-unmarked" instead of RUSTC_FORCE_UNSTABLE=1 when cmd == "clippy". Bootstrap's rustc shim adds the flag when RUSTC_FORCE_UNSTABLE is found, but cargo clippy invokes its own rustc driver.

I'm not sure how to redirect the errors in a way that would show colors; for now I was able to de-jsonize and print them (something not needed for check)

This is also due to rustc shim. It removes --message-format json from args passed to rustc, so rustc normally prints colored errors to stderr.

ljedrz · 2018-12-16T14:30:51Z

Thanks for the suggestions @sinkuu! I'll apply them soon.

Edit: I added RUSTFLAGS="-Zforce-unstable-if-unmarked" for clippy and fix, but I don't know how I can override --message-format json; any ideas?

oli-obk · 2019-04-23T08:04:46Z

@ljedrz you can now have the colors in the rendered field of json ouptut via --json-rendered=termcolor

ljedrz · 2019-04-23T08:58:52Z

Thanks! I'm currently traveling, but I'll get back to this shortly.

ljedrz · 2019-04-23T20:42:26Z

With the addition of the new flag we now have available thanks to @oli-obk, I am now able to run e.g. ./x.py clippy src/librustc_driver and see nice output.

This option doesn't seem to be available to fix, though, so now I'm wondering if it should also be added to it or if I should omit that parameter from that command altogether for now.

mark-i-m · 2019-04-24T03:43:15Z

It sounds like a nice addition, but perhaps in a followup PR?

ljedrz · 2019-04-24T09:12:33Z

Done; at this point the fix support is limited, but clippy appears to be operational.

Centril · 2019-04-25T08:58:26Z

I would kind of like to merge this actually. It would be a big step towards moving the compiler to edition 2018.

The entirety of the repo (+ relevant submodules) has been moved to 2018 so this PR is no longer needed for that.

(but it is still nice for other things...)

Centril · 2019-05-18T10:41:49Z

r? @oli-obk

src/bootstrap/builder.rs

bors · 2019-05-24T15:50:10Z

☔ The latest upstream changes (presumably #60568) made this pull request unmergeable. Please resolve the merge conflicts.

oli-obk · 2019-05-25T15:44:17Z

src/bootstrap/builder.rs

@@ -898,6 +902,11 @@ impl<'a> Builder<'a> {
            extra_args.push_str(&s);
        }

+        if cmd == "clippy" {
+            extra_args.push_str("-Zforce-unstable-if-unmarked -Zunstable-options \


have you tested that this prevents the rebuilds? I would have assumed this to still cause build thrashing. If this works, then I'm happy with it, if not, you should be able to add them in the args function you created below by putting them after a -- argument

I just checked it - there are no rebuilds. The only thing is that after throwing its warnings (a lot of stuff found in libcore) ./x.py clippy is complaining that it can't find crate for std and I no longer remember if this was an acceptable step forward or not ^^. I remember having trouble making it respect the build order.

While not great, we can solve this in follow-ups. I think this may happen whenever one tries to run clippy on a target without libstd, so it's likely that it's a clippy issue.

oli-obk · 2019-05-25T18:04:58Z

@bors r+

bors · 2019-05-25T18:04:59Z

📌 Commit 2f3533b has been approved by oli-obk

bors · 2019-05-25T22:42:20Z

⌛ Testing commit 2f3533b with merge 483567e...

@alexcrichton

Add clippy and fix commands to x.py Since they are kind of similar in nature, I have used the same approach as for `cargo check`. At least some of the boilerplate could probably be shared, but I'd prefer to gather some feedback before I decide to merge them more aggressively. This works reasonably well for `clippy`; with `-A clippy::all` and some extra `#![feature(rustc_private)]`s almost the whole codebase can be processed. There are some concerns, though: - unlike `check`, in order to be able to traverse all the crates, some of them need to be marked with the `#![feature(rustc_private)]` attribute - `-W clippy::all` breaks on any error. Is there a way to produce errors but not have them break the progress? - I'm not sure how to redirect the errors in a way that would show colors; for now I was able to de-jsonize and print them (something not needed for `check`) `cargo fix` is much more stubborn; it refuses to acknowledge crates like `core` and `std`, so it doesn't progress much at all. Since this is a bit more tricky than I have envisioned, I need some guidance: - is this the right approach or am I doing something very wrong ^^? - why are the extra `rustc_private` features necessary? I was hoping for the same treatment as `check` - are changes in `clippy` and `cargo fix` needed e.g. in order to produce errors in the same manner as `check` or did I miss something? - do we need this level of file granularity (e.g. for futureproofing) or can `check`, `clippy` and `fix` files be condensed? Hopes-to-fix: #53896 Cc @alexcrichton, @zackmdavis

bors · 2019-05-26T01:31:26Z

☀️ Test successful - checks-travis, status-appveyor
Approved by: oli-obk
Pushing 483567e to master...

@ljedrz

Emit ansi color codes in the `rendered` field of json diagnostics cc @ljedrz Implemented for rust-lang/rust#56595 (comment) (x.py clippy)

rust-highfive assigned nikomatsakis Dec 7, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 7, 2018

ljedrz mentioned this pull request Dec 7, 2018

bootstrap: fix edition #56600

Merged

rust-highfive assigned alexcrichton and unassigned nikomatsakis Dec 10, 2018

alexcrichton mentioned this pull request Dec 10, 2018

Add more rust_private attributes #56559

Closed

kennytm added a commit to kennytm/rust that referenced this pull request Dec 12, 2018

Rollup merge of rust-lang#56600 - ljedrz:fix_edition, r=Mark-Simulacrum

613f082

bootstrap: fix edition A byproduct of work on rust-lang#56595; done with `cargo fix --edition`.

kennytm added a commit to kennytm/rust that referenced this pull request Dec 13, 2018

Rollup merge of rust-lang#56600 - ljedrz:fix_edition, r=Mark-Simulacrum

3cff3f6

bootstrap: fix edition A byproduct of work on rust-lang#56595; done with `cargo fix --edition`.

kennytm added a commit to kennytm/rust that referenced this pull request Dec 14, 2018

Rollup merge of rust-lang#56600 - ljedrz:fix_edition, r=Mark-Simulacrum

4b1afad

bootstrap: fix edition A byproduct of work on rust-lang#56595; done with `cargo fix --edition`.

kennytm added a commit to kennytm/rust that referenced this pull request Dec 14, 2018

Rollup merge of rust-lang#56600 - ljedrz:fix_edition, r=Mark-Simulacrum

f7e5aef

bootstrap: fix edition A byproduct of work on rust-lang#56595; done with `cargo fix --edition`.

ljedrz force-pushed the x_py_clippy_fix branch from 9135b3e to 0e43db1 Compare December 15, 2018 08:38

ljedrz changed the title ~~[WIP] Add clippy and fix commands to x.py~~ Add clippy and fix commands to x.py Dec 15, 2018

bors added a commit that referenced this pull request Dec 15, 2018

Auto merge of #56600 - ljedrz:fix_edition, r=Mark-Simulacrum

748d354

bootstrap: fix edition A byproduct of work on #56595; done with `cargo fix --edition`.

ljedrz force-pushed the x_py_clippy_fix branch from 0e43db1 to aee9a7a Compare December 16, 2018 07:21

sinkuu reviewed Dec 16, 2018

View reviewed changes

ljedrz force-pushed the x_py_clippy_fix branch 2 times, most recently from 5176bb1 to 0820b48 Compare December 17, 2018 12:48

ljedrz force-pushed the x_py_clippy_fix branch from c0d52e7 to c8bf775 Compare April 23, 2019 20:35

oli-obk approved these changes Apr 24, 2019

View reviewed changes

oli-obk added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 24, 2019

ljedrz force-pushed the x_py_clippy_fix branch from c8bf775 to ecb42b8 Compare April 24, 2019 09:11

rust-highfive assigned oli-obk and unassigned alexcrichton May 18, 2019

oli-obk reviewed May 18, 2019

View reviewed changes

src/bootstrap/builder.rs Outdated Show resolved Hide resolved

Add clippy and fix commands to x.py

2f3533b

ljedrz force-pushed the x_py_clippy_fix branch from ecb42b8 to 2f3533b Compare May 25, 2019 13:32

oli-obk reviewed May 25, 2019

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 25, 2019

bors added the merged-by-bors This PR was explicitly merged by bors. label May 26, 2019

bors merged commit 2f3533b into rust-lang:master May 26, 2019

ljedrz deleted the x_py_clippy_fix branch May 26, 2019 02:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add clippy and fix commands to x.py #56595

Add clippy and fix commands to x.py #56595

ljedrz commented Dec 7, 2018 •

edited

Loading

rust-highfive commented Dec 7, 2018

bors commented Dec 8, 2018

nikomatsakis commented Dec 10, 2018

alexcrichton commented Dec 14, 2018

mark-i-m commented Dec 14, 2018

ljedrz commented Dec 15, 2018

ljedrz commented Dec 15, 2018 •

edited

Loading

bors commented Dec 16, 2018

ljedrz commented Dec 16, 2018

sinkuu Dec 16, 2018

ljedrz Dec 17, 2018 •

edited

Loading

sinkuu commented Dec 16, 2018 •

edited

Loading

ljedrz commented Dec 16, 2018 •

edited

Loading

oli-obk commented Apr 23, 2019

ljedrz commented Apr 23, 2019

ljedrz commented Apr 23, 2019

mark-i-m commented Apr 24, 2019

ljedrz commented Apr 24, 2019 •

edited

Loading

Centril commented Apr 25, 2019

Centril commented May 18, 2019

bors commented May 24, 2019

oli-obk May 25, 2019

ljedrz May 25, 2019

oli-obk May 25, 2019

oli-obk commented May 25, 2019

bors commented May 25, 2019

bors commented May 25, 2019

bors commented May 26, 2019

Add clippy and fix commands to x.py #56595

Add clippy and fix commands to x.py #56595

Conversation

ljedrz commented Dec 7, 2018 • edited Loading

rust-highfive commented Dec 7, 2018

bors commented Dec 8, 2018

nikomatsakis commented Dec 10, 2018

alexcrichton commented Dec 14, 2018

mark-i-m commented Dec 14, 2018

ljedrz commented Dec 15, 2018

ljedrz commented Dec 15, 2018 • edited Loading

bors commented Dec 16, 2018

ljedrz commented Dec 16, 2018

sinkuu Dec 16, 2018

Choose a reason for hiding this comment

ljedrz Dec 17, 2018 • edited Loading

Choose a reason for hiding this comment

sinkuu commented Dec 16, 2018 • edited Loading

ljedrz commented Dec 16, 2018 • edited Loading

oli-obk commented Apr 23, 2019

ljedrz commented Apr 23, 2019

ljedrz commented Apr 23, 2019

mark-i-m commented Apr 24, 2019

ljedrz commented Apr 24, 2019 • edited Loading

Centril commented Apr 25, 2019

Centril commented May 18, 2019

bors commented May 24, 2019

oli-obk May 25, 2019

Choose a reason for hiding this comment

ljedrz May 25, 2019

Choose a reason for hiding this comment

oli-obk May 25, 2019

Choose a reason for hiding this comment

oli-obk commented May 25, 2019

bors commented May 25, 2019

bors commented May 25, 2019

bors commented May 26, 2019

ljedrz commented Dec 7, 2018 •

edited

Loading

ljedrz commented Dec 15, 2018 •

edited

Loading

ljedrz Dec 17, 2018 •

edited

Loading

sinkuu commented Dec 16, 2018 •

edited

Loading

ljedrz commented Dec 16, 2018 •

edited

Loading

ljedrz commented Apr 24, 2019 •

edited

Loading