Make sure to run all mir analyses in check mode #108730

oli-obk · 2023-03-04T07:42:25Z

Only do so after checking for previous errors as otherwise the ui test suite diff is too large.

a step towards something like #107510

This was previously only done for consts since at the time, only consts needed to run checks after borrowck or drop elab. With the move of const prop lints before optimizations, we should've switched this on for all mir bodies.

r? @ghost

oli-obk · 2023-03-04T07:42:34Z

@bors try @rust-timer queue

bors · 2023-03-04T07:42:45Z

⌛ Trying commit 16ea61bc20aae8ce55d264f4dfc5946d96f6d4bb with merge ef6fb91f1639ca8eb8441f20e006ae031ff3aa98...

bors · 2023-03-04T09:55:28Z

☀️ Try build successful - checks-actions
Build commit: ef6fb91f1639ca8eb8441f20e006ae031ff3aa98 (ef6fb91f1639ca8eb8441f20e006ae031ff3aa98)

rust-timer · 2023-03-04T11:29:37Z

Finished benchmarking commit (ef6fb91f1639ca8eb8441f20e006ae031ff3aa98): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	5.7%	[0.3%, 16.8%]	75
Regressions ❌ (secondary)	3.5%	[0.7%, 18.1%]	51
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.3%	[-0.3%, -0.3%]	2
All ❌✅ (primary)	5.7%	[0.3%, 16.8%]	75

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.4%	[0.9%, 9.1%]	39
Regressions ❌ (secondary)	2.3%	[1.5%, 4.1%]	11
Improvements ✅ (primary)	-2.7%	[-4.5%, -0.8%]	2
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	2.1%	[-4.5%, 9.1%]	41

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	6.0%	[0.7%, 13.3%]	60
Regressions ❌ (secondary)	4.2%	[0.9%, 19.0%]	40
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	6.0%	[0.7%, 13.3%]	60

oli-obk · 2023-03-04T11:48:24Z

cc @tmandry this PR removes a large source of situations where check builds don't report anything but full builds do. It's mainly about reporting lints like unconditional_panic and overflowing shift rhs.

As expected, it is a major regression. This is not changing anything around when PMEs are reported, it's an entirely different issue.

rustbot · 2023-03-06T13:17:07Z

The Miri subtree was changed

cc @rust-lang/miri

Some changes occurred in src/tools/clippy

cc @rust-lang/clippy

oli-obk · 2023-03-06T13:23:46Z

src/tools/miri/tests/pass/track-alloc-1.stderr

-note: tracking was triggered
-   |
-   = note: created extern static allocation of SIZE bytes (alignment ALIGN bytes) with id 1
-   = note: (no span available)
-


I'm a bit baffled why this disappears. The test is actually starting MIR interpretation, and there are no allocations beyond the one with id 1. So my guess is MIR interpretation doesn't create it, but const prop or some other analysis pass does and we just reuse the cached one.

Did AllocIds shift? The PR might evaluate more things earlier now so const-eval uses AllocIds that previously were still available to Miri.

oh 🤦 I didn't go up far enough with the ids, because I started seeing other ids that I thought came later.

…rieb Add regression test for rust-lang#98444 cc rust-lang#108730 this will need to be changed to a `check-fail` test once it lands. Fixes rust-lang#98444

tmandry · 2023-03-07T20:17:55Z

Thanks for tagging me! To check my understanding, this is about fixing some (maybe all?) cases of optimization-dependent errors, and the affected errors we're seeing in tests happen to specifically be lints (rather than hard errors)?

This is not changing anything around when PMEs are reported, it's an entirely different issue.

I'm not 100% sure what you include when you say PMEs. Can I propose to use the terminology I wrote at the top of this doc: Non-local errors in Rust? In those terms, I think what you're talking about is probably a combination of parameter-dependent errors and use-dependent errors.

oli-obk · 2023-03-07T21:32:24Z

This PR does not affect non-local errors. It just makes sure that at least all lints that operate just on function-local information get emitted. It is a prerequisite to reporting all non-local errors in check mode, as most of these lints will start getting reported if we start reporting non-local errors in check mode.

I wanted to open this PR separately so that we can observe the performance changes and diagnostics changes on their own instead of mixing them with the changes that would occur if we reported non-local errors in check mode.

tmandry · 2023-03-08T00:48:34Z

Thanks, that makes sense.

So these lints previously depended on check vs build, anything else? They weren't dependent on whether a function was used or not?

(By the way, I don't feel that strongly about using the term "non-local errors", I care more about splitting out into sub-categories like "CLI-dependent" and "parameter-dependent". If these distinctions don't quite map onto the problem space from your point of view, that would be good to know, but my hope is that everything fits into one or more of those sub-categories.)

oli-obk · 2023-03-08T06:41:30Z

They weren't dependent on whether a function was used or not?

No, these lints are reported for dead code, too. Previously they were only reported for all const items and functions (even those in dead code)

bors · 2023-03-13T03:04:01Z

☔ The latest upstream changes (presumably #108872) made this pull request unmergeable. Please resolve the merge conflicts.

tmandry · 2023-03-21T18:28:22Z

The lang team discussed this today in our meeting (notes). We agreed that we consider this a bug fix and think it should be merged.

After that, we'd like to know how costly it is to go further, eliminating any errors that appear in check but not build.

oli-obk · 2023-03-22T14:15:42Z

r? compiler

oli-obk · 2023-03-24T07:39:48Z

This is not changing anything around when PMEs are reported, it's an entirely different issue.
I am confused by this statement, given that the PR claims to fix #99682 which very much looks like a PME to me? It's not just about "lints like unconditional_panic", at least.

Whoops, wrong issue. i'll search for the right one

RalfJung · 2023-03-24T09:35:28Z

Ah okay. So I'm trying to understand what exactly this diff does: some MIR checks like unsafety, or packed fields, were already run in check builds before. But ConstPropLint was not?

Looking at run_analysis_to_runtime_passes and the passes it calls, we are doing a lot of work here that is unnecessary. Ideally we'd run the lints but not all the optimizations. Sadly, Lint(const_prop_lint::ConstProp) runs after all the run_runtime_lowering_passes. Can this be moved up, effectively running the lint on analysis MIR and thus avoiding the need to generate runtime MIR for check builds?

RalfJung · 2023-03-24T09:36:12Z

compiler/rustc_interface/src/passes.rs

+            tcx.ensure()
+                .mir_drops_elaborated_and_const_checked(ty::WithOptConstParam::unknown(def_id));
+        }
+    });


This looks pretty similar to the existing MIR_effect_checking above. Please add comments explaining why there are two of these blocks and what each of them is doing.

oli-obk · 2023-03-24T15:44:52Z

So anyone using cargo check success to mean "this is guaranteed to compile" is already wrong.

yea... that's the question. Where do we draw the line. Right now the line is arbitrary. If we don't want to check more lints, we may want to consider relaxing a few other things in cargo check, too, just to speed things up.

RalfJung · 2023-03-24T15:50:20Z

I do think it would be valuable to get the const-prop-lints run in check mode. It's not what I would consider a PME, so not having them in check builds is quite surprising. Or put differently: I don't think we should have lints as late in the MIR pipeline as const-pro-lints currently is. We have analysis MIR and runtime MIR, and just conceptually it seems clear that lints should run on analysis MIR, no?

oli-obk · 2023-03-24T16:26:19Z

Looking at run_analysis_to_runtime_passes and the passes it calls, we are doing a lot of work here that is unnecessary. Ideally we'd run the lints but not all the optimizations. Sadly, Lint(const_prop_lint::ConstProp) runs after all the run_runtime_lowering_passes. Can this be moved up, effectively running the lint on analysis MIR and thus avoiding the need to generate runtime MIR for check builds?

Unfortunately this quickly causes cycle errors because the only query before that is mir_promoted, and I can't run it in that query, as const prop lint will try to compute layouts of things containing opaque types, which rely on borrowck having run, which relies on the result of mir_promoted. It could certainly be refactored to support this (const promotion works around that cycle after all), but it's nontrivial.

RalfJung · 2023-03-24T16:28:57Z

Hm, I see. borrowck being needed for layout is surprising (layout shouldn't depend on lifetimes) but does make this tricky indeed.

oli-obk · 2023-03-24T16:35:37Z

Hm, I see. borrowck being needed for layout is surprising (layout shouldn't depend on lifetimes) but does make this tricky indeed.

that's actually a very good point... We can grab the types from typeck for layouting, but we need type_of an opaque type to result in the type with correct lifetimes. I'll see if we can use this to eliminate all our layout cycle problems

oli-obk · 2023-03-24T16:36:41Z

Marking this PR as blocked on moving the const prop lint earlier.

bjorn3 · 2023-03-24T21:06:50Z

borrowck being needed for layout is surprising (layout shouldn't depend on lifetimes)

One of the outputs of borrowck is the list of paths captured by a closure, which affects the layout of closures.

bors · 2023-04-03T10:13:21Z

☔ The latest upstream changes (presumably #109884) made this pull request unmergeable. Please resolve the merge conflicts.

apiraino · 2023-08-15T09:23:55Z

hey 👋 what's the status for this PR? Still S-blocked? thanks for an update :)

cc @oli-obk

oli-obk · 2023-10-23T11:30:57Z

I tried moving const prop earlier, but that is a significant perf regression: #115756

I won't be able to work on this again before January.

RalfJung · 2023-10-24T08:02:10Z

We could run const-prop-lints after borrowck but still as part of analysis MIR -- or does that not help perf-wise? It could be the last pass in analysis MIR. But the point is, it shouldn't be in optimized_mir, as that query (for good reasons) is not fired in cargo check.

oli-obk · 2024-01-03T15:33:07Z

We could run const-prop-lints after borrowck but still as part of analysis MIR

that's already what's happening. run_analysis_to_runtime_passes is run right after we ensure that borrowck has been run. But we don't necessarily run mir_drops_elaborated_and_const_checked (which runs borrowck and the other passes) in check mode (I mean, that is exactly what this PR does 😆 ).

One possibility would be to turn const prop lint into a query and only run that explicitly (and make it work on top of mir_promoted + mir_borrowck). I'll try that out.

RalfJung · 2024-01-03T16:27:42Z

Ah I see, I didn't realize we have all that stuff in run_runtime_lowering_passes that shouldn't be needed for check builds but runs before ConstPropLint.

Do we know why running ConstPropLint earlier is such a perf regression? Is it because MIR gets so much bigger? Would it help to switch the lint to a different analysis? It still does the full const-prop dance; we could possibly trade some lint power for performance here.

However I think also lost track of the big picture here. The critical PR for const blocks is #112879, right? This one here is "just" some nice-to-have better alignment of check builds and full builds in terms of which diagnostics are emitted?

oli-obk · 2024-01-03T18:58:55Z

However I think also lost track of the big picture here.

same. I was sure this was a blocker, but I'm not even sure what the next one is

… r=<try> Remove all ConstPropNonsense We track all locals and projections on them ourselves within the const propagator and only use the InterpCx to actually do some low level operations or read from constants (via `OpTy` we get for said constants). This helps moving the const prop lint out from the normal pipeline and running it just based on borrowck information. This in turn allows us to make progress on rust-lang#108730 (comment) there are various follow up cleanups that can be done after this PR (e.g. not matching on Rvalue twice and doing binop checks twice), but lets try landing this one first. r? `@RalfJung`

… r=cjgillot Remove all ConstPropNonsense We track all locals and projections on them ourselves within the const propagator and only use the InterpCx to actually do some low level operations or read from constants (via `OpTy` we get for said constants). This helps moving the const prop lint out from the normal pipeline and running it just based on borrowck information. This in turn allows us to make progress on rust-lang#108730 (comment) there are various follow up cleanups that can be done after this PR (e.g. not matching on Rvalue twice and doing binop checks twice), but lets try landing this one first. r? `@RalfJung`

Remove all ConstPropNonsense We track all locals and projections on them ourselves within the const propagator and only use the InterpCx to actually do some low level operations or read from constants (via `OpTy` we get for said constants). This helps moving the const prop lint out from the normal pipeline and running it just based on borrowck information. This in turn allows us to make progress on rust-lang/rust#108730 (comment) there are various follow up cleanups that can be done after this PR (e.g. not matching on Rvalue twice and doing binop checks twice), but lets try landing this one first. r? `@RalfJung`

Run const_prop_lint in check builds, too implements rust-lang#108730 (comment) Turns the const_prop_lint pass into a query that is run alongside borrowck.

Dylan-DPC · 2024-03-22T05:35:15Z

@oli-obk what's the status of this?

oli-obk · 2024-04-17T15:19:51Z

#120687 did not succeed. 1.4% average regression with up to 20% regressions

I don't see a way to make progress here

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Mar 4, 2023

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Mar 4, 2023

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Mar 4, 2023

This comment has been minimized.

Sign in to view

oli-obk commented Mar 6, 2023

View reviewed changes

This was referenced Mar 6, 2023

Taking a shared reference of an array suppresses the unconditional_panic lint #98444

Open

Add regression test for #98444 #108812

Merged

This comment has been minimized.

Sign in to view

oli-obk force-pushed the hardening_check_builds branch from 68f22a4 to de89a48 Compare March 22, 2023 13:27

rustbot assigned davidtwco Mar 22, 2023

RalfJung reviewed Mar 24, 2023

View reviewed changes

oli-obk added the S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. label Mar 24, 2023

compiler-errors removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 24, 2023

cjgillot mentioned this pull request Apr 10, 2023

Ensure mir_drops_elaborated_and_const_checked when requiring codegen. #110107

Merged

oli-obk mentioned this pull request Jan 5, 2024

Remove all ConstPropNonsense #119627

Merged

oli-obk mentioned this pull request Feb 5, 2024

Run const_prop_lint in check builds, too #120687

Closed

oli-obk closed this Apr 17, 2024

Make sure to run all mir analyses in check mode #108730

Make sure to run all mir analyses in check mode #108730

Conversation

oli-obk commented Mar 4, 2023 • edited Loading

oli-obk commented Mar 4, 2023

This comment has been minimized.

bors commented Mar 4, 2023

bors commented Mar 4, 2023

This comment has been minimized.

rust-timer commented Mar 4, 2023

Overall result: ❌ regressions - ACTION NEEDED

Instruction count

Max RSS (memory usage)

Cycles

oli-obk commented Mar 4, 2023

This comment has been minimized.

rustbot commented Mar 6, 2023

oli-obk Mar 6, 2023

Choose a reason for hiding this comment

RalfJung Mar 7, 2023

Choose a reason for hiding this comment

oli-obk Mar 7, 2023

Choose a reason for hiding this comment

This comment has been minimized.

tmandry commented Mar 7, 2023

oli-obk commented Mar 7, 2023

tmandry commented Mar 8, 2023

oli-obk commented Mar 8, 2023

bors commented Mar 13, 2023

tmandry commented Mar 21, 2023

oli-obk commented Mar 22, 2023

oli-obk commented Mar 24, 2023

RalfJung commented Mar 24, 2023

RalfJung Mar 24, 2023

Choose a reason for hiding this comment

oli-obk commented Mar 24, 2023

RalfJung commented Mar 24, 2023 • edited Loading

oli-obk commented Mar 24, 2023

RalfJung commented Mar 24, 2023

oli-obk commented Mar 24, 2023

oli-obk commented Mar 24, 2023

bjorn3 commented Mar 24, 2023

bors commented Apr 3, 2023

apiraino commented Aug 15, 2023

oli-obk commented Oct 23, 2023 • edited Loading

RalfJung commented Oct 24, 2023 • edited Loading

oli-obk commented Jan 3, 2024

RalfJung commented Jan 3, 2024

oli-obk commented Jan 3, 2024

Dylan-DPC commented Mar 22, 2024

oli-obk commented Apr 17, 2024

oli-obk commented Mar 4, 2023 •

edited

Loading

RalfJung commented Mar 24, 2023 •

edited

Loading

oli-obk commented Oct 23, 2023 •

edited

Loading

RalfJung commented Oct 24, 2023 •

edited

Loading