Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Audit and handle remaining cases of rustc::potential_query_instability lint (iterating HashMaps) #84447

Open
Aaron1011 opened this issue Apr 22, 2021 · 11 comments
Labels
A-incr-comp Area: Incremental compilation C-cleanup Category: PRs that clean code up or issues documenting cleanup. E-easy Call for participation: Easy difficulty. Experience needed to fix: Not much. Good first issue. E-help-wanted Call for participation: Help is requested to fix this issue. E-mentor Call for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@Aaron1011
Copy link
Member

Aaron1011 commented Apr 22, 2021

The iteration order of a HashMap depends on the Hash value of the keys. In rustc, the Hash value of a key can change between compilation sessions, even if the HashStable value remains the same (e.g. DefId, Symbol, etc.). This means that iterating over a HashMap can cause a query to produce different results given the same (as determined by HashStable) input. This can be quite subtle, as demonstrated by #82272 (comment)

To eliminate this source of issues, we should make an internal rustc-only lint for iterating over a HashMap in any way. This will need to cover explicit method calls (e.g. .keys() and .values()), as well as any usage of the IntoIterator impl for HashMap.

Note that we don't want to ban the usage of HashMaps with unstable keys - as long as they're only used for lookup (and not iteration), it's much more efficient than repeatedly converting to the stable form of a key (e.g. DefId -> DefPathHash or Symbol -> String).

@Aaron1011 Aaron1011 self-assigned this Apr 22, 2021
@Aaron1011 Aaron1011 added A-incr-comp Area: Incremental compilation T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Apr 22, 2021
@bjorn3
Copy link
Member

bjorn3 commented Apr 23, 2021

Note that we don't want to ban the usage of HashMaps with unstable keys - as long as they're only used for lookup (and not iteration),

#64131 added the StableMap wrapper for FxHashMap that doesn't expose the iteration order.

@jyn514
Copy link
Member

jyn514 commented Apr 23, 2021

@bjorn3 see #84446

@cjgillot
Copy link
Contributor

cjgillot commented Jul 5, 2022

The lint exists since #89558, and has allowed on all compiler crates as rustc::potential_query_instability.
We now need to stop allowing it in the compiler.

Steps, for each crate except rustc_data_structures:

  • remove the allow(rustc::potential_query_instability) in lib.rs;
  • replace the FxHash{Map,Set} by FxIndex{Map,Set} where necessary.

Please reach out on zulip if there are any questions.

@cjgillot cjgillot added E-easy Call for participation: Easy difficulty. Experience needed to fix: Not much. Good first issue. E-mentor Call for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion. E-help-wanted Call for participation: Help is requested to fix this issue. labels Jul 5, 2022
@fee1-dead
Copy link
Member

fee1-dead commented Jul 7, 2022

@rustbot claim

@rustbot release-assignment

@rustbot rustbot assigned fee1-dead and unassigned Aaron1011 and fee1-dead Jul 7, 2022
@NiklasJonsson
Copy link
Contributor

It seems like no one is assigned to this and I'd like to give it a try so assigning myself.

@rustbot claim

Dylan-DPC added a commit to Dylan-DPC/rust that referenced this issue Jul 14, 2022
…etrochenkov

remove allow(rustc::potential_query_instability) in rustc_span

Also, avoid sorting before debug output as iteration order can now be
relied upon.

Related rust-lang#84447
bors added a commit to rust-lang-ci/rust that referenced this issue Jul 23, 2022
…ompiler-errors

rustc_expand: Switch FxHashMap to FxIndexMap where iteration is used

Relates rust-lang#84447
@NiklasJonsson
Copy link
Contributor

I was hoping to get back to this after vacation but I haven't had the time and I likely won't in the near future. I've converted some of the modules to use the stable hash containers but there is a decent amount left so it is probably possible to have multiple people working on this, assuming some coordination.

There are two outstanding PRs from me that I will finish up. The first is #99334 and seems ready to merge. Second one is #99065 which requires the get_many_mut feature to be merged for the stable hash containers indexmap-rs/indexmap#238, which in turn is waiting on the stabilization of the matching features on the standard hash containers (#97601).

@rustbot release-assignment

bors added a commit to rust-lang-ci/rust that referenced this issue Sep 12, 2022
…oli-obk

rustc_error, rustc_private: Switch to stable hash containers

Relates rust-lang#84447
flip1995 pushed a commit to flip1995/rust-clippy that referenced this issue Sep 22, 2022
rustc_error, rustc_private: Switch to stable hash containers

Relates rust-lang/rust#84447
@CastilloDel
Copy link
Contributor

I'm gonna give this a try
@rustbot claim

rust-timer added a commit to rust-lang-ci/rust that referenced this issue Dec 12, 2023
Rollup merge of rust-lang#118865 - Enselic:rustc_codegen_llvm-lint-fix, r=petrochenkov

rustc_codegen_llvm: Enforce `rustc::potential_query_instability` lint

Stop allowing `rustc::potential_query_instability` on all of `rustc_codegen_llvm` and instead allow it on a case-by-case basis if it is safe to do so. In this case, all 2 instances are safe to allow.

Part of rust-lang#84447 which is E-help-wanted.
GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this issue Dec 12, 2023
…ty, r=compiler-errors

rustc_passes: Enforce `rustc::potential_query_instability` lint

Stop allowing `rustc::potential_query_instability` in all of `rustc_passes` and instead allow it on a case-by-case basis if it is safe. In this case, all instances of the lint are safe to allow.

Part of rust-lang#84447 which is E-help-wanted.
bors added a commit to rust-lang-ci/rust that referenced this issue Dec 13, 2023
…, r=compiler-errors

rustc_passes: Enforce `rustc::potential_query_instability` lint

Stop allowing `rustc::potential_query_instability` in all of `rustc_passes` and instead allow it on a case-by-case basis if it is safe. In this case, all instances of the lint are safe to allow.

Part of rust-lang#84447 which is E-help-wanted.
github-actions bot pushed a commit to rust-lang/miri that referenced this issue Dec 14, 2023
…ler-errors

rustc_passes: Enforce `rustc::potential_query_instability` lint

Stop allowing `rustc::potential_query_instability` in all of `rustc_passes` and instead allow it on a case-by-case basis if it is safe. In this case, all instances of the lint are safe to allow.

Part of rust-lang/rust#84447 which is E-help-wanted.
GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this issue Dec 15, 2023
…ility, r=michaelwoerister

rustc_mir_build: Enforce `rustc::potential_query_instability` lint

Stop allowing `rustc::potential_query_instability` on all of `rustc_mir_build` and instead allow it on a case-by-case basis if it is safe to do so. In this crate there was only one instance of the lint, and it was safe to allow.

Part of rust-lang#84447 which is E-help-wanted.
rust-timer added a commit to rust-lang-ci/rust that referenced this issue Dec 15, 2023
Rollup merge of rust-lang#118863 - Enselic:rustc_mir-build-query-stability, r=michaelwoerister

rustc_mir_build: Enforce `rustc::potential_query_instability` lint

Stop allowing `rustc::potential_query_instability` on all of `rustc_mir_build` and instead allow it on a case-by-case basis if it is safe to do so. In this crate there was only one instance of the lint, and it was safe to allow.

Part of rust-lang#84447 which is E-help-wanted.
bors added a commit to rust-lang-ci/rust that referenced this issue Dec 31, 2023
…r=cjgillot

rustc_lint: Enforce `rustc::potential_query_instability` lint

Stop allowing `rustc::potential_query_instability` on all of `rustc_lint` and instead allow it on a case-by-case basis if it is safe to do so. In this particular crate, all lints were safe to allow.

Part of rust-lang#84447 which is E-help-wanted.
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Jan 5, 2024
…ble, r=cjgillot

rustc_mir_transform: Make DestinationPropagation stable for queries

By using `FxIndexMap` instead of `FxHashMap`, so that the order of visiting of locals is deterministic.

We also need to bless
`copy_propagation_arg.foo.DestinationPropagation.panic*.diff`. Do not review the diff of the diff. Instead look at the diff files before and after this commit. Both before and after this commit, 3 statements are replaced with nop. It's just that due to change in ordering, different statements are replaced. But the net result is the same. In other words, compare this diff (before fix):
* https://github.com/rust-lang/rust/blob/090d5eac722000906cc00d991f2bf052b0e388c3/tests/mir-opt/dest-prop/copy_propagation_arg.foo.DestinationPropagation.panic-unwind.diff

With this diff (after fix):
* https://github.com/rust-lang/rust/blob/f603babd63a607e155609dc0277806e559626ea0/tests/mir-opt/dest-prop/copy_propagation_arg.foo.DestinationPropagation.panic-unwind.diff

and you can see that both before and after the fix, we replace 3 statements with `nop`s.

I find it _slightly_ surprising that the test this PR affects did not previously fail spuriously due to the indeterminism of `FxHashMap`, but I guess in can be explained with the predictability of small `FxHashMap`s with `usize` (`Local`) keys, or something along those lines.

This should fix [this](rust-lang#119252 (comment)) comment, but I wanted to make a separate PR for this fix for a simpler development and review process.

Part of rust-lang#84447 which is E-help-wanted.

r? `@cjgillot` who is reviewer for the highly related PR rust-lang#119252.
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Jan 5, 2024
…ble, r=cjgillot

rustc_mir_transform: Make DestinationPropagation stable for queries

By using `FxIndexMap` instead of `FxHashMap`, so that the order of visiting of locals is deterministic.

We also need to bless
`copy_propagation_arg.foo.DestinationPropagation.panic*.diff`. Do not review the diff of the diff. Instead look at the diff files before and after this commit. Both before and after this commit, 3 statements are replaced with nop. It's just that due to change in ordering, different statements are replaced. But the net result is the same. In other words, compare this diff (before fix):
* https://github.com/rust-lang/rust/blob/090d5eac722000906cc00d991f2bf052b0e388c3/tests/mir-opt/dest-prop/copy_propagation_arg.foo.DestinationPropagation.panic-unwind.diff

With this diff (after fix):
* https://github.com/rust-lang/rust/blob/f603babd63a607e155609dc0277806e559626ea0/tests/mir-opt/dest-prop/copy_propagation_arg.foo.DestinationPropagation.panic-unwind.diff

and you can see that both before and after the fix, we replace 3 statements with `nop`s.

I find it _slightly_ surprising that the test this PR affects did not previously fail spuriously due to the indeterminism of `FxHashMap`, but I guess in can be explained with the predictability of small `FxHashMap`s with `usize` (`Local`) keys, or something along those lines.

This should fix [this](rust-lang#119252 (comment)) comment, but I wanted to make a separate PR for this fix for a simpler development and review process.

Part of rust-lang#84447 which is E-help-wanted.

r? ```@cjgillot``` who is reviewer for the highly related PR rust-lang#119252.
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Jan 6, 2024
…ble, r=cjgillot

rustc_mir_transform: Make DestinationPropagation stable for queries

By using `FxIndexMap` instead of `FxHashMap`, so that the order of visiting of locals is deterministic.

We also need to bless
`copy_propagation_arg.foo.DestinationPropagation.panic*.diff`. Do not review the diff of the diff. Instead look at the diff files before and after this commit. Both before and after this commit, 3 statements are replaced with nop. It's just that due to change in ordering, different statements are replaced. But the net result is the same. In other words, compare this diff (before fix):
* https://github.com/rust-lang/rust/blob/090d5eac722000906cc00d991f2bf052b0e388c3/tests/mir-opt/dest-prop/copy_propagation_arg.foo.DestinationPropagation.panic-unwind.diff

With this diff (after fix):
* https://github.com/rust-lang/rust/blob/f603babd63a607e155609dc0277806e559626ea0/tests/mir-opt/dest-prop/copy_propagation_arg.foo.DestinationPropagation.panic-unwind.diff

and you can see that both before and after the fix, we replace 3 statements with `nop`s.

I find it _slightly_ surprising that the test this PR affects did not previously fail spuriously due to the indeterminism of `FxHashMap`, but I guess in can be explained with the predictability of small `FxHashMap`s with `usize` (`Local`) keys, or something along those lines.

This should fix [this](rust-lang#119252 (comment)) comment, but I wanted to make a separate PR for this fix for a simpler development and review process.

Part of rust-lang#84447 which is E-help-wanted.

r? `@cjgillot` who is reviewer for the highly related PR rust-lang#119252.
rust-timer added a commit to rust-lang-ci/rust that referenced this issue Jan 6, 2024
Rollup merge of rust-lang#119591 - Enselic:DestinationPropagation-stable, r=cjgillot

rustc_mir_transform: Make DestinationPropagation stable for queries

By using `FxIndexMap` instead of `FxHashMap`, so that the order of visiting of locals is deterministic.

We also need to bless
`copy_propagation_arg.foo.DestinationPropagation.panic*.diff`. Do not review the diff of the diff. Instead look at the diff files before and after this commit. Both before and after this commit, 3 statements are replaced with nop. It's just that due to change in ordering, different statements are replaced. But the net result is the same. In other words, compare this diff (before fix):
* https://github.com/rust-lang/rust/blob/090d5eac722000906cc00d991f2bf052b0e388c3/tests/mir-opt/dest-prop/copy_propagation_arg.foo.DestinationPropagation.panic-unwind.diff

With this diff (after fix):
* https://github.com/rust-lang/rust/blob/f603babd63a607e155609dc0277806e559626ea0/tests/mir-opt/dest-prop/copy_propagation_arg.foo.DestinationPropagation.panic-unwind.diff

and you can see that both before and after the fix, we replace 3 statements with `nop`s.

I find it _slightly_ surprising that the test this PR affects did not previously fail spuriously due to the indeterminism of `FxHashMap`, but I guess in can be explained with the predictability of small `FxHashMap`s with `usize` (`Local`) keys, or something along those lines.

This should fix [this](rust-lang#119252 (comment)) comment, but I wanted to make a separate PR for this fix for a simpler development and review process.

Part of rust-lang#84447 which is E-help-wanted.

r? `@cjgillot` who is reviewer for the highly related PR rust-lang#119252.
compiler-errors added a commit to compiler-errors/rust that referenced this issue Jan 7, 2024
…stability, r=cjgillot

rustc_mir_transform: Enforce `rustc::potential_query_instability` lint

Stop allowing `rustc::potential_query_instability` on all of rustc_mir_transform and instead allow it on a case-by-case basis if it is safe to do so. In this particular crate, all instances were safe to allow.

Part of rust-lang#84447 which is E-help-wanted.
rust-timer added a commit to rust-lang-ci/rust that referenced this issue Jan 7, 2024
Rollup merge of rust-lang#119252 - Enselic:rustc_mir_transform-query-stability, r=cjgillot

rustc_mir_transform: Enforce `rustc::potential_query_instability` lint

Stop allowing `rustc::potential_query_instability` on all of rustc_mir_transform and instead allow it on a case-by-case basis if it is safe to do so. In this particular crate, all instances were safe to allow.

Part of rust-lang#84447 which is E-help-wanted.
bors added a commit to rust-lang-ci/rust that referenced this issue Jan 13, 2024
…bility, r=Nilstrieb

rustc_codegen_ssa: Enforce `rustc::potential_query_instability` lint

Part of rust-lang#84447.
github-actions bot pushed a commit to rust-lang/miri that referenced this issue Jan 14, 2024
…=Nilstrieb

rustc_codegen_ssa: Enforce `rustc::potential_query_instability` lint

Part of rust-lang/rust#84447.
@chenyukang chenyukang self-assigned this Jan 30, 2024
lnicola pushed a commit to lnicola/rust-analyzer that referenced this issue Apr 7, 2024
…=Nilstrieb

rustc_codegen_ssa: Enforce `rustc::potential_query_instability` lint

Part of rust-lang/rust#84447.
RalfJung pushed a commit to RalfJung/rust-analyzer that referenced this issue Apr 27, 2024
…=Nilstrieb

rustc_codegen_ssa: Enforce `rustc::potential_query_instability` lint

Part of rust-lang/rust#84447.
@chenyukang chenyukang removed their assignment May 7, 2024
@chenyukang
Copy link
Member

There may be more need to clean up, for anyone maybe interested can refer to #120485

bors added a commit to rust-lang-ci/rust that referenced this issue Jun 3, 2024
… r=<try>

Stabilize order of MonoItems in CGUs and disallow query_instability lint for rustc_monomorphize

The HashStable impl for `CodegenUnit` was incorrect as described in [MCP 533](rust-lang/compiler-team#533). This PR removes any indeterminism from the way codegen units are built. The changes are pretty straightforward.

Part of rust-lang#84447 and [MCP 533](rust-lang/compiler-team#533).
bors added a commit to rust-lang-ci/rust that referenced this issue Jun 7, 2024
… r=oli-obk

Stabilize order of MonoItems in CGUs and disallow query_instability lint for rustc_monomorphize

The HashStable impl for `CodegenUnit` was incorrect as described in [MCP 533](rust-lang/compiler-team#533). This PR removes any indeterminism from the way codegen units are built. The changes are pretty straightforward.

Part of rust-lang#84447 and [MCP 533](rust-lang/compiler-team#533).
RalfJung pushed a commit to RalfJung/miri that referenced this issue Jun 8, 2024
Stabilize order of MonoItems in CGUs and disallow query_instability lint for rustc_monomorphize

The HashStable impl for `CodegenUnit` was incorrect as described in [MCP 533](rust-lang/compiler-team#533). This PR removes any indeterminism from the way codegen units are built. The changes are pretty straightforward.

Part of rust-lang/rust#84447 and [MCP 533](rust-lang/compiler-team#533).
flip1995 pushed a commit to flip1995/rust-clippy that referenced this issue Jun 28, 2024
Stabilize order of MonoItems in CGUs and disallow query_instability lint for rustc_monomorphize

The HashStable impl for `CodegenUnit` was incorrect as described in [MCP 533](rust-lang/compiler-team#533). This PR removes any indeterminism from the way codegen units are built. The changes are pretty straightforward.

Part of rust-lang/rust#84447 and [MCP 533](rust-lang/compiler-team#533).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-incr-comp Area: Incremental compilation C-cleanup Category: PRs that clean code up or issues documenting cleanup. E-easy Call for participation: Easy difficulty. Experience needed to fix: Not much. Good first issue. E-help-wanted Call for participation: Help is requested to fix this issue. E-mentor Call for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

10 participants