misc coercion cleanups and handle safety correctly

[BoxyUwU]

r? lcnr

"remove normalize call"

Fixes #132765

If the normalization fails we would sometimes get a TypeError containing inference variables created inside of the probe used by coercion. These would then get leaked out causing ICEs in diagnostics logic

"leak check and lub for closure<->closure coerce-lubs of same defids"

Fixes rust-lang/trait-system-refactor-initiative#233

fn peculiar() -> impl Fn(u8) -> u8 {
    return |x| x + 1
}

the |x| x + 1 expr has a type of Closure(?31t) which we wind up inferring the RPIT to. The CoerceMany ret_coercion for the whole peculiar typeck has an expected type of RPIT (unnormalized). When we type check the return |x| x + 1 expr we go from the never type to Closure(?31t) which then participates in the ret_coercion giving us a coerce-lub(RPIT, Closure(?31t)).

Normalizing RPIT gives us some Closure(?50t) where ?31t and ?50t have been unified with ?31t as the root var. resolve_vars_if_possible doesn't resolve infer vars to their roots so these wind up with different structural identities so the fast path doesn't apply and we fall back to coercing to a fn ptr. cc #147193 which also fixes this

New solver probably just gets more inference variables here because canonicalization + generally different approach to normalization of opaques. Idk :3

FCP worthy stuffy

there are some other FCP worthy things but they're in my FCP comment which also contains some analysis of the breaking nature of the previously listed changes in this PR: #148602 (comment)

[rustbot]

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

Some changes occurred in compiler/rustc_codegen_ssa

cc @WaffleLapkin

Some changes occurred to the CTFE machinery

cc @RalfJung, @oli-obk, @lcnr

Some changes occurred in src/tools/clippy

cc @rust-lang/clippy

Some changes occurred to the CTFE / Miri interpreter

cc @rust-lang/miri

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

This PR changes rustc_public

cc @oli-obk, @celinval, @ouz-a

This PR changes a file inside tests/crashes. If a crash was fixed, please move into the corresponding ui subdir and add 'Fixes #' to the PR description to autoclose the issue upon merge.

Some changes occurred to constck

cc @fee1-dead

[BoxyUwU]

Hi @rust-lang/types. I've recently gone over our implementation of coercions and discovered a few things which I'd like to change which need to go through an FCP. I'm not really sure if it makes sense to split this into three separate FCPs or not so I've just kept it all together. Let me know if you'd rather me split them apart :)

I am not including lang on this FCP as this is either clear bug fixes that make our behaviour more consistent and agree with what is already in the reference, or its just incredibly minor type checking stuff :)

@rfcbot merge

Leak Checks in coerce-lub to fn-pointers

Background Context

For misc context on the leak checking see a previous FCP by lcnr: #119820. Also note that due to not having implied bounds on binders the leak check can incorrectly produce errors that would be satisfied if we took implied bounds into account.

leak check fndef lub fndef

When performing a coerce-lub between two FnDefs we first attempt to see if they have some common supertype (infcx.lub). If there is a common supertype we return that instead of coercing the FnDefs to fn-pointers.

Note that FnDefs are invariant over their generic parameters so in this case having a common supertype effectively means the FnDefs are equal.

macro_rules! lub {
    ($lhs:expr, $rhs:expr) => {
        if true { $lhs } else { $rhs }
    };
}

fn foo<T: ?Sized>() {}

// This performs a `coerce-lub(foo<?lhs>, foo<?rhs)` operation, which
// will infer `?lubbed_ty = foo<?lhs>, ?lhs=?rhs`.
let r: /* ?lubbed_ty */ = lub!(foo::<_ /* ?lhs */>, foo::<_ /* ?rhs */>);

// assert that `r` is a ZST/`FnDef`
unsafe { std::mem::transmute::<_, ()>(r) }

On stable we do not leak check after this infcx.lub operation. This can lead us to incorrectly believe two FnDefs have a mutual supertype and avoid coercing to fn-pointers.

This FCP proposes that we start leak checking there. See this test for a code example that will go from error->compile with this change: tests/ui/coercion/leak_check_fndef_lub.rs

This is theoretically breaking. There could exist deadcode which performs a coerce-lub of two unequal-by-binders FnDefs and then only typechecks due to the lubbed type being a FnDef.

With this change we would now (correctly) coerce those FnDefs to fn-pointers resulting in an error. See this test for a code example that will go from compile->error with this change: tests/ui/coercion/leak_check_fndef_lub_deadcode_breakage.rs

I believe this theoeretical breakage to be acceptable. I do not expect this to be encountered in practice let alone often, crater results agree with this showing no regressions due to this change.

If someone does wind up encountering this I still think this is acceptable. This feels like a clear bug fix to me and certainly falls under "allowable" inference breakage.

perform closure lub closure

When performing a coerce-lub between two closures we now handle it the same way we do FnDefs (including the proposed leak check addition). We first try to determine if the two closure types have a mutual supertype and if so we return that instead of coercing both to a fn-pointer.

I don't believe this is observable on stable but can be observed on nightly via type_alias_impl_trait hacks under the new trait solver: tests/ui/coercion/lub_closures_before_fnptr_coercion.rs

Safety and Target Features in coerce-lub

Background Context

See the reference header describing target features for an explanation of how target features interact with the safety of the function: r-attributes.codegen.target_feature

safe target feature fns are fn not unsafe fn

On stable when performing a coerce-lub operation between closures and FnDefs we do not treat the FnDef's signature as being safe if it has target features enabled. This differs from normal coerce operations.

This differs from normal coerce operations where when coercing an FnDef to a fn-pointer we will produce a safe fn-pointer if the target features of the FnDef are also enabled in the current body:

#[target_feature(enable = "avx512")]
fn my_fn() {}

#[target_feature(enable = "avx512")]
fn test() {
    let a: fn() = my_fn;
}

It also differs from other coerce-lub operations when coercing between a FnDef and a fn-pointer, which does correctly allow the FnDef to coerce to a safe fn-pointer.

This FCP proposes that when acquiring the signature of a FnDef during a coerce-lub operation, we give a safe signature if all of the target features of the FnDef are enabled in the current body.

See this test for examples of the proposed behaviour: tests/ui/coercion/lub_coercion_handles_safety.rs

This is theoretically a breaking change as there could be code that relies on us incorrectly coercing safe target feature functions to unsafe fnpointers when we will now coerce them to safe fn-pointers.

I expect this to be quite unlikely as safe target features are a relatively new addition to the language (1.86.0) and also a somewhat niche feature.

The crater runs confirm this showing no breakage due to this change.

If someone does wind up encountering this it still feels acceptable to me as it is a clear bug fix and certainly falls under "allowable" inference breakage.

fn->unsafe fn in coerce-lub of fndefs and closures

On stable when performing a coerce-lub operation between closures and FnDefs we do not allow coercing the closure or FnDef to an unsafe fn-pointer if it is a safe fn-pointer.

This differs from normal coerce operations where when coercing n safe FnDef/closure to an unsafe fn-pointer we will produce an unsafe fn-pointer:

fn my_fn() {}

fn test() {
    let a: unsafe fn() = my_fn;
}

It also differs from other coerce-lub operations when coercing between a safe FnDef/closure and a unsafe fn-pointer, which does correctly allow the FnDef/closure to coerce to an unsafe fn-pointer.

This FCP proposes that after acquiring the signatures of both sides of the coerce-lub operation, if they have mismatched safeties we coerce the safe signature to an unsafe signature.

See this test for examples of the proposed behaviour: tests/ui/coercion/lub_coercion_handles_safety.rs

I don't expect this to be able to break code as any code involving coerce-lubs of signatures with differing safeties is currently an error on stable.

Removing an unnecessary normalize call

We currently have a redundant normalize call inside of coercion logic to re-normalize an already normalized type. This was mostly a code tidyup but happend to fix an ICE.

This is theoretically breaking as re-normalizing already normalized types is not necessarily a no-op under the old solver due to ambiguous aliases inside of binders. I don't expect this to really be encountered in practice and this is backed up by the lack of crater regressions. If there are regressions we can always just re-add a normalize call somewhere.

Crater Regressions

There were no crater regressions detected for this set of changes. The crater run can be found here: #147565#issuecomment-3492623733. This crater run had an additional change part of it which does cause regressions and is not included in this PR. An analysis of those regressions can be found here: #147565#issuecomment-3486416652, which should hopefully show that those regressions are not a problem without the additional changes that PR has.

[rust-rfcbot]

Team member @BoxyUwU has proposed to merge this. The next step is review by the rest of the tagged team members:

• @BoxyUwU
• @jackh726
• @lcnr
• @oli-obk
• @spastorino

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[BoxyUwU]

I've gone and allowed ReifyFnPointer coercions to coerce safe fndefs to unsafe fn pointers directly instead of needing to compose two coercions. This made handling safety in coerce-lub simpler/nicer.

[rust-log-analyzer]

The job x86_64-gnu-tools failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

 Documenting proc_macro_test v0.1.0 (/checkout/tests/rustdoc-gui/src/proc_macro_test)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.79s
   Generated /checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-gui/doc/proc_macro_test/index.html
npm WARN deprecated puppeteer@22.15.0: < 24.10.2 is no longer supported
npm ERR! code 127
npm ERR! git dep preparation failed
npm ERR! command /node/bin/node /node/lib/node_modules/npm/bin/npm-cli.js install --force --cache=/home/user/.npm --prefer-offline=false --prefer-online=false --offline=false --no-progress --no-save --no-audit --include=dev --include=peer --include=optional --no-package-lock-only --no-dry-run
npm ERR! npm WARN using --force Recommended protections disabled.
npm ERR! npm WARN deprecated inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
npm ERR! npm WARN deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
npm ERR! npm WARN deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
npm ERR! npm WARN deprecated readdir-scoped-modules@1.1.0: This functionality has been moved to @npmcli/fs
npm ERR! npm WARN deprecated debuglog@1.0.1: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.
npm ERR! npm WARN deprecated read-package-json@2.1.2: This package is no longer supported. Please use @npmcli/package-json instead.
npm ERR! npm WARN deprecated read-installed@4.0.3: This package is no longer supported.
npm ERR! npm ERR! code 127
npm ERR! npm ERR! path /home/user/.npm/_cacache/tmp/git-cloneXXXXXXspIiXs/node_modules/rollup
npm ERR! npm ERR! command failed
npm ERR! npm ERR! command sh -c patch-package
npm ERR! npm ERR! sh: 1: patch-package: not found
npm ERR! 
npm ERR! npm ERR! A complete log of this run can be found in: /home/user/.npm/_logs/2025-11-07T17_05_25_027Z-debug-0.log

npm ERR! A complete log of this run can be found in: /home/user/.npm/_logs/2025-11-07T17_05_17_591Z-debug-0.log
npm install did not exit successfully

thread 'main' (61389) panicked at src/tools/rustdoc-gui-test/src/main.rs:63:10:
unable to install browser-ui-test: Custom { kind: Other, error: "npm install returned exit code exit status: 127" }
stack backtrace:
   0: __rustc::rust_begin_unwind
             at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/std/src/panicking.rs:698:5
   1: core::panicking::panic_fmt
             at /rustc/3b4dd9bf1410f8da6329baa36ce5e37673cbbd1f/library/core/src/panicking.rs:80:14