Remove ability to disable some target features

[GuentherVIII]

The documentation of TargetOptions.features claims that these features will always be passed to LLVM, but the logic implemented in #83084 allows them to be disabled with -C target-feature. Change the implementation to match the documentation.

Disabling x87 on x86 changes the way functions return floats. The second commit makes x87 mandatory on the existing i686 targets. Disabling floating point usage now requires a new target. #116344 does not fully get fixed by this (softfloat targets still have a problem with +x87), but it helps a lot.

In preparation for #115919 sse and sse2 are also made mandatory.

See https://rust-lang.zulipchat.com/#narrow/stream/136281-t-opsem/topic/Float.20ABI.20hell/near/394473707 and #116344.

Lang team: see nomination comment below.

@rustbot label: +A-abi +O-x86 +T-opsem +T-compiler

[rustbot]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @cjgillot (or someone else) soon.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

• @rustbot author: the review is finished, PR author should check the comments and take action accordingly
• @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

[rustbot]

These commits modify compiler targets.
(See the Target Tier Policy.)

[workingjubilee]

?
That should've worked.

@rustbot label: +A-abi +O-x86 +T-opsem

[cjgillot]

I'm not familiar enough with this code to review.
r? compiler

[petrochenkov]

cc @nagisa as the author of #83084.

[petrochenkov]

Change the implementation to match the documentation.

This alone is not a reason to make the change, that comment is from 2014 and it's not a public documentation, exactly.
Maybe it's the comment that needs to be changed.

Why does it make sense to have non-removable features for targets?

If yes, then are all features currently specified in target specs supposed to be non-removable?
Perhaps it makes sense to have two fields in target specs, for non-removable features, and for features that are just enabled by default?

[petrochenkov]

I'm not sure whether it needs a full MCP, but I'd like to see the "disabling x87 on x86 changes the way functions return floats" sentence to be elaborated into a more detailed proposal.
(I'll read the linked threads in the meantime.)
@rustbot author

[workingjubilee]

Why does it make sense to have non-removable features for targets?

Because some can change the call ABI, and this makes changing the features incompatible with the prescribed call ABI, especially for extern "C".

If yes, then are all features currently specified in target specs supposed to be non-removable?

Only ones that change ABI must be considered immutable. Specifically, we need to know what the "baseline" ABI is supposed to be for the target and for that knowledge to not disappear. If LLVM supported different ways of handling this problem, we could adopt them, but afaik it does not.

[chorman0773]

Only ones that change ABI must be considered immutable

This seems like it would imply that avx, avx512f, and avx10 cannot be enabled.

[workingjubilee]

It doesn't, because I was talking about the features of the target CPU, not the features of the target ISA.

[RalfJung]

With this PR, what happens when someone does -C target-features=-x87? Does it just silently get ignored? I think it'd be prudent to at least emit a warning saying that the target spec features are overwriting the -C flag.

I'm not sure whether it needs a full MCP, but I'd like to see the "disabling x87 on x86 changes the way functions return floats" sentence to be elaborated into a more detailed proposal.

This is one half of resolving #116344, so that issue should hopefully provide the necessary context.

But yeah I agree this needs some form of process, t-compiler MCP at least.

[nagisa]

Why is the feature stability mechanism insufficient for this? If you don't want to make it possible to disable them (at which point they are ABI-critical and it doesn’t make sense to make it possible to enable a disabled feature either, since it would also change the ABI) just don’t make the feature stably controllable via -Ctarget-feature in the first place?

x87 in particular is not in the X86_ALLOWED_FEATURES set, so in that respect closing down the loophole where you could pass in features that aren’t in this list would also fix the immediate issue (and possibly others)

It doesn't, because I was talking about the features of the target CPU, not the features of the target ISA.

I have trouble understanding what this sentence means. Neither LLVM nor Rust has a concept of target ISA features, as far as I know? One way or another everything comes down to a single set of flags, whether they are configured implicitly (e.g. through target-cpu metadata) or explicitly… What makes avx512 features "ISA" features and not "CPU" ones?

[nagisa]

I don’t believe this change addresses this FIXME at all. It still isn’t at all clear to me, even if we agreed that we don’t want to allow some features to be configurable outside of --target specifications, that this specific way to build up the feature set is the absolutely the correct one.

For example:

rustc -Ctarget-cpu=native --target=x86_64-unknown-linux-gnu

pretty clearly shows that the intent is for the -Ctarget-cpu=native to take precedence over whatever x86_64-unknown-linux-gnu means (including it not having any of the fancy AVX extensions.)

On the other hand

rustc -Ctarget-cpu=pentium4 --target=x86_64v4-unknown-linux-gnu

is quite a bit more ambiguous: does the user want something that would run on a pentium4? Or something that would run on an x86_64v4 class hardware, which pentium4 is not? Should the ordering of flags be taken into the account (unusual for rustc, but standard behaviour in gcc/clang style CLIs.) Perhaps it would be better to just error outright?

[petrochenkov]

I'll try to play a devil's advocate here.

-Ctarget-features is an inherently unsafe feature that relies on users understanding what they are doing.
What if by trying to make it safe you remove support for its main use cases, is it still worth doing then?

ABI differences turn into unsafety only if functions with one set of features are called as functions with another incompatible set of features.
If the user avoids such calls then everything is good and creating a new target (which is a major nuisance) is not necessary.

Maybe if we are trying to improve safety, it should be done in a more fine-grained way too.
For example, by including the feature bitset into function pointer types (internally for a start, without a surface syntax).
Then to convert function pointer types the user will have to perform an unsafe operation.

[RalfJung]

-Ctarget-features is inherently unsafe feature

I wasn't aware of that, but rustc -C help indeed says so. It does that in a completely unhelpful way though since it doesn't tell me what the safety requirements are.

-Ctarget-cpu fundamentally has the same problem and has no such comment.

[workingjubilee]

@petrochenkov: -Ctarget-features is an inherently unsafe feature that relies on users understanding what they are doing. What if by trying to make it safe you remove support for its main use cases, is it still worth doing then?

The problem is that, as Ralf highlighted, very few people actually understand the safety requirements. The majority of users only know that they can "solve a problem" by enabling/disabling features, they don't necessarily understand the larger ramifications of that. Users can and do set incompatible sets of features that fail to even pass LLVM's isel. I would even argue that very few compiler engineers understand the safety requirements, given the number of ABI bugs we've had along these lines.

@nagisa: x87 in particular is not in the X86_ALLOWED_FEATURES set, so in that respect closing down the loophole where you could pass in features that aren’t in this list would also fix the immediate issue (and possibly others)

That could work for x87, at least, but I'm not sure it works for sse and sse2.

I have trouble understanding what this sentence means. Neither LLVM nor Rust has a concept of target ISA features, as far as I know? One way or another everything comes down to a single set of flags, whether they are configured implicitly (e.g. through target-cpu metadata) or explicitly… What makes avx512 features "ISA" features and not "CPU" ones?

In the model I was implicitly referencing, a given CPU implements a specific instance of an ISA, with certain ISA extensions, and -Ctarget-cpu implicitly designates a subset of the ISA as the target by targeting that CPU. Our targets are effectively defined in terms of specific CPU instances we want code to be able to run on.

[RalfJung]

For the one use case of disabling x87 I know of, kernel code which wants to avoid saving the user mode x87 registers, linking in code that uses the x87 is a problem even if no function returning a float is ever called. In the Linux kernel, for example, calling any function working with floats is bad without calling kernel_fpu_begin() before. Which suggests that calling functions with target_feature(x87) from code without it should be unsafe?

Yes, that should either be a separate target or such calls should be disallowed. (Not just unsafe, they are always UB so allowing them makes little sense even in unsafe code.) Usually, "softfloat" is a separate target, so that would be a good pattern for x86 to follow as well.

[workingjubilee]

That's already what x86_64-unknown-none does.

Please refer to the target spec:

rust/compiler/rustc_target/src/spec/x86_64_unknown_none.rs

Lines 21 to 23 in 4b85902

	features:
	"-mmx,-sse,-sse2,-sse3,-ssse3,-sse4.1,-sse4.2,-3dnow,-3dnowa,-avx,-avx2,+soft-float"
	.into(),

[workingjubilee]

If we actually want to support this configuration being set for any given x86_64-?-? target, we can't really use the xmm registers for anything, as we have to at least assume that these features may be enabled or disabled without reference to the target. This prevents a basic optimization, like actually passing vector registers as registers, from being implemented, without it being effectively instantly UB to compile. Because, yes, ever calling that code is incorrect, even if it passes or receives no such values!

And if it is unsupported, and we're okay with it being UB to enable, then why are we allowing people to set an option that would be effectively guaranteed to make all linkage to the Rust core library virtually instantly undefined behavior, no ifs, ands, or buts?

[workingjubilee]

Setting -Ctarget-feature=-sse2,-sse as a compiler flag, as a user, then thinking that would take care of it, and then linking to libcore, will pull in... who cares about floats?... slice APIs that have been precompiled to use the target features. These will thrash the xmm register state, which means that a kernel cannot assume that the xmm registers have been preserved, even if it does "only" integer operations. In order to catch this, it will have to manually disable and reenable the vector registers each time, because the only way to protect itself from the Rust libcore will be to make sure SSE instructions trap, which is itself the kind of heavyweight operation you would probably prefer not to do with a context switch.

Or, more likely, they recompile libcore for the kernel anyways, thus reinventing x86_64-unknown-none, which already exists.

[Muon]

Sounds like it's likely some people have lurking bugs because of this. Is there any way to prevent linking against code compiled with a different ABI? Is any metadata about the target embedded in the object files?

[workingjubilee]

And... people keep rediscovering this property, again and again, after thrashing with the compiler and their setups for HOURS. The fact that these combinations of flags are de facto unviable has bitten learned academics who specialize in verification of instruction sets: https://alastairreid.github.io/verifying-vectorized-code2/

Apologies if anyone thinks I should have stated this earlier! I refrained, because I could probably go on like this for what might well be the entire rest of the actual year, until 2024 January 1. Thus, simply stating "no one actually understands this" seemed enough.

[workingjubilee]

Sounds like it's likely some people have lurking bugs because of this. Is there any way to prevent linking against code compiled with a different ABI? Is any metadata about the target embedded in the object files?

The answer to that is a familiar, joyous refrain:
🎵 It depends on the target! 🎵

[RalfJung]

Is there any way to prevent linking against code compiled with a different ABI?

Yes. Make sure that all code built for the same target is ABI-compatible, no matter the target features. Reject any code or target features where we can't make sure this is the case. Linking files with different targets is already "obviously" wrong. (I don't know if we actually prevent it on a technical level though.)

That's exactly what this PR does.

So, looks like we should get this to be decided by some team. But is it t-compiler or t-lang? Or both?

[RalfJung]

Given that stabilizing target features seems to be a t-lang thing, I am going to assume the same applies here.

@rust-lang/lang, we'd like to remove the ability to disable certain target features. The broad goal we are going for here is that code built for the same target is ABI-compatible no matter the target features. This is an easy rule to remember and a principle that it seems reasonable to enforce. This principle is currently violated in several ways (for x86, that's tracked in #116344 and #116558). This PR is one part of achieving that, this pre-RFC is another part, and then one final piece is needed to reject +x87/+sse on x86_64-unknown-none (or to reject float-taking/float-returning-functions) as that would similarly change the ABI.

This particular PR fixes #116344 for our tier 1 targets. The issue is that disabling the "x87" feature on an x86 (32bit or 64bit) target changes the ABI, so linking two object files that were compiled with different target features can lead to UB in entirely safe code. In particular, calling any float-returning function from the standard library when x87 is disabled will break (caller and callee disagree about where the return value should go), and if we want to fix #115567 by returning floats in SSE registers for "Rust" functions, then similar ABI incompatibilities will arise when sse is disabled.

Disabling these target features also does not guarantee code that does not use SSE/x87 registers, since the standard library was built with those registers available. SSE registers are used even in code that doesn't involve any floats. So if the goal is to e.g. not save SSE registers in a context switch, then disabling the SSE target feature does not suffice, you also need to rebuild the standard library.

In other words, -sse/-x87 is (a) unsuited to generate code that doesn't use SSE/x87 registers, and (b) a major footgun due to ABI incompatibilities. We have the x86_64-unknown-none target for cases like that, which guarantees that all code, including the standard library, is built without use of SSE/x87 registers. (Many other architectures have similar softfloat targets. This PR here is for x86 only, but we should do a thorough audit of all stabilized target features and keep this concern in mind when stabilizing target features in the future.)

EDIT: I have created a design meeting proposal for the wider problem space here, in case you think this needs more fundamental discussion.

[RalfJung]

Implementation-wise, will this PR also reject enabling +softfloat? That also seems necessary, since enabling that feature also seems to change the float ABI.

[RalfJung]

Implementation-wise, will this PR also reject enabling +softfloat? That also seems necessary, since enabling that feature also seems to change the float ABI.

I think we'll need a list of forbidden features for each target (complementing the list of required features), then we can use that to reject +softfloat on hardfloat targets. (And we can later use that to reject +x87 on softfloat targets.)

[RalfJung]

Hm, it still seems fairly fragile that each x86 target has to list those required/forbidden features though. Maybe we should use a different approach where features in the features list are marked based on whether they are "float-ABI-affecting" and then we reject changing those features? (We also have SIMD-ABI-affecting features but there's a different proposal for how to handle them so that we can still support toggling those features.)

The tricky part with this approach is figuring out whether -Ctarget-cpu changed any of the "bad" features... (or whether they are implied by other explicitly specified features). We might have to just compute the feature set for the baseline target and the feature set the user configured and then compare that the two are identical for all float-ABI-affecting features.

[RalfJung]

So this will silently ignore -Ctarget-features=-x87? That doesn't seem great, we shouldn't just ignore that. We should hard error or at least warn about ignoring this flag.

I think we'll want to compare that in the final feature set, certain features (for x86: x87, soft-float; for other targets we need to still determine this list) have the same state as without applying the user config. This needs to take into account -Ctarget-cpu and features implying other features.

[GuentherVIII]

Comments in rustc led me to believe that this minimal change could unblock #115919, but the code implementing the problematic design is almost entirely in LLVM. Target, target-cpu and target-feature together specify the CPU features the generated code will use, and the ABI follows from that. rustc could work around that to try to make the ABI follow only from target alone, but that requires at minimum a complete list of all ABI-affecting features in all targets, and maybe more. A clean solution would have the complete logic in rustc instead.

That's a bunch of code for a small reduction in unsoundness. To trigger the problem, you have to link an object compiled with different target-feature-flags into the program. I think "link an object" is the important part, not "different target-feature-flags". The object could also have been compiled with a different "--target" flag, or a different rustc version, or a pascal compiler, or it could have been downloaded from the internet. Making linking objects into the program safer would be nice in general, but there are far too many ways in which problematic objects can be produced.

That said, giving an error message for some known-problematic arguments to -C target-feature might be an efficient way to improve the situation?

[workingjubilee]

@GuentherVIII I want to

• acknowledge you are correct that the problem is linkage.
  • The complication is that we ship a precompiled libstd, including libcore, and link that in to every single Rust program built for a tier 1 or tier 2 target, unless exceptional measures (from the perspective of "typical or even fairly advanced Rust compiler user") are taken. So we are creating a lot of possible unsound linkages just on our own.
• Disagree on the idea that the problem is necessarily LLVM. The idea that rustc reasons about target features the way LLVM does is largely illusory, caused by an accidental conflation that we have probably also-accidentally disposed of.
  • We have several "canonizing" maps of "Rust target feature" to "LLVM target feature" already, for various reasons. This is partly because LLVM sees fit to change what a target feature means at their leisure, whereas Rust wants to expose stable interfaces, because we have made these features something you can set and unset in source code, thus subject to Rust stability expectations, at least somewhat.
  • We have adopted a framing that means "a target feature is something you can put on a single function" due to accepting and implementing the target_feature / cfg_target_feature / cfg_feature_enabled RFC, which means the (several) target features in LLVM that only work if you annotate an entire LLVM module with them, thus affecting all functions, and often only work if the entire program has them set, are not actually coherent if treated as Rust target features.
  • Thus, we have largely backed ourselves into a corner of accidentally having to redefine what a "target feature" even means, in terms of Rust, instead of being able to simply rely on LLVM to do the work.

And for what we have decided on so far, we can pretty much decide if things are being used correctly in a fairly likely way for most cases and emit errors in the known-problem cases, yes. It's true that it may be a "best effort" in reality, but that's to be expected, and it describes the "99%" case, even amongst the set of "advanced" users.

It's not necessarily about sound or unsound, it's about user experience, so that we aren't sitting smugly on an answer we already have and letting people waste hundreds of hours on trying to resolve cryptic issues.

It also is about the questions already raised re: actually being able to use optimizations in the compiler without exploding things in the face of users. We reserve the right to change the Rust ABI between versions of the compiler without notice, but that means that if setting -Ctarget-feature affects the Rust ABI for a given feature, then the soundness of setting that flag for a given feature and then linking libcore is determined by the Rust compiler version, and subject to change without notice. Thus we really already know an answer to whether it's sound or unsound.

[RalfJung]

That's a bunch of code for a small reduction in unsoundness.

That's describing a lot of rustc. ;)

To trigger the problem, you have to link an object compiled with different target-feature-flags into the program.

As Jubilee said: this happens every time someone uses -Ctarget-feature or -Ctarget-cpu, since libcore and libstd are still built with the default target features.

that requires at minimum a complete list of all ABI-affecting features in all targets

Correct. But that doesn't seem so bad? Such features are fairly rare, in particular if we assume we can exclude repr(simd) types from consideration because they will hopefully use a different mechanism.

We can start with the tier 1 targets, and then slowly work our way through the rest.

[bors]

☔ The latest upstream changes (presumably #117716) made this pull request unmergeable. Please resolve the merge conflicts.

[traviscross]

This question is scheduled for a T-lang design meeting on 2023-12-20:

rust-lang/lang-team#235

[traviscross]

@rustbot labels -I-lang-nominated

We discussed this question in the T-lang design meeting on 2023-12-20. The consensus was generally in favor of fixing this, and we were interested in seeing more work in this direction. While we considered various ways that this could be fixed, we were supportive of finding the most minimal, simplest way to fix this as the first step, assuming that such an approach proves to be feasible. We'll need to see at least the results of a crater run and further analysis to confirm that feasibility.

We appreciate @RalfJung and all the others who have been investing time to push this forward.

[Dylan-DPC]

@GuentherVIII any updates on this? thanks

[Dylan-DPC]

Closing this as inactive. Feel free to reöpen this pr or create a new pr if you get the time to work on this. Thanks