x86: Correctly pass larger structs/types in registers with -Zregparm

[sulix]

Fixes #145694. I think this should be the last compiler fix needed to unblock 32-bit x86 support for Rust-for-linux (Rust-for-Linux/linux#78)

Tracking issue for -Zregparm: #131749

The -Zregparm option for 32-bit x86 modifies the calling convention to pass a number of arguments in registers instead of on the stack. (This is primarily used by the Linux kernel.)

Currently, rustc will only pass primitive integer types in registers, which seems to match the gcc documentation[1], which says that arguments "of integral type" are passed as registers. This also matches the fastcall and vectorcall conventions on windows.

However, it seems that any type — most particularly structs — should be passed as a register if possible. This matches what clang and gcc do, and so avoids an ABI mismatch when linking with C code (which, after all, is the whole point of supporting -Zregparm).

Note that I haven't tested this particularly thoroughly outside the Linux kernel usecase, and in particular haven't tried to implement any 'assembly-llvm' tests, nor test for regressions against fastcall/vectorcall.

[rustbot]

r? @jdonszelmann

rustbot has assigned @jdonszelmann.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[rustbot]

⚠️ Warning ⚠️

• There are issue links (such as #123) in the commit messages of the following commits.
  Please move them to the PR description, to avoid spamming the issues with references to the commit, and so this bot can automatically canonicalize them to avoid issues with subtree.
  • 8118dc6

[bjorn3]

The PassMode::Ignore | PassMode::Indirect { attrs: _, meta_attrs: None, on_stack: _ } => { continue; } above prevents hitting this code for most structs.

[jdonszelmann]

If this fixes that issue, I'd love to see a test for that issue which now passes. At least some amount of testing seems wise. I'm aware this is an unstable option I know little about otherwise, but since there is a specific test failing, I think adding it as a test is the least we can do before merging this fix so we won't forget about this failure in the future.

[jdonszelmann]

@rustbot author

[rustbot]

Reminder, once the PR becomes ready for a review, use @rustbot ready.