Make NPO-optimized enums containing `struct`s with just a single "nullable" field "FFI-safe"

[yyny]

Code

#![allow(dead_code)]
use std::ptr::NonNull;
use std::marker::PhantomData;

#[repr(C)]
struct Pointer<'a, T: ?Sized> {
    pointer: NonNull<T>,
    _phantom: PhantomData<&'a T>,
}

#[repr(C)]
struct Slice<'a, T: ?Sized> {
    pointer: NonNull<T>,
    length: u64,
    _phantom: PhantomData<&'a T>,
}

#[repr(C)]
struct Value<'vm, T: ?Sized> {
    ty: &'vm (),
    value: Option<&'vm T>,
}

unsafe extern "C" {
    // Unambiguous: If `None`, the `Pointer.pointer` will be `null`.
    fn pointer<'a>(_: Option<Pointer<'a, u8>>) -> Option<Pointer<'a, u8>>;
    // Unambiguous: If `None`, the `Slice.pointer` will be `null`.
    fn slice<'a>(_: Option<Slice<'a, u8>>) -> Option<Slice<'a, u8>>;
    // Unambiguous: If `None`, the `Value.type` will be `null`.
    fn value<'a>(_: Option<Value<'a, u8>>) -> Option<Value<'a, u8>>;
}

Current output

warning: `extern` block uses type `Option<Pointer<'_, u8>>`, which is not FFI-safe
  --> src/lib.rs:26:23
   |
26 |     fn pointer<'a>(_: Option<Pointer<'a, u8>>) -> Option<Pointer<'a, u8>>;
   |                       ^^^^^^^^^^^^^^^^^^^^^^^ not FFI-safe
   |
   = help: consider adding a `#[repr(C)]`, `#[repr(transparent)]`, or integer `#[repr(...)]` attribute to this enum
   = note: enum has no representation hint
   = note: `#[warn(improper_ctypes)]` on by default

warning: `extern` block uses type `Option<Pointer<'_, u8>>`, which is not FFI-safe
  --> src/lib.rs:26:51
   |
26 |     fn pointer<'a>(_: Option<Pointer<'a, u8>>) -> Option<Pointer<'a, u8>>;
   |                                                   ^^^^^^^^^^^^^^^^^^^^^^^ not FFI-safe
   |
   = help: consider adding a `#[repr(C)]`, `#[repr(transparent)]`, or integer `#[repr(...)]` attribute to this enum
   = note: enum has no representation hint

warning: `extern` block uses type `Option<Slice<'_, u8>>`, which is not FFI-safe
  --> src/lib.rs:28:21
   |
28 |     fn slice<'a>(_: Option<Slice<'a, u8>>) -> Option<Slice<'a, u8>>;
   |                     ^^^^^^^^^^^^^^^^^^^^^ not FFI-safe
   |
   = help: consider adding a `#[repr(C)]`, `#[repr(transparent)]`, or integer `#[repr(...)]` attribute to this enum
   = note: enum has no representation hint

warning: `extern` block uses type `Option<Slice<'_, u8>>`, which is not FFI-safe
  --> src/lib.rs:28:47
   |
28 |     fn slice<'a>(_: Option<Slice<'a, u8>>) -> Option<Slice<'a, u8>>;
   |                                               ^^^^^^^^^^^^^^^^^^^^^ not FFI-safe
   |
   = help: consider adding a `#[repr(C)]`, `#[repr(transparent)]`, or integer `#[repr(...)]` attribute to this enum
   = note: enum has no representation hint

warning: `extern` block uses type `Option<Value<'_, u8>>`, which is not FFI-safe
  --> src/lib.rs:30:21
   |
30 |     fn value<'a>(_: Option<Value<'a, u8>>) -> Option<Value<'a, u8>>;
   |                     ^^^^^^^^^^^^^^^^^^^^^ not FFI-safe
   |
   = help: consider adding a `#[repr(C)]`, `#[repr(transparent)]`, or integer `#[repr(...)]` attribute to this enum
   = note: enum has no representation hint

warning: `extern` block uses type `Option<Value<'_, u8>>`, which is not FFI-safe
  --> src/lib.rs:30:47
   |
30 |     fn value<'a>(_: Option<Value<'a, u8>>) -> Option<Value<'a, u8>>;
   |                                               ^^^^^^^^^^^^^^^^^^^^^ not FFI-safe
   |
   = help: consider adding a `#[repr(C)]`, `#[repr(transparent)]`, or integer `#[repr(...)]` attribute to this enum
   = note: enum has no representation hint

warning: `playground` (lib) generated 6 warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.54s

Desired output

Compiling playground v0.0.1 (/playground)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.59s

Rationale and extra context

All enums with the following properties should be considered "FFI-safe":

• they have the "Null Pointer Optimization" (NPO) applied to them
• the value they contain is "FFI-safe"
• the value they contain has exactly one field that enables NPO, and this field has exactly one "invalid" value

In other words, an NPO-optimized enum is FFI-safe if the "null" value can be unambiguously determined.

arguably, NPO-optimized enums should also be considered FFI-safe when the "null" value can not be unambiguously determined, but that is a rabbit hole that I don't want to get into right now, and probably worthy of an RFC.

Other cases

Adding #[repr(transparent)] to struct Pointer can remove the first few warnings. However, #[repr(transparent)] also changes the calling convention, which may not be desired.

Rust Version

$ rustc --version --verbose
rustc 1.91.1 (ed61e7d7e 2025-11-07)
binary: rustc
commit-hash: ed61e7d7e242494fb7057f2657300d9e77bb4fcb
commit-date: 2025-11-07
host: x86_64-unknown-linux-gnu
release: 1.91.1
LLVM version: 21.1.2

Anything else?

No response

[hanna-kruppe]

The pattern proposed here seems like a risky shortcut to me. If the C side is treating the pointer field as nullable at least some of the time, then it's safer to also define the corresponding Rust field as a nullable pointer (e.g., Option<NonNull<T>>). Otherwise, one has to be hyper-vigilant when translating between C and Rust types -- any mention of the Rust type that's not wrapped in Option<_> becomes instant UB when a value with a null pointer is passed, even if the pointer isn't used. It's already unfortunate that a C function pointer doesn't quite correspond to unsafe extern "C" fn(...) -> _ in Rust because of nullability.

It's also worth noting that in case of additional fields making the field non-null and wrapping the entire struct in Option loses expressiveness if there's other fields. In the Slice example, the length field is "lost" if the pointer is null. That probably doesn't matter much for this specific example, but it means the "trick" is not as widely applicable as it seems on first glance.

[workingjubilee]

@yyny

Adding #[repr(transparent)] to struct Pointer can remove the first few warnings. However, #[repr(transparent)] also changes the calling convention, which may not be desired.

Uh. If you don't want a pointer to be passed as a pointer then I don't know what you want, here.

[workingjubilee]

I agree with hanna but for different reasons, I think: The specific layout of fields you are choosing here is not what we have promised to extend the null pointer layout transformation to. We are allowed to claw it back in this case. This is especially so because you are using repr(C), which means we will try to make the layout C-ABI-compliant. And I don't think we ever want to promise that Option<SomeCStruct> is FFI-safe. We could stop linting, but it would only be because the lint needs to be ripped out and redone anyways.

[workingjubilee]

If we did extend the guarantees, it would only affect Pointer, because we can't guarantee we can reason about multiple-field structs for the null-pointer layout transformation.

@yyny Basically, for 2 of the 3 cases you present, those aren't promises we've even inferrably made. For 1 of the 3 cases, we've made that promise specifically for repr(transparent), which you reject here.

You are suggesting the lint stop triggering, but here it is actually correct: you are interpreting a current condition as a promise. It isn't.

[yyny]

Uh. If you don't want a pointer to be passed as a pointer then I don't know what you want, here.

To pass the value as a structure containing a pointer. In some (legacy) calling conventions, a void* may be passed in a register, whereas a struct { void *p; } would have be passed on the stack.

[yyny]

we can't guarantee we can reason about multiple-field structs for the null-pointer layout transformation.

Why not? The compiler can and does already optimize such structs without any problems. And even if there are cases where the compiler couldn't reason about this, then I think it's fine to keep warning for them.

those aren't promises we've even inferrably made

I don't consider this an argument, because there is no Rust specification yet, Rust makes no promises yet, and especially not about ABI. I can't find a normative source about the FFI layout of NPO-optimized pointers either, for example.

Even then, the fact that there is no promise implies that the layout is currently undefined, which would make it backwards-compatible to start defining a layout, or at least to stop warning about it for now (or even yet, create a new warning specifically for unambiguous NPO-optimized enums, so you could disable them but keep the useful ones).

And regardless of all that, this argument is circular, the issue is precisely that Rust doesn't appear to make this promise, when there is no reason why it couldn't (as far as I would argue, at least).

We are allowed to claw it back in this case.

This seems reasonable, but I don't see why you would? You could argue that Rust might eventually want to promise for Option<T> to be represented as struct { u8 tag; T data; } or something like that, but I don't think this argument is very strong, because Option<SomeTransparentNPOAbleStruct> is already supposed to be promised to have another representation, so either way, you would need a way to deal with multiple layouts. or that reason, I think something like #[repr(C)] enum TaggedOption<T> { ... } makes more sense for promising those kinds of layouts rather than trying to envision a scenario where the NPO-optimization of non-#[repr(C)] enums containing #[repr(C)] types that is currently already happening would have to be clawed back.

[yyny]

any mention of the Rust type that's not wrapped in Option<_> becomes instant UB when a value with a null pointer is passed

This is no different from the current behaviour. And even if you create two different types (e.g. Slice<T> and OptionalSlice<T>), you still have the same problem.

It's also worth noting that in case of additional fields making the field non-null and wrapping the entire struct in Option loses expressiveness if there's other fields

Yes, that is precisely the point. Option<Slice<T>> specifies clearly that the remaining fields are meaningless when the nullable field is null, and enforces this at the type level, unlike OptionalSlice<T>, which still allows you to access the (invalid) fields in the null case, still does Clone and Drop, etc.

As a more concrete example, in my Value case, when value is non-null, ty is the type of that value, should be reference counted, and can be used to safely cast &Value to another struct (e.g. ArrayValue), but when value is null, ty is a garbage value that must not be garbage collected, and casting &Value to another struct in this case becomes undefined behaviour.

it means the "trick" is not as widely applicable as it seems on first glance.

Well, of course this is a niche trick just like NPO in general, however, I have several other instances of the

#[repr(C)]
struct Value<'vm, T: ?Sized> {
    ty: &'vm (),
    value: Option<&'vm T>,
}

pattern that store additional metadata. Array lengths, field names, virtual tables, etc.

All containing dummy values of course that, in the null case, will contain garbage that does not follow the normal guarantees made by Value.

And all warning about FFI-safety when I use Option<Value> instead of OptionValue, despite me much preferring the first one for safety reasons.

If the C side is treating the pointer field as nullable at least some of the time, then it's safer to also define the corresponding Rust field as a nullable pointer (e.g., Option<NonNull>). Otherwise, one has to be hyper-vigilant when translating between C and Rust types

Agreed. However, in my opinion, it is only Rust's job to define the FFI layout, not to enforce it. I want to define that FFI arguments/results are non-nullable in some cases, and nullable in others. Option<Slice> is safer and more ergonomic for this than OptionalSlice.