Specify #[repr(transparent)] #1758

nox · 2016-09-26T08:53:25Z

Summary: Extend the existing #[repr] attribute on newtypes with a transparent option
specifying that the type representation is the representation of its only field.
This matters in FFI context where struct Foo(T) might not behave the same
as T.

Cc @eddyb @Manishearth

rendered

[edited to update rendered link]

Amanieu · 2016-09-26T08:54:46Z

text/0000-repr-transparent.md

+# Alternatives
+[alternatives]: #alternatives
+
+None.


One obvious alternative would be to do this automatically for all structs that have just a single field.

It is done automatically for all newtypes, but then you can't pass that to FFI stuff without it becoming a struct on the C side too.

Is there a tooling solution here? Could a sufficiently smart bindgen work around this problem?

could you explain why please?

Because there is no tooling that can be invented to do something that is not possible without this attribute. Either you do struct Foo(f64) and it can't be used in FFI, or you do #[repr(C)] struct Foo(f64) and it won't behave as a bare f64 in FFI.

It seems like something like bindgen could pretty easily create wrappers that unbox transparent structs.

This means an additional type, additional transmutes and/or additional boxing/unboxing functions. That's code bloat to me and I don't see why I would want that. It also doesn't help for structures with UnsafeCell<T> fields.

It seems like something like bindgen could pretty easily create wrappers that unbox transparent structs.

This would mean that the fields can't be private, however. Not sure how much that matters in practice.

Amanieu · 2016-09-26T08:57:06Z

I feel that the RFC could do with a note about transmute, for example specifying that transmute<TransparentNewtype, f64> should be a no-op since the two types have the same representation.

Amanieu · 2016-09-26T08:58:46Z

I can also see this attribute being useful for atomic types such as AtomicUsize, which RFC #1649 specifies should have the same representation as the underlying type. This attribute could also be added to Cell and UnsafeCell for the same reasons.

nox · 2016-09-26T09:00:49Z

This attribute could also be added to Cell and UnsafeCell for the same reasons.

Yep the whole reason I'm writing this RFC is for UnsafeCell<T> to be used in some structures that are passed to Gecko from Servo.

nox · 2016-09-26T09:02:44Z

@Amanieu Added a note about FFI and AtomicUsize use cases.

nagisa

I feel like this RFC is quite underspecified, especially around interactions with some more obscure features.

Overall the idea seems okay to me, though.

nagisa · 2016-09-26T09:22:24Z

text/0000-repr-transparent.md

+the same type as the single field. For example on ARM64, functions returning
+a structure with a single `f64` field return nothing and take a pointer to be
+filled with the return value, whereas functions returning a `f64` return the
+floating-point number directly.


I do not see why the (unspecified) Rust ABI {w,sh}ould share the same behaviour, as opposed to being “transparent” by default.

+, transparent seems to be relevant only for repr(C) stuff, as a tweak to the default C ABI

I think @petrochenkov meant "and"

nagisa · 2016-09-26T09:23:55Z

text/0000-repr-transparent.md

+
+Syntactically, the `repr` meta list will be extended to accept a meta item
+with the name "transparent". This attribute can be placed only on newtypes,
+which means structures (and structure tuples) with a single field.


Why not univariant unions and enums?

I tried to be conservative for now given I don't have a use case for univariant unions and enums in FFI context.

nagisa · 2016-09-26T09:34:43Z

text/0000-repr-transparent.md

+
+This new representation is mostly useful when the structure it is put on must be
+used in FFI context as a wrapper to the underlying type without actually being
+affected by any ABI semantics.


Oh, so you expect to use this when passing stuff into foreign functions… Then a multiple questions arise, for example:

#[repr(transparent)] struct Transp<T>(T); extern { fn banana(x: Transp<SomeRustType>); } // does the non-c-types lint warn? probably yes, but not specified in the RFC extern { fn banana(x: Transp<SomeCType>); } // does the non-c-types lint warn? probably no, but not specified in the RFC #[repr(transparent)] struct TranspPack(PackedTy); // is packed? Probably yes. #[repr(packed,transparent)] struct TransPack(SomeTy); // is packed? No idea. #[repr(transparent, align="128")] struct TransU32(u32); // is valid? If so, is alignment 128 or `align_of::<u32>()`? What happens if native alignment of u32 is greater than alignment specified for the transparent newtype?

This RFC specifies that Transp<SomeRustType> should warn because it says the representation of Transp is whatever the representation of its single field is. So if the single field is not a proper C type, using it in FFI should warn.

But yeah, it should mention that no other repr should be used with it, so no align nor packed.

Mentioned in the RFC that these are illegal.

nox · 2016-09-26T10:16:03Z

I feel like this RFC is quite underspecified, especially around interactions with some more obscure features.

I have no idea how to specify more, feel free to amend or give me directions.

ahicks92 · 2016-09-26T15:49:07Z

I agree with the people saying this should only apply to repr(C). If this is an optimization worth having, then Rust should do it automatically for repr(Rust). This doesn't sound like it would be too terribly hard to implement, though I'm not entirely sure of that offhand. Probably just adding a Newtype variant to Layout (alternatives end up in the territory of adding flags to Univariant, which feels like a hack to me).

I am working with @eddyb on optimizing struct layout, the first PR of which just merged into the compiler. While we could apply repr(transparent) to anything containing one field, it's only useful if that field also has repr(c) on its type. Otherwise, you eliminate the outer layer but may not have the fields in the order you want or expect. This would mostly match the C ABI now, but won't in future if I have my way.

This feels really hacky and temporary, and hacky/temporary-feeling things that we have a need for seem like a good time to revisit the general pattern. There is never any reason to leave repr(transparent) off a newtype that I can see, so perhaps someone should work out the semantics of newtype Foo = Bar instead of just leaving this is an ill-defined pattern that everyone eventually learns on their own.

nox · 2016-09-26T15:51:34Z

I agree with the people saying this should only apply to repr(C).

What? No, f64 is not #[repr(C)] explicitly but it is nonetheless usable in C functions.

If I have a newtype without an explicit representation, I can't pass it to C functions. If I have a newtype with #[repr(C)], I can't use it in place of its inner type in a C function because it may be handled differently on the ABI level.

nox · 2016-09-26T15:53:19Z

There is never any reason to leave repr(transparent) off a newtype that I can see, so perhaps someone should work out the semantics of newtype Foo = Bar instead of just leaving this is an ill-defined pattern that everyone eventually learns on their own.

That doesn't help to use UnsafeCell<T> in a field of a #[repr(C)] structure to signal that the C side of the code may change this field, which is what is needed to make Gecko and Servo work hand in hand in some cases.

durka · 2016-09-26T15:58:35Z

What about extra zero-sized fields? Out of scope?

#[repr(transparent)]
struct Wrapper<T: SomeKindaMarker> {
    wrapped: i32,
    _ghost: PhantomData<T>
}

eddyb · 2016-09-26T16:06:27Z

@camlorn This is exactly the newtype unpacking optimization I had in mind, which requires writing a not-entirely-recursive algorithm to handle cycles. It's pretty nasty stuff and I was waiting for old trans to die.
@durka At least for the Rust default, I would support that, too, since it affects ~all smart pointers.

For everyone else: there's some confusion here: #[repr(C)] is not the requirement of the FFI warning.
It's "you must only use types which are either some primitive with a cross-ABI representation or an user-defined type marked with #[repr(C)]". This RFC expands that rule to recurse into newtypes marked with #[repr(transparent)].

The reason that doing it automatically is not enough, and which @nox should most definitely make it the prime motivation of this RFC is that depending on it leaks implementation details by default.

If I have:

pub struct Opaque(T);

Then why should Opaque be usable with FFI if T is a primitive, or some #[repr(C)] struct?
Why should that detail be exposed and depended upon, if that field is private?

This is why you need #[repr(transparent)], to indicate that ABI properties of the wrapped private type, although not the type itself, are exposed by the public structure.

ahicks92 · 2016-09-26T16:39:51Z

So shouldn't we just do a proper newtype, then say that repr(transparent) is only allowed on newtypes?

If things like UnsafeCell need to be exposed through this method, then we change them to newtype in the standard library and put repr(transparent) on them. This shouldn't cause a backwards compatibility problem. As things stand, the case of UnsafeCell even being a newtype isn't documented by the std docs,

My problem here is that it feels like this takes a step in the direction of enshrining the single-field struct as a newtype forever. This is workable if we then do stuff like accept #1406 in some form, which may have some uses anyway. But it feels like this is one of those decision points: accepting this without a dedicated newtype makes it more likely that we never will.

Also, unless it's already named, I hereby define FFI-capable type to refer to any type which can be safely used with the C FFI. Run with it or not as you like.

eddyb · 2016-09-26T16:42:43Z

@camlorn But these types with private fields are not newtypes in the classical sense: it's not just a type disambiguator, but the field is opaque modulo the ABI implications of #[repr(transparent)] (if present).

"FFI-capable" sounds good, cc @steveklabnik.

nagisa · 2016-09-26T16:52:24Z

@nox

No, f64 is not #[repr(C)] explicitly but it is nonetheless usable in C functions.

f64 et al are POD and none of the repr stuff really applies to it.

@eddyb

leaks implementation details by default

Sure, nobody ought to rely on default behaviour if it involves repr(Rust) stuff, whereas repr(transparent) mandates a concrete behaviour, similarly to how repr(C) does…

@nox

That doesn't help to use UnsafeCell in a field of a #[repr(C)] structure to signal that the C side

…that being said, @camlorn’s critique seems very valid to me. There’s no reason why some alternative approach (e.g. newtype Peach<T> = Banana<T>?) couldn’t be functionally equivalent to #[repr(transparent)] struct Peach<T>(Banana<T>);.

At this point I would like to note that adding (in some cases) and removing (in all cases) repr(transparent) would become a breaking change. However, it being a plain attribute (OTOH, repr(C) shares the same problem) does not really signal that weight.

Interestingly, AFAIR rustdoc does not in any way make note of any representational particularities. These are part of the API and probably should be prominently presented to everyone interested. Perhaps fixing that would help to add that weight I was speaking of in the previous paragraph.

Bikeshed time. repr(newtype). Yay? Nay? Less to type, is clearer.

arielb1 · 2016-09-26T17:01:20Z

@nagisa

You mean an ABI-breaking change? Isn't that the entire point of the repr attribute?

@eddyb

repr(Rust) is intentionally unspecified (but should be equivalent to repr(transparent) when that is valid). We want to hide that implementation detail.

OTOH, repr(transparent) only means that your type has the same ABI as the wrapped type - it's a specific kind of ABI declaration. IMO it should cause a compilation error if the wrapped type has more than 1 non-zero-sized fields.

@camlorn

The problem is that nobody agrees what a newtype is. Is an Rc a newtyped raw pointer? I can certainly imagine it being a #[repr(transparent)] wrapped pointer, for easier interoperability with C.

eddyb · 2016-09-26T17:05:22Z

@nagisa @arielb1 I agree fully. While guaranteeing pub struct Foo(pub T); as having this behavior is technically possible, it's neither sufficient nor desirable (for the various reasons you've mentioned).

The RFC does need to account for all of these requirements which lead to #[repr(transparent)].

ahicks92 · 2016-09-26T17:35:10Z

@arielb1
I can try to define newtype. Let the capability set of a type T be a set of all the logical actions a program can perform on T. A newtype U of some type T is a type whose capability set is a strict subset of T, plus any additional operations which can be written using only operations from the capability set of T. This may make sense only to me, is dense, and probably needs to be more rigorously defined and/or may have exceptions.

Under this definition, Rc is not a newtype because it also requires operations from the capability set of whatever type you use for the reference counted pointer in addition to the pointer itself.

Nevertheless, two things should be noted. A newtype meeting my definition should be adequate here because any way we do that is going to have to conceptually look like a struct with one field. And because newtypes are a subset, going from newtype Foo = Bar to some struct Foo or whatever isn't a breaking change, save that you have to lose repr(transparent) (which you would have to lose under this proposal anyway, should you suddenly need extra fields).

I don't have a problem with us keeping it as it is and never having newtype, but newtypes need machinery to forward some traits at least in order to be ergonomic. This machinery would have to apply to all structs and tuples without it. So either you gain the complexity of teaching people how to use newtype or the complexity of teaching people how to write good code that uses the trait forwarding machinery. I would personally prefer general-purpose trait forwarding machinery, but I can deal with a lot more complexity than most programmers. Consequently, I would begin by arguing against what I personally want in this case.

You could easily counter by saying "well, okay, trait forwarding will only work on single-field structs". Which is fine as far as it goes. But that's not a general feature, and I will argue for more general features over more specific ones. If we choose to leave newtypes as single-field structs then add newtype trait forwarding, someone has to answer the question of why we couldn't make it general.

All I'm trying to say is that I think it's necessary for someone with the power to make decisions to make a definitive decision as to what kind of newtypes we want longterm before we consider this RFC: single-field structs plus general-case machinery that affects more than newtypes or a dedicated mechanism for newtypes with newtype-specific functionality.

nagisa · 2016-09-26T17:35:30Z

You mean an ABI-breaking change? Isn't that the entire point of the repr attribute?

With #[repr(transparent)] and it inheriting the repr(C)-ness it would become potentially API-breaking too.

nox · 2016-09-26T17:50:00Z

@camlorn Replace every instance of "newtype" in that RFC by "structure with one field". This RFC doesn't need to be blocked by things that are completely unrelated like trait forwarding.

ahicks92 · 2016-09-26T18:16:29Z

@nox
Got into a discussion on IRC about this. The problem is that I wasn't asking what the harm is in allowing repr(transparent) on single-field structs; instead, I was asking something along the lines of why we should allow it on such if it's only useful on newtype. The former question puts this in a much better light. Then possibly needing to allow this with PhantomData came up.

So, anyway, carry on.

Mark-Simulacrum · 2016-09-27T20:48:01Z

Though the two are different in meaning as far as I can tell, it may be worth considering the similarity in naming to nrc's proposal for #[transparent] in #1744. (Just pointing it out, no suggestions).

nrc · 2016-09-27T22:55:16Z

text/0000-repr-transparent.md

+[summary]: #summary
+
+Extend the existing `#[repr]` attribute on newtypes with a `transparent` option
+specifying that the type representation is the representation of its only field.


I find this RFC confusing, I think because I am unclear what you mean by representation. Rust uses interior allocation for structs (which while not guaranteed because of no formal ABI, you can definitely rely on). So in that respect, the in-memory representation does not change at all. But you seem to be proposing changing the how newtypes interact with FFI? It might be better to write the RFC with that focus.

#[repr(C)] can also influence how values are passed to and returned from FFI functions, notably struct Foo(f64) on ARM64.

I think I agree with @nrc here. Unless I'm confused, this has nothing to do with 'repr' but is purely a calling convention issue.

However, if you change the calling convention like this then it is incompatible with FFI. Could you possibly give a more detailed example how you would use this in a FFI case?

nox · 2016-09-28T07:34:30Z

I'll write a FFI example, but calling convention is also affected by representation, cf. the example where f64 and Struct(f64) don't behave the same in C.

nox · 2016-09-28T08:23:39Z

Added the PhantomData unresolved question and a paragraph explaining why it matters, with a f64 wrapper example.

parched · 2016-09-28T12:33:01Z

but calling convention is also affected by representation

or rather, calling convention is affected by the type, f64 and Struct(f64) have the same layout but are different types so (can) have different calling conventions.

Why not just implement std::convert::{From,Into} for your struct type?

rfcbot · 2017-05-07T22:53:45Z

The final comment period is now complete.

RalfJung · 2017-06-24T21:00:57Z

An alternative that I expected to find discussed in the RFC but did not is to declare that in general, (i.e., no matter the repr), if a struct has a single non-ZST field, then it is guaranteed to have the same layout as that field. I would not be surprised if people are actually already relying on that.

In fact, #1789 does. That other RFC relies on Cell<T> having the same representation as T; it also sends a signal that we can actually rely on this kind of interconversion.

EDIT: I see this has been brought up in discussion, but not finally resolved yet. All right.

Amanieu · 2017-06-24T21:53:58Z

I agree with @RalfJung, I think this should be implicit as part of the Rust ABI. This would not apply to #[repr(C)] structs, which will have the layout of a C struct containing a single member. The main difference is that the latter will likely be passed as a function argument in a less efficient way.

sfackler · 2017-06-24T22:00:32Z

@Amanieu the main motivation for this feature is FFI use cases where it is normally very bad to pass a repr(Rust) struct into C. It seems strange for it to be bad except for this one use case. A repr(transparent) allows the programmer to explicitly identify that they are defining a C struct with one field (#[repr(C)]), or just a transparent wrapper for the inner type #[repr(transparent)].

aturon · 2017-07-03T19:09:19Z

Huzzah! This RFC has been merged. Tracking issue.

(Apologies for the delay!)

briansmith · 2017-12-16T20:25:43Z

Sorry, it's been forever since we had that conversation. Maybe I'm overlooking something, but I think #[repr(transparent, align=16)] would be very common. For example:

#[repr(transparent, align=16)]
struct Aes256Key([u8; 32]);

#[repr(transparent, align=16)]
struct AesNonce([u8; 16]);

// Keep this in sync with the C definition in aes.h
#[repr(C)]
struct Aes256Context {
    key: Aes256Key,
    ...
    nonce: AesNonce,
    ...
}

extern aes256Encrypt(context: &mut Aes256Context, ...);

I'm not sure how one would express this otherwise.

briansmith · 2017-12-16T20:28:26Z

(Sorry, the previous comment was intended for another issue.)

arielb1 · 2017-12-16T22:47:14Z

@briansmith

What C declaration is this intended to match? Aka why can't you use

#[repr(C, align=16)]
struct AesNonce([u8; 16]);

briansmith · 2017-12-17T20:39:26Z

What C declaration is this intended to match? Aka why can't you use

#[repr(C, align=16)]
struct AesNonce([u8; 16]);

Because, for example, sizeof(AesNonce) isn't guaranteed to equal sizeof(AesNonce::0) because C allows the ABI to specify extra (even useless and arbitrary) padding at the end. More generally the're not guaranteed to have the same memory layout. See rust-lang/rust#43036 (comment)

tbelaire · 2018-01-24T16:45:34Z

The RFC says that this should be illegal.

#[repr(transparent, align = "128")]
struct BogusAlign(f64);

But this should be legal

#[repr(transparent, align = "128")]
struct OverAligned(f64); // Behaves as a bare f64 with 128 bits alignment.

I think you've made an error somewhere.

willmo · 2018-07-08T17:26:33Z

@tbelaire this RFC was amended to fix that in #2271.

Amanieu reviewed Sep 26, 2016

View reviewed changes

nox force-pushed the repr-transparent branch from 4056178 to a70131c Compare September 26, 2016 09:02

nagisa reviewed Sep 26, 2016

View reviewed changes

nrc added the T-lang Relevant to the language team, which will review and decide on the RFC. label Sep 27, 2016

nrc reviewed Sep 27, 2016

View reviewed changes

nox force-pushed the repr-transparent branch from a70131c to 481d8f5 Compare September 28, 2016 08:23

nox force-pushed the repr-transparent branch from 481d8f5 to 4ad9e69 Compare September 28, 2016 08:27

crumblingstatue mentioned this pull request Jun 8, 2017

system::Bool might be unsound jeremyletang/rust-sfml#163

Closed

Mark-Simulacrum mentioned this pull request Jun 19, 2017

[Question] Can we guarantee that the memory representation of a newtype will be equal to the representation of the type it wraps? rust-lang/rust#32146

Closed

RalfJung mentioned this pull request Jun 24, 2017

RFC - Zero-Sized References #2040

Closed

aturon mentioned this pull request Jul 3, 2017

Tracking issue for RFC 1758: Specify repr(transparent) rust-lang/rust#43036

Closed

3 tasks

RFC 1758 is Specify repr(transparent)

c86cfd3

aturon merged commit 2c49bef into rust-lang:master Jul 3, 2017

cramertj mentioned this pull request Nov 10, 2017

Document pointer cast safety in std::path rust-lang/rust#45910

Closed

nox deleted the repr-transparent branch November 30, 2017 15:22

This was referenced Dec 30, 2017

Make Lisp_Object a proper struct type remacs/remacs#562

Merged

Move to #[repr(transparent)] for Lisp_Object remacs/remacs#568

Closed

cuviper mentioned this pull request Mar 16, 2018

impl IntoIterator for arrays rust-lang/rust#49000

Closed

regexident mentioned this pull request Aug 10, 2018

Added support for #[repr(transparent)] mozilla/cbindgen#185

Merged

Centril added A-data-types RFCs about data-types A-repr #[repr(...)] related proposals & ideas A-machine Proposals relating to Rust's abstract machine. labels Nov 23, 2018

pnkfelix mentioned this pull request Mar 7, 2019

Transparent Unions and Enums #2645

Merged

mheese mentioned this pull request Apr 13, 2020

lib, types: Implement string accessors for some padded fields mheese/rust-pkcs11#26

Merged

Ekleog mentioned this pull request May 7, 2021

Does T and U having the same layout mean that Foo<T> and Foo<U> have the same layout? rust-lang/reference#1019

Open

dtolnay mentioned this pull request Apr 5, 2023

RFC: result_ffi_guarantees #3391

Merged

Specify #[repr(transparent)] #1758

Specify #[repr(transparent)] #1758

Conversation

nox commented Sep 26, 2016 • edited by mbrubeck Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Amanieu commented Sep 26, 2016

Amanieu commented Sep 26, 2016

nox commented Sep 26, 2016

nox commented Sep 26, 2016

nagisa left a comment

Choose a reason for hiding this comment

nagisa Sep 26, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nagisa Sep 26, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nox commented Sep 26, 2016

ahicks92 commented Sep 26, 2016

nox commented Sep 26, 2016 • edited Loading

nox commented Sep 26, 2016

durka commented Sep 26, 2016 • edited Loading

eddyb commented Sep 26, 2016 • edited Loading

ahicks92 commented Sep 26, 2016

eddyb commented Sep 26, 2016

nagisa commented Sep 26, 2016

arielb1 commented Sep 26, 2016 • edited Loading

eddyb commented Sep 26, 2016

ahicks92 commented Sep 26, 2016

nagisa commented Sep 26, 2016

nox commented Sep 26, 2016

ahicks92 commented Sep 26, 2016

Mark-Simulacrum commented Sep 27, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nox commented Sep 28, 2016

nox commented Sep 28, 2016

parched commented Sep 28, 2016 • edited Loading

rfcbot commented May 7, 2017

RalfJung commented Jun 24, 2017 • edited Loading

Amanieu commented Jun 24, 2017

sfackler commented Jun 24, 2017

aturon commented Jul 3, 2017

briansmith commented Dec 16, 2017

briansmith commented Dec 16, 2017

arielb1 commented Dec 16, 2017 • edited Loading

briansmith commented Dec 17, 2017

tbelaire commented Jan 24, 2018 • edited Loading

willmo commented Jul 8, 2018

nox commented Sep 26, 2016 •

edited by mbrubeck

Loading

nagisa Sep 26, 2016 •

edited

Loading

nagisa Sep 26, 2016 •

edited

Loading

nox commented Sep 26, 2016 •

edited

Loading

durka commented Sep 26, 2016 •

edited

Loading

eddyb commented Sep 26, 2016 •

edited

Loading

arielb1 commented Sep 26, 2016 •

edited

Loading

parched commented Sep 28, 2016 •

edited

Loading

RalfJung commented Jun 24, 2017 •

edited

Loading

arielb1 commented Dec 16, 2017 •

edited

Loading

tbelaire commented Jan 24, 2018 •

edited

Loading