RFC: Structural Records

[Centril]

🖼️ Rendered

📝 Summary

Introduce structural records of the form { foo: 1u8, bar: true } of type { foo: u8, bar: bool } into the language. Another way to understand these sorts of objects is to think of them as "tuples with named fields", "unnamed structs", or "anonymous structs".

💖 Thanks

To @kennytm, @alexreg, @Nemo157, and @tinaun for reviewing the draft version of this RFC.
To @varkor and @pnkfelix for good and helpful discussions.

[Diggsey]

This looks like a super well thought out and comprehensive RFC.

It might be worth clarifying the behaviour of #[derive(...)] with respect to types like:

struct RectangleTidy {
    dimensions: {
        width: u64,
        height: u64,
    },
    color: {
        red: u8,
        green: u8,
        blue: u8,
    },
}

Presumably, there will be no "magic" here, and you will only be able to derive traits which are implemented for the anonymous structs themselves.

Another question is around trait implementations: at the moment, 3rd party crates can provide automatic implementations of their traits for "all" tuples by using macro expansion to implement them for 0..N element tuples. With anonymous structs this will not be possible. Especially with the likely arrival of const generics in the not too distant future, negating the need for macros entirely, anonymous structs will become second class citizens. Is this a problem, and are there any possible solutions to allow implementing traits for all anonymous structs?

[petrochenkov]

fn do_stuff_with((red, green, blue): (u8, u8, u8)) {
    some_stuff(red);
    other_stuff(green);
    yet_more_stuff(blue);
}

[Centril]

Even better, with #2522:

fn do_stuff_with((red: u8, green: u8, blue: u8)) {
    some_stuff(red);
    other_stuff(green);
    yet_more_stuff(blue);
}

However, while if you write it in this way it is clear what each tuple component means, it is not clear at the call site...

let color = (255, 0, 0);
// I can guess that this is (red, green, blue) because that's
// the usual for "color" but it isn't clear in the general case.

[petrochenkov]

This syntax looks very similar to another already accepted RFC, but semantics seems different - #2102.

[Centril]

True; I have to think about if some unification can be done in some way here.

[Centril]

@Diggsey

This looks like a super well thought out and comprehensive RFC.

Thanks!

It might be worth clarifying the behaviour of #[derive(...)] with respect to types like:

struct RectangleTidy {
    dimensions: {
        width: u64,
        height: u64,
    },
    color: {
        red: u8,
        green: u8,
        blue: u8,
    },
}

Presumably, there will be no "magic" here, and you will only be able to derive traits which are implemented for the anonymous structs themselves.

This bit is not currently well specified, but it should be so I will fix that (EDIT: fixed)... I see two different ways to do it:

1. Use the structural records in there as types; this means that #[derive(Default)] would produce:

   impl Default for RectangleTidy {
       fn default() -> Self {
           RectangleTidy {
               dimensions: Default::default(),
               color: Default::default()
           }
       }
   }

   The main benefit of this approach is that it is quite simple to implement. It will also work for all derivable standard library traits because implementations are auto-provided for structural records.

2. Use the structural records in there as syntax; this means that #[derive(Default)] would produce:

   impl Default for RectangleTidy {
       fn default() -> Self {
           RectangleTidy {
               dimensions: {
                  width: Default::default(),
                  height: Default::default(),
               },
               color: {
                   red: Default::default(),
                   green: Default::default(),
                   blue: Default::default(),
               }
           }
       }
   }

   This would need to be done recursively in the macro but it shouldn't be very hard to implement.

   The main benefit of this is flexibility. More types would be derivable.

   However, due to the automatically provided implementations as noted in 1. there is no benefit to this approach for the standard library so I think the sane behaviour is 1.

   That said, a crate like serde can benefit from using the "magical" approach in 2. and we certainly cannot mandate what a procedural macro must do in the language so it will be up to each derive macro to decide what to do.

Another question is around trait implementations: at the moment, 3rd party crates can provide automatic implementations of their traits for "all" tuples by using macro expansion to implement them for 0..N element tuples.

Yup. Usually up to 12, emulating the way the standard library does this.

Especially with the likely arrival of const generics in the not too distant future,
negating the need for macros entirely, anonymous structs will become second
class citizens.

Nit: to not have to use macros for implementing traits for tuples you need variadic generics, not const generics. :)

Is this a problem, and are there any possible solutions to allow implementing traits for all anonymous structs?

I think it might be possible technically; I've written down some thoughts about it in the RFC. However, the changes needed to make it possible might not be what folks want.

However, not solving the issue might also be good; by not solving the issue you add a certain pressure to gradually move towards nominal typing once there is enough operations and structure that you want on the type.

[petrochenkov]

That's... quite a bit of magic.

[Centril]

I agree; this is noted in the drawbacks.

[Centril]

Ideas for possible magic reduction discussed below in #2584 (comment).

[petrochenkov]

With macros fields can have names with same textual representation, but different hygienic contexts.

[Centril]

Does that change anything wrt. the implementations provided tho? I don't see any problems with hygiene intuitively, but maybe I can given elaboration?

[petrochenkov]

This is probably relevant to other places where the order of fields needs to be "normalized" as well.

[Centril]

Sure. :) But I'm not sure if you are pointing out a problem or just noting...
Any sorting / "normalization" is done post expansion.

[petrochenkov]

It's not clear to me how to sort hygienic contexts in stable order, especially in cross-crate scenarios. That probably can be figured out somehow though.

[petrochenkov]

I don't think types can start with { syntactically.
There are multiple already existing ambiguities where parser assumes that types cannot start with {, and upcoming const generics introduce one more big ambiguity ({} is used to disambiguate in favor of const generic arguments).
#2102 uses struct { field: Type, ... } to solve this (and also to discern between struct { ... } and union { ... }).

[petrochenkov]

I'm not sure about expressions/patterns.
Unlimited lookahead may also be required, need to check more carefully.

[Centril]

There are multiple already existing ambiguities where parser assumes that types cannot start with {

Such as? (examples please...)

Const generics allow literals and variables as expressions but anything else needs to be in { ... }.
This is not ambiguous with structural records (because the one-field-record requires a comma...) but requires lookahead.

Unlimited lookahead may also be required, need to check more carefully.

Scroll down ;) It's discussed in the sub-section "Backtracking".

[petrochenkov]

It's discussed in the sub-section "Backtracking".

It's a huge drawback.

[petrochenkov]

Such as? (examples please...)

Nothing that can't be solved by infinite lookahead, but here's "how to determine where where clause ends":

fn f() where PREDICATE1, PREDICATE2, { a: ...

Does { a: ... belong to the function body or to the third predicate?

[Centril]

@H2CO3 oh sure; the parser must handle it, but the perils of backtracking is that it would considerably slow down parsing or produce bad error messages. What I'm saying is that the code paths in the parser that would handle this are pathological, so the slow downs are unlikely.

There's also no ambiguity here at all, but lookahead / backtracking / GLL may be required.

If we ever use the GLL crate in the compiler then we don't even need backtracking... it provides a correct algorithm to parse context-free-grammars and can produce good error messages for these sort of things because it produces a parse forest that you can catch certain patterns on.

[petrochenkov]

The fact that advanced formalisms exist doesn't necessarily mean that we should use them.
I many respects the simpler means the better.
Officially abandoning LL(n) is a separate decision that shouldn't be done in comments to a mostly unrelated RFC.

(That said, for this RFC disambiguation can also be done using cover grammar TYPE_OR_EXPR.
We parse { IDENT: TYPE_OR_EXPR and look at the next token, if it's , then we have a struct record type.)

(Nit: GLL is a backtracking of sorts, just with fancy memoization making is O(n^3) rather than exponential).

[Centril]

The fact that advanced formalisms exist doesn't necessarily mean that we should use them.
I many respects the simpler means the better.

Well there are advantages to both, and we should decide what trade-off is best for Rust.

Officially abandoning LL(n) is a separate decision that shouldn't be done in comments to a mostly unrelated RFC.

Abandoning LL(k) must be made for a reason, and that reason would be some feature or change we would like to make that forces us to do it. I don't think (correct me if that is inaccurate) we ever stated explicitly that LL(k) is a design goal so if so it doesn't need explicit and official abandonment either.

So I think either we abandon LL(k) on this RFC or in the #2544 RFC, or we accept neither and keep LL(k).

(That said, for this RFC disambiguation can also be done using cover grammar TYPE_OR_EXPR.
We parse { IDENT: TYPE_OR_EXPR and look at the next token, if it's , then we have a struct record type.)

Yeah, that's the idea I had also.

(Nit: GLL is a backtracking of sorts, just with fancy memoization making is O(n^3) rather than exponential).

Fair enough ;)

[Laaas]

Even if you disregard the fact that this syntax would necessitate backtracking, it is confusing for readers IMO. Why not require a struct keyword or something?

[Centril]

@Laaas Because it is inconsistent with Tuples/Tuple-structs and also less ergonomic.

I don't think it's confusing for readers, the one-field case is clearly disambiguated by an extra comma and that is consistent with how tuples are treated. In other words, if { x, } is confusing, then so is (x,).

[petrochenkov]

#1595 was closed, so I'd expect this to be rejected.
This saves us from stuff like "checking that there are no gaps in the positional fields" as well.

[Centril]

Yeah probably; I've left an unresolved question about it.

[varkor]

struct A(u8);
let _ = A { 0: 0u32 };

is accepted, so this is currently inconsistent as it stands anyway.

[Centril]

@varkor yeah that was accepted to make macro writing easier (and it totally makes it easier...!)

[petrochenkov]

I think the language should be frozen for large additions like this in general, for a couple of years at least.
This is not a crucial feature and not a completion of another crucial feature, motivation seems really insufficient for a change of this scale.

[scottmcm]

The trait auto-implementation magic worries me, mostly from the point of view of things not the compiler wanting to implement traits on things. Today crates can do a macro-based implementation for the important tuples, and I can easily see a path to variadic generics where the impls could easily support any tuple. But I can imagine serde really wanting to be able to support these, so I wouldn't want to accept them without some sort of plan -- especially as we're getting closer to finally having good stories for our two other structural type families.

I'd also be tempted to block this on changing struct literal syntax to using = -- I really don't like the lookahead requirement here.

Overall, I think I feel that this is cool, but not necessary.

[clarfon]

I'm very concerned with properties of the struct explicitly relying on lexicographic ordering. I don't think that Ord or PartialOrd should be implemented for structural records-- rather, they should be treated as having unordered fields. If someone wishes to enforce Ord or PartialOrd, I feel that they should be required to either name the struct or use a tuple.

I think that defining lexicographic behaviour for Hash and Debug, however, is completely fine. Hash really only needs to uphold that x == y => hash(x) == hash(y), and any ordering (as long as it's consistent) will satisfy this. Additionally, lexicographic ordering makes the most sense for debugging.

But as a user, I would be surprised to find the below code fail:

let first = { foo: 1, bar: 2 };
let second = { foo: 2, bar: 1};
assert!(first < second);

[Centril]

@clarcharr

I've added an unresolved question to be resolved prior to merging (if we do that...) about whether (Partial)Ord should be provided or not. I'm certainly open to your suggestion.

@scottmcm

I'd also be tempted to block this on changing struct literal syntax to using = -- I really don't like the lookahead requirement here.

I would have loved if we had used = instead... but blocking on that seems like blocking on a thing that no one will agree to changing because it would cause churn in more or less every single Rust program/library ever written... ;)

The trait auto-implementation magic worries me, mostly from the point of view of things not the compiler wanting to implement traits on things.

Sure; I entirely agree with that sentiment; it is by far the biggest drawback.

Today crates can do a macro-based implementation for the important tuples, and I can easily see a path to variadic generics where the impls could easily support any tuple. But I can imagine serde really wanting to be able to support these, so I wouldn't want to accept them without some sort of plan -- especially as we're getting closer to finally having good stories for our two other structural type families.

I noted somewhere in the RFC that with the combination of const generics and variadic generics, you may be able to extend this to structural records as well. For example (please note that this is 100% a sketch), using a temporary syntax due to Yato:

impl
    <*(T: serde::Serialize, const F: meta::Field)>
    // quantify a list of pairs with:
    // - a) type variables T all bound by `serde::Serialize`,
    // - b) a const generic variables F standing in for the field of type meta::Field
    //       where `meta` is the crate we reserved and `Field` is a compile time
    //       reflection / polymorphism mechanism for fields.
    serde::Serialize
for
   { *(F: T) }
   // The structural record type; (Yes, you need to extend the type grammar *somehow*..)
{
    // logic...
}

This idea is inspired by Glasgow Haskell's Symbol.

[bbatha]

The last few comments here make me think more about the fields-in-traits proposal -- with closures you have impl Fn() + Clone; here you could hypothetically have some impl Fields{a:i32} + Clone

This also plays nicely with defaults for optional and default parameters. For instance fn foo(bar: { a: i32 = 3, b: u32 }) can desugar into something like impl Field{b: u32} + DefaultField{a: i32, fn default() -> i32 { 3 }}

[ssokolow]

@scottmcm I never intended to say that they are similar on a technical level.

My intent was to more effectively communicate the value and purpose of having structural records in addition to structs by drawing an analogy to when and why people use (and don't use) closures with only explicit arguments when we already have functions.

(ie. As an alternative or complement to throwing a list of example use cases at people.)

[graydon]

Strongly opposed. The fact that there's an un-filled box in a particular table enumerating aspects of types does not at all warrant filling every such box in. Consistency should not be considered an design criterion that exceeds thrift. Worse: we had this feature before in the past and explicitly removed it to cut things down to a more-manageable size. It should not be added back.

[Centril]

@graydon

The fact that there's an un-filled box in a particular table enumerating aspects of types does not at all warrant filling every such box in. Consistency should not be considered an design criterion that exceeds thrift.

This is not the only motivation of this proposal; indeed, the RFC explicitly states that it is a minor motivation. Perhaps consider the other motivations in the proposal?

Worse: we had this feature before in the past and explicitly removed it to cut things down to a more-manageable size. It should not be added back.

Yes I'm aware; it is addressed here; perhaps address the text there?

EDIT: I won't be able to devote too much time to this RFC since other accepted proposals and work on those take precedence.

[graydon]

It was not addressed. Those 3 reasons patrick listed are the reasons we added nominal structs. The reason we removed structural records was because they were too redundant with nominal structs. They are still too redundant with nominal structs.

[Centril]

It was not addressed. Those 3 reasons patrick listed are the reasons we added nominal structs. The reason we removed structural records was because they were too redundant with nominal structs.

Okay, this was not at all clear from what Patrick said.

They are still too redundant with nominal structs.

Well, let's agree to disagree on that. I think structural records add a large degree of convenience.

[burdges]

I do think structural records give many fun tricks for things like named function arguments. In fact, I believe structural records should be clarified as "the one true way" to do things like named function arguments. Yet, we should delay by years any serious experimentation with such tricks, or structural records, because they all incur significant strangeness cost, and far more important concerns exist.

[Stargateur]

let color = Color { red: 255, green: 0, blue: 0 }; => let color = { red: 255, green: 0, blue: 0 };

You really think it's a "Major: Improving ergonomics, readability, and maintainability" ?!?!?

[ssokolow]

You really think it's a "Major: Improving ergonomics, readability, and maintainability" ?!?!?

If anything, it illustrates how easily a newcomer might mistake such a syntax for an equivalent to Python's dict literals and JavaScript's object literals. (ie. a HashMap literal of some sort, producing an object that allows dynamic addition and removal of fields.)

I'd certainly expect a language to develop map literals before it develops something in the vein of anonymous structs.

[iopq]

@Stargateur where is Color defined? If I use libA::Color it's not compatible with libB::Color so if libA gives me a Color struct I have to convert it to libB::Color before calling libB's function

[burdges]

We noticed a handy trick for replacing structural records:

We permit types declared pub inside fn bodies to be referenced from outside the fn, which naturally prevents them from using type parameters of the fn declaration, so

impl<V> Storage for FooStore {
    pub fn insert<'o,K>(&'o mut self, i: insert::Input<K>) -> insert::Output<'o> {
        #[derive(...)]
        pub Input<K> { k: K, v: V }
        impl From<(K,V)> for Input { ... }
        ...
        pub struct Output<'oo> { old_value: Option<&'oo V> }
        Ouput { old_value }
    }
}

In essence, insert::Input/Output now serve as our "structural records" but we address all concerns like traits directly.

We might support more polymorphism this way than with structural records, but obviously structural records simplify everything sometimes. We've some complex type parameters in insert::<V>::Input<K> or whatever in this case. We'll run into worse concerns if we ever permit defining types inside impls though because at least here insert acts like a submodule. And consistency with ATCs should make this fairly canonical.

[Stargateur]

@Stargateur where is Color defined? If I use libA::Color it's not compatible with libB::Color so if libA gives me a Color struct I have to convert it to libB::Color before calling libB's function

@iopq And what if one library use { r: u8, g: u8, b: u8 } and the other { red: u8, green: u8, blue: u8 } ?

[ssokolow]

@iopq And what if one library use { r: u8, g: u8, b: u8 } and the other { red: u8, green: u8, blue: u8 } ?

From that "optimize for interoperation without conversion" perspective, the best solution is a (u8, u8, u8) tuple, which Rust already supports, since there are no names to disagree and, in the most common case (RGB component ordering), they'll Just Work™. (At the cost of risking Perl/PHP-like "do what I mean" behaviour which compiles but breaks in confusing ways when said assumptions are violated.)

Of course, it's also always possible that you'll find libA using Color { red: u8, green: u8, blue: u8 }, libB using { r: u8, g: u8, b: u8 } and libC using (u8, u8, u8) if all three are supported.

There's no magic bullet. The best you can get is to have a standard crate which just defines types like Color (similar to the http crate) and encourage everyone to depend on it rather than rolling their own types.

[jswrenn]

@Stargateur totally valid concerns, and for those reasons, I don't think we'll see ad hoc record types appear in the public API of many crates. Ad hoc records aren't really a replacement for named records; they're more closely a substitute for Rust's other ad hoc type: tuples.

We generally use tuples for the one-off packings of data that you need when you're doing transformations over that data; e.g., iterator chains, or internal helper functions. Tuples are the right tool to use when giving a name to a type is actually more cognitive load than just bunching the data all together (and there is some cognitive load to naming a type: you need to pick a name, pick a place to define it, manage imports, and look back at the definition). The big problem with tuples is they aren't really self-documenting: you need to always remember what order the components are in. If the components have different types, you'll be saved from mistakes by a compiler error. If they don't, you're hosed. I think of named records as basically just being tuples where the fields aren't named 0, 1, 2, etc.

And that's the role I need them for. In my work I do a lot of transformations over deep, tree-structured data for analysis purposes. The input types are all named structures (thanks serde!) and the output types are all named structures too. What happens in the middle is a complete mess of tuples, though. I do dozens of one-off transformations of data. The cognitive load (and code bloat) of turning all of those tuples into name structures would be totally unmanagable. But: trying to decipher iterator chains that use tuples also sucks! Ad hoc records are going to really improve my quality of life in programming Rust.

[burdges]

I think transformations over deeply structured data sounds like lenses, but structural records would often be simpler.

[iopq]

But Color has a canonical ordering

A 2d rectangle doesn't, but { top-left: (10, 10), width: 190, height: 50 } seems good

Not having to declare it somewhere seems like a win

[ssokolow]

Actually, color does not have a canonical ordering. For historical reasons, you run into BGR as a legacy detail that has persisted in various Windows file formats and APIs and it also crops up when the abstraction leaks on a little-endian system.

For example, I got tripped up when using PyOpenCV because its use of 32-bit integers as an internal representation results in BGR(A) content on a little-endian processor and (A)RGB on a big-endian processor. (I wound up very confused about why my images were coming out wrong until I finally managed to determine that it was my assumptions about the mapping between color components and indices... the exact kind of bug named struct fields prevent.)

Likewise, Google's design document for Skia has a section named "The epic battle of RGBA and BGRA" which talks about having to do conversion between RGB(A) file formats and BGRA memory representations. (Another situation where bugs are likely to creep in when using numeric indices.)

As for 2D rectangles, I've always seen them in one of two representations:

• (x, y, width, height) (the most common and what you see in the constructor for QRect and the struct definitions for GdkRectangle and cairo_rectangle_int_t)
• (x1, y1, x2, y2) (as used in things like the SetRect function in the Win32 API.)

In both cases, taking after the (x, y) ordering people learn when graphing points in high school.

[scottmcm]

We generally use tuples for the one-off packings of data that you need when you're doing transformations over that data; e.g., iterator chains, or internal helper functions.

Note that this is the same thing that C# hit, which I mentioned in #2584 (comment) -- it doesn't necessarily need structural types, just anonymous ones often work great.