Alternative metadata syntax/semantics

[mathiasgredal]

I would like to write the metadata directly on the field, instead of a separate struct like this:

struct Person
{
    Field<"name", std::string> name;
    Field<"age", int> age;
    Field<"country", std::string> country;
};

I have made an example on how this syntax could be enabled: https://godbolt.org/z/acheb5Y1v

[stephenberry]

This is a neat concept and I appreciate your effort to put together a working example.
As you've shown its totally doable, and it lends itself to making very clean macros like FIELD(std::string, name);

I'm a bit hesitant of the design though. For one, while we could make these strings entirely static constexpr, I'd be afraid the compiler wouldn't be as efficient with packing and optimizing the struct. It's also annoying to have to call get_value() or use a conversion operator for basic types. It is much cleaner to just call Person.name or &Person.name to get the address. I find it best to use simple structures in coding with basic parameters for simple and clean code, and this approach adds a layer of indirection. I expect you could still see nearly the same performance, but I don't feel like the current glaze metadata approach is much more complicated than this approach, especially if you use the macros.

I've been thinking about creating a simple tool to generate the glaze metadata structs for a given C++ struct or header file. This way users would easily be able to automate their glz::meta generation.

Allowing users to not have to change their structures and still serialize/deserialize without any performance degradation is a huge benefit of the current design. The benefit of your design is that there is a clear connection between the metadata and the variable itself, but I'm not sure that the proximity is worth the added complexity to generally program with the structs. I feel that the API should just be an additional layer that you can remove or include without affecting any of the core programming logic.

Anyway, that's my philosophy, but I would love to hear more thoughts about this approach, pros and cons, and I'm always open to ways to improve the library. I just think this is a problem that should be solved by reflection, so whether that is creating a tool to generate the metadata, using compile builtins like __builtin_dump_struct, or patiently waiting for reflection to be added to C++, I think these are better long term solutions.

Sincerely,
Stephen

[mathiasgredal]

Thanks for your response,

I totally understand if you don't want to implement this in your library, it was just a suggestion.

To address the points you raised:

• I definitely don't want any macros, you could say I'm allergic to them (although I understand they sometimes are a necessary evil). In this case the code I provided for the struct should just work verbatim.
• You don't have to make any get_value() calls, it should just transparently forward the value, when you access a field as shown here: https://godbolt.org/z/TMEW6691x
• I agree that it adds a layer of indirection, but so does the current separate metadata declaration struct. If implemented correctly these names should be shared across all object instances adding minimal overhead.
• I don't think there is any performance degradation with this design, and it should be able to work ontop of the existing system. I imagine the system working like this:
  • When inferring the types it would start by reading from the metadata registry, if it can't find it there then it would use boost::pfr or a custom implementation to read the field names and then add them into the registry.
  • The next time it encounters this type, it is already cached in the registry, adding no additional overhead from that point onward. I don't know if the metadata registry can be expanded at runtime, although I can't imagine that being much more overhead.
• I agree having built-in reflection would be a lot cleaner, but I don't expect WG21 to ever agree on something and c++ reflection has become a bit of a meme at this point. Secondly having a separate tool or compiler plugin is just as bad as macros, and I would prefer that it just works with standard c++, in clang and gcc. I don't think there is anything to lose by adding this now and if we get reflection in c++26 or c++29 then you can just switch to that then.

I might try and add this feature to the library if you are open to it, although I have looked a bit at the metadata stuff and it looks quite complex since I am not that familiar with the latest c++ and constexpr(I am also quite busy with work and school, but I guess you are too). Anyways your library is awesome, and I appreciate the effort you have put into it.

Best regards,
Mathias

[stephenberry]

Mathias,

Thanks for your additional thoughts. I'm also quite allergic to macros, and even when we get reflection users will still have to use metadata for non-default behaviors. So, I am open to this approach, but cautious.

To get this to work well with the architecture, and thus optimal performance, we would need to use structured binding magic to access the struct parameters, decode into a tuple, and then build the glz::object with member variable pointers. This would all happen at compile time, but a limiting factor is that this would only work for a limited number of elements (we could make it large like 64 or 128). Look up struct to tuple conversion to get an idea of what I'm talking about. There are also some quirks to structured bindings, but I think we could work through them.

All the metadata would be stored as static constexpr values, so the struct should be nearly as efficient when defined this way.

The biggest challenge I currently see is how do we get a member variable pointer from a normal variable? We have the class type, but I don't know of a means in C++ of merging the class type with the variable type to produce a member variable pointer at compile time.

The reason member variable pointers are critical is that it allows us to do much more at compile time than normal pointers, because normal pointers are only known at runtime, but member variable pointers are just offsets that are known at compile time. It also allows us to produce much more efficient code by storing limited variants of member variable pointers and building compile time maps to these. So, I wouldn't want to implement this approach without figuring out how to get member variable pointers out at compile time. If you can help figure this out, that would be great. But, I'm not sure it's even possible.

Thanks for the discussion!

[stephenberry]

Okay, after thinking more about this we would require reflection on the name of the variable in order to know the proper offset and thus generate a member variable pointer.

So, we could solve this by adding the member variable pointer to the field: Field<"name", &Person::name> name;. This would work.

How do you feel about this?

[stephenberry]

Oh, but wait, I don't know if this would work because it would reference itself.

[mathiasgredal]

Is this what you mean? https://godbolt.org/z/Ges7b7vn6

[stephenberry]

While this does get a value at an offset, it doesn't provide a member variable pointer. I've realized that it's not possible without reflection.

[mathiasgredal]

I see that it is impossible to have this type of syntax for this library, since you cannot get a member pointer without proper reflection. You could probably write directly to the structured binding reference, with a similar level of optimization, but that wouldn't fit into the glz::object design. So in order to support this syntax, it would require major modifications and add extra complexity to the library, for a very small benefit.

[stephenberry]

Yeah, the design is also gaining more compile time wrappers like glz::quoted, which also work with member variable and function pointers. So, while it would be nice to add metadata in a struct like Field, I don't think it is worth it without carefully considering how it could play well with the current architecture. Thanks for this interesting discussion though. It's helpful to consider alternative approaches.

[stephenberry]

@mathiasgredal

After considering this syntax more, I've come to the conclusion that it isn't a good design for a variety of reasons. These features are all supported by the current approach of glaze, using member variable pointers, but are not supported with the field approach without additional code from the user and/or additional runtime cost.

The field approach has the following issues:

• Has issues/complexities with non-default constructible classes
• Doesn't support third party libraries (would need to be edited)
• Difficult to use references in a struct
• Makes it harder to remove serialization when it isn't wanted
• Makes classes more complex for programmers with additional interfaces for basic types
• Adds additional runtime overhead to mapping pointers (since they are only known at runtime)
• Doesn't allow the structure to be mapped differently to match JSON, whereas in the glaze approach we can remap using lambda functions without overhead

[mathiasgredal]

1. Yes, i was intending this syntax to primarily be used for json specific dataclasses, which could be referenced/passed around by other classes.
2. That use case inherently requires the mapping be located outside the class, which you can use the existing system for
3. Again see point 1
4. True, there are tradeoffs with both of the solutions, by providing them both the developer can make these tradeoffs and pick what best fits their usecase
5. I disagree, the current system is also very complex. I tried to read to code to see how it works but had to give up
6. With structured binding references everything should be able to be done at compile time.
7. See point 4, but that sounds like a cool feature

i just want to reiterate that this is your library and i can totally understand if you don’t want to implement this feature. I think there are positives and negatives to both approaches and it would be best if the developer/user had the options to pick which version they felt best fit their usecase.

Edit: As an example, if you had 30 dataclasses to model some complex json structure, then it would be a lot more maintainable to do with the proposed syntax, since each field is only defined in one place.

[stephenberry]

Thanks for the feedback. I have been seriously considering adding this syntax, because it wouldn't be too hard. But, I also have to consider that giving a codebase more user options often adds confusion, and I want to make sure that the core features of the current approach would also be supported in this approach. I'm cautious, because I don't want to run into a case where users are often posting issues because they can't use this syntax for something that would seem intuitive.

But, I have been considering this carefully, and just because I think it has downsides doesn't mean it isn't worthwhile.

I'll note that concerning point 4 I was talking about the user's classes, not internal classes to glaze. struct S { double x{}; }; is simply cleaner to use than struct S { Field<"x", double> x{}; }; This is especially true when the type is something like std::shared_ptr<double> or something even more complex.

I've started working on experimental support for this as I have a path to making it work cleanly with the rest of glaze. But, I'm not sure when it will be completed.