Distinguish between 'null' and missing keys (undefined)

[justend29]

Currently, (de)serializing from/to a struct treats missing keys as 'null' if glz::opts::error_on_missing_keys is false.
However, there are plenty of situations where a null value of a field is a valid value, and is very different from this field not being present.

Other JSON libraries allow searching the underlying maps for the presence of a given key with something like find() or hasMember().
With the understanding that this approach is inoperable when going directly to/from structs, what kind of lift would it be to differentiate null and undefined?

Would a new glaze meta type be a good approach - similar to glz::quoted?
Unlike quoted, which deals exclusively with (de)serializing a value directly, something like glz::undefinable would read/write just like its stored type, but would instead need to affect the read/write of objects instead.

I'm happy to help, too.

[stephenberry]

Glaze doesn't quite treat missing keys as null, it simply doesn't change the state of anything missing. So a std::optional would remain its default value, which might be null, or could be something else as defined in your class.

So you could use something like std::numeric_limits<int>::max() in your optional to be able to determine if the user set its value to null. But, that would have to assume that std::numeric_limits<int>::max() wouldn't be input by the user.

It seems like you want to be able to determine if a certain key was provided in the input JSON where a valid value for that key could be null. We could make a type named glz::undefinable:

glz::undefinable<int> x{};

An undefinable would behave much like a std::optional, but with the added state of being able to be undefined and not simply null. But, this double handling of boolean logic seems like it will be confusing.

I think a better approach might be for the library to allow the user to add introspection variables to their class:

struct my_struct
{
   std::optional<int> x{};
   bool has_x{};
};

template <>
struct glz::meta<my_struct>
{
   using T = my_struct;
   static constexpr auto value = glz::object("x", glz::has_key<&T::x, &T::has_x>());
};

This would set the has_x boolean to true if x were seen in the input.

This is more typing, but it allows you to use your original variables without needing to use a glz::undefinable structure.

@justend29
What are your thoughts?

[justend29]

This is interesting. I've also been thinking about alternatives on the side.

Options

There seem to be various options for this double handling of boolean logic (realistically, tri-state), each with varying capabilities and coherence with the current glaze. For all of these options, the default is undefined and glaze inserts 'null' when present as a value.

1. could be done with nested "nullable" types:

std::optional<std::optional<bool>> undefinable_or_null; 
//   ^ undefined   ^ null 

std::unique_ptr<std::optional<bool>> other_undefinable_or_null;
//   ^ undefined     ^ null

3. glaze type to distinguish the undefinable value

glz::undefineable<bool> my_undefineable;                        // can be present or not, just not 'null'
glz::undefineable<std::unique_ptr<bool>>  undefinable_or_null;  // present as bool or null, or not at all 

4. glaze type to handle the tri-state, including the null

glz::undefineable<bool> my_undefinable;    // present as bool or null, or not at all 
assert(my_undefinable.is_undefined() || my_undefinable.is_null() == !my_undefinable)

5. a separate flag in the client structure, as you suggest above with glz::has_key. This is basically "teaching" glaze how to represent a value as undefined, but exclusively with an added field.

Extending to null

What I was considering on the side, after reading through more glaze code since posting this, is allowing users to not only "teach" glaze how to represent undefinable value, but also null values. I'll explain why this thought came to me, and you can tell me if this makes sense or not.

glaze currently handles null for very specific, hard-coded types. I think this is because there's no single 'null' value to share among these types; they each represent 'null' uniquely. A snippet from glz::from_json<nullable_t>::op:

if constexpr (is_specialization_v<T, std::optional>)
   value = std::make_optional<typename T::value_type>();
else if constexpr (is_specialization_v<T, std::unique_ptr>)
   value = std::make_unique<typename T::element_type>();
else if constexpr (is_specialization_v<T, std::shared_ptr>)
   value = std::make_shared<typename T::element_type>();
else if constexpr (constructible<T>) {
   value = meta_construct_v<T>();
}
else {
   ctx.error = error_code::invalid_nullable_read;
   return;
   // Cannot read into unset nullable that is not std::option
}

Suggestion

So, assuming the above isn't insane, I believe what would be the most flexible is:

• offer a mechanism for users to represent what a nullable value is
• offer a mechanism for users to represent what an undefinable value is
• remove the hard-coded overloads for null values, and instead provide specializations of the above "mechanism" for known nullable types

This is the same process that's currently provided with glaze's (de)serialization abilities. A user can specialize meta<> or use the specialization for common types.

Magical Mechanism

So, what would this magical mechanism look like? I haven't thought that far. But, going through the existing meta specializations would probably be best in combination with some concepts.

disclosure: I'm making this code up on the fly in GH - likely invalid. Exposition only

Provided with glaze to generically handle nullable types:

namespace glz {

// could probably consolidate these with some concepts, but for the example
  
template<typename T>
struct meta<std::optional<T>> {
  [[nodiscard]] static std::optional<T> make_null() { return {}; }
  [[nodiscard]] static bool is_null(const std::optional<T> v) { return v; }
  [[nodiscard]] static auto& value(const std::optional<T> v) { return *v; }
}

template<typename T, typename D>
struct meta<std::unique_ptr<T, D>> {
  [[nodiscard]] static std::unique_ptr<T, D> make_null { return {}; }
  [[nodiscard]] static bool is_null(const std::unique_ptr<T, D> v) { return v; }
  [[nodiscard]] static auto& value(const std::unique_ptr<T, D> v) { return *v; }
}
  
}

Then, glaze's nullable_t concept becomes a check for all these members on meta,
and the provided from_json::op() functions go from constexpr branches to:

  value = meta_wrapper_v<T>::make_null();

With this, undefined follows suit - both for values that can be undefined/defined and those that can be undefined/null/defined. Example with some arbitrary type that uses separate flags:

namespace justen {

template<typename T>
struct Custom_with_flag {
  T value{}; 
  bool has_field{false};
  bool is_null{false};
};

}

namespace glz {

template<>
struct meta<justen::Custom_with_flag> {
  [[nodiscard]] static justen::Custom_with_flag make_null() { return {.is_null = true}; }
  [[nodiscard]] static bool is_null(const justen::Custom_with_flag& v) { return v.is_null; }
  
   // could enforce the type's default constructor as `undefined` instead of a `make_undefined()`
  [[nodiscard]] static justen::Custom_with_flag make_undefined() { return {}; }
  [[nodiscard]] static bool is_undefined(const justen::Custom_with_flag& v) { return !v.has_field; }
  
  [[nodiscard]] static auto& value(const std::optional<T> v) { return *v; }
};

}

And, for nested "nullable" types, which are currently unproductive with glaze, glaze could provide:

namespace glz{

template<typename T>
 requires (
   (nullable_t<T> || undefinable_t<T>)        
    && requires { typename T::value_type; }
    && nullable_t<typename T::value_type>
    && std::constrictible_from<T, typename T::value_type>
    && std::is_invocable_r_v<typename T::value_type, decltype(&meta_wrapper_t<typename T::value_type>::value)>)
struct meta<T> {
  using Inner = typename T::value_type;
  [[nodiscard]] static T make_null() { return meta_wrapper_t<Inner>::make_null(); }
  [[nodiscard]] static bool is_null(const T& v) {
    return meta_wrapper_t<Inner>::is_null(meta_wrapper_t<T>::value());
  }
  
  [[nodiscard]] static T make_undefined() {
    if constexpr (undefinable_t<T>) {
        return meta_wrapper_t<T>::make_undefined();
    }
    return meta_wrapper_t<T>::make_null();
  }
  
  [[nodiscard]] static bool is_undefined(const T& v) {
    if constexpr (undefinable_t<T>) {
        return meta_wrapper_t<T>::is_undefined(v);
    }
    return meta_wrapper_t<T>::is_undefined(v);
  }
  
  [[nodiscard]] static auto& value(const T& v) { return meta_wrapper_t<T>::value(); }
};

}

@stephenberry What are your thoughts?

[justend29]

Update: In my last example, T::value_type should just be replaced with the result of T's associated meta specialization, as requiring value_type isn't reasonable, especially when considering unique_ptr/shared_ptr offer element_type.

[justend29]

@stephenberry I'm leisurely tackling this on my fork: here
The branch is currently behind the tip of glaze, and isn't nearly ready for a PR. However, I just want to make it known that I'm not dumping this issue without input. I haven't forgotten!

[stephenberry]

@justend29 Thanks for continuing to work on this issue! If I get some time I'll try to give some feedback, but I also trust that a pull request from you will be well thought out, so just submit one when you're ready.

[stephenberry]

@justend29, I'm moving this issue to discussions because it isn't a pressing issue. Feel free to submit a pull request if you ever get a working implementation (or just merge it in), but no pressure. I'm a bit uncertain as to the best approach, so I probably won't try tackling this until I have an actual use case.