Support flattening `Option<T>` into the `UnionSchema` of `T`

[Kriskras99]

Original issues #449 and #458 by @PookieBuns

Currently the SchemaAware(De)Serializer implementations only support one representation of Option<T>: ["null", T::get_schema()] (order of null and T does not matter). This means that the following schema:

[
    "null",
    "int",
    "string"
]

can only map to the following enum:

enum Example {
    Null,
    Int(i32),
    String(String),
}

It is requested that the following Option would also match this schema:

enum Example {
    Int(i32),
    String(String),
}
Option<Example>

This would require the following changes to the serializer:

1. In serialize_some and serialize_none: Only require that the current schema is a Schema::Union with a Schema::Null variant
2. In serialize_some there are two cases to keep in mind:

• If there are only two variants, serialize using the current implementation.
• If there are more variants, serialize T using the current union schema but provide a flag to the serializer that the null variant cannot be used. It would be preferable to modify the union schema/create a new union schema so that the null variant does not exist, but as we need a reference to the schema that would mean using a Cow<Schema> everywhere.

3. In get_resolved_union_variant the logic needs to be modified to correct the index:

• If the index is before the null variant, it can be used as is
• If the index is at or after the null variant, the index needs to be incremented by one

4. UnionSerializer would also need to be changed to ignore the null variant when the flag is set

This would require the following changes to the deserializer:

1. In deserialize_option:

• Only require that the current schema is a Schema::Union with a Schema::Null variant
• If a null is read, then the current logic can be used
• Otherwise, store the index that was read and visit_some with the current deserializer

2. In with_union reading the index must be skipped if the index is already set.
3. In deserialize_enum the UnionEnumDeserializer must be provided with the stored index
4. In UnionEnumDeserializer::variant_seed the stored index must be used if available

This would require the AvroSchemaComponent implementation for Option<T> to only panic if T::get_schema is a union with a Schema::Null variant, otherwise modify that schema to include Schema::Null.

There is probably more changes needed, but this is what I can think of right now.

[PookieBuns]

Given that now we have access to the schema for schema aware serialization and deserialization, do you think we should still use the variant index with bumping / decrementing for se / deserialization, or would you be open to using names for variant matching? i.e. variant_name to find the serialization branch, and using IdentifierDeserializer::string for visiting identifier? For primitives, it would be Int(i32), Long(i64) etc... Note that this would be breaking but it seems like #424 is already primed with a large set of breaking changes so if we do want to adopt this it could be a good time.

[Kriskras99]

Your suggestion unfortunately breaks as much as it fixes. With your suggestion an enum variant with a primitive type must be named like the primitive type (empty variant must be named Null, variant with i32 must be named Int). The suggestion I made, does not require that.

If you have trouble keeping your schemas in sync with your types, I do suggest using the derive or rsgen-avro. I'm updating the derive in my next PR to support enums, and I'm doing rsgen-avro next (or potentially add similar functionality direct to this project). That way you don't have to worry about the order of your enum variants.

[PookieBuns]

Thanks for the clarification. I will update the draft MR to follow your suggestion as the method of resolving Nullable Unions then!