feat(biome_deserialize_derive): adding a rest attribute

[NicholasLYang]

Summary

We want the ability to put any unknown fields into a catch-all field, like serde's flatten attribute. This allows for parsing a package.json file while preserving the unknown fields.

Discussion here

This is a first draft, I'd love some guidance on the correct naming/API. Perhaps we could add a new option to the unknown_fields container attribute?

Test Plan

Would love some guidance here!

[Conaclos]

This looks like a good first shot :)
We also need to update the derive macro documentation.

Have you tested this change with your use case?

[NicholasLYang]

To work with my use case, we need to add a catch all type that can implement Deserializable. I was thinking either AnyJsonValue or serde_json::Value. Is there an easy way to construct AnyJsonValue?

[Conaclos]

To work with my use case, we need to add a catch all type that can implement Deserializable. I was thinking either AnyJsonValue or serde_json::Value. Is there an easy way to construct AnyJsonValue?

Sorry I completely miss your reply.
I took some time today to think about that. I think it is not a good idea of relying on AnyJsonValue or serde_json::Value for two main reasons:

1. This will leak implementations details. biome_eeserialize was designed as a way of abstracting over any text format (JSON, YAML, ...). By using one of these types we will break this.
2. It is currently not possible to convert AnyJsonValue into serde_json::Value or the opposite. Of course, we could implement the conversion, however I am unsure of the difficulty of doing this and this requires more work.

I think a better solution could be of keeping the JSON tree around and to update the tree based on the changes made on the deserialized data. This could look like:

JSON text  --parse--> JSON tree  --deserialize--> (JSON tree, structs) --update structs--> (JSON tree, structs) --update tree--> JSON tree  --print-->  JSON text

I think, for now, this represents too much work.
Thus, I thought about another approach:

We could keep the idea of the flatten attribute and provide generic structures to represent any data. We could have this kind of structure:

enum AnyValue {
  Null,
  Bool(bool),
  Number(TextNumber),
  Str(Text),
  Array(Vec<GenericValue>),
  Map(HshMap<Text, GenericValue>),
}

The disadvantage of this approach is that we deserialize data we don't care about. However, this should be enough to cover your use case for now. What do you think?

[NicholasLYang]

Hmm, I was thinking it's okay to implement Deserializable on serde_json::Value, since that's what serde allows as well, even though serde also abstracts over the specific format. While the type is JSON specific, there's nothing that constrains the format that it's being deserialized from. Like if I implement Deserialize on a PackageJson type, obviously it's meant to be deserialized from JSON, but I could deserialize it from YAML too, even though that wouldn't make much sense.

Ultimately though it's up to you! I have a PR that implements Deserializable on serde_json::Value. But if you wish to make a more generic value I can switch it to that.

I think a better solution could be of keeping the JSON tree around and to update the tree based on the changes made on the deserialized data.

We actually do have a limited version of the in place tree modification in the turborepo codebase. And ofc that would be doable with rowan/immutable trees, but agreed that this is out of scope.

[Conaclos]

Ultimately though it's up to you! I have a PR that implements Deserializable on serde_json::Value. But if you wish to make a more generic value I can switch it to that.

serde_json::Value seems "generic". Thus, I think it is fine. Feel free to submit your other PR.

[NicholasLYang]

@Conaclos, any more feedback? I wrote up a little documentation on the attribute

[Conaclos]

Looks good to me! Thanks!

[NicholasLYang]

@Conaclos, thanks so much! Do you think it would be possible to do a release for the crates?

[Conaclos]

thanks so much! Do you think it would be possible to do a release for the crates?

Sure. I'll have a look later today.

[Conaclos]

@NicholasLYang

I published biome_deserialize@0.6.0 and biome_deserialize_macros@0.6.0.