Open Variant Type for semi-structured data

[wjones127]

I've been starting to experiment with implementing the Open Variant Type ¹ in Rust / DataFusion. There is a specification and Java library for this, and Spark will release this type in 4.0. There are also plans to integrate this into table formats such as Delta Lake ² and Iceberg ³. This would be a high-performance data type for semi-structured data, designed for better OLAP performance than JSON or BSON (discussed in #7845). I've discussed a little bit in the Arrow repo about it's potential as an Arrow extension type ⁴.

I'm working on creating an extension similar to datafusion-functions-json. If we could create a new repo datafusion-functions-variant, I'd be happy to develop that in the open.

Footnotes

1. https://github.com/apache/spark/tree/master/common/variant ↩

2. https://www.databricks.com/blog/introducing-open-variant-data-type-delta-lake-and-apache-spark ↩

3. https://lists.apache.org/thread/xnyo1k66dxh0ffpg7j9f04xgos0kwc34 ↩

4. https://github.com/apache/arrow/issues/42069 ↩

[alamb]

@samuelcolvin and @adriangb and I were speaking about datafusion-functions-json just today. They may be interested too

I have created https://github.com/datafusion-contrib/datafusion-functions-variant and added you as an admin 🚀

[samuelcolvin]

Interesting, this is exactly what we need.