Writing StructType to ClickHouse

[MohsenIT]

I want to write a dataframe with struct type column into clickhouse. But it raises an error:
Can't convert 'StructType([StructField('addon_id', StringType(), True), StructField('app_slug', StringType(), True), StructField('token', StringType(), True), StructField('uri', StringType(), True), StructField('in_browser', BooleanType(), True), StructField('addon_position', StringType(), True)])' datatype from spark to clickhouse.

1. Why struct type does not convert to NestedType/Tuple/Json?
2. How can we create NestedType/Tuple/ Json Column using spark-clickhouse-connector ?

[pan3793]

The current implementation has very limited support for complex data types.

[BentsiLeviav]

ClickHouse Target Type

When mapping Spark’s struct type, we need to preserve its “fixed set of named fields with potentially different types” semantics. ClickHouse offers a few candidates, each with trade-offs:

CH Map Type

A key–value pairs with a single type for keys and a single type for values.
Good for dynamic, schema-less key–value data, but not suitable because all values must share the same type

CH Nested Type

Internally stored as multiple parallel arrays (array per field) - always representing arrays of structs.
Efficient for repeated substructures per row, with schema enforcement but not ideal for a single struct per row (forces array semantics)

CH JSON Type

Binary-encoded JSON with native JSON functions for querying. Based on the Variant type. Flexible, easy ingestion from Spark with or without schema evolution preserves nested structure. No strict schema enforcement (though it is achievable with some implementation).

Recommendation:

IMHO, JSON is the most practical choice.

CC @mzitnik @pan3793 @mshustov

[ShimonSte]

After further consideration and investigation, we've determined that the Arrow struct type is already mapped to the ClickHouse Tuple data type. To maintain consistency, we will map the Spark struct type to the ClickHouse Tuple type as well.

Action Items:

• Map Spark struct type -> ClickHouse Tuple type
• Add tests for default values in both JSON and Arrow writers

CC @mzitnik @pan3793 @mshustov @BentsiLeviav

[mshustov]

Map Spark struct type -> ClickHouse Tuple type

@ShimonSte does it mean that Tuple will be the only supported destination type for now?

[ShimonSte]

does it mean that Tuple will be the only supported destination type for now?

@mshustov Yes, for Spark StructType, we will map to ClickHouse Tuple type as the supported destination type.

For Spark Variant type, we will likely map to either ClickHouse JSON or Variant type (pending discussion with the core team on ingestion support via Arrow/JSON formats).