Presumably when this returns false for any reason, we fallback to the default cache driver, that should be made clear in the docs if it isn't already

e.g. the doc says supports all Spark SQL data types but this implementation would seem to falsify that claim ;)

Well, I guess the claim may be true, and the fallthrough at the end might be defensive... In which case maybe we'd want to log a surprisingly unsupported type :)

Good catch -- "supports all Spark SQL data types" did overstate it, and the type list was also incomplete (it omitted Time, intervals, Geometry/Geography, Variant, Null, and UDTs, which are all supported). Fixed the doc to list the actually-supported set and to describe the unsupported-type behavior.

One clarification on the mechanism, re: your fallback question: isSupportedByArrow here only gates supportsColumnarInput, which the cache framework uses to choose the columnar-vs-row input path into this same serializer -- it isn't a fallback to the default cache driver (there's no such per-type fallback). A truly unsupported type isn't silently dropped either: toArrowSchema throws UNSUPPORTED_DATATYPE when the cache is materialized. The docs now state this explicitly.

cc @MaxGekk to take a look ^^

[P2] Please reconcile this capability whitelist with current master before merging. master now maps and reads/writes TimestampNTZNanosType and TimestampLTZNanosType through Arrow, but this predicate still rejects them. The serializer then takes the row path, falls through to ObjectColumnStats (whose ColumnType has no nanos-timestamp case), and fails materialization with UNSUPPORTED_DATATYPE; row output also lacks a typed reader or fallback. The synthetic merge is conflict-free but still has this behavioral incompatibility.

Good catch that master now maps these through Arrow. I looked at the cache paths, though, and the physical value for TimestampNTZNanosType/TimestampLTZNanosType is a TimestampNanosVal, not a plain Long, so the cache's stats collector and fast columnar reader can't treat them as long-backed without a dedicated, precision-aware path -- that's net-new support rather than a predicate tweak. Importantly there's no parity regression: the default cache serializer doesn't support these types either. I verified on current master that df.cache() of a TIMESTAMP_NTZ(9) column throws not support type: TimestampNTZNanosType(9) with the default serializer (its ColumnBuilder has no case for them). The Arrow serializer now rejects them with a clear checkSupportedSchema error at materialization. I'd rather add real support as a focused follow-up than land a half-correct fast path here -- let me know if you'd prefer it gated differently in the meantime.

Follow-up filed: SPARK-57735 / #56842 adds nanosecond-timestamp support to the default in-memory cache (DefaultCachedBatchSerializer), which is the prerequisite -- once that lands, the Arrow cache can route these types through the same statistics machinery rather than rejecting them here.

[P2] The prerequisite cited here has now landed in this exact tree: DefaultCachedBatchSerializer has nanosecond timestamp builders in ColumnBuilder.scala, matching ColumnType cases and TimestampNanosColumnStats. This predicate still rejects both TimestampNTZNanosType and TimestampLTZNanosType, so the Arrow cache is now less capable than the default cache even though the guide says it covers every default-supported type. Since #56842 was the stated gate, please reconcile this before merging by completing Arrow-cache support or narrowing the compatibility claim.

[ 🤖 posted by Codex on behalf of sunchao using the code-review-for-me skill 🤖 ]

[P2] Please do not advertise CalendarIntervalType as fully supported unless the cache representation is lossless over Spark's full value domain. Arrow's IntervalMonthDayNanoWriter multiplies CalendarInterval.microseconds by 1000 with Math.multiplyExact, but Spark permits the full Long range. I reproduced this with new CalendarInterval(0, 0, Long.MaxValue / 1000L + 1L): the default cache returns COUNT=1, while the Arrow cache aborts materialization with ArithmeticException: long overflow. Please use a lossless representation or reject and document this type rather than claiming parity with the default serializer.

Fixed with a clear diagnostic. Caching a CalendarInterval whose microseconds exceed +/-(Long.MaxValue / 1000) now throws an explanatory error (naming the type and the nanosecond-conversion limit) instead of an opaque ArithmeticException: long overflow. The check is installed only when the schema actually contains a CalendarInterval column, so there is no per-row cost for other schemas. Arrow's IntervalMonthDayNano is nanosecond-based and cannot losslessly hold the full Long microsecond domain, so I documented the value-range limit rather than changing the shared Arrow writer; the default serializer's lack of this restriction is noted in the docs.

[P2] The diagnostic fix still covers only top-level intervals. hasCalendarInterval checks attr.dataType == CalendarIntervalType, while isSupportedByArrow accepts CalendarInterval recursively inside arrays, structs, maps, and UDTs, and their recursive Arrow writers still reach Math.multiplyExact(microseconds, 1000L). For example, an array<interval> containing Long.MaxValue / 1000 + 1 still escapes withIntervalOverflowTranslation and fails with raw ArithmeticException: long overflow, contrary to the guide and this reply. Please recurse through complex types and UserDefinedType.sqlType, with nested overflow coverage.

[ 🤖 posted by Codex on behalf of sunchao using the code-review-for-me skill 🤖 ]

[P2] This capability check accepts any UDT whose sqlType is supported, but the cache statistics path does not unwrap UDTs. A valid UDT backed by VariantType passes here, then createColumnStats constructs ObjectColumnStats; ColumnType(udt.sqlType) has no VariantType case and throws UNSUPPORTED_DATATYPE during materialization. The default cache serializer does unwrap the same UDT and supports it, so this also contradicts the documented claim that Arrow covers every default-supported type. Please unwrap UDTs when selecting statistics collectors, or narrow this predicate and the documentation.

Fixed. createColumnStats now unwraps UDTs (case udt: UserDefinedType[_] => createColumnStats(udt.sqlType)), so a Variant- or Geometry-backed UDT gets the right collector instead of falling through to ObjectColumnStats and throwing UNSUPPORTED_DATATYPE. This keeps the capability check and the statistics path in agreement.