Search before asking
Paimon version
master, 2788fe596 (2.1-SNAPSHOT).
Compute Engine
Flink and Spark, reading a format table with 'file.format' = 'json' that has a column whose type has no cast from string.
Minimal reproduce step
Read a JSON format table with a MULTISET<STRING> column:
CREATE TABLE t (f0 MULTISET<STRING>) WITH (
'type' = 'format-table',
'file.format' = 'json',
'path' = '...'
);
SELECT * FROM t;
The query fails with a NullPointerException that has no message. JsonFileReader.convertPrimitiveStringToType:
default:
BinaryString binaryString = BinaryString.fromString(str);
CastExecutor cast = CastExecutors.resolve(DataTypes.STRING(), dataType);
return cast.cast(binaryString);
CastExecutors.resolve is documented as returning null when no rule can be resolved, and there is no rule from STRING to MULTISET, VARIANT or BLOB. The result is dereferenced on the next line.
What doesn't meet your expectations?
The format already has a sentence for this case: JsonFileFormat.validateDataType throws UnsupportedOperationException("Unsupported data type for JSON format: X") at create time. Reading the same type should say the same thing rather than producing a stack trace with no message, which sends the reader looking at the data rather than at the column type.
A managed table never gets here, since SchemaValidation rejects those types when the table is created. A format table is created through CatalogUtils.validateCreateTable, which does not call validateDataFields, so the column exists and the failure surfaces on read.
Anything else?
The same shape, dereferencing CastExecutors.resolve without a null check, also exists in ArrayToStringCastRule, MapToStringCastRule, RowToStringCastRule and InternalRowPartitionComputer. I have not checked whether those are reachable, so I am not proposing to change them here.
Are you willing to submit a PR?
Search before asking
Paimon version
master,
2788fe596(2.1-SNAPSHOT).Compute Engine
Flink and Spark, reading a format table with
'file.format' = 'json'that has a column whose type has no cast from string.Minimal reproduce step
Read a JSON format table with a
MULTISET<STRING>column:The query fails with a
NullPointerExceptionthat has no message.JsonFileReader.convertPrimitiveStringToType:CastExecutors.resolveis documented as returning null when no rule can be resolved, and there is no rule from STRING toMULTISET,VARIANTorBLOB. The result is dereferenced on the next line.What doesn't meet your expectations?
The format already has a sentence for this case:
JsonFileFormat.validateDataTypethrowsUnsupportedOperationException("Unsupported data type for JSON format: X")at create time. Reading the same type should say the same thing rather than producing a stack trace with no message, which sends the reader looking at the data rather than at the column type.A managed table never gets here, since
SchemaValidationrejects those types when the table is created. A format table is created throughCatalogUtils.validateCreateTable, which does not callvalidateDataFields, so the column exists and the failure surfaces on read.Anything else?
The same shape, dereferencing
CastExecutors.resolvewithout a null check, also exists inArrayToStringCastRule,MapToStringCastRule,RowToStringCastRuleandInternalRowPartitionComputer. I have not checked whether those are reachable, so I am not proposing to change them here.Are you willing to submit a PR?