Data: Add TCK for Writer builder in FileFormat API#16575
Conversation
| /** Write with engine type T without explicit engineSchema, read with Generic Record */ | ||
| @ParameterizedTest | ||
| @FieldSource("FORMAT_AND_GENERATOR") | ||
| void testDataWriterEngineWriteWithoutEngineSchema( |
There was a problem hiding this comment.
We have fix #15688. testDataWriterEngineWriteWithoutEngineSchema is the same testDataWriterEngineWriteGenericRead. Remove this ut.
There was a problem hiding this comment.
Could we remove it in a different PR pls?
There was a problem hiding this comment.
Revert this change. I will raise a new pr to fix this when this pr is merge.
|
|
||
| @ParameterizedTest | ||
| @FieldSource("FILE_FORMATS") | ||
| void testDataWriterSet(FileFormat fileFormat) throws IOException { |
There was a problem hiding this comment.
For now I can found the same property for all format is comress type, so test it in this part.
Guosmilesmile
force-pushed
the
writebuilder_tck
branch
from
c24a562 to
ae7ae61
Compare
| .isInstanceOf(AlreadyExistsException.class) | ||
| .hasMessageContaining("Already exists"); | ||
|
|
||
| genericRecords = dataGenerator.generateRecords(); |
There was a problem hiding this comment.
I would guess that this is the same records as previously. We might want to generate new records
There was a problem hiding this comment.
Yes, you are right. My original intent was to use different data, but I didn't realize I was using the same seed. This time I'm generating a different number of records to produce different data.
| return dataFile; | ||
| } | ||
|
|
||
| private static Reader newOrcReader(InputFile inputFile, Configuration conf) throws IOException { |
There was a problem hiding this comment.
Can we create an interface for the FileFormats to collect these methods?
So every FF needs to implement these for testing, and BaseFormatModelTests doesn't include file format specific code
There was a problem hiding this comment.
Introduce a new interface FileFormatTestSupport and move file-format-specific logic into its implementations, such as compressionProperties, writeRecordsWithoutFieldIds, and supportsFeature.
Guosmilesmile
force-pushed
the
writebuilder_tck
branch
from
528b613 to
eca95ae
Compare
2 participants
Add TCK coverage for the missing parts of the Writer builder in the File Format API:
withFileEncryptionKey(...) and withAADPrefix(...) are Parquet-only and related to encryption. They will be covered in a follow-up PR.