Task Summary
Texera reads CSV, JSONL, Arrow and plain text off disk, but not Parquet, which is the format most tables in a data-science workflow are already stored in. A user holding one has to convert it first, and a conversion to CSV loses the types the file was written with: the column that was an INTEGER comes back as text for the schema to guess at again.
A scan source that reads Parquet closes that. It is the same shape as the CSV and Arrow sources, a ScanSourceOpDesc declaring fileTypeName = "Parquet", and it reads its schema off the file's own footer rather than inferring one, so the types survive. pyarrow is already a declared dependency, and Arrow's source shows how a binary format is read.
The parity harness covers it the moment the format is registered: SourceCategoryRunner maps a declared fileTypeName to an encoder that writes a fixture in it, and Parquet needs one entry added to encoderByFileType. This is the first operator to exercise that branch, the four already there being covered by an encoder that existed.
Not part of #8325: nothing here makes a workflow exportable. It does ship standalone code, so it comes after the trait #8327 introduces, and its fixture needs the harness that issue builds.
Task Type
Task Summary
Texera reads CSV, JSONL, Arrow and plain text off disk, but not Parquet, which is the format most tables in a data-science workflow are already stored in. A user holding one has to convert it first, and a conversion to CSV loses the types the file was written with: the column that was an INTEGER comes back as text for the schema to guess at again.
A scan source that reads Parquet closes that. It is the same shape as the CSV and Arrow sources, a
ScanSourceOpDescdeclaringfileTypeName = "Parquet", and it reads its schema off the file's own footer rather than inferring one, so the types survive.pyarrowis already a declared dependency, and Arrow's source shows how a binary format is read.The parity harness covers it the moment the format is registered:
SourceCategoryRunnermaps a declaredfileTypeNameto an encoder that writes a fixture in it, and Parquet needs one entry added toencoderByFileType. This is the first operator to exercise that branch, the four already there being covered by an encoder that existed.Not part of #8325: nothing here makes a workflow exportable. It does ship standalone code, so it comes after the trait #8327 introduces, and its fixture needs the harness that issue builds.
Task Type