Parent Issue
Part of #124 (support partitioned table)
Depends on #128 (fix TableScan partition paths)
Background
TableRead::to_arrow() explicitly rejects partitioned tables:
// read_builder.rs:88-95
if !self.table.schema.partition_keys().is_empty() {
return Err(Error::Unsupported {
message: format!(
"Reading partitioned tables is not yet supported. Partition keys: {:?}",
self.table.schema.partition_keys()
),
});
}
Once TableScan generates correct partition paths (#128), the ArrowReader will be able to read data files under partition directories without additional changes (since it reads from the bucket_path in DataSplit).
What needs to be done
- Remove the partition key check in
TableRead::to_arrow() (lines 88-95 of read_builder.rs)
- Verify ArrowReader works with the corrected paths — it reads
{bucket_path}/{file_name}, so as long as bucket_path is correct, no ArrowReader changes should be needed
Affected files
crates/paimon/src/table/read_builder.rs — TableRead::to_arrow() method
Parent Issue
Part of #124 (support partitioned table)
Depends on #128 (fix TableScan partition paths)
Background
TableRead::to_arrow()explicitly rejects partitioned tables:Once TableScan generates correct partition paths (#128), the ArrowReader will be able to read data files under partition directories without additional changes (since it reads from the
bucket_pathin DataSplit).What needs to be done
TableRead::to_arrow()(lines 88-95 ofread_builder.rs){bucket_path}/{file_name}, so as long asbucket_pathis correct, no ArrowReader changes should be neededAffected files
crates/paimon/src/table/read_builder.rs—TableRead::to_arrow()method