Description
Flink's CommonExecSink.applyRowtimeTransformation inserts a row-mode StreamRecordTimestampInserter on the sink input chain when the sink table declares a rowtime attribute. This operator reads the rowtime column from each RowData and stamps it onto the StreamRecord on a per-row basis.
When the downstream sink is offloaded to Velox via Gluten-Flink, this row-mode inserter forces the columnar chain to be broken right before the sink: data is converted to RowData, the inserter iterates row by row to set the timestamp, and the result is then handed back to the (columnar) sink input. This adds two row↔column conversions per batch.
Offload Flink's row-mode StreamRecordTimestampInserter to a native Velox operator, so the sink input chain stays columnar end-to-end when the sink is offloaded to Velox.
Gluten version
main branch
Description
Flink's
CommonExecSink.applyRowtimeTransformationinserts a row-modeStreamRecordTimestampInserteron the sink input chain when the sink table declares a rowtime attribute. This operator reads the rowtime column from eachRowDataand stamps it onto theStreamRecordon a per-row basis.When the downstream sink is offloaded to Velox via Gluten-Flink, this row-mode inserter forces the columnar chain to be broken right before the sink: data is converted to
RowData, the inserter iterates row by row to set the timestamp, and the result is then handed back to the (columnar) sink input. This adds two row↔column conversions per batch.Offload Flink's row-mode
StreamRecordTimestampInserterto a native Velox operator, so the sink input chain stays columnar end-to-end when the sink is offloaded to Velox.Gluten version
main branch