Projection Pushdown with Newly Added Columns Fails for Old Batches

[loserwang1024]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

0.9.0 (latest release)

Please describe the bug 🐞

Our system supports schema evolution: when a new column is added to the table schema, existing data files are not rewritten—instead, the query engine handles missing columns by injecting NULL or default values at read time (client-side compatibility). This works correctly in most scenarios.

However, when projection pushdown is applied and the query explicitly selects the newly added column, the scan operator may attempt to read the column directly from storage—despite its absence in old data files—leading to a failure because the column vector is missing in the batch.

Solution

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[Kaixuan-Duan]

@loserwang1024 Hi, I would like to work on this issue. Could you please assign it to me?

[loserwang1024]

@Kaixuan-Duan Would you like to share your thoughts before that? The most important thing prior to it is how to fix it and whether a change in the RPC proto is needed.

[Kaixuan-Duan]

@loserwang1024 Thanks for the feedback! Here's my proposed fix:

1. No RPC proto change needed: The client already sends the expected projection indices.
2. Server-side alignment: In createProjectionInfo(), we split selectedFieldPositions into existing fields and missing fields. Then in projectSingleBatch(), we reconstruct the Arrow batch by copying existing vectors and generating all-NULL Arrow vectors for the missing columns at their correct target positions.

[loserwang1024]

2. Server-side alignment: In createProjectionInfo(), we split selectedFieldPositions into existing fields and missing fields. Then in projectSingleBatch(), we reconstruct the Arrow batch by copying existing vectors and generating all-NULL Arrow vectors for the missing columns at their correct target positions.

@wuchong , I have been talked this with you , WDYT? (You have adviced that server only return existing fields (and with filed id), then append null in client side.)