Search before asking
Motivation
Fluss's Flink lookup join doesn't custom-shuffle the probe stream, so probe rows are distributed to lookup subtasks arbitrarily. This hurts lookup.cache (partial) hit rate and enlarges RPC fan-out, since each subtask may hit any bucket / any TabletServer.
Flink 2.0 added SupportsLookupCustomShuffle (FLIP-462), which lets a source define the probe-stream distribution. We can shuffle by bucket key so each subtask handles a stable set of buckets → better cache locality and lower RPC fan-out.
Solution
- Implement SupportsLookupCustomShuffle and provide an InputDataPartitioner that maps the lookup key to a Fluss bucket using the existing bucketing logic (consistent with the write path / PrimaryKeyLookuper).
- Only enable when the lookup key covers the bucket key (PK lookup / prefix lookup on bucket key); otherwise fall back to default distribution.
- Note: the interface is Flink 2.0+, so this targets fluss-flink-2.2 only (FlinkTableSource lives in fluss-flink-common which must stay Flink 1.18 compatible).
Anything else?
No response
Willingness to contribute
Search before asking
Motivation
Fluss's Flink lookup join doesn't custom-shuffle the probe stream, so probe rows are distributed to lookup subtasks arbitrarily. This hurts lookup.cache (partial) hit rate and enlarges RPC fan-out, since each subtask may hit any bucket / any TabletServer.
Flink 2.0 added SupportsLookupCustomShuffle (FLIP-462), which lets a source define the probe-stream distribution. We can shuffle by bucket key so each subtask handles a stable set of buckets → better cache locality and lower RPC fan-out.
Solution
Anything else?
No response
Willingness to contribute