[Feature] Strategically Engineering Wide Tables with Foreign Key-Based Joins

[MonsterChenzhuo]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

When performing a join operation with Flink's dual stream join, there is an issue with excessive state storage. Flink Lookup Join only concerns itself with changes in the primary stream; changes in the dimension table cannot update data that has already been joined. With Partial Update association, there is a problem where the joined tables need to have a common primary key to ensure matching between two or more data sources. All the aforementioned solutions for generating wide tables through multi-table associations have their limitations.

I would like to implement an entirely new method to overcome these shortcomings: a dynamic dimension table-driven Lookup Join, which I also refer to as 'ForeignKey Widening'.
https://cwiki.apache.org/confluence/display/PAIMON/PIP-12%3A+Strategically+Engineering+Wide+Tables+with+Foreign+Key-Based+Joins

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[polyzos]

@MonsterChenzhuo whats the status of this? It's a pretty amazing and useful feature

[qinjunjerry]

Indeed amazing! Exactly what I am looking for :)

[eric666666]

@MonsterChenzhuo whats the status of this? It's a pretty amazing and useful feature

+1，Are there any plans for release version？