-
Notifications
You must be signed in to change notification settings - Fork 419
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement Flink ScanTableSource and LookupTableSource #213
Conversation
<groupId>org.apache.flink</groupId> | ||
<artifactId>flink-csv</artifactId> | ||
<version>${flink.version}</version> | ||
</dependency> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is flink-csv neccesary?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is flink-csv neccesary?
flink sqlclient exports parquet as csv file by flink-csv for k8s test
<artifactId>spark-sql_2.12</artifactId> | ||
<version>3.3.1</version> | ||
<scope>test</scope> | ||
</dependency> | ||
</dependencies> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is is necessary for importing a full spark-sql_2.12 package for test?
…bleSource && mock lookup cache for test
…tition value pushdown
…& LookupTableSource supports reading latest N partitions, sort order is partition name
…ionReader.read && update CHBenchmark、LookupJoinCase
FlinkUtil.setFSConfigs(conf, reader); | ||
reader.initializeReader(); | ||
this.reader = new LakeSoulArrowReader(reader, 10000); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
partition value handling can be pushed down to Native
this.conf = null; | ||
this.awaitTimeout = 10000; | ||
this.curPartitionId = -1; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
parameter of LakeSoulPartitionReader should be configurable
…ds && fix no pk table select
|
||
@Override | ||
public void addReader(int subtaskId) { | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why do nothing here?
73887d1
to
6f945e4
Compare
4282887
to
c7da8bd
Compare
Close #194