[FLINK-23115][python] Expose new APIs about TableDescriptor in PyFlink #16516

HuangXingBo · 2021-07-16T11:27:30Z

What is the purpose of the change

This pull request will expose new APIs about TableDescriptor in PyFlink

Brief change log

Add ConfigOptions/TableDescriptor/FormatDescriptor/Schema
Expose create_table/create_temporary_table/from_descriptor of TableEnvironment
Expose Table#execute_insert and StatementSet#add_insert in Python Table API

Verifying this change

This change added tests and can be verified as follows:

test_schema.py , test_table_descriptor.py, test_table_environment_api.py

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

flinkbot · 2021-07-16T11:31:03Z

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 59da1ae (Thu Sep 23 18:07:16 UTC 2021)

Warnings:

No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

❓ 1. The [description] looks good.
❓ 2. There is [consensus] that the contribution should go into to Flink.
❓ 3. Needs [attention] from.
❓ 4. The change fits into the overall [architecture].
❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
@flinkbot approve all to approve all aspects
@flinkbot approve-until architecture to approve everything until architecture
@flinkbot attention @username1 [@username2 ..] to require somebody's attention
@flinkbot disapprove architecture to remove an approval you gave earlier

flinkbot · 2021-07-16T11:47:42Z

CI report:

59da1ae Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run travis re-run the last Travis build
@flinkbot run azure re-run the last Azure build

Airblader

Thank you @HuangXingBo for taking on this issue, I appreciate it. Will someone from your team review it? I'm not too familiar with PyFlink, but if needed I can also review this. Just let me know. Thanks!

Airblader · 2021-07-16T16:42:19Z

flink-python/pyflink/table/catalog.py

@@ -733,9 +734,22 @@ def get_schema(self) -> TableSchema:
        Get the schema of the table.

        :return: Schema of the table/view.
+
+        . note:: Deprecated in 1.14. This method returns the deprecated ableSchema class. The old


nit: missing "T" in TableSchema"

Airblader · 2021-07-16T16:42:39Z

flink-python/pyflink/common/config_options.py

+class ConfigOptions(object):
+    """
+    {@code ConfigOptions} are used to build a :class:`~pyflink.table.ConfigOption`. The option is
+    typically built in one of the following pattern:


nit: "pattern" → "patterns"

HuangXingBo · 2021-07-17T02:30:33Z

Thanks @Airblader for the review. @dianfu is familiar with PyFlink and he will help review this PR too.

dianfu

@HuangXingBo Thanks for the PR! Have left a few minor comments. Besides, is there any documentation needs to be updated?

dianfu · 2021-07-19T04:47:30Z

flink-python/pyflink/common/config_options.py

+
+class ConfigOptions(object):
+    """
+    {@code ConfigOptions} are used to build a :class:`~pyflink.table.ConfigOption`. The option is


Suggested change

{@code ConfigOptions} are used to build a :class:`~pyflink.table.ConfigOption`. The option is

{@code ConfigOptions} are used to build a :class:`~pyflink.common.ConfigOption`. The option is

dianfu · 2021-07-19T04:48:40Z

flink-python/pyflink/common/config_options.py

+
+T = TypeVar('T')
+
+__all__ = ['ConfigOptions', 'ConfigOption']


What about also adding it in init.py? otherwise, I'm afraid that it will not appear in the Python doc

dianfu · 2021-07-19T04:53:03Z

flink-python/pyflink/table/schema.py

+from pyflink.table.types import DataType, _to_java_data_type
+from pyflink.util.java_utils import to_jarray
+
+__all__ = ['Schema']


also added in init.py?

dianfu · 2021-07-19T04:54:29Z

flink-python/pyflink/table/schema.py

+            """
+            Adopts all members from the given unresolved schema.
+            """
+            self._j_builder.fromSchema(unresolved_schema)


Suggested change

self._j_builder.fromSchema(unresolved_schema)

self._j_builder.fromSchema(unresolved_schema._j_schema)

dianfu · 2021-07-19T04:59:34Z

flink-python/pyflink/table/schema.py

+            """
+            Adopts all fields of the given row as physical columns of the schema.
+            """
+            self._j_builder.fromRowDataType(data_type)


It would be great to improve the test case to also cover this method

dianfu · 2021-07-19T05:02:53Z

flink-python/pyflink/table/statement_set.py

+            Examples:
+            ::
+
+                >>>stmt_set = table_env.create_statement_set()


Suggested change

>>>stmt_set = table_env.create_statement_set()

>>> stmt_set = table_env.create_statement_set()

HuangXingBo · 2021-07-19T12:20:47Z

@dianfu Thanks a lot for the review. I have addressed the comments at the corresponding commits. About the documentation part, I plan to add the content after the java doc has been updated in https://issues.apache.org/jira/browse/FLINK-23116

…tor/Schema in Python Table API

…tor/Schema in Python Table API-fix-1

…tor/Schema in Python Table API-fix-2

…_descriptor of TableEnvironment in Python Table API

…_descriptor of TableEnvironment in Python Table API-fix-1

…d_insert in Python Table API

…d_insert in Python Table API-fix-1

dianfu

LGTM. Just two minor comments and +1 from my side.

dianfu · 2021-07-20T07:32:36Z

flink-python/pyflink/common/config_options.py

+            """
+            Defines that the value of the option should be of float type
+            (4-byte single precision floating point number).
+            :return:


Suggested change

:return:

dianfu · 2021-07-20T07:37:39Z

flink-python/pyflink/table/statement_set.py

-    def add_insert(self, target_path: str, table, overwrite: bool = False) -> 'StatementSet':
+    def add_insert(self,
+                   target_path_or_descriptor: Union[str, TableDescriptor],
+                   table,


Suggested change

table,

table: Table,

…_descriptor of TableEnvironment in Python Table API This closes #16516.

…d_insert in Python Table API This closes #16516.

…tor/Schema in Python Table API This closes apache#16516.

…_descriptor of TableEnvironment in Python Table API This closes apache#16516.

…d_insert in Python Table API This closes apache#16516.

…tor/Schema in Python Table API This closes apache#16516.

…_descriptor of TableEnvironment in Python Table API This closes apache#16516.

…d_insert in Python Table API This closes apache#16516.

rmetzger added review=description? component=API/Python labels Jul 16, 2021

Airblader reviewed Jul 16, 2021

View reviewed changes

HuangXingBo force-pushed the FLINK-23115 branch from d1de6d7 to 570dd81 Compare July 19, 2021 02:14

dianfu reviewed Jul 19, 2021

View reviewed changes

HuangXingBo force-pushed the FLINK-23115 branch from 570dd81 to 1eeef1f Compare July 19, 2021 12:16

HuangXingBo added 7 commits July 20, 2021 10:28

[FLINK-23115][python] Add ConfigOptions/TableDescriptor/FormatDescrip…

cd7dbb7

…tor/Schema in Python Table API

[FLINK-23115][python] Add ConfigOptions/TableDescriptor/FormatDescrip…

a210104

…tor/Schema in Python Table API-fix-1

[FLINK-23115][python] Add ConfigOptions/TableDescriptor/FormatDescrip…

09d22a5

…tor/Schema in Python Table API-fix-2

[FLINK-23115][python] Expose create_table/create_temporary_table/from…

2b2d4ef

…_descriptor of TableEnvironment in Python Table API

[FLINK-23115][python] Expose create_table/create_temporary_table/from…

e725edd

…_descriptor of TableEnvironment in Python Table API-fix-1

[FLINK-23115][python] Expose Table#execute_insert and StatementSet#ad…

ac30907

…d_insert in Python Table API

[FLINK-23115][python] Expose Table#execute_insert and StatementSet#ad…

59da1ae

…d_insert in Python Table API-fix-1

HuangXingBo force-pushed the FLINK-23115 branch from 1eeef1f to 59da1ae Compare July 20, 2021 02:28

dianfu reviewed Jul 20, 2021

View reviewed changes

dianfu approved these changes Jul 20, 2021

View reviewed changes

HuangXingBo closed this in 7bf453d Jul 20, 2021

HuangXingBo added a commit that referenced this pull request Jul 20, 2021

[FLINK-23115][python] Expose create_table/create_temporary_table/from…

9beb59b

…_descriptor of TableEnvironment in Python Table API This closes #16516.

HuangXingBo added a commit that referenced this pull request Jul 20, 2021

[FLINK-23115][python] Expose Table#execute_insert and StatementSet#ad…

93aab0a

…d_insert in Python Table API This closes #16516.

srinipunuru pushed a commit to srinipunuru/flink that referenced this pull request Jul 21, 2021

[FLINK-23115][python] Add ConfigOptions/TableDescriptor/FormatDescrip…

a612ab4

…tor/Schema in Python Table API This closes apache#16516.

srinipunuru pushed a commit to srinipunuru/flink that referenced this pull request Jul 21, 2021

[FLINK-23115][python] Expose create_table/create_temporary_table/from…

d9f9e80

…_descriptor of TableEnvironment in Python Table API This closes apache#16516.

srinipunuru pushed a commit to srinipunuru/flink that referenced this pull request Jul 21, 2021

[FLINK-23115][python] Expose Table#execute_insert and StatementSet#ad…

306096e

…d_insert in Python Table API This closes apache#16516.

hhkkxxx133 pushed a commit to hhkkxxx133/flink that referenced this pull request Aug 25, 2021

[FLINK-23115][python] Add ConfigOptions/TableDescriptor/FormatDescrip…

3db170f

…tor/Schema in Python Table API This closes apache#16516.

hhkkxxx133 pushed a commit to hhkkxxx133/flink that referenced this pull request Aug 25, 2021

[FLINK-23115][python] Expose create_table/create_temporary_table/from…

ac86943

…_descriptor of TableEnvironment in Python Table API This closes apache#16516.

hhkkxxx133 pushed a commit to hhkkxxx133/flink that referenced this pull request Aug 25, 2021

[FLINK-23115][python] Expose Table#execute_insert and StatementSet#ad…

26e3fdb

…d_insert in Python Table API This closes apache#16516.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-23115][python] Expose new APIs about TableDescriptor in PyFlink #16516

[FLINK-23115][python] Expose new APIs about TableDescriptor in PyFlink #16516

HuangXingBo commented Jul 16, 2021

flinkbot commented Jul 16, 2021 •

edited

flinkbot commented Jul 16, 2021 •

edited

Airblader left a comment

Airblader Jul 16, 2021

Airblader Jul 16, 2021

HuangXingBo commented Jul 17, 2021

dianfu left a comment

dianfu Jul 19, 2021

dianfu Jul 19, 2021

dianfu Jul 19, 2021

dianfu Jul 19, 2021

dianfu Jul 19, 2021

dianfu Jul 19, 2021

dianfu Jul 19, 2021

HuangXingBo commented Jul 19, 2021

dianfu left a comment

dianfu Jul 20, 2021

dianfu Jul 20, 2021

	{@code ConfigOptions} are used to build a :class:`~pyflink.table.ConfigOption`. The option is
	{@code ConfigOptions} are used to build a :class:`~pyflink.common.ConfigOption`. The option is

	self._j_builder.fromSchema(unresolved_schema)
	self._j_builder.fromSchema(unresolved_schema._j_schema)

	>>>stmt_set = table_env.create_statement_set()
	>>> stmt_set = table_env.create_statement_set()

[FLINK-23115][python] Expose new APIs about TableDescriptor in PyFlink #16516

[FLINK-23115][python] Expose new APIs about TableDescriptor in PyFlink #16516

Conversation

HuangXingBo commented Jul 16, 2021

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

flinkbot commented Jul 16, 2021 • edited

Automated Checks

Review Progress

flinkbot commented Jul 16, 2021 • edited

CI report:

Airblader left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HuangXingBo commented Jul 17, 2021

dianfu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HuangXingBo commented Jul 19, 2021

dianfu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flinkbot commented Jul 16, 2021 •

edited

flinkbot commented Jul 16, 2021 •

edited