[FLINK-21732][python] Bundle Java libraries into apache-flink-libraries #15386

HuangXingBo · 2021-03-26T10:10:42Z

What is the purpose of the change

This pull request will bundle Java libraries into apache-flink-libraries

Brief change log

Add a new distributed package named apache-flink-libraries and bundle Java libraries into it
Change the pipeline of building apache-flink wheel packages in Azure Pipeline

Verifying this change

This change added tests and can be verified as follows:

original IT tests and UT tests
Manual test installing apache-flink-libraries and apache-flink in a machine with old version of apache-flink and run some python udf test
trigger building apache-flink wheel packages in private Azure pipeline

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

flinkbot · 2021-03-26T10:29:02Z

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 58978e2 (Sat Aug 28 11:11:50 UTC 2021)

Warnings:

1 pom.xml files were touched: Check for build and licensing issues.

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

❓ 1. The [description] looks good.
❓ 2. There is [consensus] that the contribution should go into to Flink.
❓ 3. Needs [attention] from.
❓ 4. The change fits into the overall [architecture].
❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

Details

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
@flinkbot approve all to approve all aspects
@flinkbot approve-until architecture to approve everything until architecture
@flinkbot attention @username1 [@username2 ..] to require somebody's attention
@flinkbot disapprove architecture to remove an approval you gave earlier

flinkbot · 2021-03-26T10:49:44Z

CI report:

54b8c72 Azure: SUCCESS
c90e7a0 UNKNOWN
203779a Azure: PENDING
58978e2 Azure: PENDING

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run travis re-run the last Travis build
@flinkbot run azure re-run the last Azure build

WeiZhong94

@HuangXingBo Thanks for your PR. I have left some comments for you. In addition I think the content in docs/flinkDev/building#build_pyflink also need to be changed because it does not work on current situation.

WeiZhong94 · 2021-03-31T03:18:57Z

flink-python/apache-flink-libraries/setup.py

+
+try:
+    exec(open(version_file).read())
+except IOError:


We can check if the in_flink_source is true instead of try .. except.

WeiZhong94 · 2021-03-31T03:21:24Z

flink-python/apache-flink-libraries/setup.py

+    To build Flink with maven you can run:
+      mvn -DskipTests clean package
+    Building the source dist is done in the flink-python directory:
+      cd flink-python


These scripts need to change.

WeiZhong94 · 2021-03-31T03:30:20Z

flink-python/apache-flink-libraries/setup.py

+        PACKAGE_DATA['pyflink.licenses'] = ['*']
+
+    setup(
+        name='apache_flink_libraries',


use - instead of _

WeiZhong94 · 2021-03-31T03:30:48Z

flink-python/apache-flink-libraries/setup.py

+        author='Apache Software Foundation',
+        author_email='dev@flink.apache.org',
+        python_requires='>=3.6',
+        description='Apache Flink Python API',


The short description need to be changed.

WeiZhong94 · 2021-03-31T03:36:34Z

flink-python/pyflink/pyflink_gateway_server.py

+    flink_lib_directory = None
+    flink_opt_directory = None
+    if 'FLINK_LIB_DIR' in os.environ:
+        flink_lib_directory = os.path.realpath(os.environ['FLINK_LIB_DIR'])


else flink_lib_directory = os.path.join(real_flink_home, "lib")?

WeiZhong94 · 2021-03-31T03:36:51Z

flink-python/pyflink/pyflink_gateway_server.py

+    if 'FLINK_LIB_DIR' in os.environ:
+        flink_lib_directory = os.path.realpath(os.environ['FLINK_LIB_DIR'])
+    if 'FLINK_OPT_DIR' in os.environ:
+        flink_opt_directory = os.path.realpath(os.environ['FLINK_OPT_DIR'])


WeiZhong94 · 2021-03-31T04:06:12Z

flink-python/apache-flink-libraries/setup.py

@@ -0,0 +1,229 @@
+################################################################################


We need to add clean logic for this package in flink-python/pom.xml

WeiZhong94 · 2021-03-31T06:49:56Z

flink-python/apache-flink-libraries/README.md

@@ -0,0 +1,5 @@
+# Apache Flink


add some description about this library package?

WeiZhong94 · 2021-03-31T07:21:47Z

flink-python/setup.py

+    file_elements = (root_data.getElementsByTagName("files")[0]).getElementsByTagName("file")
+    # extracted <files><file></file></files>
+    for file_element in file_elements:
+        source = ((file_element.getElementsByTagName('source')[0]).childNodes[0]).data


Currently this function does not support the <includes></includes> tag and <excludes></excludes> tag. I think we need to raise Exception when the file element contains them to ensure the correctness.

HuangXingBo · 2021-03-31T09:57:26Z

@WeiZhong94 Thanks a lot for the review. I have addressed the comments at the latest commit.

WeiZhong94

LGTM, +1 to Merge

HuangXingBo · 2021-03-31T16:36:28Z

https://dev.azure.com/hxbks2ks/FLINK-TEST/_build/results?buildId=982&view=results

rmetzger added the review=description? label Mar 26, 2021

rmetzger added the component=API/Python label Mar 26, 2021

[FLINK-21732][python] Bundle Java libraries into apache-flink-libraries

5735eb1

WeiZhong94 reviewed Mar 31, 2021

View reviewed changes

HuangXingBo force-pushed the FLINK-21732 branch from 54b8c72 to c90e7a0 Compare March 31, 2021 09:56

HuangXingBo force-pushed the FLINK-21732 branch 2 times, most recently from 64c4dfa to 203779a Compare March 31, 2021 10:13

WeiZhong94 approved these changes Mar 31, 2021

View reviewed changes

address comments

58978e2

HuangXingBo force-pushed the FLINK-21732 branch from 1316598 to 58978e2 Compare March 31, 2021 13:03

HuangXingBo closed this in af681c6 Mar 31, 2021

		@@ -0,0 +1,229 @@
		################################################################################

[FLINK-21732][python] Bundle Java libraries into apache-flink-libraries #15386

[FLINK-21732][python] Bundle Java libraries into apache-flink-libraries #15386

Uh oh!

Conversation

HuangXingBo commented Mar 26, 2021

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

flinkbot commented Mar 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Automated Checks

Review Progress

Uh oh!

flinkbot commented Mar 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

WeiZhong94 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuangXingBo commented Mar 31, 2021

Uh oh!

WeiZhong94 left a comment

Choose a reason for hiding this comment

Uh oh!

HuangXingBo commented Mar 31, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

flinkbot commented Mar 26, 2021 •

edited

Loading

flinkbot commented Mar 26, 2021 •

edited

Loading