[FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable #21770

Samrat002 · 2023-01-28T05:55:30Z

What is the purpose of the change

Currently, below are the ways Python Worker gets the Python Flink Dependencies.

Worker Node's System Python Path (/usr/local/lib64/python3.7/site-packages)
Client passes the python Dependencies through -pyfs and --pyarch which is localised into PYTHONPATH of Python Worker.
Client passes the requirements through -requirement.txt which gets installed on Worker Node and added into PYTHONPATH of Python Worker.

This change allow PYTHONPATH of Python Worker configurable where Admin/Service provider can install the required python Flink dependencies on a custom path (/usr/lib/pyflink/lib/python3.7/site-packages) on all Worker Nodes and then set the path in the client machine configuration flink-conf.yaml. This way it works without any configurations from the Application Users and also without affecting any other components dependent on System Python Path.

Brief change log

add an option to add in flink conf to pick configurable python path in worker and client.

Verifying this change

In flink-conf.yaml add the following configs:

python.client.executable: python3
python.executable: python3
python.pythonpath: /tmp/PYTHONPATH/lib64/python3.7/site-packages/:/tmp/PYTHONPATH/lib/python3.7/site-packages/

Install PyFlink libraries on Client and Worker nodes

[hadoop@ip-172-1-2-3 flink]$ cat  /etc/flink/conf/req.txt
apache-flink==1.16.0

Install python in all worker nodes

python3 -m pip install --ignore-installed -r /tmp/req.txt --prefix /tmp/PYTHONPATH/

Run a job

flink run -py /usr/lib/flink/examples/python/table/word_count.py

LOG

jobmanager log

obmanager.log:2023-02-23 07:09:30,931 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: python.pythonpath, /tmp/PYTHONPATH/lib64/python3.7/site-packages/:/tmp/PYTHONPATH/lib/python3.7/site-packages/
jobmanager.log:2023-02-23 07:09:33,070 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: python.pythonpath, /tmp/PYTHONPATH/lib64/python3.7/site-packages/:/tmp/PYTHONPATH/lib/python3.7/site-packages/

taskmanager log

taskmanager.log:2023-02-23 07:14:50,549 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: python.pythonpath, /tmp/PYTHONPATH/lib64/python3.7/site-packages/:/tmp/PYTHONPATH/lib/python3.7/site-packages/
taskmanager.log:2023-02-23 07:14:53,978 INFO  org.apache.flink.python.env.AbstractPythonEnvironmentManager [] - PYTHONPATH of python worker: /tmp/PYTHONPATH/lib64/python3.7/site-packages/:/tmp/PYTHONPATH/lib/python3.7/site-packages/
taskmanager.log:2023-02-23 07:14:56,039 INFO  /tmp/PYTHONPATH/lib64/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py:84 [] - Logging handler created.
taskmanager.log:2023-02-23 07:14:56,039 INFO  /tmp/PYTHONPATH/lib64/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py:107 [] - semi_persistent_directory: /tmp
taskmanager.log:2023-02-23 07:14:56,039 WARN  /tmp/PYTHONPATH/lib64/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py:281 [] - No session file found: /tmp/staged/pickled_main_session. Functions defined in __main__ (interactive session) may fail.
taskmanager.log:2023-02-23 07:14:56,039 WARN  /tmp/PYTHONPATH/lib64/python3.7/site-packages/apache_beam/options/pipeline_options.py:335 [] - Discarding unparseable args: ['--options_id=0.0', '--app_name=BeamPythonFunctionRunner']
taskmanager.log:2023-02-23 07:14:56,039 INFO  /tmp/PYTHONPATH/lib64/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py:125 [] - Pipeline_options: {'experiments': ['state_cache_size=1000']}
taskmanager.log:2023-02-23 07:14:56,039 INFO  /tmp/PYTHONPATH/lib64/python3.7/site-packages/apache_beam/runners/worker/statecache.py:172 [] - Creating state cache with size 1000
taskmanager.log:2023-02-23 07:14:56,039 INFO  /tmp/PYTHONPATH/lib64/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py:164 [] - Creating insecure control channel for localhost:43865.
taskmanager.log:2023-02-23 07:14:56,040 INFO  /tmp/PYTHONPATH/lib64/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py:172 [] - Control channel established.
taskmanager.log:2023-02-23 07:14:56,040 INFO  /tmp/PYTHONPATH/lib64/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py:215 [] - Initializing SDKHarness with unbounded number of workers.
taskmanager.log:2023-02-23 07:14:56,040 INFO  /tmp/PYTHONPATH/lib64/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py:179 [] - Python sdk harness starting.
taskmanager.log:2023-02-23 07:14:56,174 INFO  /tmp/PYTHONPATH/lib64/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py:840 [] - Creating insecure state channel for localhost:35489.
taskmanager.log:2023-02-23 07:14:56,174 INFO  /tmp/PYTHONPATH/lib64/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py:847 [] - State channel established.
taskmanager.log:2023-02-23 07:14:56,178 INFO  /tmp/PYTHONPATH/lib64/python3.7/site-packages/apache_beam/runners/worker/data_plane.py:750 [] - Creating client data channel for localhost:43911

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no) no
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no) no
The serializers: (yes / no / don't know) no
The runtime per-record code paths (performance sensitive): (yes / no / don't know) don't know
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) no
The S3 file system connector: (yes / no / don't know) no

Documentation

Does this pull request introduce a new feature? (yes / no) yes
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) docs

flinkbot · 2023-01-28T06:03:28Z

CI report:

b739eef Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

flink-python/src/main/java/org/apache/flink/python/PythonOptions.java

flink-python/src/main/java/org/apache/flink/python/env/AbstractPythonEnvironmentManager.java

Samrat002 · 2023-02-09T09:12:59Z

@HuangXingBo
Please review this small change in free time .

HuangXingBo

@Samrat002 Thanks a lot for the feature. I have left some comments.

HuangXingBo · 2023-02-22T07:56:35Z

flink-python/src/main/java/org/apache/flink/client/python/PythonEnvUtils.java

        if (pythonEnv.pythonPath != null) {
-            String defaultPythonPath = env.get("PYTHONPATH");
+            String defaultPythonPath =
+                    config.getOptional(PYTHON_PATH).orElse(env.get("PYTHON_PATH"));


Suggested change

config.getOptional(PYTHON_PATH).orElse(env.get("PYTHON_PATH"));

config.getOptional(PYTHON_PATH).orElse(env.get("PYTHONPATH"));

Will the PYTHON_PATH configuration work on both the client and the cluster side?

And I prefer these logic is in preparePythonEnvironment rather than adding another param ReadableConfig to startPythonProcess

Will the PYTHON_PATH configuration work on both the client and the cluster side?

Yes this is for both client side and cluster side.

HuangXingBo · 2023-02-22T08:01:38Z

docs/static/generated/rest_v1_dispatcher.yml

@@ -6,7 +6,7 @@ info:
  license:
    name: Apache 2.0
    url: https://www.apache.org/licenses/LICENSE-2.0.html
-  version: v1/1.17-SNAPSHOT
+  version: v1/1.18-SNAPSHOT


Modifications not related to this feature?

This is related to this feature change only.

This doc update is generated after running the below command

./mvnw package -Dgenerate-rest-docs -pl flink-docs -am -nsu -DskipTests

Samrat002 · 2023-02-23T08:56:13Z

Thank you for initially reviewing the changes. @HuangXingBo
I have made updates in the PR as per review comments, also updated the description section with logs and steps done to verify the changes in a sample cluster.
Please review the changes whenever time .

HuangXingBo

@Samrat002 Thanks a lot for the update. I have left some comments.

docs/layouts/shortcodes/generated/execution_config_configuration.html

docs/static/generated/rest_v1_dispatcher.yml

flink-python/src/main/java/org/apache/flink/python/PythonOptions.java

flink-python/src/main/java/org/apache/flink/python/env/PythonDependencyInfo.java

flink-python/src/main/java/org/apache/flink/python/env/AbstractPythonEnvironmentManager.java

flink-python/src/main/java/org/apache/flink/python/env/PythonDependencyInfo.java

flink-python/src/main/java/org/apache/flink/python/PythonOptions.java

HuangXingBo

@Samrat002 Thanks a lot for the update. Only a comment left which I will update during merging.

flink-python/src/main/java/org/apache/flink/client/python/PythonEnvUtils.java

Co-authored-by: HuangXingBo <hxbks2ks@gmail.com>

Samrat002 changed the title ~~[DRAFT][FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable~~ [FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable Jan 28, 2023

Samrat002 force-pushed the FLINK-30277 branch from d10e432 to c596d8c Compare January 28, 2023 06:20

flinkbot added the component=API/Python label Jan 28, 2023

Samrat002 force-pushed the FLINK-30277 branch 2 times, most recently from 649e1e4 to 1431e69 Compare February 3, 2023 14:37

hlteoh37 reviewed Feb 9, 2023

View reviewed changes

flink-python/src/main/java/org/apache/flink/python/PythonOptions.java Outdated Show resolved Hide resolved

flink-python/src/main/java/org/apache/flink/python/env/AbstractPythonEnvironmentManager.java Outdated Show resolved Hide resolved

Samrat002 force-pushed the FLINK-30277 branch from dc50cfb to b07163a Compare February 21, 2023 05:00

HuangXingBo reviewed Feb 22, 2023

View reviewed changes

Samrat002 force-pushed the FLINK-30277 branch from 68a18b1 to 29704c4 Compare February 23, 2023 08:14

Samrat002 closed this Feb 23, 2023

Samrat002 force-pushed the FLINK-30277 branch from 29704c4 to db6e01e Compare February 23, 2023 08:23

Samrat002 reopened this Feb 23, 2023

Samrat002 requested review from HuangXingBo and hlteoh37 and removed request for HuangXingBo and hlteoh37 February 23, 2023 08:49

Samrat002 force-pushed the FLINK-30277 branch 4 times, most recently from 7684db4 to 29821f3 Compare February 24, 2023 03:59

HuangXingBo reviewed Feb 24, 2023

View reviewed changes

Samrat002 force-pushed the FLINK-30277 branch from 217826e to 5722222 Compare February 24, 2023 17:37

Samrat002 requested a review from HuangXingBo February 24, 2023 17:40

HuangXingBo approved these changes Feb 25, 2023

View reviewed changes

flink-python/src/main/java/org/apache/flink/client/python/PythonEnvUtils.java Outdated Show resolved Hide resolved

[FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable

b739eef

Co-authored-by: HuangXingBo <hxbks2ks@gmail.com>

Samrat002 force-pushed the FLINK-30277 branch from 13b09c4 to b739eef Compare February 26, 2023 04:26

HuangXingBo closed this in 2e0efe4 Feb 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable #21770

[FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable #21770

Samrat002 commented Jan 28, 2023 •

edited

flinkbot commented Jan 28, 2023 •

edited

Samrat002 commented Feb 9, 2023 •

edited

HuangXingBo left a comment

HuangXingBo Feb 22, 2023

HuangXingBo Feb 22, 2023

HuangXingBo Feb 22, 2023

Samrat002 Feb 22, 2023 •

edited

HuangXingBo Feb 22, 2023

Samrat002 Feb 22, 2023 •

edited

Samrat002 commented Feb 23, 2023 •

edited

HuangXingBo left a comment

HuangXingBo left a comment

	config.getOptional(PYTHON_PATH).orElse(env.get("PYTHON_PATH"));
	config.getOptional(PYTHON_PATH).orElse(env.get("PYTHONPATH"));

[FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable #21770

[FLINK-30277][python]Allow PYTHONPATH of Python Worker configurable #21770

Conversation

Samrat002 commented Jan 28, 2023 • edited

What is the purpose of the change

Brief change log

Verifying this change

Install PyFlink libraries on Client and Worker nodes

LOG

Does this pull request potentially affect one of the following parts:

Documentation

flinkbot commented Jan 28, 2023 • edited

CI report:

Samrat002 commented Feb 9, 2023 • edited

HuangXingBo left a comment

Choose a reason for hiding this comment

HuangXingBo Feb 22, 2023

Choose a reason for hiding this comment

HuangXingBo Feb 22, 2023

Choose a reason for hiding this comment

HuangXingBo Feb 22, 2023

Choose a reason for hiding this comment

Samrat002 Feb 22, 2023 • edited

Choose a reason for hiding this comment

HuangXingBo Feb 22, 2023

Choose a reason for hiding this comment

Samrat002 Feb 22, 2023 • edited

Choose a reason for hiding this comment

Samrat002 commented Feb 23, 2023 • edited

HuangXingBo left a comment

Choose a reason for hiding this comment

HuangXingBo left a comment

Choose a reason for hiding this comment

Samrat002 commented Jan 28, 2023 •

edited

flinkbot commented Jan 28, 2023 •

edited

Samrat002 commented Feb 9, 2023 •

edited

Samrat002 Feb 22, 2023 •

edited

Samrat002 Feb 22, 2023 •

edited

Samrat002 commented Feb 23, 2023 •

edited