Initial support for python executable #165
Conversation
| - **className** (required): Fully qualified Java class name of the job. | ||
| - **jarFile** (optional): JAR file of the job. It could be a local file or remote URI, depending on which | ||
| protocols (e.g., `https://`, `gs://`) are supported by the Flink image. `jarFile` or `python` is required | ||
| - **className** (optional): Fully qualified Java class name of the job. |
There was a problem hiding this comment.
className is optional but it is not yet updated.
https://github.com/spotify/flink-on-k8s-operator/blob/master/api/v1beta1/flinkcluster_types.go#L371
|
@regadas could you review it? |
There was a problem hiding this comment.
Thank you for this @moonsub-kim; I'll have a better look at this later today/
api/v1beta1/flinkcluster_types.go
Outdated
|
|
||
| // Fully qualified Java class name of the job. | ||
| ClassName *string `json:"className,omitempty"` | ||
|
|
||
| // Python file of the job. | ||
| Python *string `json:"python,omitempty"` |
There was a problem hiding this comment.
wdyt of PythonFile inline with JarFile
There was a problem hiding this comment.
It seems good to apply your suggestion
There was a problem hiding this comment.
Overall the looks good! @moonsub-kim Could you add an example to the examples folder?
|
Due to flink does not provide an image including python, Users must be set up thier own dockerfile and upload to any docker registry, and use it in the FlinkCluster. In my case, the Dockerfile copied from Docker Setup - Enabling Python is uploaded into my docker-hub registry. apiVersion: flinkoperator.k8s.io/v1beta1
kind: FlinkCluster
metadata:
name: flinkjobcluster-sample
spec:
flinkVersion: "1.14"
image:
name: gos16052/pink-squirrel:0.1.0 # customized flink image with python
jobManager:
accessScope: Cluster
ports:
ui: 8081
resources:
limits:
memory: "1024Mi"
cpu: "200m"
taskManager:
replicas: 2
resources:
limits:
memory: "1024Mi"
cpu: "200m"
job:
pythonFile: "apps/example/word_count.py" # The image contains example python file
parallelism: 2
restartPolicy: "Never"
flinkProperties:
taskmanager.numberOfTaskSlots: "1"
taskmanager.memory.flink.size: "560mb"
taskmanager.memory.process.size: "1024mb" |
There was a problem hiding this comment.
@moonsub-kim I think it would be cool if we include support for these 2 use cases, wdyt?
Run a PyFlink job which will reference Java UDF or external connectors. JAR file specified in --jarfile will be uploaded to the cluster.
$ ./bin/flink run \
--python examples/python/table/word_count.py \
--jarfile <jarFile>
Run a PyFlink job with pyFiles and the main entry module specified in --pyModule:
$ ./bin/flink run \
--pyModule table.word_count \
--pyFiles examples/python/table
|
It's good to support |
|
would this work with Beam python jobs? |
|
@eddiewang |
|
I have extended to support additional arguments and added the guide doc. |
|
@regadas could you review this? |
There was a problem hiding this comment.
LGTM, it will be good to have remote file support though. I think we can tackle it on separate PR.
@moonsub-kim Thanks for this.
regadas
changed the title
#164
I implemented to support python executable in the CRD.
To run python or jarFile, the JobSpec.JarFile is changed with optional variable
and validate that only one of them is specified.