-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature] submit Flink CDC pipeline job to k8s cluster. #2934
Comments
I think it can be regarded as an ordinary Flink application and can be submitted to Flink Operator or Flink Cluster. I am willing to take on this work. Can you assign it to me? @lvyanquan |
This is a comment requirement from many users, looking forward to your code 😊 @czy006 |
After discussion, we produced a version of relevant design documents : https://docs.google.com/document/d/1HlvRS5aVNqPtOx14mynEL3RFaZORuTFkWZAjvl4k998/edit?usp=sharing |
I don’t see any discussion on yarn deployment in the document. Are you planning to support yarn mode? @czy006 |
hello,i have the same problem,is there any plans to support yarn mode?All of our flink jobs are running on yarn flink session. |
Flink on YARN is the main deployment mode, why is there no discussion about the YARN deployment mode? @czy006 |
@czy006 @lvyanquan Compared with submitting using flink-cdc.sh, I think reuse the flink client to submit flink cdc job is more reasonable and easier to maintain, while flink client already supports all kinds of submitting way, including k8s session/application mode and yarn session/application mode, and widely used in production environments. |
YARN will be implemented at a later date and is not discussed in this thread. |
From another perspective, when we use Flink CDC as an application, we need to support deployment target in the script, which essentially uses flink-client to submit jobs |
Closing this issue as it has been migrated to Apache Jira. |
Search before asking
Motivation
Currently, there is no a clear description to tell how to run a pipeline on k8s.
So If there is no need to modify the code, please add some docs to guide users how to submit job to a Kubernetes cluster.
If it's necessary to modify the code, you can submit a pr and add doc to fix it.
Solution
No response
Alternatives
No response
Anything else?
No response
Are you willing to submit a PR?
The text was updated successfully, but these errors were encountered: