Add Flink application mode support

[regadas]

No description provided.

[hjwalt]

@regadas for clarification, is this an issue for supporting flink native kubernetes deployment? As per this documentation.

[regadas]

Right so the idea is to support https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/overview/#application-mode as alternative way to the current way of submitting jobs: Per-job and session.

Adding Flink Kubernetes native is smth we can also have a look at.

[elanv]

It would be good to discuss ways to improve the API design when considering new features.

It would probably be nice to add a new CRD to support Flink-native k8s cluster. I think also implementing the application mode with new CRD will also help lower the complexity.

API improvements, for example:

• Rename current FlinkCluster to FlinkSessionCluster
• New CRD for native k8s Flink cluster: FlinkCluster
• New CRDs for various job types: FlinkJob, BeamJob (spec includes FlinkCluster or FlinkSessionCluster template for dedicated use or link existing cluster as clusterName to share it). I think new FlinkJob could support various modes like application, per-job on native k8s Flink cluster.

There are existing issues that had been created. I hope you review it.

• Issues in gcp repo
  • CRD design improvement: Define a new CRD to submit job to session cluster GoogleCloudPlatform/flink-on-k8s-operator#49
  • Motivation for new CRD: [Feautre] New CRD for job submission to Flink session cluster GoogleCloudPlatform/flink-on-k8s-operator#303

[elanv]

There is some interference between my PR and current controller. Therefore if new CRD is going to be added soon, it would be efficient to apply my PR there than reworking the PR.

@regadas In that sense, any specific work or plan for this issue? If not, I would like to proceed with this work first. How is it?
The basic idea is to keep the current FlinkCluster but create new v1beta2 FlinkJob CRD with the job routines extracted from it. If we need Flink k8s native cluster, we can leave the current FlinkCluster as it is and add a new CRD, and among them, we can select the Flink cluster CRD where FlinkJob will run.

[canghai118]

It would probably be nice to add a new CRD to support Flink-native k8s cluster. I think also implementing the application mode with new CRD will also help lower the complexity.

API improvements, for example:

• Rename current FlinkCluster to FlinkSessionCluster
• New CRD for native k8s Flink cluster: FlinkCluster
• New CRDs for various job types: FlinkJob, BeamJob (spec includes FlinkCluster or FlinkSessionCluster template for dedicated use or link existing cluster as clusterName to share it). I think new FlinkJob could support various modes like application, per-job on native k8s Flink cluster.

Is there any progress or plan on this feature @elanv thanks

[haoxins]

It would probably be nice to add a new CRD to support Flink-native k8s cluster. I think also implementing the application mode with new CRD will also help lower the complexity.
API improvements, for example:

• Rename current FlinkCluster to FlinkSessionCluster
• New CRD for native k8s Flink cluster: FlinkCluster
• New CRDs for various job types: FlinkJob, BeamJob (spec includes FlinkCluster or FlinkSessionCluster template for dedicated use or link existing cluster as clusterName to share it). I think new FlinkJob could support various modes like application, per-job on native k8s Flink cluster.

Is there any progress or plan on this feature @elanv thanks

Is there any one working on this?
I think we can split this issue into multi issues/steps so that others can help with it.