Skip to content

Commit 15d51a5

Browse files
gunnarmorlinggunnarmorling
authored
Adding source code for blog post "Get Running With Flink on Kubernetes" (#34)
1 parent 37e287d commit 15d51a5

23 files changed

+1590
-0
lines changed

‎README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,12 @@ _Decodable provides a managed PyFlink service. Learn more [here](https://docs.de
9999
| [Decodable CI/CD](declarative-cicd) | An example of using Decodable with GitHub Actions|
100100
| [Decodable CLI Docker image](cli-docker) | An example Dockerfile for running the Decodable CLI under Docker.|
101101

102+
### Kubernetes
103+
104+
| Example | Description |
105+
|-------------------------------------------------------|-------------|
106+
| [Fink on Kubernetes](flink-on-kubernetes) | An example of running Flink via the Flink Kubernetes Operator|
107+
102108
## License
103109

104110
This code base is available under the Apache License, version 2.

‎flink-on-kubernetes/README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Get Running With Flink on Kubernetes
2+
3+
An extensive example showing how to run Flink on Kubernetes.
4+
5+
Touching on a range of topics such as installation and set-up, creating container images for your own Flink jobs, fault tolerance and high availability, savepoint management, observability, and more.

‎flink-on-kubernetes/flink/basic.yaml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
################################################################################
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
################################################################################
18+
19+
apiVersion: flink.apache.org/v1beta1
20+
kind: FlinkDeployment
21+
metadata:
22+
name: basic-example
23+
spec:
24+
image: flink:1.20-java17
25+
flinkVersion: v1_20
26+
flinkConfiguration:
27+
taskmanager.numberOfTaskSlots: "2"
28+
serviceAccount: flink
29+
jobManager:
30+
resource:
31+
memory: "2048m"
32+
cpu: 1
33+
taskManager:
34+
resource:
35+
memory: "2048m"
36+
cpu: 1
37+
job:
38+
jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
39+
parallelism: 2
40+
upgradeMode: stateless
41+
ingress:
42+
template: "localhost/{{name}}(/|$)(.*)"
43+
className: "nginx"
44+
annotations:
45+
nginx.ingress.kubernetes.io/rewrite-target: "/$2"

‎flink-on-kubernetes/flink/custom-job-ha.yaml‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
################################################################################
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
################################################################################
18+
19+
apiVersion: flink.apache.org/v1beta1
20+
kind: FlinkDeployment
21+
metadata:
22+
name: custom-job-ha
23+
spec:
24+
image: decodable-examples/hello-world-job:1.0
25+
flinkVersion: v1_20
26+
flinkConfiguration:
27+
taskmanager.numberOfTaskSlots: "2"
28+
29+
s3.access.key: minio
30+
s3.secret.key: minio123
31+
s3.endpoint: http://minio-service.default.svc.cluster.local:9000
32+
s3.path.style.access: "true"
33+
s3.entropy.key: _entropy_
34+
s3.entropy.length: "4"
35+
36+
execution.checkpointing.interval: "5000"
37+
38+
state.backend: rocksdb
39+
state.backend.incremental: "true"
40+
state.checkpoints.dir: s3://flink-data/_entropy_/checkpoints
41+
state.savepoints.dir: s3://flink-data/savepoints
42+
43+
high-availability.type: kubernetes
44+
high-availability.storageDir: s3://flink-data/ha
45+
serviceAccount: flink
46+
jobManager:
47+
resource:
48+
memory: "2048m"
49+
cpu: 1
50+
taskManager:
51+
resource:
52+
memory: "2048m"
53+
cpu: 1
54+
podTemplate:
55+
spec:
56+
containers:
57+
- name: flink-main-container
58+
env:
59+
- name: ENABLE_BUILT_IN_PLUGINS
60+
value: "flink-s3-fs-presto-1.20.0.jar"
61+
job:
62+
jarURI: local:///opt/flink-jobs/hello-world-job-1.0.jar
63+
parallelism: 2
64+
upgradeMode: savepoint
65+
state: running
66+
ingress:
67+
template: "localhost/{{name}}(/|$)(.*)"
68+
className: "nginx"
69+
annotations:
70+
nginx.ingress.kubernetes.io/rewrite-target: "/$2"

‎flink-on-kubernetes/flink/custom-job-s3.yaml‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
################################################################################
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
################################################################################
18+
19+
apiVersion: flink.apache.org/v1beta1
20+
kind: FlinkDeployment
21+
metadata:
22+
name: custom-job-s3
23+
spec:
24+
image: decodable-examples/hello-world-job:1.0
25+
flinkVersion: v1_20
26+
flinkConfiguration:
27+
taskmanager.numberOfTaskSlots: "2"
28+
29+
s3.access.key: minio
30+
s3.secret.key: minio123
31+
s3.endpoint: http://minio-service.default.svc.cluster.local:9000
32+
s3.path.style.access: "true"
33+
s3.entropy.key: _entropy_
34+
s3.entropy.length: "4"
35+
36+
execution.checkpointing.interval: "5000"
37+
38+
state.backend: rocksdb
39+
state.backend.incremental: "true"
40+
state.checkpoints.dir: s3://flink-data/_entropy_/checkpoints
41+
state.savepoints.dir: s3://flink-data/savepoints
42+
43+
high-availability.type: kubernetes
44+
high-availability.storageDir: s3://flink-data/ha
45+
serviceAccount: flink
46+
jobManager:
47+
resource:
48+
memory: "2048m"
49+
cpu: 2
50+
taskManager:
51+
resource:
52+
memory: "2048m"
53+
cpu: 2
54+
podTemplate:
55+
spec:
56+
containers:
57+
- name: flink-main-container
58+
env:
59+
- name: ENABLE_BUILT_IN_PLUGINS
60+
value: "flink-s3-fs-presto-1.20.0.jar"
61+
job:
62+
jarURI: s3://job-files/hello-world-job-1.0.jar
63+
parallelism: 2
64+
upgradeMode: savepoint
65+
state: running
66+
ingress:
67+
template: "localhost/{{name}}(/|$)(.*)"
68+
className: "nginx"
69+
annotations:
70+
nginx.ingress.kubernetes.io/rewrite-target: "/$2"

‎flink-on-kubernetes/flink/custom-job.yaml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
################################################################################
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
################################################################################
18+
19+
apiVersion: flink.apache.org/v1beta1
20+
kind: FlinkDeployment
21+
metadata:
22+
name: custom-job
23+
spec:
24+
image: decodable-examples/hello-world-job:1.0
25+
flinkVersion: v1_20
26+
flinkConfiguration:
27+
taskmanager.numberOfTaskSlots: "2"
28+
serviceAccount: flink
29+
jobManager:
30+
resource:
31+
memory: "2048m"
32+
cpu: 1
33+
taskManager:
34+
resource:
35+
memory: "2048m"
36+
cpu: 1
37+
job:
38+
jarURI: local:///opt/flink-jobs/hello-world-job-1.0.jar
39+
parallelism: 1
40+
upgradeMode: stateless
41+
ingress:
42+
template: "localhost/{{name}}(/|$)(.*)"
43+
className: "nginx"
44+
annotations:
45+
nginx.ingress.kubernetes.io/rewrite-target: "/$2"

‎flink-on-kubernetes/flink/logging-job.yaml‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
################################################################################
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
################################################################################
18+
19+
apiVersion: flink.apache.org/v1beta1
20+
kind: FlinkDeployment
21+
metadata:
22+
name: logging-job
23+
spec:
24+
image: decodable-examples/hello-world-job:1.0
25+
flinkVersion: v1_20
26+
flinkConfiguration:
27+
taskmanager.numberOfTaskSlots: "2"
28+
serviceAccount: flink
29+
jobManager:
30+
resource:
31+
memory: "2048m"
32+
cpu: 1
33+
taskManager:
34+
resource:
35+
memory: "2048m"
36+
cpu: 1
37+
job:
38+
jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
39+
parallelism: 2
40+
upgradeMode: stateless
41+
logConfiguration:
42+
log4j-console.properties: |+
43+
################################################################################
44+
# Licensed to the Apache Software Foundation (ASF) under one
45+
# or more contributor license agreements. See the NOTICE file
46+
# distributed with this work for additional information
47+
# regarding copyright ownership. The ASF licenses this file
48+
# to you under the Apache License, Version 2.0 (the
49+
# "License"); you may not use this file except in compliance
50+
# with the License. You may obtain a copy of the License at
51+
#
52+
# http://www.apache.org/licenses/LICENSE-2.0
53+
#
54+
# Unless required by applicable law or agreed to in writing, software
55+
# distributed under the License is distributed on an "AS IS" BASIS,
56+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
57+
# See the License for the specific language governing permissions and
58+
# limitations under the License.
59+
################################################################################
60+
61+
# This affects logging for both user code and Flink
62+
rootLogger.level = INFO
63+
rootLogger.appenderRef.console.ref = ConsoleAppender
64+
rootLogger.appenderRef.rolling.ref = RollingFileAppender
65+
66+
# Uncomment this if you want to _only_ change Flink's logging
67+
#logger.flink.name = org.apache.flink
68+
#logger.flink.level = INFO
69+
70+
# The following lines keep the log level of common libraries/connectors on
71+
# log level INFO. The root logger does not override this. You have to manually
72+
# change the log levels here.
73+
logger.akka.name = akka
74+
logger.akka.level = INFO
75+
logger.kafka.name= org.apache.kafka
76+
logger.kafka.level = INFO
77+
logger.hadoop.name = org.apache.hadoop
78+
logger.hadoop.level = INFO
79+
logger.zookeeper.name = org.apache.zookeeper
80+
logger.zookeeper.level = INFO
81+
82+
# Log all infos to the console
83+
appender.console.name = ConsoleAppender
84+
appender.console.type = CONSOLE
85+
appender.console.layout.type = JsonTemplateLayout
86+
87+
# Log all infos in the given rolling file
88+
appender.rolling.name = RollingFileAppender
89+
appender.rolling.type = RollingFile
90+
appender.rolling.append = false
91+
appender.rolling.fileName = ${sys:log.file}
92+
appender.rolling.filePattern = ${sys:log.file}.%i
93+
appender.rolling.layout.type = PatternLayout
94+
appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n
95+
appender.rolling.policies.type = Policies
96+
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
97+
appender.rolling.policies.size.size=100MB
98+
appender.rolling.strategy.type = DefaultRolloverStrategy
99+
appender.rolling.strategy.max = 10
100+
101+
# Suppress the irrelevant (wrong) warnings from the Netty channel handler
102+
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
103+
logger.netty.level = OFF
104+
105+
# The monitor interval in seconds to enable log4j automatic reconfiguration
106+
# monitorInterval = 30
107+
ingress:
108+
template: "localhost/{{name}}(/|$)(.*)"
109+
className: "nginx"
110+
annotations:
111+
nginx.ingress.kubernetes.io/rewrite-target: "/$2"

‎flink-on-kubernetes/flink/savepoint.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
apiVersion: flink.apache.org/v1beta1
2+
kind: FlinkStateSnapshot
3+
metadata:
4+
name: example-savepoint
5+
spec:
6+
backoffLimit: 1
7+
jobReference:
8+
kind: FlinkDeployment
9+
name: custom-job-ha
10+
savepoint: {}

0 commit comments

Comments
 (0)