Skip to content

[FLINK-15794][Kubernetes] Generate the Kubernetes default image version #11245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
nielsbasjes wants to merge 2 commits into from
Closed

[FLINK-15794][Kubernetes] Generate the Kubernetes default image version #11245

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bb6a970
[FLINK-15794][Kubernetes] Generate the Kubernetes default image version
nielsbasjes May 28, 2020
4e43566
fixup! [FLINK-15794][Kubernetes] Generate the Kubernetes default imag…
azagrebin Jun 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/_includes/generated/kubernetes_config_configuration.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,9 @@
</tr>
<tr>
<td><h5>kubernetes.container.image</h5></td>
<td style="word-wrap: break-word;">"flink:latest"</td>
<td style="word-wrap: break-word;">The default value depends on the actually running version. In general it looks like "flink:&lt;FLINK_VERSION&gt;-scala_&lt;SCALA_VERSION&gt;"</td>
<td>String</td>
<td>Image to use for Flink containers.</td>
<td>Image to use for Flink containers. The specified image must be based upon the same Apache Flink and Scala versions as used by the application. Visit https://hub.docker.com/_/flink?tab=tags for the images provided by the Flink project.</td>
</tr>
<tr>
<td><h5>kubernetes.container.image.pull-policy</h5></td>
Expand Down
5 changes: 5 additions & 0 deletions docs/ops/deployment/docker.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,11 @@ For example, you can use the following aliases:
* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`

<span class="label label-info">Note</span> It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
versions (for example `flink:1.11-scala_2.12`).
This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
from the versions provided by the docker image.

<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
Expand Down
5 changes: 5 additions & 0 deletions docs/ops/deployment/docker.zh.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,11 @@ For example, you can use the following aliases:
* `flink:latest` → `flink:<latest-flink>-scala_<latest-scala>`
* `flink:1.11` → `flink:1.11.<latest-flink-1.11>-scala_2.11`

<span class="label label-info">Note</span>It is recommended to always use an explicit version tag of the docker image that specifies both the needed Flink and Scala
versions (for example `flink:1.11-scala_2.12`).
This will avoid some class conflicts that can occur if the Flink and/or Scala versions used in the application are different
from the versions provided by the docker image.

<span class="label label-info">Note</span> Prior to Flink 1.5 version, Hadoop dependencies were always bundled with Flink.
You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop28`).
Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink
Expand Down
8 changes: 4 additions & 4 deletions docs/ops/deployment/kubernetes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,7 @@ spec:
spec:
containers:
- name: jobmanager
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %}
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest # The 'latest' tag contains the latest released version of Flink for a specific Scala version. Do not use the 'latest' tag in production as it will break your setup automatically when a new version is released.{% endif %}
args: ["jobmanager"]
ports:
- containerPort: 6123
Expand Down Expand Up @@ -314,7 +314,7 @@ spec:
spec:
containers:
- name: taskmanager
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %}
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest # The 'latest' tag contains the latest released version of Flink for a specific Scala version. Do not use the 'latest' tag in production as it will break your setup automatically when a new version is released.{% endif %}
args: ["taskmanager"]
ports:
- containerPort: 6122
Expand Down Expand Up @@ -363,7 +363,7 @@ spec:
restartPolicy: OnFailure
containers:
- name: jobmanager
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %}
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest # The 'latest' tag contains the latest released version of Flink for a specific Scala version. Do not use the 'latest' tag in production as it will break your setup automatically when a new version is released.{% endif %}
env:
args: ["standalone-job", "--job-classname", "com.job.ClassName", ["--job-id", "<job id>",] ["--fromSavepoint", "/path/to/savepoint", ["--allowNonRestoredState",]] [job arguments]]
ports:
Expand Down Expand Up @@ -421,7 +421,7 @@ spec:
spec:
containers:
- name: taskmanager
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %}
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest # The 'latest' tag contains the latest released version of Flink for a specific Scala version. Do not use the 'latest' tag in production as it will break your setup automatically when a new version is released.{% endif %}
env:
args: ["taskmanager"]
ports:
Expand Down
8 changes: 4 additions & 4 deletions docs/ops/deployment/kubernetes.zh.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,7 @@ spec:
spec:
containers:
- name: jobmanager
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %}
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest # The 'latest' tag contains the latest released version of Flink for a specific Scala version. Do not use the 'latest' tag in production as it will break your setup automatically when a new version is released.{% endif %}
args: ["jobmanager"]
ports:
- containerPort: 6123
Expand Down Expand Up @@ -314,7 +314,7 @@ spec:
spec:
containers:
- name: taskmanager
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %}
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest # The 'latest' tag contains the latest released version of Flink for a specific Scala version. Do not use the 'latest' tag in production as it will break your setup automatically when a new version is released.{% endif %}
args: ["taskmanager"]
ports:
- containerPort: 6122
Expand Down Expand Up @@ -363,7 +363,7 @@ spec:
restartPolicy: OnFailure
containers:
- name: jobmanager
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %}
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest # The 'latest' tag contains the latest released version of Flink for a specific Scala version. Do not use the 'latest' tag in production as it will break your setup automatically when a new version is released.{% endif %}
env:
args: ["standalone-job", "--job-classname", "com.job.ClassName", ["--job-id", "<job id>",] ["--fromSavepoint", "/path/to/savepoint", ["--allowNonRestoredState",]] [job arguments]]
ports:
Expand Down Expand Up @@ -421,7 +421,7 @@ spec:
spec:
containers:
- name: taskmanager
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %}
image: flink:{% if site.is_stable %}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest # The 'latest' tag contains the latest released version of Flink for a specific Scala version. Do not use the 'latest' tag in production as it will break your setup automatically when a new version is released.{% endif %}
env:
args: ["taskmanager"]
ports:
Expand Down
17 changes: 15 additions & 2 deletions ...etes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesConfigOptions.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,13 @@
package org.apache.flink.kubernetes.configuration;

import org.apache.flink.annotation.PublicEvolving;
import org.apache.flink.annotation.docs.Documentation;
import org.apache.flink.configuration.ConfigOption;
import org.apache.flink.configuration.ExternalResourceOptions;
import org.apache.flink.runtime.util.EnvironmentInformation;

import java.util.List;
import java.util.Locale;
import java.util.Map;

import static org.apache.flink.configuration.ConfigOptions.key;
Expand Down Expand Up @@ -137,11 +140,14 @@ public class KubernetesConfigOptions {
.withDescription("The cluster-id, which should be no more than 45 characters, is used for identifying " +
"a unique Flink cluster. If not set, the client will automatically generate it with a random ID.");

@Documentation.OverrideDefault("The default value depends on the actually running version. In general it looks like \"flink:<FLINK_VERSION>-scala_<SCALA_VERSION>\"")
public static final ConfigOption<String> CONTAINER_IMAGE =
key("kubernetes.container.image")
.stringType()
.defaultValue("flink:latest")
.withDescription("Image to use for Flink containers.");
.defaultValue(getDefaultFlinkImage())
.withDescription("Image to use for Flink containers. " +
"The specified image must be based upon the same Apache Flink and Scala versions as used by the application. " +
"Visit https://hub.docker.com/_/flink?tab=tags for the images provided by the Flink project.");

/**
* The following config options need to be set according to the image.
Expand Down Expand Up @@ -229,6 +235,13 @@ public class KubernetesConfigOptions {
.withDescription("If configured, Flink will add \"resources.limits.<config-key>\" and \"resources.requests.<config-key>\" " +
"to the main container of TaskExecutor and set the value to the value of " + ExternalResourceOptions.EXTERNAL_RESOURCE_AMOUNT.key() + ".");

private static String getDefaultFlinkImage() {
// The default container image that ties to the exact needed versions of both Flink and Scala.
boolean snapshot = EnvironmentInformation.getVersion().toLowerCase(Locale.ENGLISH).contains("snapshot");
String tag = snapshot ? "latest" : EnvironmentInformation.getVersion() + "-scala_" + EnvironmentInformation.getScalaVersion();
return "flink:" + tag;
}

/**
* The flink rest service exposed type.
*/
Expand Down