diff --git a/docs/content/docs/deployment/overview.md b/docs/content/docs/deployment/overview.md index 2442cc4da3ef6..d77fb34abce87 100644 --- a/docs/content/docs/deployment/overview.md +++ b/docs/content/docs/deployment/overview.md @@ -170,7 +170,7 @@ Flink can execute applications in one of three ways: {{< img class="img-fluid" width="80%" style="margin: 15px" src="/fig/deployment_modes.svg" alt="Figure for Deployment Modes" >}} -#### Application Mode +### Application Mode In all the other modes, the application's `main()` method is executed on the client side. This process includes downloading the application's dependencies locally, executing the `main()` to extract a representation @@ -180,18 +180,21 @@ network bandwidth to download dependencies and ship binaries to the cluster, and `main()`. This problem can be more pronounced when the Client is shared across users. Building on this observation, the *Application Mode* creates a cluster per submitted application, but this time, -the `main()` method of the application is executed on the JobManager. Creating a cluster per application can be +the `main()` method of the application is executed by the *JobManager*. Creating a cluster per application can be seen as creating a session cluster shared only among the jobs of a particular application, and torn down when the application finishes. With this architecture, the *Application Mode* provides the same resource isolation -and load balancing guarantees as the *Per-Job* mode, but at the granularity of a whole application. Executing -the `main()` on the JobManager allows for saving the CPU cycles required, but also save the bandwidth required -for downloading the dependencies locally. Furthermore, it allows for more even spread of the network load for -downloading the dependencies of the applications in the cluster, as there is one JobManager per application. +and load balancing guarantees as the *Per-Job* mode, but at the granularity of a whole application. + +The *Application Mode* builds on an assumption that the user jars are already available on the classpath (`usrlib` folder) +of all Flink components that needs access to it (*JobManager*, *TaskManager*). In other words, your application comes +bundled with the Flink distribution. This allows the application mode to speed up the deployment / recovery process, by +not having to distribute the user jars to the Flink components via RPC as the other deployment modes do. {{< hint info >}} -In the Application Mode, the `main()` is executed on the cluster and not on the client, -as in the other modes. This may have implications for your code as, for example, any paths you register in -your environment using the `registerCachedFile()` must be accessible by the JobManager of your application. +The application mode assumes that the user jars are bundled with the Flink distribution. + +Executing the `main()` method on the cluster may have other implications for your code, such as any paths you register +in your environment using the `registerCachedFile()` must be accessible by the JobManager of your application. {{< /hint >}} Compared to the *Per-Job* mode, the *Application Mode* allows the submission of applications consisting of @@ -210,7 +213,7 @@ Additionally, when any of multiple running jobs in Application Mode (submitted f Regular job completions (by the sources shutting down) are supported. {{< /hint >}} -#### Per-Job Mode +### Per-Job Mode Aiming at providing better resource isolation guarantees, the *Per-Job* mode uses the available resource provider framework (e.g. YARN, Kubernetes) to spin up a cluster for each submitted job. This cluster is available to @@ -220,7 +223,7 @@ TaskManagers. In addition, it spreads the load of book-keeping across multiple J one per job. For these reasons, the *Per-Job* resource allocation model is the preferred mode by many production reasons. -#### Session Mode +### Session Mode *Session mode* assumes an already running cluster and uses the resources of that cluster to execute any submitted application. Applications executed in the same (session) cluster use, and consequently compete @@ -233,7 +236,7 @@ Additionally, having a single cluster running multiple jobs implies more load fo is responsible for the book-keeping of all the jobs in the cluster. -#### Summary +### Summary In *Session Mode*, the cluster lifecycle is independent of that of any job running on the cluster and the resources are shared across all jobs. The *Per-Job* mode pays the price of spinning up a cluster diff --git a/docs/content/docs/deployment/resource-providers/native_kubernetes.md b/docs/content/docs/deployment/resource-providers/native_kubernetes.md index 5a05d8d8b7d93..34b953f946101 100644 --- a/docs/content/docs/deployment/resource-providers/native_kubernetes.md +++ b/docs/content/docs/deployment/resource-providers/native_kubernetes.md @@ -83,6 +83,10 @@ For production use, we recommend deploying Flink Applications in the [Applicatio ### Application Mode +{{< hint info >}} +For high-level intuition behind the application mode, please refer to the [deployment mode overview]({{< ref "docs/deployment/overview#application-mode" >}}). +{{< /hint >}} + The [Application Mode]({{< ref "docs/deployment/overview" >}}#application-mode) requires that the user code is bundled together with the Flink image because it runs the user code's `main()` method on the cluster. The Application Mode makes sure that all Flink components are properly cleaned up after the termination of the application. @@ -122,12 +126,20 @@ $ ./bin/flink cancel --target kubernetes-application -Dkubernetes.cluster-id=my- You can override configurations set in `conf/flink-conf.yaml` by passing key-value pairs `-Dkey=value` to `bin/flink`. -### Per-Job Cluster Mode +### Per-Job Mode + +{{< hint info >}} +For high-level intuition behind the per-job mode, please refer to the [deployment mode overview]({{< ref "docs/deployment/overview#per-job-mode" >}}). +{{< /hint >}} Flink on Kubernetes does not support Per-Job Cluster Mode. ### Session Mode +{{< hint info >}} +For high-level intuition behind the session mode, please refer to the [deployment mode overview]({{< ref "docs/deployment/overview#session-mode" >}}). +{{< /hint >}} + You have seen the deployment of a Session cluster in the [Getting Started](#getting-started) guide at the top of this page. The Session Mode can be executed in two modes: diff --git a/docs/content/docs/deployment/resource-providers/standalone/docker.md b/docs/content/docs/deployment/resource-providers/standalone/docker.md index 07907fddbaae4..e55e5a7269284 100644 --- a/docs/content/docs/deployment/resource-providers/standalone/docker.md +++ b/docs/content/docs/deployment/resource-providers/standalone/docker.md @@ -108,7 +108,11 @@ Once you've started Flink on Docker, you can access the Flink Webfrontend on [lo We recommend using [Docker Compose](#flink-with-docker-compose) or [Docker Swarm](#flink-with-docker-swarm) for deploying Flink in Session Mode to ease system configuration. -### Application Mode on Docker +### Application Mode + +{{< hint info >}} +For high-level intuition behind the application mode, please refer to the [deployment mode overview]({{< ref "docs/deployment/overview#application-mode" >}}). +{{< /hint >}} A *Flink Application cluster* is a dedicated cluster which runs a single job. In this case, you deploy the cluster with the job as one step, thus, there is no extra job submission needed. @@ -198,15 +202,22 @@ You can provide the following additional command line arguments to the cluster e If the main function of the user job main class accepts arguments, you can also pass them at the end of the `docker run` command. -### Per-Job Mode on Docker +### Per-Job Mode + +{{< hint info >}} +For high-level intuition behind the per-job mode, please refer to the [deployment mode overview]({{< ref "docs/deployment/overview#per-job-mode" >}}). +{{< /hint >}} + +Per-Job Mode is not supported by Flink on Docker. -[Per-Job Mode]({{< ref "docs/deployment/overview" >}}#per-job-mode) is not supported by Flink on Docker. +### Session Mode -### Session Mode on Docker +{{< hint info >}} +For high-level intuition behind the session mode, please refer to the [deployment mode overview]({{< ref "docs/deployment/overview#session-mode" >}}). +{{< /hint >}} Local deployment in the Session Mode has already been described in the [Getting Started](#starting-a-session-cluster-on-docker) section above. - {{< top >}} ## Flink on Docker Reference diff --git a/docs/content/docs/deployment/resource-providers/standalone/kubernetes.md b/docs/content/docs/deployment/resource-providers/standalone/kubernetes.md index c81b7fa684e42..17c7926d2809d 100644 --- a/docs/content/docs/deployment/resource-providers/standalone/kubernetes.md +++ b/docs/content/docs/deployment/resource-providers/standalone/kubernetes.md @@ -93,7 +93,11 @@ You can tear down the cluster using the following commands: ## Deployment Modes -### Deploy Application Cluster +### Application Mode + +{{< hint info >}} +For high-level intuition behind the application mode, please refer to the [deployment mode overview]({{< ref "docs/deployment/overview#application-mode" >}}). +{{< /hint >}} A *Flink Application cluster* is a dedicated cluster which runs a single application, which needs to be available at deployment time. @@ -129,11 +133,20 @@ with the `kubectl` command: $ kubectl delete -f jobmanager-job.yaml ``` -### Per-Job Cluster Mode +### Per-Job Mode + +{{< hint info >}} +For high-level intuition behind the per-job mode, please refer to the [deployment mode overview]({{< ref "docs/deployment/overview#per-job-mode" >}}). +{{< /hint >}} + Flink on Standalone Kubernetes does not support the Per-Job Cluster Mode. ### Session Mode +{{< hint info >}} +For high-level intuition behind the session mode, please refer to the [deployment mode overview]({{< ref "docs/deployment/overview#session-mode" >}}). +{{< /hint >}} + Deployment of a Session cluster is explained in the [Getting Started](#getting-started) guide at the top of this page. {{< top >}} diff --git a/docs/content/docs/deployment/resource-providers/standalone/overview.md b/docs/content/docs/deployment/resource-providers/standalone/overview.md index 4ccab51e32880..44136a033a902 100644 --- a/docs/content/docs/deployment/resource-providers/standalone/overview.md +++ b/docs/content/docs/deployment/resource-providers/standalone/overview.md @@ -77,6 +77,10 @@ In step `(3)`, we are starting a Flink Client (a short-lived JVM process) that s ### Application Mode +{{< hint info >}} +For high-level intuition behind the application mode, please refer to the [deployment mode overview]({{< ref "docs/deployment/overview#application-mode" >}}). +{{< /hint >}} + To start a Flink JobManager with an embedded application, we use the `bin/standalone-job.sh` script. We demonstrate this mode by locally starting the `TopSpeedWindowing.jar` example, running on a single TaskManager. @@ -110,10 +114,18 @@ $ ./bin/standalone-job.sh stop ### Per-Job Mode +{{< hint info >}} +For high-level intuition behind the per-job mode, please refer to the [deployment mode overview]({{< ref "docs/deployment/overview#per-job-mode" >}}). +{{< /hint >}} + Per-Job Mode is not supported by the Standalone Cluster. ### Session Mode +{{< hint info >}} +For high-level intuition behind the session mode, please refer to the [deployment mode overview]({{< ref "docs/deployment/overview#session-mode" >}}). +{{< /hint >}} + Local deployment in Session Mode has already been described in the [introduction](#starting-a-standalone-cluster-session-mode) above. ## Standalone Cluster Reference diff --git a/docs/content/docs/deployment/resource-providers/yarn.md b/docs/content/docs/deployment/resource-providers/yarn.md index 25bedd49d8906..2426d9549674f 100644 --- a/docs/content/docs/deployment/resource-providers/yarn.md +++ b/docs/content/docs/deployment/resource-providers/yarn.md @@ -87,6 +87,10 @@ For production use, we recommend deploying Flink Applications in the [Per-job or ### Application Mode +{{< hint info >}} +For high-level intuition behind the application mode, please refer to the [deployment mode overview]({{< ref "docs/deployment/overview#application-mode" >}}). +{{< /hint >}} + Application Mode will launch a Flink cluster on YARN, where the main() method of the application jar gets executed on the JobManager in YARN. The cluster will shut down as soon as the application has finished. You can manually stop the cluster using `yarn application -kill ` or by cancelling the Flink job. @@ -120,7 +124,11 @@ The above will allow the job submission to be extra lightweight as the needed Fl are going to be picked up by the specified remote locations rather than be shipped to the cluster by the client. -### Per-Job Cluster Mode +### Per-Job Mode + +{{< hint info >}} +For high-level intuition behind the per-job mode, please refer to the [deployment mode overview]({{< ref "docs/deployment/overview#per-job-mode" >}}). +{{< /hint >}} The Per-job Cluster mode will launch a Flink cluster on YARN, then run the provided application jar locally and finally submit the JobGraph to the JobManager on YARN. If you pass the `--detached` argument, the client will stop once the submission is accepted. @@ -144,6 +152,10 @@ Note that cancelling your job on an Per-Job Cluster will stop the cluster. ### Session Mode +{{< hint info >}} +For high-level intuition behind the session mode, please refer to the [deployment mode overview]({{< ref "docs/deployment/overview#session-mode" >}}). +{{< /hint >}} + We describe deployment with the Session Mode in the [Getting Started](#getting-started) guide at the top of the page. The Session Mode has two operation modes: