From 6386983239bd3024b395c865ec4fd33e232ca5a3 Mon Sep 17 00:00:00 2001 From: Bowen Li Date: Wed, 30 Aug 2017 09:35:03 -0700 Subject: [PATCH 1/3] FLINK-7422 Upgrade Kinesis Client Library (KCL) and AWS SDK in flink-connector-kinesis --- flink-connectors/flink-connector-kinesis/pom.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/flink-connectors/flink-connector-kinesis/pom.xml b/flink-connectors/flink-connector-kinesis/pom.xml index c5a91b178c2a09..83934f64608be5 100644 --- a/flink-connectors/flink-connector-kinesis/pom.xml +++ b/flink-connectors/flink-connector-kinesis/pom.xml @@ -33,8 +33,8 @@ under the License. flink-connector-kinesis_${scala.binary.version} flink-connector-kinesis - 1.10.71 - 1.6.2 + 1.11.171 + 1.8.1 0.12.5 From ae4b5647e25d6e6915486c6ac4a42e887a53101c Mon Sep 17 00:00:00 2001 From: Bowen Li Date: Tue, 19 Sep 2017 10:44:16 -0700 Subject: [PATCH 2/3] FLINK-7603 Document how to take a savepoint on YARN --- docs/ops/cli.md | 21 +++++++++++++++++---- docs/ops/state/savepoints.md | 18 +++++++++++++++--- 2 files changed, 32 insertions(+), 7 deletions(-) diff --git a/docs/ops/cli.md b/docs/ops/cli.md index 7b36177223f394..8fa4536b16084a 100644 --- a/docs/ops/cli.md +++ b/docs/ops/cli.md @@ -138,17 +138,30 @@ This allows the job to finish processing all inflight data. [Savepoints]({{site.baseurl}}/ops/state/savepoints.html) are controlled via the command line client: -#### Trigger a savepoint +#### Trigger a Savepoint {% highlight bash %} ./bin/flink savepoint [savepointDirectory] {% endhighlight %} -Returns the path of the created savepoint. You need this path to restore and dispose savepoints. +This will trigger a savepoint for the job with ID `jobId`, and returns the path of the created savepoint. You need this path to restore and dispose savepoints. -You can optionally specify a `savepointDirectory` when triggering the savepoint. If you don't specify one here, you need to configure a default savepoint directory for the Flink installation (see [Savepoints]({{site.baseurl}}/ops/state/savepoints.html#configuration)). -##### Cancel with a savepoint +Furthermore, you can optionally specify a target file system directory to store the savepoint in. The directory needs to be accessible by the JobManager. + +If you don't specify a target directory, you need to have [configured a default directory](#configuration) (see [Savepoints]({{site.baseurl}}/ops/state/savepoints.html#configuration)). Otherwise, triggering the savepoint will fail. + +#### Trigger a Savepoint with YARN + +{% highlight bash %} +./bin/flink savepoint [savepointDirectory] -yid +{% endhighlight %} + +This will trigger a savepoint for the job with ID `jobId` and YARN application ID `yarnAppId`, and returns the path of the created savepoint. + +Everything else is the same as described in the above **Trigger a Savepoint** section. + +#### Cancel with a savepoint You can atomically trigger a savepoint and cancel a job. diff --git a/docs/ops/state/savepoints.md b/docs/ops/state/savepoints.md index 1d82d2b10d3e56..6029cc28086051 100644 --- a/docs/ops/state/savepoints.md +++ b/docs/ops/state/savepoints.md @@ -103,12 +103,24 @@ Note that if you use the `MemoryStateBackend`, metadata *and* savepoint state wi #### Trigger a Savepoint ```sh -$ bin/flink savepoint :jobId [:targetDirectory] +$ bin/flink savepoint :jobId [:savepointDirectory] ``` -This will trigger a savepoint for the job with ID `:jobid`. Furthermore, you can specify a target file system directory to store the savepoint in. The directory needs to be accessible by the JobManager. +This will trigger a savepoint for the job with ID `:jobId`, and returns the path of the created savepoint. You need this path to restore and dispose savepoints. -If you don't specify a target directory, you need to have [configured a default directory](#configuration). Otherwise, triggering the savepoint will fail. +Furthermore, you can optionally specify a target file system directory to store the savepoint in. The directory needs to be accessible by the JobManager. + +If you don't specify a target directory, you need to have [configured a default directory](#configuration) (see [Savepoints]({{site.baseurl}}/ops/state/savepoints.html#configuration)). Otherwise, triggering the savepoint will fail. + +#### Trigger a Savepoint with YARN + +```sh +$ bin/flink savepoint :jobId [:savepointDirectory] -yid :yarnAppId +``` + +This will trigger a savepoint for the job with ID `:jobId` and YARN application ID `:yarnAppId`, and returns the path of the created savepoint. + +Everything else is the same as described in the above **Trigger a Savepoint** section. #### Cancel Job with Savepoint From f82db156e6cabab9f43b7c3ec658b0ddc1a0637c Mon Sep 17 00:00:00 2001 From: Bowen Li Date: Tue, 19 Sep 2017 10:48:48 -0700 Subject: [PATCH 3/3] update doc --- docs/ops/cli.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/ops/cli.md b/docs/ops/cli.md index 8fa4536b16084a..99c7c9a6928957 100644 --- a/docs/ops/cli.md +++ b/docs/ops/cli.md @@ -141,7 +141,7 @@ This allows the job to finish processing all inflight data. #### Trigger a Savepoint {% highlight bash %} -./bin/flink savepoint [savepointDirectory] +./bin/flink savepoint [savepointDirectory] {% endhighlight %} This will trigger a savepoint for the job with ID `jobId`, and returns the path of the created savepoint. You need this path to restore and dispose savepoints. @@ -154,7 +154,7 @@ If you don't specify a target directory, you need to have [configured a default #### Trigger a Savepoint with YARN {% highlight bash %} -./bin/flink savepoint [savepointDirectory] -yid +./bin/flink savepoint [savepointDirectory] -yid {% endhighlight %} This will trigger a savepoint for the job with ID `jobId` and YARN application ID `yarnAppId`, and returns the path of the created savepoint.