[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide #10999

azagrebin · 2020-02-03T13:30:51Z

What is the purpose of the change

In release 1.10, with FLIP-49, we introduced significant changes to the TaskExecutor memory model and it's related configuration options / logics.

It is very important that we clearly state the changes and potential effects, and guide our users to tune their clusters with the new configuration for both new setups and migrations of previous setups.

Brief change log

Add memory model description and configuration guide
Add tuning guide for various features
Add troubleshooting guide
Add migration guide from pre-FLIP-49 configuration

Verifying this change

cd flink/docs/docker
./run.sh
./build_docs.sh -p

open localhost:4000

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (docs)

azagrebin · 2020-02-03T13:32:39Z

I will duplicate files for the later Chinese translation before merging and reference old model (#11004).
cc @xintongsong @sjwiesman

flinkbot · 2020-02-03T13:34:16Z

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 3ed9750 (Mon Feb 03 13:34:16 UTC 2020)

✅no warnings

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

❓ 1. The [description] looks good.
❓ 2. There is [consensus] that the contribution should go into to Flink.
❓ 3. Needs [attention] from.
❓ 4. The change fits into the overall [architecture].
❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
@flinkbot approve all to approve all aspects
@flinkbot approve-until architecture to approve everything until architecture
@flinkbot attention @username1 [@username2 ..] to require somebody's attention
@flinkbot disapprove architecture to remove an approval you gave earlier

flinkbot · 2020-02-03T14:11:43Z

CI report:

3ed9750 Travis: PENDING Azure: FAILURE
83e8781 UNKNOWN
a38bb3f UNKNOWN
f2ec29a Travis: FAILURE Azure: FAILURE
94175b9 Travis: SUCCESS Azure: FAILURE
09865b1 UNKNOWN
4ff8243 Travis: SUCCESS Azure: FAILURE
1dafbfa Travis: FAILURE Azure: FAILURE
bbc0d2c Travis: SUCCESS Azure: FAILURE
b7df600 Travis: SUCCESS Azure: SUCCESS
1178014 UNKNOWN
16eeb75 Travis: SUCCESS Azure: SUCCESS
e34dafa Travis: FAILURE Azure: FAILURE
bac3201 Travis: SUCCESS Azure: FAILURE
0cb8834 UNKNOWN

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run travis re-run the last Travis build
@flinkbot run azure re-run the last Azure build

tillrohrmann

Thanks for the PR @azagrebin. I finished the first part of my review (general documentation) and had some comments. I will continue with the remaining parts.

docs/ops/memory/mem_setup.md

tillrohrmann · 2020-02-03T14:28:31Z

docs/ops/memory/mem_setup.md

+The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>


Inconsistent casing. Sometimes Task Executor is written with capital letters and sometimes with lower case letters. I'd suggest to stick to one way.

Here I wanted to emphasise that it is for TM not JM. I can try to use closer to TM.

tillrohrmann · 2020-02-03T14:28:44Z

docs/ops/memory/mem_setup.md

+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.


Same for "Job Manager"

tillrohrmann · 2020-02-03T14:29:27Z

docs/ops/memory/mem_setup.md

+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note: This memory setup guide is relevant only for Task Executors!</strong>
+Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.


Suggested change

Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.

Check [Job Manager related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.

docs/ops/memory/mem_setup.md

tillrohrmann · 2020-02-03T14:58:55Z

docs/ops/memory/mem_setup.md

+a bigger JVM limit in this case.
+
+<strong>Note:</strong> The *network memory* is also part of JVM *direct memory* but it is managed by Flink and guaranteed
+that it is always allocated within its configured size. Therefore, resizing network memory will not necessarily help in this situation.


Suggested change

that it is always allocated within its configured size. Therefore, resizing network memory will not necessarily help in this situation.

to never exceed its configured size. Therefore, resizing network memory will not necessarily help in this situation.

tillrohrmann · 2020-02-03T14:59:35Z

docs/ops/memory/mem_setup.md

+
+See also [detailed Memory Model](#detailed-memory-model).
+
+## Detailed Memory Model


I would suggest to make this a separate page. This will shorten the page a bit and make it easier to digest for the reader.

sjwiesman · 2020-02-03T17:46:15Z

docs/ops/memory/mem_migration.md

+under the License.
+-->
+
+* toc


nit: Can you move the TOC below the first paragraph, I think it looks nicer

sjwiesman · 2020-02-03T17:47:28Z

docs/ops/memory/mem_tuning.md

+-->
+
+* toc
+{:toc}


Again, please move to line 30 below the first paragraph

docs/ops/memory/index.md

sjwiesman · 2020-02-03T18:03:48Z

docs/ops/memory/mem_tuning.md

+[Managed memory](../memory/mem_setup.html#managed-memory) helps Flink to run the batch operators efficiently.
+Therefore, the size of the [managed memory](mem_setup.html#managed-memory) can affect the performance of [batch jobs](../../dev/batch).
+If it makes sense, Flink will try to allocate and use as much *managed memory* as configured for batch jobs but not beyond its limit.
+It prevents OutOfMemoryExceptions because Flink knows how much memory it can use to execute operations.
+If Flink runs out of [managed memory](../memory/mem_setup.html#managed-memory), it utilizes disk space.
+Using [managed memory](../memory/mem_setup.html#managed-memory), some operations can be performed directly
+on the raw data without having to deserialize the data to convert it into Java objects. All in all,
+[managed memory](../memory/mem_setup.html#managed-memory) improves the robustness and speed of the system.


Flink's batch operators leverage managed memory to run more efficiently. In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects. This means that managed memory configurations have practical effects on the performance of your applications. Flink will attempt to allocate and use as much managed memory as configured for batch jobs but not go beyond its limits. This prevents OutOfMemoryException's because Flink knows precisely how much memory it has to leverage. If the managed memory is not sufficient, Flink will gracefully spill to disk.

docs/ops/memory/mem_tuning.md

sjwiesman · 2020-02-03T18:08:38Z

@azagrebin I have a few small comments on the placement of the TOC and other minutiae. I'm relying on Till and others to validate this for correctness but overall I think this looks good.

azagrebin · 2020-02-03T19:05:18Z

Thanks for the review @sjwiesman ! I have addressed the comments.

sjwiesman · 2020-02-03T19:13:11Z

ok, please tag me if anything is dramatically changed and you’d like me to check the language otherwise tentative +1

xintongsong

Thanks for opening the PR, @azagrebin.
I've also left some comments, most of them related to the links.
In addition, since state_backends.zh.md is already translated into Chinese, I translated the new added contents in this file in my comments. Maybe @carp84 or @Myasuka can help verify my translation.

xintongsong · 2020-02-04T02:22:58Z

docs/ops/memory/mem_setup.md

+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<strong>Note:</strong> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager) for the memory setup of job manager.


It seems the linked page does not have the anchor #jobmanager. This link jumps to the beginning of the configuration page.

Indeed, the docs have been recently restructured. I will find a new link

xintongsong · 2020-02-04T02:31:09Z

docs/ops/memory/mem_setup.md

+* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
+* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))


These two links do not work for me. Why do we have -1 in the urls?
Is it a mistake or something wrong with my building?

The options, which have -1 at the end of their links, were duplicated in the main configuration page. They were listed as common options and then as TM or TM mem specific. After docs restructure, they got deduplicated. I will remove -1 for them.

xintongsong · 2020-02-04T02:35:21Z

docs/ops/memory/mem_setup.md

+[Here](#detailed-memory-model) are more details about the other memory components.
+
+Configuring *total Flink memory* is better suited for standalone deployments where you want to declare how much memory
+is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.


Suggested change

is given to Flink itself. The *total Flink memory* splits up into [managed memory size](#managed-memory) and JVM heap.

is given to Flink itself. The *total Flink memory* splits up into JVM heap, [managed memory](#managed-memory), and direct memory.

I think it would be better to align the description with the figure above, to avoid potential confusing.

xintongsong · 2020-02-04T02:46:59Z

docs/ops/memory/mem_setup.md

+* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
+* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)


Same problem of '-1' in urls for these two lines.

xintongsong · 2020-02-04T02:48:17Z

docs/ops/memory/mem_setup.md

+This more fine-grained approach is described in more detail [here](#configure-heap-and-managed-memory).
+
+<strong>Note:</strong> One of the three mentioned ways has to be used to configure Flink’s memory (except for local execution), or the Flink startup will fail.
+This means that one of the following option subsets, which do not have default values, have to be configured explicitly in *flink-conf.yaml*:


Not necessarily in flink-conf.yaml. It can also be '-D' parameters.

True, I will just remove in *flink-conf.yaml*

xintongsong · 2020-02-04T05:40:43Z

docs/ops/state/state_backends.md

@@ -74,6 +74,8 @@ The MemoryStateBackend is encouraged for:
  - Local development and debugging
  - Jobs that do hold little state, such as jobs that consist only of record-at-a-time functions (Map, FlatMap, Filter, ...). The Kafka Consumer requires very little state.

+It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero.


Suggested change

It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero.

It is also recommended to set [managed memory](../memory/mem_setup.html#managed-memory) to zero.

xintongsong · 2020-02-04T05:43:38Z

docs/ops/state/state_backends.zh.md

+It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.


Translation:
建议同时将 [managed memory](../memory/mem_setup.html#managed-memory) 设为0，以保证将最大限度的内存分配给 JVM 上的用户代码。

xintongsong · 2020-02-04T05:45:02Z

docs/ops/state/state_backends.md

@@ -92,6 +94,9 @@ The FsStateBackend is encouraged for:
  - Jobs with large state, long windows, large key/value states.
  - All high-availability setups.

+It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero.


Suggested change

It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero.

It is also recommended to set [managed memory](../memory/mem_setup.html#managed-memory) to zero.

xintongsong · 2020-02-04T05:49:19Z

docs/ops/state/state_backends.zh.md

+It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero.
+This will ensure that the maximum amount of memory is allocated for user code on the JVM.


Translation:
建议同时将 [managed memory](../memory/mem_setup.html#managed-memory) 设为0，以保证将最大限度的内存分配给 JVM 上的用户代码。

xintongsong · 2020-02-04T05:53:31Z

docs/ops/state/state_backends.zh.md

@@ -115,6 +120,8 @@ RocksDBStateBackend 的适用场景：
 然而，这也意味着使用 RocksDBStateBackend 将会使应用程序的最大吞吐量降低。
 所有的读写都必须序列化、反序列化操作，这个比基于堆内存的 state backend 的效率要低很多。

+Check also recommendations about the [task executor memory configuration](../memory/mem_tuning.html#rocksdb-state-backend) for the RocksDBStateBackend.


Translation:
请同时参考 [Task Executor 内存配置](../memory/mem_tuning.html#rocksdb-state-backend) 中关于 RocksDBStateBackend 的建议。

xintongsong · 2020-02-04T06:28:17Z

docs/ops/memory/mem_tuning.md

+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
+on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)
+as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryException`'s because Flink knows precisely


The class name is OutOfMemoryError

azagrebin · 2020-02-04T13:29:45Z

Thanks for the review @tillrohrmann
I addressed the comments

xintongsong

Thanks for addressing my comments, @azagrebin. LGTM.

GJL · 2020-02-06T17:24:31Z

docs/ops/memory/mem_tuning.md

+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
+on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)
+as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely


Plural is without ' just s
Maybe it's easier to just write OutOfMemoryErrors

@sjwiesman what's your opinion? is it typo?

GJL · 2020-02-06T17:28:42Z

docs/ops/memory/mem_migration.md

+
+This section describes the changes of the default ‘flink-conf.yaml’ shipped with Flink.
+
+The total memory (‘taskmanager.heap.size’) is replaced by [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)


Why no backticks here?

Indeed, I overlooked old options w/o links

GJL · 2020-02-06T17:29:12Z

docs/ops/memory/mem_migration.md

+This section describes the changes of the default ‘flink-conf.yaml’ shipped with Flink.
+
+The total memory (‘taskmanager.heap.size’) is replaced by [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+in the default ‘flink-conf.yaml’. The value is also increased from 1024Mb to 1568Mb.


Here as well, why no backticks? There are a couple of these.

tillrohrmann

Thanks for updating this PR @azagrebin. I had some additional comments.

tillrohrmann · 2020-02-07T17:38:34Z

docs/ops/memory/mem_setup.md

+The further described memory configuration is applicable starting with the release version *1.10*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.


Nit: In the first sentence we use italics for the release version and in the next sentence we don't use italics.

tillrohrmann · 2020-02-07T17:39:17Z

docs/ops/memory/mem_setup.md

+from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.
+
+<span class="label label-info">Note</span> This memory setup guide is relevant <strong>only for task executors</strong>!
+Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager.


Suggested change

Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of job manager.

Check [job manager related configuration options](../config.html#jobmanager-heap-size) for the memory setup of a job manager.

tillrohrmann · 2020-02-07T17:40:13Z

docs/ops/memory/mem_setup.md

+
+The *total process memory* of Flink JVM processes consists of memory consumed by Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption includes usage of JVM heap,
+*managed memory* (managed by Flink) and other direct (or native) memory.


should direct memory be italic similar to the other components?

tillrohrmann · 2020-02-07T17:40:38Z

docs/ops/memory/mem_setup.md

+<br />
+
+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](mem_detail.html#local-execution) for more details.


Suggested change

options are relevant, see also [local execution](mem_detail.html#local-execution) for more details.

options are relevant. See [local execution](mem_detail.html#local-execution) for more details.

tillrohrmann · 2020-02-07T17:41:00Z

docs/ops/memory/mem_setup.md

+If you run FIink locally (e.g. from your IDE) without creating a cluster, then only a subset of the memory configuration
+options are relevant, see also [local execution](mem_detail.html#local-execution) for more details.
+
+Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:


Suggested change

Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:

Otherwise, the simplest way to setup memory in Flink is to configure either of the following two options:

tillrohrmann · 2020-02-07T23:36:34Z

docs/ops/memory/mem_detail.md

+<br/>
+
+All of the components listed above can be but do not have to be explicitly configured for the local execution.
+If they are not configured they are set to their default values. [Task heap memory](mem_setup.html#task-operator-heap-memory) and


Suggested change

If they are not configured they are set to their default values. [Task heap memory](mem_setup.html#task-operator-heap-memory) and

If they are not configured they are set to their default values. [*Task heap memory*](mem_setup.html#task-operator-heap-memory) and

tillrohrmann · 2020-02-07T23:37:10Z

docs/ops/memory/mem_detail.md

+
+All of the components listed above can be but do not have to be explicitly configured for the local execution.
+If they are not configured they are set to their default values. [Task heap memory](mem_setup.html#task-operator-heap-memory) and
+*task off-heap memory* are considered to be infinite (*Long.MAX_VALUE* bytes) and [managed memory](mem_setup.html#managed-memory)


Suggested change

*task off-heap memory* are considered to be infinite (*Long.MAX_VALUE* bytes) and [managed memory](mem_setup.html#managed-memory)

*task off-heap memory* are considered to be infinite (*Long.MAX_VALUE* bytes) and [*managed memory*](mem_setup.html#managed-memory)

tillrohrmann · 2020-02-07T23:38:18Z

docs/ops/memory/mem_detail.md

+has a default value of 128Mb only for the local execution mode.
+
+<span class="label label-info">Note</span> The task heap size is not related in any way to the real heap size in this case.
+It can become relevant for future optimizations coming with next releases. The actual JVM heap size of the started


Suggested change

It can become relevant for future optimizations coming with next releases. The actual JVM heap size of the started

The actual JVM heap size of the started

tillrohrmann · 2020-02-07T23:38:57Z

docs/ops/memory/mem_detail.md

+<span class="label label-info">Note</span> The task heap size is not related in any way to the real heap size in this case.
+It can become relevant for future optimizations coming with next releases. The actual JVM heap size of the started
+local process is not controlled by Flink and depends on how you start the process.
+If you want to control the JVM heap size you have to explicitly pass the corresponding JVM arguments, e.g. *-Xmx*, *-Xms*.


Suggested change

If you want to control the JVM heap size you have to explicitly pass the corresponding JVM arguments, e.g. *-Xmx*, *-Xms*.

If you want to control the JVM heap size, then you have to explicitly pass the corresponding arguments *-Xmx* and *-Xms* to the JVM.

tillrohrmann · 2020-02-07T23:39:32Z

docs/ops/memory/mem_migration.md

@@ -0,0 +1,234 @@
+---
+title: "Migration from old configuration (before 1.10 release)"


Not consistent capitalization wrt the other titles.

Suggested change

title: "Migration from old configuration (before 1.10 release)"

title: "Migration Guide"

tillrohrmann

Some more comments.

tillrohrmann · 2020-02-08T22:16:23Z

docs/ops/memory/mem_trouble.md

+
+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates


Suggested change

If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates

If you see an `IllegalConfigurationException` thrown from `TaskExecutorProcessUtils`, it usually indicates

tillrohrmann · 2020-02-08T22:16:41Z

docs/ops/memory/mem_tuning.md

@@ -0,0 +1,86 @@
+---
+title: "Memory tuning guide"


Suggested change

title: "Memory tuning guide"

title: "Memory Tuning Guide"

tillrohrmann · 2020-02-08T22:16:48Z

docs/ops/memory/mem_tuning.md

+under the License.
+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors


Suggested change

In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors

In addition to the [main memory setup guide](mem_setup.html), this section explains how to configure the task executor's memory for different workloads and deployments.

tillrohrmann · 2020-02-08T22:16:59Z

docs/ops/memory/mem_tuning.md

+-->
+
+In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
+depending on the use case and which options are important in which case.


Suggested change

depending on the use case and which options are important in which case.

tillrohrmann · 2020-02-08T22:17:12Z

docs/ops/memory/mem_tuning.md

+* toc
+{:toc}
+
+## Configure memory for standalone deployment


Capitalization of heading is not consistent with the headings of the other pages.

tillrohrmann · 2020-02-08T22:21:38Z

docs/ops/memory/mem_tuning.md

+
+Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently.
+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects


Suggested change

This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects

This means that the [*managed memory*](../memory/mem_setup.html#managed-memory) configuration affects

tillrohrmann · 2020-02-08T22:21:57Z

docs/ops/memory/mem_tuning.md

+Flink's batch operators leverage [managed memory](../memory/mem_setup.html#managed-memory) to run more efficiently.
+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
+on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)


Suggested change

on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)

the performance of your applications. Flink will use as much [managed memory](../memory/mem_setup.html#managed-memory)

tillrohrmann · 2020-02-08T22:22:09Z

docs/ops/memory/mem_tuning.md

+In doing so, some operations can be performed directly on raw data without having to be deserialized into Java objects.
+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
+on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)
+as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely


Suggested change

as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely

as configured for the execution of batch jobs. This prevents `OutOfMemoryErrors` because Flink knows precisely

tillrohrmann · 2020-02-08T22:22:15Z

docs/ops/memory/mem_tuning.md

+This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
+on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)
+as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely
+how much memory it has to leverage. If the [managed memory](../memory/mem_setup.html#managed-memory) is not sufficient,


Suggested change

how much memory it has to leverage. If the [managed memory](../memory/mem_setup.html#managed-memory) is not sufficient,

how much memory it can use. If the [*managed memory*](../memory/mem_setup.html#managed-memory) is not large enough,

tillrohrmann · 2020-02-08T22:22:28Z

docs/ops/memory/mem_tuning.md

+on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)
+as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely
+how much memory it has to leverage. If the [managed memory](../memory/mem_setup.html#managed-memory) is not sufficient,
+Flink will gracefully spill to disk.


Suggested change

Flink will gracefully spill to disk.

then Flink will gracefully spill to disk.

tillrohrmann

Last round of review. Thanks a lot for creating this documentation @azagrebin. I had a couple of comments which we can resolve before merging this PR. Once this is done +1 for merging.

tillrohrmann · 2020-02-09T07:43:53Z

docs/ops/memory/mem_trouble.md

+## IllegalConfigurationException
+
+If you see an *IllegalConfigurationException* thrown from *TaskExecutorProcessUtils*, it usually indicates
+that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)


Suggested change

that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)

that there is either an invalid configuration value (e.g. negative memory size, fraction that is greater than 1, etc.)

tillrohrmann · 2020-02-09T07:44:40Z

docs/ops/memory/mem_trouble.md

+
+## OutOfMemoryError: Java heap space
+
+The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size


Suggested change

The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size

The exception usually indicates that the JVM heap is too small. You can try to increase the JVM heap size

tillrohrmann · 2020-02-09T07:44:58Z

docs/ops/memory/mem_trouble.md

+## OutOfMemoryError: Java heap space
+
+The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size
+by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).


Suggested change

by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).

by increasing [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).

tillrohrmann · 2020-02-09T07:45:31Z

docs/ops/memory/mem_trouble.md

+The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size
+by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).
+
+<span class="label label-info">Note</span> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option


Suggested change

<span class="label label-info">Note</span> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option

<span class="label label-info">Note</span> You can also increase the [framework heap memory](mem_setup.html#framework-memory) but this option

tillrohrmann · 2020-02-09T07:46:12Z

docs/ops/memory/mem_trouble.md

+by configuring larger [total memory](mem_setup.html#configure-total-memory) or [task heap memory](mem_setup.html#task-operator-heap-memory).
+
+<span class="label label-info">Note</span> You can also increase [framework heap memory](mem_setup.html#framework-memory) but this option
+is advanced and recommended to be changed if you are sure that the Flink framework itself needs more memory.


Suggested change

is advanced and recommended to be changed if you are sure that the Flink framework itself needs more memory.

is advanced and should only be changed if you are sure that the Flink framework itself needs more memory.

tillrohrmann · 2020-02-09T08:28:16Z

docs/ops/memory/mem_migration.md

+Additionally, you can now have more direct control over the JVM heap assigned to the operator tasks
+([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)),
+see also [Task (Operator) Heap Memory](mem_setup.html#task-operator-heap-memory).
+The same memory has to be accounted for the heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)


Suggested change

The same memory has to be accounted for the heap state backend ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)

The JVM heap memory is also used by the heap state backends ([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)

tillrohrmann · 2020-02-09T08:28:45Z

docs/ops/memory/mem_migration.md

+
+See also [how to configure managed memory now](mem_setup.html#managed-memory).
+
+### Explicit size


Capitalization

tillrohrmann · 2020-02-09T08:33:42Z

docs/ops/memory/mem_migration.md

+
+If the [RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) is chosen for a streaming job,
+its native memory consumption should now be accounted for in [managed memory](mem_setup.html#managed-memory).
+The RocksDB memory allocation is limited by the [managed memory](mem_setup.html#managed-memory) size.


Maybe we could add that one could turn this behaviour off if one wants the old behaviour.

tillrohrmann · 2020-02-09T08:34:45Z

docs/ops/memory/mem_migration.md

+## Container Cut-Off Memory
+
+For containerized deployments, you could previously specify a cut-off memory. This memory could accommodate for unaccounted memory allocations.
+Dependencies which were not directly controlled by Flink were the main source of those allocations, e.g. RocksDB, internals of JVM etc.


Suggested change

Dependencies which were not directly controlled by Flink were the main source of those allocations, e.g. RocksDB, internals of JVM etc.

Dependencies which were not directly controlled by Flink were the main source of those allocations, e.g. RocksDB, internals of JVM, etc.

tillrohrmann · 2020-02-09T08:37:12Z

docs/ops/memory/mem_migration.md

+The RocksDB memory allocation is also limited by the configured size of the [managed memory](mem_setup.html#managed-memory).
+See also [migrating managed memory](#managed-memory) and [how to configure managed memory now](mem_setup.html#managed-memory).
+
+The other specific consumption of direct or native off-heap memory can be now addressed by the following new configuration options:


Suggested change

The other specific consumption of direct or native off-heap memory can be now addressed by the following new configuration options:

The other direct or native off-heap memory consumers can now be addressed by the following new configuration options:

…configuration

azagrebin · 2020-02-10T18:43:23Z

Thanks for the review @tillrohrmann
I addressed the comments
merged into master by 01a9ff9 - 0cb8834
merged into 1.10 by ff7e25c - 1212f66

azagrebin requested a review from tillrohrmann February 3, 2020 13:32

rmetzger added review=description? component=Documentation labels Feb 3, 2020

tillrohrmann self-assigned this Feb 3, 2020

tillrohrmann requested changes Feb 3, 2020

View reviewed changes

azagrebin mentioned this pull request Feb 3, 2020

[FLINK-15143][docs] Add new memory configuration guide before FLIP-49 #11004

Merged

azagrebin force-pushed the FLINK-15143 branch 3 times, most recently from f2ec29a to 8cb09c8 Compare February 3, 2020 17:26

azagrebin mentioned this pull request Feb 3, 2020

[FLINK-14495][docs] Add documentation for memory control of RocksDB state backend #10498

Closed

sjwiesman reviewed Feb 3, 2020

View reviewed changes

docs/ops/memory/index.md Outdated Show resolved Hide resolved

sjwiesman reviewed Feb 3, 2020

View reviewed changes

docs/ops/memory/mem_tuning.md Outdated Show resolved Hide resolved

azagrebin force-pushed the FLINK-15143 branch from 09865b1 to 4ff8243 Compare February 3, 2020 19:04

xintongsong requested changes Feb 4, 2020

View reviewed changes

xintongsong reviewed Feb 4, 2020

View reviewed changes

carp84 mentioned this pull request Feb 4, 2020

[FLINK-15743][docs] Extend Flink 1.10 release notes #10997

Closed

azagrebin force-pushed the FLINK-15143 branch from bac3201 to 16eeb75 Compare February 4, 2020 13:52

xintongsong approved these changes Feb 6, 2020

View reviewed changes

GJL reviewed Feb 6, 2020

View reviewed changes

tillrohrmann requested changes Feb 8, 2020

View reviewed changes

tillrohrmann approved these changes Feb 9, 2020

View reviewed changes

azagrebin added 3 commits February 10, 2020 19:25

[FLINK-15143][docs] Add new memory configuration guide for FLIP-49

01a9ff9

[FLINK-15143][docs] Add tuning and troubleshooting guides for memory …

d4aa8fa

…configuration

[FLINK-15143][docs] Add migration guide from pre-FLIP-49 memory config

0cb8834

azagrebin force-pushed the FLINK-15143 branch from e34dafa to 0cb8834 Compare February 10, 2020 18:37

azagrebin merged commit 0cb8834 into apache:master Feb 10, 2020

	Check Job Manager [related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.
	Check [Job Manager related configuration options](../config.html#jobmanager) for the memory setup of Job Manager.

	that it is always allocated within its configured size. Therefore, resizing network memory will not necessarily help in this situation.
	to never exceed its configured size. Therefore, resizing network memory will not necessarily help in this situation.


		See also [detailed Memory Model](#detailed-memory-model).

		## Detailed Memory Model

+              -->
+              * toc
+              {:toc}

		* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1))
		* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1))

	is given to Flink itself. The total Flink memory splits up into [managed memory size](#managed-memory) and JVM heap.
	is given to Flink itself. The total Flink memory splits up into JVM heap, [managed memory](#managed-memory), and direct memory.

		* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
		* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)

	It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero.
	It is also recommended to set [managed memory](../memory/mem_setup.html#managed-memory) to zero.

		It is also recommended to set [managed memory](mem_setup.html#managed-memory) to zero.
		This will ensure that the maximum amount of memory is allocated for user code on the JVM.


		This section describes the changes of the default ‘flink-conf.yaml’ shipped with Flink.

		The total memory (‘taskmanager.heap.size’) is replaced by [`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)

		The further described memory configuration is applicable starting with the release version 1.10. If you upgrade Flink
		from earlier versions, check the [migration guide](mem_migration.html) because many changes were introduced with the 1.10 release.

	options are relevant, see also [local execution](mem_detail.html#local-execution) for more details.
	options are relevant. See [local execution](mem_detail.html#local-execution) for more details.

	Otherwise, the simplest way to setup memory in Flink is to configure either of the two following options:
	Otherwise, the simplest way to setup memory in Flink is to configure either of the following two options:

	If they are not configured they are set to their default values. [Task heap memory](mem_setup.html#task-operator-heap-memory) and
	If they are not configured they are set to their default values. [Task heap memory](mem_setup.html#task-operator-heap-memory) and

	task off-heap memory are considered to be infinite (Long.MAX_VALUE bytes) and [managed memory](mem_setup.html#managed-memory)
	task off-heap memory are considered to be infinite (Long.MAX_VALUE bytes) and [managed memory](mem_setup.html#managed-memory)

	It can become relevant for future optimizations coming with next releases. The actual JVM heap size of the started
	The actual JVM heap size of the started

	If you want to control the JVM heap size you have to explicitly pass the corresponding JVM arguments, e.g. -Xmx, -Xms.
	If you want to control the JVM heap size, then you have to explicitly pass the corresponding arguments -Xmx and -Xms to the JVM.

		@@ -0,0 +1,234 @@
		---
		title: "Migration from old configuration (before 1.10 release)"

	title: "Migration from old configuration (before 1.10 release)"
	title: "Migration Guide"


		## IllegalConfigurationException

		If you see an IllegalConfigurationException thrown from TaskExecutorProcessUtils, it usually indicates

[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide #10999

[FLINK-15143] Docs for FLIP-49 TM memory model and configuration guide #10999

Conversation

azagrebin commented Feb 3, 2020

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

azagrebin commented Feb 3, 2020 • edited

flinkbot commented Feb 3, 2020

Automated Checks

Review Progress

flinkbot commented Feb 3, 2020 • edited

CI report:

tillrohrmann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjwiesman commented Feb 3, 2020

azagrebin commented Feb 3, 2020

sjwiesman commented Feb 3, 2020

xintongsong left a comment

Choose a reason for hiding this comment

xintongsong Feb 4, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

azagrebin commented Feb 4, 2020

xintongsong left a comment

Choose a reason for hiding this comment

GJL Feb 6, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tillrohrmann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tillrohrmann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

azagrebin commented Feb 3, 2020 •

edited

flinkbot commented Feb 3, 2020 •

edited

xintongsong Feb 4, 2020 •

edited

GJL Feb 6, 2020 •

edited

	If you see an IllegalConfigurationException thrown from TaskExecutorProcessUtils, it usually indicates
	If you see an `IllegalConfigurationException` thrown from `TaskExecutorProcessUtils`, it usually indicates

	In addition to the [main memory setup guide](mem_setup.html), this section explains how to setup memory of task executors
	In addition to the [main memory setup guide](mem_setup.html), this section explains how to configure the task executor's memory for different workloads and deployments.

	This means that [managed memory](../memory/mem_setup.html#managed-memory) configurations have practical effects
	This means that the [managed memory](../memory/mem_setup.html#managed-memory) configuration affects

	on the performance of your applications. Flink will attempt to allocate and use as much [managed memory](../memory/mem_setup.html#managed-memory)
	the performance of your applications. Flink will use as much [managed memory](../memory/mem_setup.html#managed-memory)

	as configured for batch jobs but not go beyond its limits. This prevents `OutOfMemoryError`'s because Flink knows precisely
	as configured for the execution of batch jobs. This prevents `OutOfMemoryErrors` because Flink knows precisely

	how much memory it has to leverage. If the [managed memory](../memory/mem_setup.html#managed-memory) is not sufficient,
	how much memory it can use. If the [managed memory](../memory/mem_setup.html#managed-memory) is not large enough,

	Flink will gracefully spill to disk.
	then Flink will gracefully spill to disk.

	that there is either an invalid configuration value (e.g., negative memory size, fraction that is greater than 1, etc.)
	that there is either an invalid configuration value (e.g. negative memory size, fraction that is greater than 1, etc.)


		## OutOfMemoryError: Java heap space

		The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size

	The exception usually indicates that JVM heap is not configured with enough size. You can try to increase the JVM heap size
	The exception usually indicates that the JVM heap is too small. You can try to increase the JVM heap size