Skip to content

Commit

Permalink
fixup! [FLINK-15143][docs] Add migration guide from pre-FLIP-49 memor…
Browse files Browse the repository at this point in the history 
…y config
  • Loading branch information
@azagrebin
azagrebin committed Feb 4, 2020
1 parent 74c92f3 commit bbc0d2c
Showing 1 changed file with 19 additions and 19 deletions.
38 changes: 19 additions & 19 deletions docs/ops/memory/mem_migration.md
View file
Expand Up @@ -37,11 +37,11 @@ were removed or their semantics changed. This guide will help you to understand
<strong>Note:</strong> The previous memory configuration allows that no memory related options are set at all
as they all have default values. The [new memory configuration](mem_setup.html#configure-total-memory) requires
that at least one subset of the following options is configured explicitly, otherwise the configuration will fail:
* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)
* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
* [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size)
* [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size)
* [taskmanager.memory.task.heap.size](../config.html#taskmanager-memory-task-heap-size) and [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size)

The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
The [default ‘flink-conf.yaml’](#default-configuration-in-flink-confyaml) shipped with Flink sets [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size)
to make the default memory configuration consistent.

This [spreadsheet](https://docs.google.com/spreadsheets/d/1mJaMkMPfDJJ-w6nMXALYmTc4XxiV30P5U7DzgwLkSoE) can also help
Expand All @@ -57,26 +57,26 @@ The following options are completely removed. If they are still used, they will
| **Removed option** | **Note** |
| :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| taskmanager.memory.fraction | Check the description of the new option [taskmanager.memory.managed.fraction](../config.html#taskmanager-memory-managed-fraction). The new option has different semantics and the value of the deprecated option usually has to be adjusted. See also [how to migrate managed memory](#managed-memory). |
| taskmanager.memory.off-heap | on-heap managed memory is no longer supported, see also [how to migrate managed memory](#managed-memory) |
| taskmanager.memory.preallocate | pre-allocation is no longer supported and managed memory is always allocated lazily, see also [how to migrate managed memory](#managed-memory) |
| taskmanager.memory.off-heap | On-heap managed memory is no longer supported, see also [how to migrate managed memory](#managed-memory) |
| taskmanager.memory.preallocate | Pre-allocation is no longer supported and managed memory is always allocated lazily, see also [how to migrate managed memory](#managed-memory) |

The following options are deprecated but if they are still used they will be interpreted as new options for backwards compatibility:

| **Deprecated option** | **Interpreted as** |
| :---------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| taskmanager.heap.size | - [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1) for standalone deployment <br/> - [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1) for containerised deployments <br/> See also [how to migrate total memory](#total-memory-previously-heap-memory). |
| taskmanager.memory.size | taskmanager.memory.managed.size, see also [how to migrate managed memory](#managed-memory) |
| taskmanager.network.memory.min | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) |
| taskmanager.network.memory.max | [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) |
| taskmanager.network.memory.fraction | [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) |
| **Deprecated option** | **Interpreted as** |
| :---------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| taskmanager.heap.size | - [taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size) for standalone deployment <br/> - [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size) for containerised deployments <br/> See also [how to migrate total memory](#total-memory-previously-heap-memory). |
| taskmanager.memory.size | [taskmanager.memory.managed.size](../config.html#taskmanager-memory-managed-size), see also [how to migrate managed memory](#managed-memory) |
| taskmanager.network.memory.min | [taskmanager.memory.network.min](../config.html#taskmanager-memory-network-min) |
| taskmanager.network.memory.max | [taskmanager.memory.network.max](../config.html#taskmanager-memory-network-max) |
| taskmanager.network.memory.fraction | [taskmanager.memory.network.fraction](../config.html#taskmanager-memory-network-fraction) |

Although, the network memory configuration has not changed too much it is recommended to verify its configuration.
It can change if other memory components have new sizes, e.g. the total memory which the network can be a fraction of.
See also [new detailed memory model](mem_setup.html#detailed-memory-model).

The container cut-off configuration options, ‘containerized.heap-cutoff-ratio’ and ‘containerized.heap-cutoff-min’,
have no effect for task manager processes anymore but they still have the same semantics for the job manager process.
See also [how to migrate container cut-off](#container-cut-off-memory).
The container cut-off configuration options, [containerized.heap-cutoff-ratio](config.html#containerized-heap-cutoff-ratio)
and [containerized.heap-cutoff-min](config.html#containerized-heap-cutoff-min), have no effect for task manager processes anymore
but they still have the same semantics for the job manager process. See also [how to migrate container cut-off](#container-cut-off-memory).

## Total Memory (previously Heap Memory)

Expand All @@ -87,8 +87,8 @@ The Mesos integration also had a separate option with the same semantics: ‘mes

If the mentioned legacy options are used without specifying the corresponding new options,
they will be directly translated into the following new options:
* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size-1)) for standalone deployments
* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)) for containerized deployments (Yarn or Mesos)
* Total Flink memory ([taskmanager.memory.flink.size](../config.html#taskmanager-memory-flink-size)) for standalone deployments
* Total process memory ([taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size)) for containerized deployments (Yarn or Mesos)

It is also recommended to use these new options instead of the legacy ones as they might be completely removed in the following releases.

Expand Down Expand Up @@ -152,7 +152,7 @@ Additionally, the following changes have been made:

For containerized deployments, you could previously specify a cut-off memory. This memory could accommodate for unaccounted memory allocations.
Dependencies which were not directly controlled by Flink were the main source of those allocations, e.g. RocksDB, internals of JVM etc.
This is no longer available and the related configuration options (‘containerized.heap-cutoff-ratio’ and ‘containerized.heap-cutoff-min’)
This is no longer available and the related configuration options (`containerized.heap-cutoff-ratio` and `containerized.heap-cutoff-min`)
will have no effect on the task manager process anymore. The new memory model introduced more specific memory components,
described further, to address these concerns.

Expand All @@ -174,7 +174,7 @@ The mentioned configuration options remain valid for the job manager in the same

This section describes the changes of the default ‘flink-conf.yaml’ shipped with Flink.

The total memory (‘taskmanager.heap.size’) is replaced by [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size-1)
The total memory (‘taskmanager.heap.size’) is replaced by [taskmanager.memory.process.size](../config.html#taskmanager-memory-process-size)
in the default ‘flink-conf.yaml’. The value is also increased from 1024Mb to 1568Mb.
See also [how to configure total memory now](mem_setup.html#configure-total-memory).

Expand Down

0 comments on commit bbc0d2c

Please sign in to comment.