[HUDI-9473] Set memory config properly for FileGroup reader in Flink …#13372
[HUDI-9473] Set memory config properly for FileGroup reader in Flink …#13372danny0405 merged 2 commits intoapache:masterfrom
Conversation
|
PATL when you have time. cc @danny0405 |
| instantRangeOption); | ||
| final TypedProperties typedProps = FlinkClientUtil.getMergedTableAndWriteProps(metaClient.getTableConfig(), writeConfig); | ||
| typedProps.put(HoodieReaderConfig.MERGE_TYPE.key(), mergeType); | ||
| typedProps.put(HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE.key(), maxMemoryInBytes); |
There was a problem hiding this comment.
shouldn't it already be include in the writeConfig?
There was a problem hiding this comment.
And there are 5 options related with the spillable map:
.withMaxMemorySizeInBytes(metadataConfig.getMaxReaderMemory())
.withBufferSize(metadataConfig.getMaxReaderBufferSize())
.withSpillableMapBasePath(metadataConfig.getSplliableMapDir())
.withDiskMapType(commonConfig.getSpillableDiskMapType())
.withBitCaskDiskMapCompressionEnabled(commonConfig.isBitCaskDiskMapCompressionEnabled())There was a problem hiding this comment.
shouldn't it already be include in the writeConfig?
Yes, config MAX_MEMORY_FOR_MERGE is already in writeConfig, and is configured as the value of FlinkOptions#WRITE_MERGE_MAX_MEMORY, but Flink reader before actually use FlinkOptions#COMPACTION_MAX_MEMORY as the size of merging memory (stored in MergeOnReadInputSplit#maxCompactionMemoryInBytes), so this pr is align the memory configuration as before.
And there are 5 options related with the spillable map:
These configurations are in writeConfig, and are not set explicitly through flink conf (except for MaxMemorySizeInBytes), so value for there configurations are aligned with before.
3 participants
…reader
Change Logs
Set memory config properly for FileGroup reader in Flink reader
Impact
Correct the memory conf for FileGroup reader
Risk level (write none, low medium or high below)
low
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist