[AMORO-3329] Add MemorySize for parsing memory value and unit#3409
Conversation
df49415 to
9334783
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3409 +/- ##
============================================
+ Coverage 21.59% 27.72% +6.13%
- Complexity 2309 3566 +1257
============================================
Files 426 596 +170
Lines 39719 48528 +8809
Branches 5624 6263 +639
============================================
+ Hits 8577 13456 +4879
- Misses 30414 34133 +3719
- Partials 728 939 +211
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
zhoujinsong
left a comment
There was a problem hiding this comment.
Thanks a lot for the contribution!
I think this is a good start to make our configuration value more readable.
We may improve our configuration format in other place, like the AMS configuration after this PR been merged.
I left a minor suggestion, you may want have a look!
klion26
left a comment
There was a problem hiding this comment.
Thanks for the contribution, LGTM
This patch is back-compatible, but it contains some user behavior change, maybe we need to add this in the release notes, @Jzjsnow could you please create an issue to track the release notes update, thanks.
Apart the above, do we need to add some documentation about how to configure this in the documentation? if this is yes could you please add an issue to track it. thanks.
…#3409) * [AMORO-3329] Add MemorySize for parsing memory value and unit * fixup! [AMORO-3329] Add MemorySize for parsing memory value and unit --------- Co-authored-by: jzjsnow <snow.jiangzj@gmail.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com>
Why are the changes needed?
Currently, when we use flink optimizer groups, the memory parameters
flink-conf.jobmanager.memory.process.sizeandflink-conf.taskmanager.memory.process.sizeonly support plain numeric strings or strings withm/gsuffix. If we configure them with mb/gb suffix, we'll get wrong parsed results. And the default unit for plain numeric strings ismb, which is inconsistent with the default unit of byte in the flink native parameter and may cause inconvenience for users familiar with flink when using this parameter.Close #3329.
Brief change log
MemorySizeto theamoro-commonmodule for parsing flink's memory parameters, consistent with flink's native parsing logic.propertiesof ams metadata tablesresource_groupandresourceto automatically add theMBsuffix to the flink jm and tm memory parameters that have been set and saved as plain numeric stringsHow was this patch tested?
Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request
Documentation