Introduce optional LSM Segment Size limit

[etiennedi]

What's being changed:

• By default, LSM segments have no size limit
• During a compaction, the necessary disk space is roughly size(left_segment)+size(right_segment)
• In billion-scale setups, this could easily lead to temporary disk usage spikes of 1TB or more
• This PR adds an optional env var PERSISTENCE_LSM_MAX_SEGMENT_SIZE
  • It defaults to unlimited for backward compatibility
  • Possible values are
    • Any positive number, will be parsed into an int64 and interpreted as bytes
    • A resource string using IEC units, such as 4GiB
    • A resource string using SI units, such as 4GB

Some examples of the setting in action:

PERSISTENCE_LSM_MAX_SEGMENT_SIZE=2GiB

PERSISTENCE_LSM_MAX_SEGMENT_SIZE=8GiB

PERSISTENCE_LSM_MAX_SEGMENT_SIZE=unlimited or not set at all

Review checklist

• Documentation has been updated, if necessary. Link to changed documentation:
• Chaos pipeline run or not necessary. Link to pipeline:
• All new code is covered by tests where it is reasonable.
• Performance tests have been run or not necessary.

[sonarcloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud