Skip to content

WAL buffer queue memory control#14329

Merged
jt2594838 merged 16 commits intomasterfrom
Wal_mem_control
Dec 11, 2024
Merged

WAL buffer queue memory control#14329
jt2594838 merged 16 commits intomasterfrom
Wal_mem_control

Conversation

@HTHou
Copy link
Copy Markdown
Contributor

@HTHou HTHou commented Dec 5, 2024

Description

This PR introduces memory control for the WAL buffer queue to ensure efficient resource utilization and prevent memory overuse.

  • Default Memory Limit: The maximum memory size for the WAL entry queue is set to 1/10 of the memtable memory by default.
  • Blocking Behavior: If the total memory size of the WAL entry queue exceeds this limit, calls to the put method of the WAL entry queue will be blocked until sufficient memory is freed.

This enhancement improves stability by aligning WAL entry queue memory usage with the system’s memory management strategy.

@HTHou HTHou changed the title Wal mem control Wal queue mem control Dec 5, 2024
@codecov
Copy link
Copy Markdown

codecov bot commented Dec 6, 2024

Codecov Report

Attention: Patch coverage is 81.18812% with 19 lines in your changes missing coverage. Please review.

Project coverage is 39.68%. Comparing base (05ea473) to head (7d5666c).
Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...pache/iotdb/db/service/metrics/WritingMetrics.java 17.64% 14 Missing ⚠️
.../db/storageengine/dataregion/wal/node/WALNode.java 0.00% 2 Missing ⚠️
...rageengine/dataregion/wal/buffer/WALInfoEntry.java 92.30% 1 Missing ⚠️
...egion/wal/utils/MemoryControlledWALEntryQueue.java 96.42% 1 Missing ⚠️
...tdb/db/storageengine/rescon/memory/SystemInfo.java 90.90% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #14329      +/-   ##
============================================
+ Coverage     39.63%   39.68%   +0.04%     
  Complexity      185      185              
============================================
  Files          4325     4326       +1     
  Lines        274708   274792      +84     
  Branches      33366    33369       +3     
============================================
+ Hits         108883   109040     +157     
+ Misses       165825   165752      -73     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@HTHou HTHou changed the title Wal queue mem control WAL entry queue memory control Dec 9, 2024
@HTHou HTHou marked this pull request as ready for review December 9, 2024 09:09
@HTHou HTHou changed the title WAL entry queue memory control WAL buffer queue memory control Dec 9, 2024
@sonarqubecloud
Copy link
Copy Markdown

@jt2594838 jt2594838 merged commit f2a5854 into master Dec 11, 2024
HTHou added a commit that referenced this pull request Dec 11, 2024
* init

* dev more

* remove comment

* fix UT

* fix concurrent issue

* add metric

* add config template

* Fix review

* Fix review

* fix name
jt2594838 pushed a commit that referenced this pull request Dec 11, 2024
* WAL buffer queue memory control (#14329)

* init

* dev more

* remove comment

* fix UT

* fix concurrent issue

* add metric

* add config template

* Fix review

* Fix review

* fix name

* rm useless code
@HTHou HTHou deleted the Wal_mem_control branch December 11, 2024 13:31
Caideyipi pushed a commit to Caideyipi/iotdb that referenced this pull request Mar 25, 2026
* WAL buffer queue memory control (apache#14329)

* init

* dev more

* remove comment

* fix UT

* fix concurrent issue

* add metric

* add config template

* Fix review

* Fix review

* fix name

* rm useless code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants