Skip to content

[Fix] Reconcile actual aligned TVList memory usage - #18330

Merged
jt2594838 merged 1 commit into
apache:masterfrom
Caideyipi:fix/aligned-bitmap-actual-memory-accounting
Jul 29, 2026
Merged

[Fix] Reconcile actual aligned TVList memory usage#18330
jt2594838 merged 1 commit into
apache:masterfrom
Caideyipi:fix/aligned-bitmap-actual-memory-accounting

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Problem

#18249 made aligned MemTable bitmaps lazy, but the write-memory estimator still defensively reserved a bitmap object and bitmap-list reference for every value block. That reservation can make memory accounting larger than the objects that are actually materialized and may reject writes unnecessarily.

A newly extended column also keeps its historical value arrays as null. Therefore, this accounting model should not charge the additional 4-byte object reference for each historical null placeholder; the reference is charged only when the value array is materialized.

Fix

  • Remove the unconditional per-block bitmap reservation and account for bitmap references/objects only when they are actually created.
  • Track materialized value-array and bitmap costs incrementally in AlignedTVList, with an O(1) getRamSize() for the write path.
  • Reconcile aligned TVList RAM after row, rows, and tablet writes, including mixed-alignment rows, multi-device tablets, and sorted TVLists.

The correction is:

actual aligned TVList RAM delta - estimated aligned MemTable increment

A positive correction is reserved in both DataRegionInfo and the MemTable; a negative correction is released from both and refreshes the storage-group status. Chunk metadata and variable-length Binary payload accounting are intentionally unchanged.

Performance

The new performance UT is disabled by default and can be enabled with:

mvn -pl iotdb-core/datanode test \
  -Dtest=AlignedBitmapMemoryAccountingPerformanceTest \
  -Diotdb.aligned.bitmap.accounting.perf.enabled=true

Default benchmark setup: 64 columns, 256 rows per batch, 200 warmups, 2,000 batches per round, and 5 rounds. Reconciliation is repeated internally to avoid Windows CPU timer granularity and then normalized per batch.

Scenario Reconciliation CPU Write CPU CPU ratio Reconciliation allocation
Dense 0.038 us/batch 23.438 us/batch 0.16% 0 B/batch
Null-heavy 0.031 us/batch 39.063 us/batch 0.08% 0 B/batch

The measured reconciliation overhead is therefore negligible for both dense and null-heavy writes.

Verification

  • AlignedTVListTest: 15 passed
  • TsFileProcessorTest: 20 passed
  • Performance UT default-off behavior: 1 skipped as expected
  • Performance UT enabled: 1 passed
  • IoTDBAlignedTVListLazyAllocationIT: 1 passed on a packaged 1C1D cluster
  • Datanode and integration-test Checkstyle: 0 violations
  • Spotless and git diff --check: passed

This PR has:

  • been self-reviewed.
  • added comments explaining the intent of the accounting and reconciliation logic.
  • added or updated unit tests to cover the new code paths.
  • added/updated an integration test.
  • been tested in a test IoTDB cluster.

Key changed/added classes
  • AlignedTVList: actual materialized value-array/bitmap RAM tracking and constant-time RAM lookup.
  • TsFileProcessor: post-write aligned RAM reconciliation.
  • AlignedBitmapMemoryAccountingPerformanceTest: opt-in reconciliation overhead benchmark.

@jt2594838
jt2594838 merged commit 9e909d3 into apache:master Jul 29, 2026
40 of 41 checks passed
@jt2594838
jt2594838 deleted the fix/aligned-bitmap-actual-memory-accounting branch July 29, 2026 02:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants