Skip to content

[To dev/1.3] Reconcile actual aligned TVList memory usage (#18330) - #18339

Open
Caideyipi wants to merge 1 commit into
apache:dev/1.3from
Caideyipi:cp-18330-dev-1.3
Open

[To dev/1.3] Reconcile actual aligned TVList memory usage (#18330)#18339
Caideyipi wants to merge 1 commit into
apache:dev/1.3from
Caideyipi:cp-18330-dev-1.3

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Backport of #18330 (9e909d3ce26002507efd7a72cc53e7bf40c145bc) to dev/1.3.

Changes

  • Remove unconditional aligned bitmap reservation and account for bitmap references and objects only when they are materialized.
  • Track actual aligned TVList RAM with an O(1) lookup.
  • Reconcile estimated and actual aligned TVList RAM after row, rows, and tablet writes, including mixed-alignment batches and sorted TVLists.
  • Add the explicit-RAM TVList.RamInfo constructor needed by this backport.

dev/1.3 adaptation

dev/1.3 retains eager value primitive-array allocation, so this backport applies the actual bitmap accounting and post-write reconciliation without importing master's lazy value-array model. The master-only IoTDBAlignedTVListLazyAllocationIT was omitted because its supporting implementation and test do not exist on this branch. The performance test was adapted to the dev/1.3 APIs.

Verification

mvn -o -nsu spotless:apply -pl iotdb-core/datanode
mvn -o -nsu -pl iotdb-core/datanode -DskipTests compile
mvn -o -nsu -pl iotdb-core/datanode surefire:test '-Dtest=AlignedTVListTest,TsFileProcessorTest,AlignedBitmapMemoryAccountingPerformanceTest'

Result: 27 tests run, 0 failures, 0 errors, and 1 expected skip for the default-disabled performance test.


This PR has:

  • been self-reviewed.
  • added comments explaining the intent of the accounting and reconciliation logic.
  • added or updated unit tests to cover the backported code paths.

Key changed/added classes
  • AlignedTVList: actual bitmap RAM tracking and constant-time RAM lookup.
  • TsFileProcessor: post-write aligned RAM reconciliation.
  • TVList.RamInfo: explicit actual-RAM construction for reconciliation.
  • AlignedBitmapMemoryAccountingPerformanceTest: opt-in reconciliation overhead benchmark.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant