Improve pipe runtime memory allocation

[Caideyipi]

Summary

• Keep PR 18051's dynamic pipe memory allocation changes for TsFile parser, sink batch, and TsFile read buffers.
• Fix the TsFile -> Tablet parser memory self-lock seen in multi-pipe large TsFile sync: when tablet memory allocation fails, the same parser no longer keeps retrying while holding parser chunk/page/tablet memory.
• Bound the OOM recovery path so the TsFile event is retried by the outer pipe subtask after parser memory is released, instead of spinning forever inside one parsed tablet.

Root cause fixed

In the failing path, PipeTsFileInsertionEvent.consumeTabletInsertionEventsWithRetry caught PipeRuntimeOutOfMemoryCriticalException and retried the same parsed tablet indefinitely. The parser still held temporary tablet/chunk/page memory while the consumer was trying to allocate memory for the downstream PipeRawTabletInsertionEvent. With multiple pipes parsing large TsFiles concurrently, this could saturate the tablet memory pool and form a live-lock: pipe stays RUNNING, but sub_remain_event does not decrease and no tablet is transferred.

Implementation details

• Release parser temporary tablet memory after scan/query/table parsers hand a Tablet to a raw tablet event.
• On pipe OOM during TsFile tablet consumption:
  • release the parsed raw tablet event if it has not been queued/referenced;
  • close the current TsFile parser to release parser-side tablet/chunk/page/modification memory;
  • rethrow OOM so the outer pipe subtask keeps the original TsFile event as lastEvent and retries later.
• Ensure OOM is not hidden by processor lambdas that temporarily store exceptions in AtomicReference (PipeProcessorSubtask, DownSamplingProcessor, AggregateProcessor).
• Wake memory waiters when PipeMemoryManager.resize(..., 0) releases memory.
• Fix zero-size memory block registration so a failed expansion does not leave an unallocated block in allocatedBlocks; also register zero-size blocks expanded through tryAllocate(block, ...).
• Update the OOM log message to describe the new release-and-retry behavior instead of infinite in-place retry.

Tests

• Added UT coverage in TsFileInsertionEventParserTest:
  • testScanParserReleasesTabletMemoryAfterRawTabletGenerated
  • testConsumeTabletInsertionEventsWithRetryReleasesParserOnOutOfMemory
• mvn spotless:apply -pl iotdb-core/datanode
• git diff --check
• Attempted targeted UT:
  • mvn -pl iotdb-core/datanode '-Dtest=TsFileInsertionEventParserTest#testScanParserReleasesTabletMemoryAfterRawTabletGenerated+testConsumeTabletInsertionEventsWithRetryReleasesParserOnOutOfMemory' '-Dspotless.check.skip=true' '-Dcheckstyle.skip=true' test
  • blocked before test execution by existing unrelated main compile errors in ConsensusSubscriptionSetupHandler: missing TopicConstant.COLUMN_KEY and TopicConstant.COLUMN_DEFAULT_VALUE.
• Attempted module compile with reduced JVM memory pressure:
  • mvn -pl iotdb-core/datanode -DskipTests '-Dspotless.check.skip=true' '-Dcheckstyle.skip=true' compile
  • blocked by the same unrelated TopicConstant.COLUMN_* compile errors.

No new IT is added because this is a low-level parser/memory-manager recovery path and the deterministic UTs cover the parser-memory release and OOM cleanup behavior directly. The original 10-pipe/2kw scenario should be used as the regression workload for end-to-end validation.

[codecov]

Codecov Report

❌ Patch coverage is 21.67183% with 253 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.84%. Comparing base (71cb6e6) to head (1370353).
⚠️ Report is 8 commits behind head on master.

Files with missing lines	Patch %	Lines
.../protocol/thrift/sync/IoTDBDataRegionSyncSink.java	0.00%	55 Missing ⚠️
...otdb/db/pipe/agent/task/PipeDataNodeTaskAgent.java	0.00%	50 Missing ⚠️
...builder/IoTConsensusV2TransferBatchReqBuilder.java	0.00%	47 Missing ⚠️
...oad/evolvable/batch/PipeTabletEventPlainBatch.java	14.28%	18 Missing ⚠️
...ler/IoTConsensusV2TsFileInsertionEventHandler.java	0.00%	18 Missing ⚠️
...hrift/async/handler/PipeTransferTsFileHandler.java	0.00%	18 Missing ⚠️
.../payload/evolvable/batch/PipeTabletEventBatch.java	60.60%	13 Missing ⚠️
...rotocol/iotconsensusv2/IoTConsensusV2SyncSink.java	0.00%	10 Missing ⚠️
...pipe/sink/protocol/legacy/IoTDBLegacyPipeSink.java	0.00%	9 Missing ⚠️
...ad/evolvable/batch/PipeTabletEventTsFileBatch.java	44.44%	5 Missing ⚠️
... and 5 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18051      +/-   ##
============================================
+ Coverage     41.81%   41.84%   +0.02%     
  Complexity      318      318              
============================================
  Files          5296     5296              
  Lines        373384   373585     +201     
  Branches      48210    48204       -6     
============================================
+ Hits         156134   156324     +190     
- Misses       217250   217261      +11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sonarqubecloud]

Quality Gate passed

Issues
19 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.8% Duplication on New Code

See analysis details on SonarQube Cloud