Skip to content

[Pipe] Fix TsFile rate limit accounting - #18344

Merged
jt2594838 merged 1 commit into
apache:masterfrom
Caideyipi:fix/pipe-tsfile-rate-limit-accounting
Jul 29, 2026
Merged

[Pipe] Fix TsFile rate limit accounting#18344
jt2594838 merged 1 commit into
apache:masterfrom
Caideyipi:fix/pipe-tsfile-rate-limit-accounting

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Problem

Pipe TsFile send paths accounted for the configured read buffer size before reading from the file. As a result:

  • The final partial piece was charged as a full buffer.
  • The final EOF probe was charged as another full buffer.
  • Files no larger than one read buffer were charged twice their actual size.

This reduced the effective global TsFile send rate below the configured limit and inflated the PIPE_TSFILE_SEND_DISK_IO metric.

Fix

Read each file piece first and account for the actual readLength only when data is returned. The change covers:

  • Common sync and air-gap file transfer paths.
  • Data-region sync and async TsFile transfer paths.
  • The batched data-region air-gap TsFile transfer path.

The async handler now uses a small overridable helper so the accounting behavior can be tested without invoking the global limiter.

Tests

mvn -o -nsu -Ddevelocity.off=true -DskipTests compile -pl iotdb-core/node-commons,iotdb-core/datanode

mvn -o -nsu -Ddevelocity.off=true test -pl iotdb-core/datanode -Dtest=IoTDBDataRegionAirGapSinkTest#testTransferTsFileBatchOverAirGap,PipeTransferTsFileHandlerRateLimitTest

Both targeted tests verify that the accounted bytes equal the actual transferred/read bytes and that EOF is not accounted for.


This PR has:

  • been self-reviewed.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage.

Key changed/added classes (or packages if there are too many classes) in this PR
  • IoTDBSslSyncSink
  • IoTDBAirGapSink
  • IoTDBDataRegionSyncSink
  • PipeTransferTsFileHandler
  • IoTDBDataRegionAirGapSink

@jt2594838
jt2594838 merged commit f3f7d12 into apache:master Jul 29, 2026
41 checks passed
@Caideyipi
Caideyipi deleted the fix/pipe-tsfile-rate-limit-accounting branch July 29, 2026 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants