Skip to content

[improve][offload] PIP-461: Add queued latency metrics for offloader executors#25321

Draft
BewareMyPower wants to merge 1 commit intoapache:masterfrom
BewareMyPower:bewaremypower/offloader-queued-metrics
Draft

[improve][offload] PIP-461: Add queued latency metrics for offloader executors#25321
BewareMyPower wants to merge 1 commit intoapache:masterfrom
BewareMyPower:bewaremypower/offloader-queued-metrics

Conversation

@BewareMyPower
Copy link
Contributor

Motivation

BlobStoreManagedLedgerOffloader uses OrderedScheduler instances for both offload/write work and read work, but those executors currently do not expose an offloader-specific queued-latency metric.

The jcloud offloader runs blocking blob-store operations on these executors:

  • scheduler handles synchronous offload and delete work.
  • readExecutor handles blocking offloaded-ledger open/read/close work.

When these tasks queue up behind other blocking work, it is hard to tell from the existing offloader metrics whether the executors themselves are saturated.

Modifications

This change adds two new offloader metrics:

  • brk_ledgeroffloader_offload_executor_queue_latency
  • brk_ledgeroffloader_read_offload_executor_queue_latency

The implementation:

  • Extends LedgerOffloaderStats and LedgerOffloaderStatsImpl with the two queue-latency summaries.
  • Adds a small wrapper utility to record executor queue wait time when a task actually starts running.
  • Instruments the jcloud offloader paths that submit blocking work to scheduler and readExecutor.
  • Instruments the offloaded read-handle implementations so queued latency is also tracked for read and close tasks after the handle is opened.
  • Extends existing tests to assert the new metrics are recorded.

The queue-latency metrics use the same topic/namespace label behavior as the existing ledger offloader metrics.

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

  • ./mvnw -pl managed-ledger -Dmaven.test.skip=true install
  • ./mvnw -pl tiered-storage/jcloud -Dtest=BlobStoreManagedLedgerOffloaderTest,BlobStoreBackedReadHandleImplTest -Dsurefire.failIfNoSpecifiedTests=false test

Notes:

  • The jcloud targeted test run passed locally: 26 tests, 0 failures.
  • The updated pulsar-broker test class was not run locally because this checkout currently has unrelated compile failures in pulsar-broker / pulsar-metadata.

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: N/A

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Mar 14, 2026
@BewareMyPower BewareMyPower marked this pull request as draft March 14, 2026 12:18
@BewareMyPower
Copy link
Contributor Author

I will write a proposal for it since it adds new metrics

@BewareMyPower BewareMyPower changed the title [improve][offload] Add queued latency metrics for offloader executors [improve][offload] PIP-461: Add queued latency metrics for offloader executors Mar 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant