[Bug] OlapScanner does not propagate query cancellation to S3 I/O, causing scanner thread pool exhaustion

[mortalBibo]

Search before asking

• I had searched in the issues and found no similar issues.

Version

3.0.6

What's Wrong

When executing a large partition scan query on internal tables stored on S3/OSS (cloud storage-compute separation mode), if the query times out or is cancelled, the S3 read operations are not terminated. This causes scanner threads to remain blocked on S3 I/O, eventually exhausting the scanner thread pool and blocking all subsequent queries and Stream Load jobs.

Root cause: OlapScanner (internal table scanner) does not propagate query cancellation to the IOContext used by the remote I/O layer. Unlike FileScanner (external table scanner) which overrides try_stop() to set io_ctx->should_stop = true, the internal table data scan path has no cancellation check at the remote I/O boundary.

Two blocking boundaries lack cancellation checks:

1. S3FileReader::read_at_impl() completely ignores the io_ctx parameter (parameter name is commented out: const IOContext* /*io_ctx*/). The S3 429 retry loop sleeps with exponential backoff (up to ~7s total) without checking cancellation.
2. CachedRemoteFileReader::read_at_impl() does not check cancellation in the FileBlock wait loop (each wait round blocks ~1s by default, up to 10 rounds = ~10s).

Because the data scan path issues thousands to tens-of-thousands of S3 GET requests per query (one per data page per column per segment), many scanner threads can become simultaneously stuck in these blocking boundaries after cancellation, leading to thread pool exhaustion.

Cascade failure chain

Large cold scan on S3 → query timeout → cancel signal sent
  → Scanner threads blocked in page-by-page S3 data reads, cannot respond to cancel
  → S3 429 retry loop sleeps up to ~7s per read without checking cancel
  → FileBlock wait loop blocks up to ~10s per read without checking cancel
  → Scanner thread pool exhausted
  → All subsequent queries blocked
  → Stream Load jobs blocked
  → Cluster appears frozen (low CPU, high I/O wait)

How to Reproduce

1. Set up a Doris cluster in cloud storage-compute separation mode (S3/OSS backend)
2. Create a partitioned table with several years of historical data
3. Execute a large scan: SELECT sum(col) FROM table WHERE dt >= '2021-01-01'
4. Wait for the query to timeout or manually KILL QUERY
5. Immediately try to execute a small query — it will be blocked
6. Check Stream Load — jobs will also be blocked

Expected Behavior

After query cancellation/timeout, scanner threads should detect the cancellation at the next wait/retry boundary and release back to the thread pool promptly.

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[mortalBibo]

Fix PR submitted. See PR description for design details, alternatives considered, and scope boundary.

[mortalBibo]

Update: Root cause analysis and PR scope adjustment

After further investigation, the cluster-wide freeze described in this issue has a more nuanced root cause than originally identified.

Cluster freeze resolution:
The freeze was resolved by increasing pipeline_executor_size (from default CPU core count to 2x). Source code analysis confirmed that ScanOperator::close() does not block on scanner threads — it sets _should_stop and returns immediately, releasing the pipeline executor thread. This suggests the bottleneck was pipeline executor thread exhaustion under mixed workloads (large cold queries + Stream Load), rather than scanner threads being stuck in S3 I/O.

The root cause is likely related to segment footer parsing requiring remote S3 reads during cold reads, which blocks pipeline executor threads. PR #59476 (persist segment rows in rowset meta pb) may have optimized this from the source by reducing the need to open segment footers remotely.

PR #62278 scope:
PR #62278 has been re-scoped from [fix] to [opt]. It addresses a separate, valid concern: after query cancellation, scanner threads continue executing the page-by-page S3 read chain (potentially hundreds of pages per get_block() call) without detecting cancellation, wasting S3 bandwidth, scanner thread pool resources, and network connections. The optimization adds cancellation checks at S3 retry and cache-wait boundaries so that cancelled scanners exit promptly rather than reading all remaining pages.

Summary:

• Cluster freeze: resolved by increasing pipeline_executor_size; root cause likely related to cold segment footer reads blocking pipeline threads ([Opt](meta)persist segment rows in rowse meta pb #59476 may address this)
• Post-cancellation resource waste: addressed by [opt](storage) Propagate query cancellation to S3 I/O for OlapScanner (#62230) #62278 (propagate cancellation to S3 I/O layer)