Skip to content

fix: Retry incomplete HTTP responses - #348

Merged
mheffner merged 1 commit into
rotel-dev:mainfrom
brightsparc:agent-retry-incomplete-http-response
Aug 1, 2026
Merged

fix: Retry incomplete HTTP responses#348
mheffner merged 1 commit into
rotel-dev:mainfrom
brightsparc:agent-retry-incomplete-http-response

Conversation

@brightsparc

Copy link
Copy Markdown
Contributor

Fixes #347.

What changed

  • Treat nested hyper::Error values for which is_incomplete_message() is true as retryable.
  • Preserve the existing exporter backoff and maximum elapsed-time behavior.
  • Add a socket-level regression test that accepts an HTTP request, closes without returning a response, verifies Hyper reports IncompleteMessage, and verifies RetryPolicy retries it.

Why

During a staging ClickHouse load test with Rotel 0.2.2, ten turns were missing and Rotel logged permanent exporter drops with this error shape:

TransportErrorWithMetadata {
    original_error: hyper_util::client::legacy::Error(
        SendRequest,
        hyper::Error(IncompleteMessage),
    ),
    metadata: None,
}

Three drops occurred during the affected run at 2026-07-30 00:50:45Z, 01:10:58Z, and 01:33:58Z. Rotel's retry policy already walks the error source chain, but only recognized timeouts and ConnectError, so this transport failure bypassed the configured retry budget and immediately reached Exporting failed, dropping data.

ClickHouse query and asynchronous-insert logs showed successful inserts continuously through all three windows, with status = Ok, exception_code = 0, and no corresponding ClickHouse warning/error entries. That points to an incomplete response caused by a connection reset rather than malformed telemetry or a broad ClickHouse failure.

Delivery semantics

An incomplete response is an ambiguous commit: the destination might have committed the request before the response was truncated. Retrying therefore provides at-least-once delivery and may produce duplicates unless the destination deduplicates requests. For telemetry export, that is preferable to confirmed permanent loss; stable request identity/deduplication remains a worthwhile follow-up for destinations that need stronger guarantees.

Validation

  • cargo fmt --all --check
  • cargo test -p rotel exporters::http::retry::tests — 18 passed
  • cargo clippy -p rotel --lib --tests -- -D warnings — blocked by existing repository-wide warnings in unrelated files; this change introduces no Clippy diagnostic in the output

@brightsparc
brightsparc marked this pull request as ready for review July 31, 2026 18:35
@brightsparc brightsparc changed the title Retry incomplete HTTP responses fix(clickhouse): Retry incomplete HTTP responses Jul 31, 2026
@brightsparc brightsparc changed the title fix(clickhouse): Retry incomplete HTTP responses fix: Retry incomplete HTTP responses Jul 31, 2026

@mheffner mheffner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks for fixing!

@mheffner
mheffner merged commit 8eebe58 into rotel-dev:main Aug 1, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Retry incomplete HTTP responses instead of dropping export batches

2 participants