Skip to content

feat(sinks): introduce RetryStrategy / Config for http based sinks#25057

Merged
pront merged 22 commits intovectordotdev:masterfrom
ndrsg:ndrsg/10870
May 1, 2026
Merged

feat(sinks): introduce RetryStrategy / Config for http based sinks#25057
pront merged 22 commits intovectordotdev:masterfrom
ndrsg:ndrsg/10870

Conversation

@ndrsg
Copy link
Copy Markdown
Contributor

@ndrsg ndrsg commented Mar 27, 2026

Summary

This PR introduces a configurable retry_strategy for Vector's shared HTTP retry logic and exposes it on the http sink plus the sinks built on the same HTTP retry helpers.

It adds support for default, none, all, and custom retry modes, wires that behavior through the affected HTTP-based sinks, and adds documentation plus test coverage for custom status-code retries. The default strategy now consistently retries 408, 429, and 5xx responses across the shared HTTP retry implementations.

Affected sinks

Configurable sinks using HttpStatusRetryLogic:

  • http
  • axiom
  • opentelemetry
  • appsignal
  • azure_logs_ingestion
  • azure_monitor_logs
  • datadog_events
  • gcp_stackdriver_logs
  • gcp_stackdriver_metrics
  • honeycomb
  • keep
  • prometheus_remote_write

Sinks that still use HttpRetryLogic and are not made configurable by this PR:

  • clickhouse
  • greptimedb logs
  • influxdb_metrics
  • sematext_metrics
  • splunk_hec logs
  • splunk_hec metrics

Vector configuration

I used the new HTTP sink example configuration to validate the custom retry behavior (also checked in in examples folder):

data_dir: "/var/lib/vector"

sources:
  demo_logs:
    type: "demo_logs"
    format: "json"
    interval: 1

sinks:
  http_out:
    type: "http"
    inputs: [ "demo_logs" ]
    uri: "https://example.com/ingest"
    method: "post"

    healthcheck:
      enabled: false

    framing:
      method: "newline_delimited"
    encoding:
      codec: "json"

    request:
      timeout_secs: 60
      retry_attempts: 8
      retry_initial_backoff_secs: 2
      retry_max_duration_secs: 30

    retry_strategy:
      type: "custom"
      status_codes: [ 408, 425, 429, 503 ]

How did you test this PR?

  • Added unit coverage for the shared RetryStrategy behavior in src/sinks/util/http.rs.
  • Added HTTP sink tests covering both retryable configured status codes and non-retryable unconfigured status codes in src/sinks/http/tests.rs.
  • Built the Vector binary locally and validated config/examples/http_sink_custom_retry.yaml against it.
  • Ran make check-clippy locally.
  • Validated the changelog fragment with ./scripts/check_changelog_fragments.sh.

Change Type

  • Bug fix
  • New feature
  • Dependencies
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

@ndrsg ndrsg requested a review from a team as a code owner March 27, 2026 16:12
@github-actions github-actions Bot added the domain: sinks Anything related to the Vector's sinks label Mar 27, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 27, 2026

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@ndrsg
Copy link
Copy Markdown
Contributor Author

ndrsg commented Mar 27, 2026

I have read the CLA Document and I hereby sign the CLA

@pront
Copy link
Copy Markdown
Member

pront commented Apr 2, 2026

@codex review - draw context from #10870 (comment)

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 27feabc878

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/sinks/util/http.rs Outdated
Comment thread src/sinks/util/http.rs
@pront
Copy link
Copy Markdown
Member

pront commented Apr 3, 2026

Will come back to this PR once the open codex comments are resolved (feel free to dispute those if they don't make sense)

@pront pront added the meta: awaiting author Pull requests that are awaiting their author. label Apr 3, 2026
@github-actions github-actions Bot removed the meta: awaiting author Pull requests that are awaiting their author. label Apr 7, 2026
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6b7885db44

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/sinks/util/http.rs
@ndrsg
Copy link
Copy Markdown
Contributor Author

ndrsg commented Apr 7, 2026

@pront I addressed codex suggestions. The failing spell-check in the workflow does not seem to be related to my changes.

@pront
Copy link
Copy Markdown
Member

pront commented Apr 7, 2026

@pront I addressed codex suggestions. The failing spell-check in the workflow does not seem to be related to my changes.

Thank you @ndrsgm, you can ignore the spell-check workflow failures. I will take another look at this PR shortly.

Copy link
Copy Markdown
Member

@pront pront left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good for the most part. Thanks!

Comment thread src/sinks/util/http.rs Outdated
Comment thread src/sinks/util/http.rs
Comment thread src/sinks/util/http.rs Outdated
Comment thread src/sinks/util/http.rs
@pront pront added the meta: awaiting author Pull requests that are awaiting their author. label Apr 8, 2026
@github-actions github-actions Bot removed the meta: awaiting author Pull requests that are awaiting their author. label Apr 9, 2026
@ndrsg ndrsg requested a review from pront April 9, 2026 08:24
Comment thread src/sinks/util/http.rs Outdated
Comment thread src/sinks/util/http.rs
@ndrsg ndrsg requested a review from pront April 17, 2026 05:21
@ndrsg ndrsg requested a review from a team as a code owner April 17, 2026 21:08
@github-actions github-actions Bot added domain: external docs Anything related to Vector's external, public documentation work in progress labels Apr 17, 2026
@ndrsg
Copy link
Copy Markdown
Contributor Author

ndrsg commented Apr 29, 2026

Hello @pront, any news about this PR? I saw that some checks failed so I pushed additional changes for the doc generation.
The workflow assigned the work-in-progress label, but I don't know what that means. From my side there is no work in progress at the moment an I am waiting for the next feedback.

@pront
Copy link
Copy Markdown
Member

pront commented Apr 29, 2026

Hi @ndrsg, this PR is in great shape. The work in progress label is only for our internal coordination. It is remove once the PR is approved by a maintainer. I will do one final review. I am optimistic that we can include this in the next release.

Copy link
Copy Markdown
Member

@pront pront left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome PR, thanks!

@pront pront enabled auto-merge April 29, 2026 21:32
@pront pront added this pull request to the merge queue Apr 29, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Apr 29, 2026
@pront pront added this pull request to the merge queue May 1, 2026
Merged via the queue into vectordotdev:master with commit bd8ab1a May 1, 2026
77 of 80 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators May 1, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants