-
Notifications
You must be signed in to change notification settings - Fork 321
fix: updates write reponses, suggests exponential backoffs #6574
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
sanderson
wants to merge
3
commits into
master
Choose a base branch
from
dist-write-resp
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+144
−128
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,6 +5,7 @@ Learn how to avoid unexpected results and recover from errors when writing to {{ | |
| - [Troubleshoot failures](#troubleshoot-failures) | ||
| - [Troubleshoot rejected points](#troubleshoot-rejected-points) | ||
| - [Report write issues](#report-write-issues) | ||
| {{% show-in "cloud-dedicated,clustered" %}}- [Implement an exponential backoff strategy](#implement-an-exponential-backoff-strategy){{% /show-in %}} | ||
|
|
||
| ## Handle write responses | ||
|
|
||
|
|
@@ -39,7 +40,7 @@ The `message` property of the response body may contain additional details about | |
| | `404 "Not found"` | A requested **resource type** (for example, "database"), and **resource name** | A requested resource wasn't found | | ||
| | `422 "Unprocessable Entity"` | `message` contains details about the error | The data isn't allowed (for example, falls outside of the database's retention period). | | ||
| | `500 "Internal server error"` | Empty | Default status for an error | | ||
| | `503 "Service unavailable"` | Empty | The server is temporarily unavailable to accept writes. The `Retry-After` header contains the number of seconds to wait before trying the write again. | | ||
| | `503 "Service unavailable"` | Empty | The server is temporarily unavailable or the requested service is resource constrained. [Implement an exponential backoff strategy](#implement-an-exponential-backoff-strategy). | | ||
| {{% /show-in %}} | ||
|
|
||
| {{% show-in "cloud-serverless" %}} | ||
|
|
@@ -346,3 +347,121 @@ Include the support package when contacting InfluxData support through your stan | |
| - Business context if the issue affects production systems | ||
|
|
||
| This comprehensive information will help InfluxData engineers identify root causes and provide targeted solutions for your write issues. | ||
|
|
||
| {{% show-in "cloud-dedicated,clustered" %}} | ||
| ## Implement an exponential backoff strategy | ||
|
|
||
| Use exponential backoff with jitter for retrying requests that return `429` or `503`. | ||
| This reduces load spikes and avoids thundering-herd problems. | ||
|
|
||
| **Recommended parameters**: | ||
|
|
||
| - Base delay: 1s | ||
| - Multiplier: 2 (double each retry) | ||
| - Max delay: 30s | ||
| - Max retries: 5 (increase only with care) | ||
| - Jitter: use "full jitter" (random between 0 and computed delay) | ||
|
|
||
| ### Exponential backoff examples | ||
|
|
||
| {{< code-tabs-wrapper >}} | ||
| {{% code-tabs %}} | ||
| [cURL](#) | ||
| [Python](#) | ||
| [JavaScript](#) | ||
| {{% /code-tabs %}} | ||
| {{% code-tab-content %}} | ||
| <!--------------------------------- BEGIN cURL --------------------------------> | ||
| <!--pytest.mark.skip--> | ||
| ```sh | ||
| base=1 | ||
| max_delay=30 | ||
| max_retries=5 | ||
|
|
||
| for attempt in $(seq 0 $max_retries); do | ||
| resp_code=$(curl -s -o /dev/null -w "%{http_code}" --request POST "https://{{< influxdb/host >}}/write?db=DB" ...) | ||
| if [ "$resp_code" -eq 204 ]; then | ||
| echo "Write succeeded" | ||
| break | ||
| fi | ||
|
|
||
| if [ "$resp_code" -ne 429 ] && [ "$resp_code" -ne 503 ]; then | ||
| echo "Non-retryable response: $resp_code" | ||
| break | ||
| fi | ||
|
|
||
| # compute exponential delay and apply full jitter | ||
| delay=$(awk -v b=$base -v a=$attempt -v m=$max_delay 'BEGIN{d=b*(2^a); if(d>m) d=m; print d}') | ||
| sleep_seconds=$(awk -v d=$delay 'BEGIN{srand(); printf "%.3f", rand()*d}') | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @copilot Should |
||
| sleep $sleep_seconds | ||
| done | ||
| ``` | ||
| <!---------------------------------- END cURL ---------------------------------> | ||
| {{% /code-tab-content %}} | ||
|
|
||
| {{% code-tab-content %}} | ||
| <!-------------------------------- BEGIN Python -------------------------------> | ||
| <!--pytest.mark.skip--> | ||
| ```python | ||
| import random | ||
| import time | ||
| import requests | ||
|
|
||
| base = 1.0 | ||
| max_delay = 30.0 | ||
| max_retries = 5 | ||
|
|
||
| for attempt in range(max_retries + 1): | ||
| r = requests.post(url, headers=headers, data=body, timeout=10) | ||
| if r.status_code == 204: | ||
| break | ||
| if r.status_code not in (429, 503): | ||
| raise RuntimeError(f"Non-retryable: {r.status_code} {r.text}") | ||
|
|
||
| # exponential backoff with full jitter | ||
| retry_delay = min(base * (2 ** attempt), max_delay) | ||
| sleep = random.random() * retry_delay # full jitter | ||
| time.sleep(sleep) | ||
| else: | ||
| raise RuntimeError("Max retries exceeded") | ||
| ``` | ||
| <!--------------------------------- END Python --------------------------------> | ||
| {{% /code-tab-content %}} | ||
|
|
||
| {{% code-tab-content %}} | ||
| <!------------------------------ BEGIN JavaScript -----------------------------> | ||
| <!--pytest.mark.skip--> | ||
| ```js | ||
| const base = 1000; | ||
| const maxDelay = 30000; | ||
| const maxRetries = 5; | ||
|
|
||
| async function sleep(ms) { return new Promise(r => setTimeout(r, ms)); } | ||
|
|
||
| for (let attempt = 0; attempt <= maxRetries; attempt++) { | ||
| const res = await fetch(url, { method: 'POST', body }); | ||
| if (res.status === 204) break; | ||
| if (![429, 503].includes(res.status)) throw new Error(`Non-retryable ${res.status}`); | ||
|
|
||
| let delay = base * 2 ** attempt; | ||
| delay = Math.min(delay, maxDelay); | ||
|
|
||
| const sleepMs = Math.random() * delay; // full jitter | ||
| await sleep(sleepMs); | ||
| } | ||
| ``` | ||
| <!------------------------------- END JavaScript ------------------------------> | ||
| {{% /code-tab-content %}} | ||
| {{< /code-tabs-wrapper >}} | ||
|
|
||
| ### Exponential backoff best practices | ||
|
|
||
| - Only retry on idempotent or safe request semantics your client supports. | ||
| - Retry only for `429` (Too Many Requests) and `503` (Service Unavailable). | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we include "Too Many Requests" in the spec (as Copilot recommended)? |
||
| - Do not retry on client errors like `400`, `401`, `404`, `422`. | ||
| - Cap the delay with `max_delay` to avoid excessively long waits. | ||
| - Limit total retries to avoid infinite loops and provide meaningful errors. | ||
| - Log retry attempts and backoff delays for observability and debugging. | ||
| - Combine backoff with bounded concurrency to avoid overwhelming the server. | ||
|
|
||
| {{% /show-in %}} | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is $max-delay used?