Retry failed status batches instead of failing the whole send - #7
Open
Elimeshi1 wants to merge 2 commits into
Open
Retry failed status batches instead of failing the whole send#7Elimeshi1 wants to merge 2 commits into
Elimeshi1 wants to merge 2 commits into
Conversation
Sending a status to a large contact list failed with "timed out waiting for message send response". Shrinking the participants batch from 5000 to 500 did not fix it: it multiplied the number of stanzas by ten and traded the timeouts for "server returned error 429". There were two independent causes. The per-batch ack window. whatsmeow defaults to 75s, which a big status batch can exceed even when it is delivered, so the send is reported as a timeout. Batch size was never the problem: what trips WhatsApp's rate limiting is the number of stanzas, not the participants inside one. Set an explicit timeout (default 180s, WAHA_GOWS_STATUS_BATCH_TIMEOUT) and restore the 5000 default. Database lock contention. The sqlite device store was opened in the default rollback-journal mode, where one writer holds an exclusive lock. During a status broadcast the outgoing send and the flood of incoming decryptions (session/identity/sender-key writes) contend for it, exhaust the 30s busy_timeout and surface as "database is locked" - turning a ~4 minute send into a ~30 minute stall ending in a gRPC DEADLINE_EXCEEDED. Open it with WAL + synchronous=NORMAL + txlock=immediate, mirroring the GContainer store. Also detach the status send from the caller's context. A deadline on the caller side was aborting a send already live on WhatsApp, leaving the later batches undelivered while the caller saw a plain failure and re-sent the whole status - posting it two or three times. Fixes devlikeapro/waha#2096
Three related changes to how a status broadcast handles a batch that fails. Retry transient failures. An ack timeout or a 429 is a temporary condition, but the loop gave up on the batch immediately. Retry it (default 2 attempts, 5s then 15s). Only those two conditions are retried: any other server error is permanent and would just burn the budget and delay the batches behind it. Retrying is safe because the message ID is generated once for the whole status and reused by every batch and attempt, and WhatsApp deduplicates by (sender, message ID). Deliver best effort. A single failed batch failed the entire call, even when the other batches were already delivered - and a status that reached anyone is live on WhatsApp. The caller read that error as "reached nobody" and sent the status again, posting it two or three times. Now the call fails only when no batch got through, and reports which batch numbers were delivered. Space the batches out. They were sent back to back with no gap; a fixed delay (default 1.5s) keeps the cadence steady rather than bursty. All values are configurable: WAHA_GOWS_STATUS_BATCH_DELAY, WAHA_GOWS_STATUS_BATCH_MAX_RETRIES, WAHA_GOWS_STATUS_BATCH_RETRY_BACKOFF.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #6, and built on that branch, so this PR carries its commit too — GitHub cannot use a fork branch as the base of a cross-fork PR. Only the second commit belongs to this PR (
Retry failed status batches instead of failing the whole send); the diff reduces to it once #6 is merged.Once a large status send works at all, what is left is how it behaves when one batch fails.
An explicit participant list bypassed batching
When the caller supplied the recipients, the batch size became the size of that list — so the whole thing went out as a single batch. That is exactly the large send that needs splitting. The batch size now always comes from the config.
Transient failures were not retried
An ack timeout or a 429 is a temporary condition, but the loop gave up on the batch immediately. Batches are now retried (default 2 attempts, 5s then 15s,
WAHA_GOWS_STATUS_BATCH_MAX_RETRIES/WAHA_GOWS_STATUS_BATCH_RETRY_BACKOFF).Only those two conditions are retried. Any other server error is permanent, and retrying it would burn the budget and delay the batches behind it.
Retrying is safe:
extra.IDis generated once for the whole status and reused by every batch and every attempt, and WhatsApp deduplicates by (sender, message ID), so a recipient that already received the batch does not see the status twice.One failed batch failed the whole send
A single failed batch made the call return an error even when every other batch had been delivered — and a status that reached anyone is already live on WhatsApp. The caller read that error as "reached nobody" and sent the status again, posting it two or three times.
The call now fails only when no batch got through, and logs a report at batch granularity:
Batch numbers, not phone numbers — the log stays readable on a 48k list.
Batches were sent back to back
A fixed delay between batches (default 1.5s,
WAHA_GOWS_STATUS_BATCH_DELAY) keeps the cadence steady rather than bursty. At the 5000 default this is about 15s across a 48k list.Open question: a partial delivery is not visible to the caller
Returning success on a partial delivery is the lesser of two evils, not a complete answer. Before this change the caller got a wrong answer — an error for a send that had partly succeeded — and re-sent the whole status, posting it twice. Now it gets an incomplete one: the send succeeded, without saying how much of it.
There is nowhere to put that. gRPC has no partial-success status, and
MessageResponseisid/timestamp/message, so a caller cannot distinguish 48k delivered from 5k delivered. Only the log line above carries it.An additive, backwards-compatible option would be:
Deliberately not part of this PR — it touches the proto and the API side, and the shape is your call. Happy to open it separately if you want it.
Testing
Built and tested with
make all(go test ./...passes).The behaviour here has been running in production on a ~48k contact list — batching an explicit list, spacing the batches out, retrying, and returning success on a partial delivery. Two details were tightened for this PR and are not covered by that: the retry budget is 2 attempts rather than 3, and only a 429 is retried where the deployed version retried every server error.