fix(providers): rate-limit resilience + non-fatal ingest failures - #57
Open
dkoosis wants to merge 2 commits into
Open
fix(providers): rate-limit resilience + non-fatal ingest failures#57dkoosis wants to merge 2 commits into
dkoosis wants to merge 2 commits into
Conversation
…ng 30 search() hardcoded limit: 30, silently overriding whatever limit the caller passed in SearchOptions — so a benchmark requesting a deeper retrieval pool (e.g. limit 50) still capped at 30. Honor options.limit, keeping 30 as the default. Flagged by @sohamd22 in PR supermemoryai#44 review.
Running a full 500-question LongMemEval ingest surfaced three ways the mem0/supermemory providers and the ingest phase aborted whole runs: - mem0 + supermemory: default ingest concurrency (50/100) blew past each paid tier's per-second cap, triggering an immediate 429 storm. Add exponential-backoff retry (supermemory honors server retryAfterSeconds) and lower the provider-default ingest concurrency to 8. Per-run overrides via checkpoint.concurrency still win over these defaults. - mem0: skip sessions with no usable messages instead of erroring (abstention questions can carry empty haystacks; mem0 rejects empty adds). - ingest phase: a single failed question no longer throws and kills the run. The checkpoint records the failure, downstream phases skip it, and resume retries it. One bad question in 500 must not abort a multi-hour benchmark.
dkoosis
force-pushed
the
upstream-pr/provider-rate-limit
branch
from
August 2, 2026 15:29
cdfeab1 to
da9c8fb
Compare
dkoosis
added a commit
to dkoosis/memorybench
that referenced
this pull request
Aug 2, 2026
…rop dead failed prop, sharpen comment)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Running a full 500-question LongMemEval ingest surfaced three ways the mem0/supermemory providers and the ingest phase abort an entire multi-hour run over transient or isolated failures.
Changes
1. mem0 + supermemory: 429 retry with backoff.
Both SDKs surface rate limits as thrown errors, so a burst above the per-key RPS cap kills the run. Each provider gets a
withRetrywrapper (kept per-provider deliberately — the two APIs signal 429 differently: supermemory returns aretryAfterSecondshint, which the retry honors; mem0 uses "too frequently"-style messages). Non-rate-limit errors still rethrow immediately.2. Provider-default ingest concurrency lowered (supermemory 100→8, indexing 200→50; mem0 ingest 8, search 20).
The old defaults exceed the per-second caps on standard paid tiers, producing an immediate 429 storm at run start. These are only provider defaults —
checkpoint.concurrencyper-run overrides still take precedence (resolveConcurrency), so runs with higher-tier keys can keep the old throughput by passing their own values.3. mem0: skip empty adds.
Abstention questions can carry empty haystacks; mem0 rejects empty message arrays. Filter empty-content messages and skip sessions with nothing usable instead of erroring.
4. Ingest phase: per-question failure no longer aborts the run.
Previously one failed question threw and killed the whole run. Now the checkpoint records
status: "failed"and the run continues. This is safe because downstream already gates on checkpoint state: indexing only picks up questions whose ingest completed, and search gates on indexing — so failed questions are cleanly skipped, andresume(which filters onstatus !== "completed") retries exactly those questions under the same run ID.Typechecks clean. No public API changes.