fix(appfigures): stop gracefully at the reviews page-depth cap - #72489
Merged
Conversation
Appfigures caps offset pagination on /reviews at page*count <= 10000. Once
a sync's backlog crosses that depth it returns HTTP 400 ("page * count must
be less than or equal to 10000"), which we let bubble up as a fatal
HTTPError and killed the whole sync.
Mirrors the existing pattern in the GitHub source for its pagination-cap
403: detect the benign cap response and stop paginating, keeping the rows
already gathered, instead of raising. Because reviews sync incrementally
off the `date` watermark, the next scheduled sync picks up where this one
stopped.
Generated-By: PostHog Code
Task-Id: 27f1c31a-731e-446a-8085-85305f54b168
Contributor
|
Hey @Gilbert09! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
Generated-By: PostHog Code Task-Id: 84af0109-67b4-481d-82b5-13e6bafa6bcc
MarconLP
approved these changes
Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Error tracking surfaced a fatal
HTTPErrorfrom the Appfigures reviews sync (issue):Appfigures'
/reviewsendpoint caps offset pagination atpage * count <= 10000. Once a sync's backlog is deep enough to cross that boundary (e.g. a large historical backfill), our pagination loop keeps incrementingpagepast the cap and the resulting 400 is treated as a fatal error, failing the whole sync.Changes
This is an upstream API limit, not something we can request around, so I applied the same pattern already used for GitHub's pagination-cap 403 (see #68292): detect the benign cap response and stop paginating gracefully, keeping the rows already gathered, instead of letting it raise via
raise_for_status._is_page_limit_responsedetects the stable "must be less than or equal to 10000" message (surfaced via the HTTP reason phrase, with a body-text fallback)._fetchraises a newAppfiguresPageLimitErrorcontrol-flow signal when it sees one, before the genericraise_for_statusfallback._iter_pagedcatches it and breaks the loop, keeping whatever pages were already yielded.Reviews sync incrementally off the
datewatermark, so on the next scheduled run the source resumes just past the last row it successfully synced, effectively walking a very large backlog forward in capped chunks across multiple syncs rather than failing outright.How did you test this code?
Added tests in
test_appfigures.py:TestIsPageLimitResponse— parameterized detector cases, including that a different 400 (or the same message on a different status) is not treated as the cap, so real errors stay fatal.test_page_limit_stops_and_keeps_rows_gathered_so_far— end-to-end throughget_rows/_iter_paged: page 1 succeeds, page 2 hits the cap, and the rows gathered from page 1 are yielded without raising.Ran the full
test_appfigures.pysuite (36 tests, all passing) plusruff check/ruff formatand a repo-widemypycheck (no new errors).Automatic notifications
Docs update
N/A — internal pagination/error-handling fix, no user-facing behavior or documented workflow changed.
🤖 Agent context
Autonomy: Fully autonomous
Authored by Claude Code (Sonnet 5) from an error-tracking alert for the Appfigures reviews sync. I confirmed the stack trace originates in
products/warehouse_sources/backend/temporal/data_imports/sources/appfigures/appfigures.py, read the pagination logic in_iter_paged, and classified this as a fixable bug (our pagination doesn't respect the API's documented depth cap) rather than a user/credentials error, since nothing about the customer's config caused it.I found PR #68292 (GitHub source's pagination-cap 403 fix) as a close precedent and mirrored its shape — a detector function, a control-flow exception, and a graceful loop break — rather than inventing a new mechanism. I invoked the
/writing-testsskill before adding tests.