get_job resolves a job in one call via GET /jobs/{id}#7
Merged
Conversation
Mirrors the JS SDK change, for continuity across both clients.
get_job used to call GET /bulk/{job_id} purely to read `kind` off the
response, then re-fetch GET /crawl/{job_id} for the fields the bulk shape
omits (truncated, truncated_reason, per-item depth).
That cost two round trips per crawl poll, issued back-to-back with no
spacing — fast enough to trip the API's own per-plan min-spacing limiter
(observed against the dev API: rate_limit_too_fast, Retry-After-Ms 38).
And because /bulk/* requires the `bulk` scope, a key scoped to `crawl`
alone got a 403 on the discovery call and could never poll its own job.
GET /jobs/{id} answers for either kind and requires neither scope, so
get_job and wait_for_job now issue exactly one request per poll and read
`kind` off each body. Both the sync and async clients change; wait_for_job
drops its is_crawl routing, since every iteration hits the same path.
Requires the API change that adds /jobs/{id}; that must be deployed
before this SDK is released.
69 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mypy runs with disallow_any_generics, so the bare `dict` I annotated failed the type-check job on every Python version while the tests passed. Match the surrounding style (`dict[str, Any]`; the module already has `from __future__ import annotations`, so the builtin generic is fine on 3.9). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
get_jobcalledGET /bulk/{job_id}purely to readkindoff the response, then re-fetchedGET /crawl/{job_id}for the fields the bulk shape omits (truncated,truncated_reason, per-itemdepth).Two costs, both measured against the dev API:
1. A crawl-scoped key could not poll its own job.
/bulk/*requires thebulkscope, and the discovery call 403s in middleware before the route runs. Withscopes=["crawl"],GET /bulk/{id}→ 403 whileGET /jobs/{id}→ 200.2. Two back-to-back requests per crawl poll, with no spacing — fast enough to trip the API's own per-plan min-spacing limiter:
The SDK was 429ing itself against our own API on a normal crawl poll.
The change
GET /jobs/{id}answers for either kind and requires neither scope.get_jobandwait_for_jobnow issue exactly one request per poll and readkindoff each body, on both the sync and async clients.wait_for_jobdrops itsis_crawlrouting — every iteration hits the same path.This mirrors the JS SDK change so the two clients stay in step.
Verification
69 tests pass. The suite previously pinned the two-call behaviour, so those tests changed with it. The new ones assert the request path, not just the parsed result, so a regression back to
/bulk+/crawlfails loudly rather than silently costing a round trip:The equivalent change was additionally confirmed end-to-end against a live local API through the JS client (same endpoint, same contract): one HTTP call for
get_jobandwait_for_jobon a real crawl job, withkind,truncated, and per-itemdepthintact.Deploy ordering
Requires the API change that adds
/jobs/{id}to be deployed first (WellMarkedAPI/WellMarked#192)./bulk/{id}and/crawl/{id}are unchanged there, so already-released SDK versions keep working — but this version will 404 against an API that predates the endpoint. No fallback was added; happy to add one if you'd prefer.Not verified here
The Python client wasn't exercised against the live dev API in this pass — its tests are respx-mocked, and the live end-to-end run went through the JS client against the same endpoint. Worth one live call before release.
Companion PR: JS-TS-SDK#12.
🤖 Generated with Claude Code