Skip to content

get_job resolves a job in one call via GET /jobs/{id}#7

Merged
gradill22 merged 2 commits into
mainfrom
feat/single-call-get-job
Jul 26, 2026
Merged

get_job resolves a job in one call via GET /jobs/{id}#7
gradill22 merged 2 commits into
mainfrom
feat/single-call-get-job

Conversation

@gradill22

Copy link
Copy Markdown
Contributor

The problem

get_job called GET /bulk/{job_id} purely to read kind off the response, then re-fetched GET /crawl/{job_id} for the fields the bulk shape omits (truncated, truncated_reason, per-item depth).

Two costs, both measured against the dev API:

1. A crawl-scoped key could not poll its own job. /bulk/* requires the bulk scope, and the discovery call 403s in middleware before the route runs. With scopes=["crawl"], GET /bulk/{id}403 while GET /jobs/{id}200.

2. Two back-to-back requests per crawl poll, with no spacing — fast enough to trip the API's own per-plan min-spacing limiter:

GET /bulk/{id}  -> 200
GET /crawl/{id} -> 429  rate_limit_too_fast   Retry-After-Ms: 38

The SDK was 429ing itself against our own API on a normal crawl poll.

The change

GET /jobs/{id} answers for either kind and requires neither scope. get_job and wait_for_job now issue exactly one request per poll and read kind off each body, on both the sync and async clients. wait_for_job drops its is_crawl routing — every iteration hits the same path.

This mirrors the JS SDK change so the two clients stay in step.

Verification

69 tests pass. The suite previously pinned the two-call behaviour, so those tests changed with it. The new ones assert the request path, not just the parsed result, so a regression back to /bulk + /crawl fails loudly rather than silently costing a round trip:

assert respx.calls.call_count == 1
assert respx.calls[0].request.url.path == f"/jobs/{job_id}"

The equivalent change was additionally confirmed end-to-end against a live local API through the JS client (same endpoint, same contract): one HTTP call for get_job and wait_for_job on a real crawl job, with kind, truncated, and per-item depth intact.

Deploy ordering

Requires the API change that adds /jobs/{id} to be deployed first (WellMarkedAPI/WellMarked#192). /bulk/{id} and /crawl/{id} are unchanged there, so already-released SDK versions keep working — but this version will 404 against an API that predates the endpoint. No fallback was added; happy to add one if you'd prefer.

Not verified here

The Python client wasn't exercised against the live dev API in this pass — its tests are respx-mocked, and the live end-to-end run went through the JS client against the same endpoint. Worth one live call before release.

Companion PR: JS-TS-SDK#12.

🤖 Generated with Claude Code

gradill22 and others added 2 commits July 26, 2026 01:38
Mirrors the JS SDK change, for continuity across both clients.

get_job used to call GET /bulk/{job_id} purely to read `kind` off the
response, then re-fetch GET /crawl/{job_id} for the fields the bulk shape
omits (truncated, truncated_reason, per-item depth).

That cost two round trips per crawl poll, issued back-to-back with no
spacing — fast enough to trip the API's own per-plan min-spacing limiter
(observed against the dev API: rate_limit_too_fast, Retry-After-Ms 38).
And because /bulk/* requires the `bulk` scope, a key scoped to `crawl`
alone got a 403 on the discovery call and could never poll its own job.

GET /jobs/{id} answers for either kind and requires neither scope, so
get_job and wait_for_job now issue exactly one request per poll and read
`kind` off each body. Both the sync and async clients change; wait_for_job
drops its is_crawl routing, since every iteration hits the same path.

Requires the API change that adds /jobs/{id}; that must be deployed
before this SDK is released.

69 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mypy runs with disallow_any_generics, so the bare `dict` I annotated
failed the type-check job on every Python version while the tests passed.
Match the surrounding style (`dict[str, Any]`; the module already has
`from __future__ import annotations`, so the builtin generic is fine on
3.9).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gradill22 gradill22 self-assigned this Jul 26, 2026
@gradill22
gradill22 merged commit 8204aa4 into main Jul 26, 2026
6 checks passed
@gradill22
gradill22 deleted the feat/single-call-get-job branch July 26, 2026 06:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant