Skip to content

Fix SDK to working state (port gaps vs sdk-ts): validation, batch, target_meta, packaging, CI - #4

Merged
JuliusTva merged 17 commits into
mainfrom
fix/working-sdk-port-gaps
Aug 3, 2026
Merged

Fix SDK to working state (port gaps vs sdk-ts): validation, batch, target_meta, packaging, CI#4
JuliusTva merged 17 commits into
mainfrom
fix/working-sdk-port-gaps

Conversation

@DonatasDecodo

Copy link
Copy Markdown
Member

What

Makes the Python SDK actually work and brings it in line with the reference sdk-ts. Julius ported the TS SDK but a few things were left incomplete/broken: 4 unit tests failing on main, batch unusable, package not type-marked, CI not running tests, and the publish workflow misconfigured. All fixes below were cross-checked against Decodo/sdk-ts as the ground truth.

Fixes

Correctness (were failing tests / broken features)

  • Request-schema validation wired into scrape / scrape_async. Ported from sdk-ts web-scraping-api.ts validate() — the Python port had stored self._schema but never used it. Now accepts a typed *Params model or a plain dict, validates against the target's JSON schema via jsonschema, raises ValidationError before the HTTP call. scrape_batch does not validate (matches TS). Fixes 3 tests.
  • target_meta emitted as plain dicts, not a Pydantic model. Codegen was generating a TargetMeta(BaseModel) that collided with the TargetMeta TypedDict in schema/types.py and broke meta["group"] subscripting. TS emits plain objects — codegen now matches. Fixes 1 test.
  • Real BatchRequest. Was aliased BatchRequest = ScrapeRequest, so query/url stayed str and batch rejected list input client-side. Codegen now generates per-target *BatchParams (query/url as list[str]) and a proper discriminated BatchRequest, mirroring the TS mapped type. The bundled batch example now runs.
  • __init__.__all__: removed phantom TargetTargetParams (ImportError), exported the 52 *BatchParams + BatchRequest.

Packaging / release

  • py.typed marker added + shipped via package-data (SDK was untyped to consumers despite full annotations).
  • jsonschema>=4.0 promoted to a runtime dependency (validation needs it); types-jsonschema added to dev.
  • Publish workflow: added contents: read to worklfow.yml. This is the confirmed root cause of the failed PyPI publish — on a private repo, declaring only id-token: write zeroes contents, so actions/checkout fails with "Repository not found". (Filename typo left as-is to avoid breaking the future Trusted Publisher binding.)
  • Removed the decodo-codegen console script — it had no argparse and would silently run a full codegen (network + overwrite generated files, into site-packages when installed) on any invocation incl. --help. Codegen stays runnable via python -m decodo.codegen.codegen for maintainers.

Tooling / docs

  • CI now runs pytest (test.yml) on push + PR — the gap that let the failing tests ship unnoticed.
  • README fixed: constructor snippets used a signature that doesn't exist (DecodoClient(web_scraping_api={...})DecodoConfig(...)), scrape snippets now use typed params, removed a stray git merge-conflict marker, documented that token is the base64 user:pass basic-auth token, added a git-install fallback while the package isn't on PyPI.

Verification

  • pytest: 21 passed (was 4 failed / 17 passed)
  • ruff check: clean · mypy (strict, repo config): clean
  • Wheel build: py.typed present in artifact
  • Batch: GoogleSearchBatchParams(query=[...]) accepted; GoogleSearchParams(query=[...]) still correctly rejected
  • Every __all__ entry importable

Deliberately NOT changed (flagging for a call, not silently altering)

  • TimeoutError does not inherit DecodoError, and AuthenticationError/ValidationError hardcode 401/422 even for 403/400 responses. Both are faithful to sdk-ts so I left them, but they're real usability sharp edges in both SDKs — worth deciding cross-SDK (a TimeoutError that escapes except DecodoError will surprise users).
  • requires-python >=3.12: no 3.12-only syntax in the code; deps allow older. Could relax to >=3.10 to widen adoption — left as-is pending a support-policy decision.

Still needed outside this PR

  • PyPI Trusted Publisher must be configured on pypi.org for this repo/workflow before publish succeeds — the contents: read fix unblocks checkout, but the project also doesn't exist on PyPI yet. This is a PyPI-side action, not a code change.

🤖 Generated with Claude Code

Donatas Kasparavicius and others added 6 commits July 22, 2026 10:23
- scrape() and scrape_async() now accept ScrapeRequest | Mapping[str, Any]
- Added _to_payload() helper: model_dump(by_alias=True) for Pydantic models, dict() for mappings
- Added _validate() method using jsonschema, raises decodo.errors.ValidationError on failure
- scrape_batch() accepts BatchRequest | Mapping[str, Any] but skips validation (mirrors TS)
- Constructor default changed to BundledSchema.shared (mirrors TS)
- Removed now-unnecessary type: ignore[arg-type] comments in tests

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- generate_targets.py: removed generated TargetMeta pydantic class; target_meta is
  now emitted as dict[str, dict[str, Any]] (plain dict literals, matching TS behavior)
- For each target, generate a *BatchParams class identical to *Params except url/query
  are typed list[str] | None instead of str | None
- BatchRequest is now a proper discriminated union of all *BatchParams classes (was
  wrongly aliased to ScrapeRequest)
- Regenerated src/decodo/generated/targets.py and request_schemas.py from IR

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Removed TargetTargetParams from __all__ (never existed, caused ImportError)
- Added all 52 *BatchParams classes and BatchRequest to imports and __all__
- GoogleSearchBatchParams and BatchRequest now publicly accessible for examples

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e script

- src/decodo/py.typed: empty marker file so mypy and PEP 561 consumers find types
- pyproject.toml: added jsonschema>=4.0 to runtime dependencies (scrape() uses it)
- pyproject.toml: added types-jsonschema>=4.0 to dev deps for mypy
- pyproject.toml: added [tool.setuptools.package-data] to ship py.typed in wheel
- pyproject.toml: removed [project.scripts] decodo-codegen entry (footgun: runs
  full network fetch + file overwrite on any invocation including --help; maintainers
  run via python -m decodo.codegen.codegen instead)
- pyproject.toml: added ^build/ to mypy exclude to avoid duplicate-module error

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- .github/workflows/test.yml: new workflow running pytest on push+PR, Python 3.12,
  installs .[dev]; mirrors lint.yml style
- .github/workflows/worklfow.yml: added contents: read permission so actions/checkout
  can access the private repo (id-token: write alone zeros contents, causing
  'Repository not found' on checkout)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
README.md:
- All DecodoClient(...) calls now use DecodoConfig(web_scraping_api=WebScrapingApiConfig(token=...))
- Quick start and all API snippets use typed Params objects (GoogleSearchParams etc.)
- Batch snippet uses GoogleSearchBatchParams with query as list
- Error handling snippet fixed to use typed params
- Documented that token is base64-encoded user:password from dashboard
- Added git install fallback (pip install from GitHub) until PyPI publish
- Removed git conflict marker line (>>>>>>> 03b68da)

examples/web_scraping_api/batch/google_search_batch.py:
- Import GoogleSearchBatchParams instead of GoogleSearchParams
- Removed duplicate print('Polling for results...') before the loop

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@JuliusTva
JuliusTva force-pushed the fix/working-sdk-port-gaps branch 6 times, most recently from 04ee7bc to f33d39e Compare July 27, 2026 10:57
@JuliusTva
JuliusTva force-pushed the fix/working-sdk-port-gaps branch from f33d39e to 8bb8e4a Compare July 27, 2026 11:00
@JuliusTva
JuliusTva force-pushed the fix/working-sdk-port-gaps branch from c8ab557 to e4b4da4 Compare July 30, 2026 12:30
@JuliusTva
JuliusTva force-pushed the fix/working-sdk-port-gaps branch 2 times, most recently from 9b31d3a to c7cb001 Compare July 31, 2026 07:21
@JuliusTva
JuliusTva force-pushed the fix/working-sdk-port-gaps branch from c7cb001 to 0a188f7 Compare July 31, 2026 07:23
@JuliusTva
JuliusTva merged commit a180c25 into main Aug 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants