Skip to content

Releases

v0.3.1

Choose a tag to compare

@github-actions github-actions released this 15 Nov 11:23
Immutable release. Only release title and notes can be modified.
v0.3.1
a1a5e2d

Added

  • The Homebrew formula installs bash, zsh and fish completions, generated by
    sdo completion <shell>.

Changed

  • Built on sdo-go v0.2.1.

Fixed

  • --zip-code is sent as the documented zipcode query parameter (it was sent
    as zipCode).
  • sdo mcp serve names itself sdo in the MCP handshake (serverInfo); it
    reported the standalone server's name, sdo-mcp, with the CLI's version.
  • The scrape_plugin tool description lists the documented
    google/search/ai-overview, google/shopping/product/stores, google/lens
    and google/goto routes.

Full changelog: https://github.com/scrape-do/sdo-cli/blob/v0.3.1/CHANGELOG.md

v0.3.0

Choose a tag to compare

@github-actions github-actions released this 14 Nov 22:12
v0.3.0
afe79e4

Added

  • Homebrew and Scoop: brew install scrape-do/tap/sdo (macOS and Linux) and
    scoop bucket add scrape-do https://github.com/scrape-do/scoop-bucket then
    scoop install scrape-do/sdo (Windows). Each final release updates the
    formula and the manifest, with the hashes from the release's signed
    checksums.txt; release candidates stay on GitHub only. The
    package-managers release job that commits them runs on the org's
    self-hosted runners and runs no repository code.
  • sdo version --check asks GitHub, once and anonymously, whether a newer
    release exists, and prints the upgrade commands (brew upgrade sdo,
    scoop update sdo, go install ...@latest) and the release page when one
    does. It exits 1 only when the check itself fails, never because a newer
    release exists. Nothing checks for updates without this flag.

Full changelog: https://github.com/scrape-do/sdo-cli/blob/v0.3.0/CHANGELOG.md

v0.2.0

Choose a tag to compare

@github-actions github-actions released this 04 Nov 14:00
v0.2.0
f4b41ac

Added

  • sdo jobs create-plugin <key> submits an async plugin job: one task per
    entry of a JSON params array given with --params, or --params-file
    (a path, or - for stdin), up to 1000 entries; --option adds job body
    options such as GeoCode or WebhookURL.
  • sdo jobs list --page --page-size --status --from --to --sort: the job list
    is paginated (10 jobs a page by default). The output is the page object
    (Jobs, TotalCount, PageSize, PageNumber, TotalPages) and stderr
    shows page N of M. Job statuses also carry AcquiredConcurrency and
    LimitConcurrency.
  • The bundled MCP server (sdo mcp serve) gains the async job tools:
    async_job_create, async_plugin_job_create, async_job_status,
    async_task_result, async_jobs_list, async_job_cancel and
    async_account.

Changed

  • Breaking: sdo jobs list prints a page object instead of a bare
    array of jobs (read .Jobs, page with --page; see MIGRATING.md). It used
    to show only the first 10 jobs while describing itself as every job.
  • Built on sdo-go v0.2.0.

Full changelog: https://github.com/scrape-do/sdo-cli/blob/v0.2.0/CHANGELOG.md

v0.1.0

Choose a tag to compare

@github-actions github-actions released this 01 Nov 19:21
v0.1.0
89efc9b

The first release of sdo, the Scrape.do CLI, built on the sdo-go SDK.

Added

  • sdo scrape <url> — scrape a URL; the body goes to stdout (or
    --output-file, written with mode 0600), a metadata line to stderr (the
    cost only when the gateway reports it). A flag for every documented scrape
    parameter (--render, --super, --geo-code, --wait-selector,
    --set-cookies, --zip-code,
    --transparent-response, …), and --param key=value (repeatable) for any
    gateway parameter without a flag of its own. --method, --data,
    --data-json and -H/--header 'Key: Value' shape the request sent to the
    target; an invalid header is rejected before anything is sent.
  • sdo scrape --help describes each flag as the gateway documents it:
    --pure-cookies returns the target's Set-Cookie header instead of
    Scrape.do-Cookies (the body is unchanged); --return-json returns the
    network requests and content as JSON; --session-id is a sticky proxy
    session (the same IP), an integer 0–1000000; --regional-geo-code requires
    --super and cannot be combined with --geo-code; --extra-headers sends
    Sd-<Name> headers; the screenshot, frame and WebSocket flags need
    --return-json; --width and --height need --render.
  • sdo plugin <route> --param key=value — call a documented plugin
    endpoint; a repeated --param key sends a list. Like sdo scrape, it prints
    a metadata line with the status and cost on stderr (status=200 cost=10).
    The route amazon calls the Amazon raw-HTML plugin at /plugin/amazon/.
    The examples use gl (Google's country parameter), not geoCode.
  • sdo usage — the account's usage statistics from GET /info as JSON
    (IsActive, ConcurrentRequest, MaxMonthlyRequest,
    RemainingConcurrentRequest, RemainingMonthlyRequest); no credits. The
    endpoint's 429 (10 requests a minute) is not retried.
  • Parameter rules. The SDK's documented dependency checks reach the CLI as
    usage errors (exit 2) in flag terms, e.g. --screenshot requires --return-json, --regional-geo-code requires --super, --session-id must be an integer from 0 to 1000000. --render with a --method other than
    GET, or with a request body, is refused before anything is sent (the
    gateway answers it with a 400).
  • With --disable-redirection, sdo scrape adds the target's redirect
    (Scrape.do-Target-Redirected-Location) to the metadata line as
    location=....
  • Target statuses. Scrape.do-Request-Cost is authoritative: a target 404
    or 410, or any status of 400 or above the gateway billed (a positive cost —
    the target's own 400, 401, 403, 422, ...), is a page: printed with its status
    and cost in the metadata line, exit 0. The gateway's own errors carry no
    cost and keep their exit codes (its 400, a rejected parameter, exits 1; its
    401 exits 3) with the gateway's (or a plugin's {"error", "message"})
    message; an error's cost is appended when the gateway reported one.
  • sdo jobs — the async job API (q.scrape.do): create <url...> [--option Key=Value ...], status <jobID>, result <jobID> <taskID> [--json], list, cancel <jobID>. result writes the task's content byte
    for byte (base64 content decoded) and exits 0 only for a successful task:
    1 when it failed or was canceled, 4 while it has not finished.
  • sdo account — account concurrency and remaining credits.
  • sdo auth login / sdo auth status — store the token in a per-user
    config file (mode 0600, written atomically, symlinks refused) so it need
    not live in shell history or in an exported environment variable. Token
    precedence: --token > SDO_TOKEN > config file (SDO_CONFIG_FILE
    overrides the path). Surrounding whitespace is trimmed from every source.
  • sdo mcp serve — a bundled Model Context Protocol server over stdio
    with two tools, scrape and scrape_plugin (which reports the status and
    cost like scrape, lists the documented plugin routes and takes scalar
    params only: a multi-value plugin param is one comma-separated string). The
    scrape tool's returnJson returns the page's network requests and content
    as JSON, and its sessionId (a sticky proxy session, the same IP) is an
    integer 0–1000000, given as a number or a string. Arguments are validated
    against each tool's input schema (-32602 on an unknown or mistyped one);
    tool calls run concurrently (up to 4) and honour notifications/cancelled;
    initialize negotiates the protocol version (2025-06-18, 2025-03-26,
    2024-11-05).
  • Global flags --client-timeout (the client's per-attempt timeout, in
    seconds; else SDO_TIMEOUT, default 70: the gateway's 60-second default
    plus 10), --max-response-bytes (default
    32 MiB) and --insecure-http (allow a plain-http base URL off loopback).
  • Exit codes a script can branch on: 0 success, 1 failure, 2 usage
    error or input rejected before sending, 3 no token or token refused, 4
    async task not finished. Every failure is printed once as sdo: <reason>,
    naming the CLI's flags and environment variables.
  • sdo version (the CLI's and the SDK's version) and sdo --version.
  • api-surface.txt, the command, flag and MCP tool surface, generated by
    go run ./tools/apisurface and checked in CI.

Security

  • Everything the CLI prints as its own message (errors, metadata lines) is
    redacted: the token, literal and percent-encoded, and credential-shaped
    values. Data you asked for (page and task content, JSON output) is written as
    delivered except for the token itself.
  • Every MCP tool result is redacted. Page content, plugin output and error
    text are fenced between untrusted-content markers carrying a per-response
    random nonce; marker look-alikes, including ones built with Unicode tricks
    (compatibility forms, zero-width characters, Unicode spaces and dashes,
    homoglyphs), are neutralized; output is capped at 128 KiB; binary responses
    are described, not returned. Plugin routes from the model are validated, and
    plugin parameters cannot carry a token.
  • An MCP tool result is cut to its 128 KiB cap plus a margin before it is
    redacted and its marker look-alikes neutralized, so a 32 MiB page is cut in
    under 50 MiB of memory; a credential that straddles the cut is still masked
    whole.
  • CI: code from a fork's pull request never reaches the org's persistent
    self-hosted runners. Every job runs on GitHub-hosted runners except CodeQL
    (for same-repository events; a fork's pull request is analysed once
    merged), OpenSSF Scorecard (on main and weekly) and the release's
    github-release job (which checks out no code): the org's IP allow list
    accepts their GitHub API calls only from the self-hosted runners. Every
    checkout drops its token
    (persist-credentials: false); and gitleaks scans with a config and ignore
    list from the workflow, so a pull request cannot allowlist its own secret with
    a .gitleaks.toml or .gitleaksignore.
  • The token is sent only over https (plain http only to loopback, or with
    --insecure-http), and redirects are never followed.
  • The CLI and the MCP server identify as User-Agent: sdo-cli/<version> and
    nothing else. No telemetry, no update check.
  • Control characters in API- or target-supplied text on stderr (URLs, task
    states, error messages) are printed as escapes (\x1b, \u009b), so a
    page cannot drive the terminal with an escape sequence.
  • A body with NUL or ESC bytes is not written to a terminal stdout (sdo scrape, sdo jobs result, a raw sdo plugin body): the command fails and
    says to redirect, pipe or use --output-file, as curl does; --raw
    overrides.
  • --output-file refuses a symlink or non-regular file and writes through a
    0600 temporary file renamed over the target, so an existing file does not
    keep wider permissions.
  • The config file is opened with O_NOFOLLOW (Unix) and its type and mode are
    checked on the opened file, closing the window between the check and the
    read; sdo auth login narrows an existing default config directory to
    0700 when it is yours and wider.
  • sdo mcp serve stops on a cancelled context even while stdin is open,
    answers a request line over 16 MiB with a JSON-RPC error instead of exiting,
    and refuses a tool call with -32000 once 64 are running or waiting.
  • Releases: archives built by GoReleaser in a read-only job and published only
    after approval, for tags on main, with SPDX SBOMs, cosign-signed checksums,
    NOTICE and the third-party license texts (THIRD_PARTY_LICENSES.txt).

Full changelog: https://github.com/scrape-do/sdo-cli/blob/v0.1.0/CHANGELOG.md