Skip to content

v0.1.0

Choose a tag to compare

@github-actions github-actions released this 01 Nov 19:21
· 16 commits to main since this release
v0.1.0
89efc9b

The first release of sdo, the Scrape.do CLI, built on the sdo-go SDK.

Added

  • sdo scrape <url> — scrape a URL; the body goes to stdout (or
    --output-file, written with mode 0600), a metadata line to stderr (the
    cost only when the gateway reports it). A flag for every documented scrape
    parameter (--render, --super, --geo-code, --wait-selector,
    --set-cookies, --zip-code,
    --transparent-response, …), and --param key=value (repeatable) for any
    gateway parameter without a flag of its own. --method, --data,
    --data-json and -H/--header 'Key: Value' shape the request sent to the
    target; an invalid header is rejected before anything is sent.
  • sdo scrape --help describes each flag as the gateway documents it:
    --pure-cookies returns the target's Set-Cookie header instead of
    Scrape.do-Cookies (the body is unchanged); --return-json returns the
    network requests and content as JSON; --session-id is a sticky proxy
    session (the same IP), an integer 0–1000000; --regional-geo-code requires
    --super and cannot be combined with --geo-code; --extra-headers sends
    Sd-<Name> headers; the screenshot, frame and WebSocket flags need
    --return-json; --width and --height need --render.
  • sdo plugin <route> --param key=value — call a documented plugin
    endpoint; a repeated --param key sends a list. Like sdo scrape, it prints
    a metadata line with the status and cost on stderr (status=200 cost=10).
    The route amazon calls the Amazon raw-HTML plugin at /plugin/amazon/.
    The examples use gl (Google's country parameter), not geoCode.
  • sdo usage — the account's usage statistics from GET /info as JSON
    (IsActive, ConcurrentRequest, MaxMonthlyRequest,
    RemainingConcurrentRequest, RemainingMonthlyRequest); no credits. The
    endpoint's 429 (10 requests a minute) is not retried.
  • Parameter rules. The SDK's documented dependency checks reach the CLI as
    usage errors (exit 2) in flag terms, e.g. --screenshot requires --return-json, --regional-geo-code requires --super, --session-id must be an integer from 0 to 1000000. --render with a --method other than
    GET, or with a request body, is refused before anything is sent (the
    gateway answers it with a 400).
  • With --disable-redirection, sdo scrape adds the target's redirect
    (Scrape.do-Target-Redirected-Location) to the metadata line as
    location=....
  • Target statuses. Scrape.do-Request-Cost is authoritative: a target 404
    or 410, or any status of 400 or above the gateway billed (a positive cost —
    the target's own 400, 401, 403, 422, ...), is a page: printed with its status
    and cost in the metadata line, exit 0. The gateway's own errors carry no
    cost and keep their exit codes (its 400, a rejected parameter, exits 1; its
    401 exits 3) with the gateway's (or a plugin's {"error", "message"})
    message; an error's cost is appended when the gateway reported one.
  • sdo jobs — the async job API (q.scrape.do): create <url...> [--option Key=Value ...], status <jobID>, result <jobID> <taskID> [--json], list, cancel <jobID>. result writes the task's content byte
    for byte (base64 content decoded) and exits 0 only for a successful task:
    1 when it failed or was canceled, 4 while it has not finished.
  • sdo account — account concurrency and remaining credits.
  • sdo auth login / sdo auth status — store the token in a per-user
    config file (mode 0600, written atomically, symlinks refused) so it need
    not live in shell history or in an exported environment variable. Token
    precedence: --token > SDO_TOKEN > config file (SDO_CONFIG_FILE
    overrides the path). Surrounding whitespace is trimmed from every source.
  • sdo mcp serve — a bundled Model Context Protocol server over stdio
    with two tools, scrape and scrape_plugin (which reports the status and
    cost like scrape, lists the documented plugin routes and takes scalar
    params only: a multi-value plugin param is one comma-separated string). The
    scrape tool's returnJson returns the page's network requests and content
    as JSON, and its sessionId (a sticky proxy session, the same IP) is an
    integer 0–1000000, given as a number or a string. Arguments are validated
    against each tool's input schema (-32602 on an unknown or mistyped one);
    tool calls run concurrently (up to 4) and honour notifications/cancelled;
    initialize negotiates the protocol version (2025-06-18, 2025-03-26,
    2024-11-05).
  • Global flags --client-timeout (the client's per-attempt timeout, in
    seconds; else SDO_TIMEOUT, default 70: the gateway's 60-second default
    plus 10), --max-response-bytes (default
    32 MiB) and --insecure-http (allow a plain-http base URL off loopback).
  • Exit codes a script can branch on: 0 success, 1 failure, 2 usage
    error or input rejected before sending, 3 no token or token refused, 4
    async task not finished. Every failure is printed once as sdo: <reason>,
    naming the CLI's flags and environment variables.
  • sdo version (the CLI's and the SDK's version) and sdo --version.
  • api-surface.txt, the command, flag and MCP tool surface, generated by
    go run ./tools/apisurface and checked in CI.

Security

  • Everything the CLI prints as its own message (errors, metadata lines) is
    redacted: the token, literal and percent-encoded, and credential-shaped
    values. Data you asked for (page and task content, JSON output) is written as
    delivered except for the token itself.
  • Every MCP tool result is redacted. Page content, plugin output and error
    text are fenced between untrusted-content markers carrying a per-response
    random nonce; marker look-alikes, including ones built with Unicode tricks
    (compatibility forms, zero-width characters, Unicode spaces and dashes,
    homoglyphs), are neutralized; output is capped at 128 KiB; binary responses
    are described, not returned. Plugin routes from the model are validated, and
    plugin parameters cannot carry a token.
  • An MCP tool result is cut to its 128 KiB cap plus a margin before it is
    redacted and its marker look-alikes neutralized, so a 32 MiB page is cut in
    under 50 MiB of memory; a credential that straddles the cut is still masked
    whole.
  • CI: code from a fork's pull request never reaches the org's persistent
    self-hosted runners. Every job runs on GitHub-hosted runners except CodeQL
    (for same-repository events; a fork's pull request is analysed once
    merged), OpenSSF Scorecard (on main and weekly) and the release's
    github-release job (which checks out no code): the org's IP allow list
    accepts their GitHub API calls only from the self-hosted runners. Every
    checkout drops its token
    (persist-credentials: false); and gitleaks scans with a config and ignore
    list from the workflow, so a pull request cannot allowlist its own secret with
    a .gitleaks.toml or .gitleaksignore.
  • The token is sent only over https (plain http only to loopback, or with
    --insecure-http), and redirects are never followed.
  • The CLI and the MCP server identify as User-Agent: sdo-cli/<version> and
    nothing else. No telemetry, no update check.
  • Control characters in API- or target-supplied text on stderr (URLs, task
    states, error messages) are printed as escapes (\x1b, \u009b), so a
    page cannot drive the terminal with an escape sequence.
  • A body with NUL or ESC bytes is not written to a terminal stdout (sdo scrape, sdo jobs result, a raw sdo plugin body): the command fails and
    says to redirect, pipe or use --output-file, as curl does; --raw
    overrides.
  • --output-file refuses a symlink or non-regular file and writes through a
    0600 temporary file renamed over the target, so an existing file does not
    keep wider permissions.
  • The config file is opened with O_NOFOLLOW (Unix) and its type and mode are
    checked on the opened file, closing the window between the check and the
    read; sdo auth login narrows an existing default config directory to
    0700 when it is yours and wider.
  • sdo mcp serve stops on a cancelled context even while stdin is open,
    answers a request line over 16 MiB with a JSON-RPC error instead of exiting,
    and refuses a tool call with -32000 once 64 are running or waiting.
  • Releases: archives built by GoReleaser in a read-only job and published only
    after approval, for tags on main, with SPDX SBOMs, cosign-signed checksums,
    NOTICE and the third-party license texts (THIRD_PARTY_LICENSES.txt).

Full changelog: https://github.com/scrape-do/sdo-cli/blob/v0.1.0/CHANGELOG.md