Repository navigation
v0.1.0
The first release of sdo, the Scrape.do CLI, built on the sdo-go SDK.
Added
sdo scrape <url>— scrape a URL; the body goes to stdout (or
--output-file, written with mode0600), a metadata line to stderr (the
cost only when the gateway reports it). A flag for every documented scrape
parameter (--render,--super,--geo-code,--wait-selector,
--set-cookies,--zip-code,
--transparent-response, …), and--param key=value(repeatable) for any
gateway parameter without a flag of its own.--method,--data,
--data-jsonand-H/--header 'Key: Value'shape the request sent to the
target; an invalid header is rejected before anything is sent.sdo scrape --helpdescribes each flag as the gateway documents it:
--pure-cookiesreturns the target'sSet-Cookieheader instead of
Scrape.do-Cookies(the body is unchanged);--return-jsonreturns the
network requests and content as JSON;--session-idis a sticky proxy
session (the same IP), an integer 0–1000000;--regional-geo-coderequires
--superand cannot be combined with--geo-code;--extra-headerssends
Sd-<Name>headers; the screenshot, frame and WebSocket flags need
--return-json;--widthand--heightneed--render.sdo plugin <route> --param key=value— call a documented plugin
endpoint; a repeated--paramkey sends a list. Likesdo scrape, it prints
a metadata line with the status and cost on stderr (status=200 cost=10).
The routeamazoncalls the Amazon raw-HTML plugin at/plugin/amazon/.
The examples usegl(Google's country parameter), notgeoCode.sdo usage— the account's usage statistics fromGET /infoas JSON
(IsActive,ConcurrentRequest,MaxMonthlyRequest,
RemainingConcurrentRequest,RemainingMonthlyRequest); no credits. The
endpoint's 429 (10 requests a minute) is not retried.- Parameter rules. The SDK's documented dependency checks reach the CLI as
usage errors (exit 2) in flag terms, e.g.--screenshot requires --return-json,--regional-geo-code requires --super,--session-id must be an integer from 0 to 1000000.--renderwith a--methodother than
GET, or with a request body, is refused before anything is sent (the
gateway answers it with a 400). - With
--disable-redirection,sdo scrapeadds the target's redirect
(Scrape.do-Target-Redirected-Location) to the metadata line as
location=.... - Target statuses.
Scrape.do-Request-Costis authoritative: a target 404
or 410, or any status of 400 or above the gateway billed (a positive cost —
the target's own 400, 401, 403, 422, ...), is a page: printed with its status
and cost in the metadata line, exit 0. The gateway's own errors carry no
cost and keep their exit codes (its 400, a rejected parameter, exits 1; its
401 exits 3) with the gateway's (or a plugin's{"error", "message"})
message; an error's cost is appended when the gateway reported one. sdo jobs— the async job API (q.scrape.do):create <url...> [--option Key=Value ...],status <jobID>,result <jobID> <taskID> [--json],list,cancel <jobID>.resultwrites the task's content byte
for byte (base64 content decoded) and exits0only for a successful task:
1when it failed or was canceled,4while it has not finished.sdo account— account concurrency and remaining credits.sdo auth login/sdo auth status— store the token in a per-user
config file (mode0600, written atomically, symlinks refused) so it need
not live in shell history or in an exported environment variable. Token
precedence:--token>SDO_TOKEN> config file (SDO_CONFIG_FILE
overrides the path). Surrounding whitespace is trimmed from every source.sdo mcp serve— a bundled Model Context Protocol server over stdio
with two tools,scrapeandscrape_plugin(which reports the status and
cost likescrape, lists the documented plugin routes and takes scalar
params only: a multi-value plugin param is one comma-separated string). The
scrapetool'sreturnJsonreturns the page's network requests and content
as JSON, and itssessionId(a sticky proxy session, the same IP) is an
integer 0–1000000, given as a number or a string. Arguments are validated
against each tool's input schema (-32602on an unknown or mistyped one);
tool calls run concurrently (up to 4) and honournotifications/cancelled;
initializenegotiates the protocol version (2025-06-18, 2025-03-26,
2024-11-05).- Global flags
--client-timeout(the client's per-attempt timeout, in
seconds; elseSDO_TIMEOUT, default 70: the gateway's 60-second default
plus 10),--max-response-bytes(default
32 MiB) and--insecure-http(allow a plain-http base URL off loopback). - Exit codes a script can branch on:
0success,1failure,2usage
error or input rejected before sending,3no token or token refused,4
async task not finished. Every failure is printed once assdo: <reason>,
naming the CLI's flags and environment variables. sdo version(the CLI's and the SDK's version) andsdo --version.api-surface.txt, the command, flag and MCP tool surface, generated by
go run ./tools/apisurfaceand checked in CI.
Security
- Everything the CLI prints as its own message (errors, metadata lines) is
redacted: the token, literal and percent-encoded, and credential-shaped
values. Data you asked for (page and task content, JSON output) is written as
delivered except for the token itself. - Every MCP tool result is redacted. Page content, plugin output and error
text are fenced between untrusted-content markers carrying a per-response
random nonce; marker look-alikes, including ones built with Unicode tricks
(compatibility forms, zero-width characters, Unicode spaces and dashes,
homoglyphs), are neutralized; output is capped at 128 KiB; binary responses
are described, not returned. Plugin routes from the model are validated, and
plugin parameters cannot carry a token. - An MCP tool result is cut to its 128 KiB cap plus a margin before it is
redacted and its marker look-alikes neutralized, so a 32 MiB page is cut in
under 50 MiB of memory; a credential that straddles the cut is still masked
whole. - CI: code from a fork's pull request never reaches the org's persistent
self-hosted runners. Every job runs on GitHub-hosted runners except CodeQL
(for same-repository events; a fork's pull request is analysed once
merged), OpenSSF Scorecard (onmainand weekly) and the release's
github-releasejob (which checks out no code): the org's IP allow list
accepts their GitHub API calls only from the self-hosted runners. Every
checkout drops its token
(persist-credentials: false); and gitleaks scans with a config and ignore
list from the workflow, so a pull request cannot allowlist its own secret with
a.gitleaks.tomlor.gitleaksignore. - The token is sent only over https (plain http only to loopback, or with
--insecure-http), and redirects are never followed. - The CLI and the MCP server identify as
User-Agent: sdo-cli/<version>and
nothing else. No telemetry, no update check. - Control characters in API- or target-supplied text on stderr (URLs, task
states, error messages) are printed as escapes (\x1b,\u009b), so a
page cannot drive the terminal with an escape sequence. - A body with NUL or ESC bytes is not written to a terminal stdout (
sdo scrape,sdo jobs result, a rawsdo pluginbody): the command fails and
says to redirect, pipe or use--output-file, as curl does;--raw
overrides. --output-filerefuses a symlink or non-regular file and writes through a
0600temporary file renamed over the target, so an existing file does not
keep wider permissions.- The config file is opened with
O_NOFOLLOW(Unix) and its type and mode are
checked on the opened file, closing the window between the check and the
read;sdo auth loginnarrows an existing default config directory to
0700when it is yours and wider. sdo mcp servestops on a cancelled context even while stdin is open,
answers a request line over 16 MiB with a JSON-RPC error instead of exiting,
and refuses a tool call with-32000once 64 are running or waiting.- Releases: archives built by GoReleaser in a read-only job and published only
after approval, for tags onmain, with SPDX SBOMs, cosign-signed checksums,
NOTICEand the third-party license texts (THIRD_PARTY_LICENSES.txt).
Full changelog: https://github.com/scrape-do/sdo-cli/blob/v0.1.0/CHANGELOG.md