urlpipe

Turn any URL into clean Markdown, rendered HTML, a full-page screenshot, a PDF, metadata, a summary, keywords, data in a schema you define, console errors, a Lighthouse audit or the technologies it is built with, from Ruby. Pages are rendered in real Chrome, so JavaScript-heavy sites come back complete.

This is the official Ruby client for URLpipe. It has no runtime dependencies and runs on Ruby 3.1 and later.

bundle add urlpipe

or gem install urlpipe.

Get an API key from a project in the dashboard (the Free plan includes 1,000 credits a month, no card) and put it in URLPIPE_API_KEY:

require "urlpipe"

client = Urlpipe::Client.new # reads ENV["URLPIPE_API_KEY"]

response = client.markdown("https://example.com")
puts response.data           # "# Example Domain\n\n..."
response.meta.quota.remaining # => 943

Every call waits for the result and hands it back in response.data. Cached results and failed calls cost nothing.

Urlpipe::Client.new(
  api_key: nil,          # falls back to ENV["URLPIPE_API_KEY"]; raises Urlpipe::ConfigurationError if neither is set
  base_url: "https://urlpipe.dev",
  timeout: 90,           # seconds per HTTP request; the API holds a sync request up to 60 s
  max_retries: 2,        # see "Retries and idempotency"
  wait_timeout: 300,     # how long a long analysis is polled before Urlpipe::WaitTimeoutError
  poll_interval: 2,      # seconds between polls of GET /result/:token
  user_agent: nil        # your program's product token, e.g. "my-app/1.2"; it goes in front of the client's own
)

A client holds no connection state, so one instance can be shared across threads.

Every method takes the URL first. These options work on all of them and are sent only when you give them:

Option Hashes go to the API as given; it validates them and tells you what to change.

data is a String.

client.markdown("https://example.com", page_options: { block_cookie_banners: true }).data
client.html("https://example.com").data      # the rendered document
client.summarize("https://example.com").data # a Markdown summary

data is a Urlpipe::Screenshot: the decoded image bytes, its mime_type (image/png, image/jpeg or image/webp) and result_url, a link to the image that needs no API key.

shot = client.screenshot("https://example.com",
                         screenshot_options: { viewport_width: 390, format: "webp" }).data
shot.save("example.webp")
shot.mime_type  # => "image/webp"
shot.result_url # => "https://..." - put it straight in an <img> tag

data is a Urlpipe::Pdf: the whole rendered page as decoded PDF bytes, and result_url, a link to the PDF that needs no API key. pdf_options: choose paper_size, landscape, margin, scale, print_background, media and single_page.

pdf = client.pdf("https://example.com/invoice", pdf_options: { paper_size: "letter", landscape: true }).data
pdf.save("invoice.pdf")
pdf.result_url # => "https://..."

data is a Hash with the same keys for every page: final_url (where the page ended up, after its redirects; relative URLs are resolved against it), title, description, site_name, type, language, locale, authors, published_at, modified_at, image, video, favicon_url, icons, logo_url, feeds, keywords, canonical_url, robots, alternates, oembed_url, structured_data_types, and the raw open_graph and twitter tags. A value the page doesn't declare is nil or an empty Array. URLs are absolute, dates ISO 8601.

page = client.meta("https://example.com").data
page["title"]                   # => "Example Domain"
page.dig("image", "url")        # => "https://example.com/cover.jpg", or nil
page["authors"].map { _1["name"] }

data is an Array of Strings, most relevant first.

client.keywords("https://example.com").data # => ["example domain", "documentation", ...]

Data in a schema you define. Describe the fields you want and data is a Hash with exactly those keys, in that order, nil where the page gives no value. A field's type is string, number, integer, boolean, date, datetime, url, email, object (with its own fields) or array (with items).

product = client.extract(
  "https://example.com/products/trail-shoe",
  [
    { name: "name", type: "string" },
    { name: "price", type: "number", description: "The current price, without the currency" },
    { name: "in_stock", type: "boolean" },
    { name: "sizes", type: "array", items: "string" }
  ],
  instructions: "Ignore the related products."
).data
product["price"] # => 89.5

data is an Array of {"type" => "error" | "warning" | "exception", "text" => "..."}.

client.console("https://example.com").data.select { |entry| entry["type"] == "exception" }

data is a Hash: "final_url" and "technologies", an Array of what the page is built with, most certain first.

client.technologies("https://example.com").data["technologies"].map { |tech| [tech["name"], tech["version"]] }
# => [["WordPress", "6.4.2"], ["PHP", nil], ...]

data is the audit as a Hash. device: is "mobile" (the default) or "desktop"; include_audits: true adds the full audits object.

report = client.lighthouse("https://example.com", device: "desktop").data
report.dig("categories", "performance", "score")

Several operations off one page visit, so you get them all much sooner. data is {"url" => ..., "operations" => {"markdown" => {"success", "result", "error", "cached"}, ...}}. Each operation succeeds or fails on its own. A screenshot or a PDF inside a scrape stays Base64, as the API returns it. scrape also takes screenshot_options:, pdf_options:, and fields: and instructions: for its extract operation.

page = client.scrape("https://example.com", %w[markdown meta screenshot],
                     screenshot_options: { format: "jpeg" }).data
page.dig("operations", "meta", "result", "title")

Every method returns a Urlpipe::Response:

A value the API didn't send is nil; idempotent_replayed is always true or false. On an unlimited plan quota.limit, quota.remaining and concurrency_limit are the String "unlimited".

A sync request is held for up to 60 seconds. When an analysis takes longer (a Lighthouse audit, a slow site), the client polls GET /result/:token every 2 seconds and returns the result when it lands, so your code sees one call. After wait_timeout it raises Urlpipe::WaitTimeoutError, whose token you can collect later: the analysis keeps running.

With sync: false the call returns at once with a token, and the work continues in the background. Collect it with wait, or with a webhook:

accepted = client.lighthouse("https://example.com", sync: false)
accepted.status # => "accepted"

report = client.wait(accepted.token, operation: "lighthouse").data

wait(token, operation: nil, timeout: wait_timeout, interval: poll_interval) polls until the result is ready. result(token, operation: nil) checks once and returns a "processing" Response while it is still running.

The API doesn't say which operation produced a token, so pass operation: to get data typed the way that method returns it. Without it JSON comes back parsed and text stays a String, which means a screenshot or a PDF stays Base64: client.wait(token, operation: "screenshot") decodes it into a Urlpipe::Screenshot, and operation: "pdf" into a Urlpipe::Pdf.

Turn on webhook signing for the project (Settings → Webhook Signing) and verify each delivery before trusting it. Urlpipe::Webhook.verify needs no client:

# app/controllers/urlpipe_webhooks_controller.rb
class UrlpipeWebhooksController < ActionController::API
  def create
    payload = Urlpipe::Webhook.verify(request.raw_post, request.headers,
                                      ENV.fetch("URLPIPE_WEBHOOK_SECRET"))
    ProcessResultJob.perform_later(payload["token"])
    head :ok
  rescue Urlpipe::WebhookVerificationError
    head :unauthorized
  end
end

verify(raw_body, headers, secret, tolerance: 300) returns the payload Hash (token, operation, labels, success, result, result_url, final_url, error, meta), or raises Urlpipe::WebhookVerificationError saying why. It accepts any signature in the header that matches, so deliveries keep verifying through a secret rotation, and it refuses a timestamp more than tolerance seconds from now.

Pass the raw request body, exactly the bytes received: request.raw_post in Rails, request.body.read in Rack. Parsed and re-serialized JSON has different bytes and will not verify. headers can be Rails' request.headers, a Rack env or a plain Hash.

Make the handler idempotent on token: a delivery can be retried.

Everything raised is a Urlpipe::Error, with status, code (the API's error code, when there is one), message, body (parsed JSON, or the raw text) and token.

begin
  client.markdown(url)
rescue Urlpipe::AnalysisFailedError => e
  logger.info("Skipped #{url}: #{e.message}") # "The requested page was not found."
rescue Urlpipe::QuotaExceededError => e
  retry_at(e.resets_at)
end

The client retries, up to max_retries times, what a retry can fix: connection errors, 500/502/503, rate_limited (after Retry-After, capped at 60 s) and concurrency_limit (after 1 s, 2 s, ...). It never retries 401, 403, 404, 410, 422 or quota_exceeded.

Every analysis request that may be retried carries an Idempotency-Key: yours if you pass idempotency_key:, otherwise a fresh UUID for that call, reused on each of its retries. The API answers a repeated key with the first request's token and result, so a retry after a dropped connection never runs or bills the work twice. Pass your own key (a job id, say) to get the same guarantee across process restarts:

client.markdown(url, idempotency_key: "import-#{job.id}")

Set max_retries: 0 to turn retries off; no key is generated then.

• API docs: https://urlpipe.dev/docs
• Pricing: https://urlpipe.dev/pricing
• MCP server, to give an AI assistant the same tools: https://github.com/URLpipe/mcp

bundle install
bundle exec rake test

The tests run against a local stub server and never touch the network. Set URLPIPE_API_KEY to also run one live smoke test against the real API.

MIT. Copyright (c) 2026 Aliat Partner S.L.