Skip to content

v1.3.0

Choose a tag to compare

@al2m4n al2m4n released this 19 Oct 11:00
4e39b9d

TheYgent 1.3 adds crawler nodes. Browse thousands of hosted scrapers and crawlers (Apify Actors), add one to an agent, and run it as a step, or wire it to an llm so the model can search the web, read a site, or find places. You decide what each call may do, not the model.

Crawler nodes (#90)

  • A Crawlers browser.
    • It searches the Apify Store through your control plane. Your browser never calls Apify.
    • Each card shows the Actor's pricing, users, and 30-day run success rate, so an Actor that usually times out shows as such before you install it.
    • Pick the fields the model may fill and press Add crawler. The node lands on the canvas, already wired into the llm's tools when you add it from one.
  • The builder decides what a call does.
    • The model fills only the fields you expose, usually one or two.
    • Result counts, paid add-ons and proxies stay fixed on the node, and arguments the model invents are dropped.
    • Offered every field, a local model in testing asked for 50 places per search and turned on paid review scraping. A crawler node keeps those fixed.
  • Small prompts for local models. Three Actors offered as crawler nodes cost about 780 prompt tokens on Qwen3.6-35B, against about 13,500 when every field is offered. List fields default to one value per call, the shape local models fill reliably.
  • Spend and size caps.
    • Each node has a per-run spend cap (maxChargeUsd), held under a platform ceiling: crawler.max_charge_usd on Settings → Crawlers, default $1.00.
    • Results are trimmed to the fields you choose, within a character budget.
  • Steps or tools, in both runtimes.
    • Interactive runs make one synchronous call, capped at 300 seconds.
    • Durable runs start the Actor, poll it, and read its dataset as separate steps, so a long crawl survives a restart. A resumed run polls the run it already started instead of paying for a second.
  • Cost in the trace. Crawler spans record the Actor, the item count, the cap and, on durable runs, what the run cost (crawler.usage_total_usd).
  • Your token stays a secret. The Apify token is stored once as an encrypted connection and resolved at call time. It never appears in the agent, a trace, or the durable journal.
  • A new sample: Web researcher. A local model with a web_search crawler, capped at three results and $0.05 a run. Install it from Settings → Samples and set your token.

Start with the Crawler node docs, or read the overview and the blog post. Prefer giving a capable model every Actor field? Apify through the MCP hub works too, with no crawler node.

Also in this release

  • Website.
  • Docs.
    • The crawler node and the MCP-hub alternative.
    • The privacy page, which notes that a crawler node sends its input to the provider it names.
    • The API and environment references.
    • The observability page's crawler span attributes.

Upgrading

  • No database migrations.
  • No changes to existing agents or their content hashes.
  • New IR node type: crawler, with CrawlerConfig and CrawlerLimits.
  • New endpoints: GET /crawlers/catalog, GET /crawlers/catalog/actor and POST /crawlers/catalog/convert.
  • New error codes: invalid_actor, crawler_provider_unknown, crawler_not_found, crawler_schema_unavailable and crawler_catalog_error.
  • New setting: crawler.max_charge_usd, with the environment cap THEYGENT_CRAWLER_MAX_CHARGE_USD.
  • Samples: a sample connection may now be http_auth.

Known limits

  • Apify only, for now. The provider is a closed list so each new provider is a deliberate addition. Apify is a hosted service and needs an Apify account; its free plan includes $5 of usage a month.
  • Not run against real Apify. Runs were tested against a local stand-in for Apify's run API, because no Apify token was available. Browsing was checked against the real Store.
  • No cost figure on interactive runs. Apify's synchronous endpoint names no run, so only durable runs record what a call cost.

Full changelog: v1.2.1...v1.3.0