Repository navigation
v1.3.0
TheYgent 1.3 adds crawler nodes. Browse thousands of hosted scrapers and crawlers (Apify Actors), add one to an agent, and run it as a step, or wire it to an llm so the model can search the web, read a site, or find places. You decide what each call may do, not the model.
Crawler nodes (#90)
- A Crawlers browser.
- It searches the Apify Store through your control plane. Your browser never calls Apify.
- Each card shows the Actor's pricing, users, and 30-day run success rate, so an Actor that usually times out shows as such before you install it.
- Pick the fields the model may fill and press Add crawler. The node lands on the canvas, already wired into the llm's tools when you add it from one.
- The builder decides what a call does.
- The model fills only the fields you expose, usually one or two.
- Result counts, paid add-ons and proxies stay fixed on the node, and arguments the model invents are dropped.
- Offered every field, a local model in testing asked for 50 places per search and turned on paid review scraping. A crawler node keeps those fixed.
- Small prompts for local models. Three Actors offered as crawler nodes cost about 780 prompt tokens on Qwen3.6-35B, against about 13,500 when every field is offered. List fields default to one value per call, the shape local models fill reliably.
- Spend and size caps.
- Each node has a per-run spend cap (
maxChargeUsd), held under a platform ceiling:crawler.max_charge_usdon Settings → Crawlers, default $1.00. - Results are trimmed to the fields you choose, within a character budget.
- Each node has a per-run spend cap (
- Steps or tools, in both runtimes.
- Interactive runs make one synchronous call, capped at 300 seconds.
- Durable runs start the Actor, poll it, and read its dataset as separate steps, so a long crawl survives a restart. A resumed run polls the run it already started instead of paying for a second.
- Cost in the trace. Crawler spans record the Actor, the item count, the cap and, on durable runs, what the run cost (
crawler.usage_total_usd). - Your token stays a secret. The Apify token is stored once as an encrypted connection and resolved at call time. It never appears in the agent, a trace, or the durable journal.
- A new sample: Web researcher. A local model with a
web_searchcrawler, capped at three results and $0.05 a run. Install it from Settings → Samples and set your token.
Start with the Crawler node docs, or read the overview and the blog post. Prefer giving a capable model every Actor field? Apify through the MCP hub works too, with no crawler node.
Also in this release
- Website.
- A crawlers page and the blog post.
- The Web researcher template, bringing the templates to 13.
- The home page announcement.
- Docs.
- The crawler node and the MCP-hub alternative.
- The privacy page, which notes that a crawler node sends its input to the provider it names.
- The API and environment references.
- The observability page's crawler span attributes.
Upgrading
- No database migrations.
- No changes to existing agents or their content hashes.
- New IR node type:
crawler, withCrawlerConfigandCrawlerLimits. - New endpoints:
GET /crawlers/catalog,GET /crawlers/catalog/actorandPOST /crawlers/catalog/convert. - New error codes:
invalid_actor,crawler_provider_unknown,crawler_not_found,crawler_schema_unavailableandcrawler_catalog_error. - New setting:
crawler.max_charge_usd, with the environment capTHEYGENT_CRAWLER_MAX_CHARGE_USD. - Samples: a sample connection may now be
http_auth.
Known limits
- Apify only, for now. The provider is a closed list so each new provider is a deliberate addition. Apify is a hosted service and needs an Apify account; its free plan includes $5 of usage a month.
- Not run against real Apify. Runs were tested against a local stand-in for Apify's run API, because no Apify token was available. Browsing was checked against the real Store.
- No cost figure on interactive runs. Apify's synchronous endpoint names no run, so only durable runs record what a call cost.
Full changelog: v1.2.1...v1.3.0