Fetch and query external HTML, XML-ish markup, or JSON straight from your Twig templates. ScrapeKit is a small, dependency-light toolkit for Craft CMS 5: CSS selectors, XPath, node traversal, spec-compliant HTML5 parsing, and cached HTTP fetching - all behind one Twig variable.
{% set page = craft.scrapekit.get('https://example.com') %}
<h1>{{ page.text('h1') }}</h1>
<img src="{{ page.attr('meta[property="og:image"]', 'content') }}">
{% for item in page.find('.news li') %}
<article>
<h2>{{ item.find('h3').first().text() }}</h2>
<a href="{{ item.find('a').first().href }}">Read more</a>
</article>
{% endfor %}- One-liners for the common cases -
page.text('h1'),page.html('.intro'),page.attr('a.cta', 'href'),page.first('h2') - Full node API -
text(),html(),outerHtml(),attr(),nodeName(),classes(),hasClass(),parent(),children(), scopedfind(), and magic attribute access (node.href,node.src) - Smart lists -
find()returns aNodeListthat works like an array in Twig (|length,[0],for,|slice) and addsfirst(),last(),eq(i),texts(),attrs('href') - CSS and XPath -
find('div.item > a')orfindXPath('//item/title') - JSON too -
craft.scrapekit.json(url)fetches and decodes JSON APIs - Concurrent batches - fetch multiple pages with
getMany()while preserving input keys and isolating failures - HTML5 parsing - documents are parsed with a spec-compliant HTML5 parser (via Symfony DomCrawler), so real-world markup behaves like it does in a browser
- Cached and polite - responses are cached (1 hour by default) with a dedicated
cache tag; clear them from Craft's Caches utility or
php craft scrapekit/cache/clear - Safe by default - private networks, unsafe redirects, and oversized responses are blocked; cache variants include all effective request headers
- Per-request options - lower the cache duration or timeout, choose a parser, override the user agent, or add safe headers for a single call
- Never breaks your page - fetch and selector errors are logged and degrade to
empty results; check
page.ok/page.statusCodewhen you want to know
- Craft CMS 5.0.0 or later
- PHP 8.2 or later
From your project's Composer root:
composer require viesrood/scrapekit
php craft plugin/install scrapekit{% set page = craft.scrapekit.get('https://example.com') %}
{# per-request options #}
{% set page = craft.scrapekit.get('https://example.com', {
cacheDuration: 600, {# seconds; 0 disables caching for this call #}
timeout: 10,
userAgent: 'MyBot/1.0',
headers: { 'Accept-Language': 'nl' },
}) %}
{% if page.ok %} ... {% else %} status: {{ page.statusCode }} {% endif %}Request metadata is available as page.url, page.contentType,
page.fromCache, and page.errorCode.
getMany() fetches cache misses concurrently (five at a time by default), keeps
the input keys, and returns a failed Crawler for an individual error:
{% set pages = craft.scrapekit.getMany({
news: 'https://example.com/news',
events: 'https://example.com/events',
}) %}
{% if pages.news.ok %}
{{ pages.news.text('h1') }}
{% endif %}{# quick single-value reads (first match, '' when nothing matches) #}
{{ page.text('h1') }}
{{ page.html('.article-body') }}
{{ page.attr('link[rel="canonical"]', 'href') }}
{# lists #}
{% set rows = page.find('table.prices tr') %}
{{ rows | length }} rows, first: {{ rows.first().text() }}
{% for row in rows | slice(1, 10) %}
{{ row.find('td').texts() | join(' | ') }}
{% endfor %}
{{ page.find('a.download').attrs('href') | join(', ') }}
{# XPath #}
{% for title in page.findXPath('//rss/channel/item/title') %}
{{ title.text() }}
{% endfor %}{% set button = page.first('button.buy') %}
{{ button.nodeName() }} {# 'button' #}
{{ button.hasClass('primary') }}
{{ button.parent().attr('id') }}
{% for child in button.parent().children('span') %}
{{ child.text() }}
{% endfor %}{% set data = craft.scrapekit.json('https://api.example.com/items') %}
{% if data %}
{% for item in data.items %} {{ item.name }} {% endfor %}
{% endif %}json() remains backwards compatible and returns associative arrays or null.
Use jsonResult() when the API can return any valid JSON value or when response
metadata is needed:
{% set result = craft.scrapekit.jsonResult('https://api.example.com/value') %}
{% if result.ok %}
{{ result.data|json_encode }}
{% else %}
{{ result.errorCode }}
{% endif %}get() parses as HTML (lenient). For XML use craft.scrapekit.xml(url) or
{format: 'xml'}; {format: 'auto'} picks the parser from the content type or
XML declaration.
Defaults are editable under Settings → Plugins → ScrapeKit, or overridden per
environment with a config/scrapekit.php file (which wins over the stored settings):
<?php
return [
'cacheDuration' => 3600,
'timeout' => 15,
'connectTimeout' => 5,
'maxResponseBytes' => 5 * 1024 * 1024,
'maxRedirects' => 3,
'retryCount' => 1,
'concurrency' => 5,
'userAgent' => 'Mozilla/5.0 (compatible; CraftCMS ScrapeKit)',
'allowedHosts' => '', // comma/newline separated; supports *.example.com
'allowedPorts' => '80,443',
'allowPrivateNetworks' => false,
];An empty allowedHosts value permits all public hosts. For a fixed integration,
set an allowlist. Private, loopback, link-local, multicast, and reserved IP
ranges remain blocked unless allowPrivateNetworks is explicitly enabled.
Per-request options cannot raise the configured timeout, concurrency, response size, redirect, port, host, or network limits.
ScrapeKit fetches trusted integration URLs from developer-authored templates; do
not concatenate untrusted visitor input into a URL. html() and outerHtml()
return unsanitized remote markup. Twig escapes it normally—only add |raw when
the remote source is trusted and its markup is intentionally rendered.
Only http and https are supported. URL credentials and unsafe Host,
proxy, and hop-by-hop headers are rejected. Logs omit query strings and
credentials.
Responses are cached in Craft's cache with the scrapekit tag. Invalidate them via:
- Utilities → Caches → "ScrapeKit responses"
php craft scrapekit/cache/clear- per call:
craft.scrapekit.get(url, { cacheDuration: 0 })
Built on Symfony DomCrawler and CssSelector by Fabien Potencier and contributors.