Skip to content
3.x.dev
Go to file
Code

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
src
 
 
 
 
 
 
 
 

README.md

Scraper

Easily fetch, slice, dice, and output HTML (or XML) content from anywhere.

A Top Shelf Craft creation
Michael Rog, Proprietor


Installation

  1. From your project directory, use Composer to require the plugin package:

    composer require topshelfcraft/scraper
    
  2. In the Control Panel, go to Settings → Plugins and click the “Install” button for Scraper.

  3. There is no Step 3.

Scraper is also available for installation via the Craft CMS Plugin Store.

Usage

The Scraper plugin exposes a full-featured crawler object to your Twig template, allowing you to fetch, parse, and filter DOM elements from a remote source document.

Instantiating a client

When invoking the plugin, you can choose whether to use SimpleHtmlDom or Symfony components to instantiate your crawler:

{% set crawler = craft.scraper.using('symfony').get('https://zombo.com') %}
{% set crawler = craft.scraper.using('simplehtmldom').get('https://zombo.com') %}

I generally recommend using the Symfony components; they are more powerful and resilient to malformed source code. (The SimpleHtmlDom crawler is included to provide backwards compatibility with Craft 2 projects.)

Using the Symfony client

When you opt for Symfony components, the get method instantiates a full BrowserKit client, giving you access to all the BrowserKit and DomCrawler methods.

You can iterate over the DOM elements from your source document like this:

{% for node in crawler.filter('h2 > a') %}
    {{ node.text() }}
{% endfor %}

Using the SimpleHtmlDom client

When you opt for the SimpleHtmlDom crawler, the get method instantiates a SimpleHtmlDom client, giving you access to all the SimpleHtmlDom methods.

You can iterate over the DOM elements from your source document like this:

{% for node in crawler.find('h1') %}
    {{ node.innertext() }}
{% endfor %}

This is great! I still have questions.

Ask a question on StackExchange, and ping me with a URL via email or Discord.

What are the system requirements?

Craft 3.0+ and PHP 7.0+

I found a bug.

Please open a GitHub Issue, submit a PR to the 3.x.dev branch, or just email me.


Contributors:

About

Easily fetch, slice, dice, and output HTML content from remote pages in your CraftCMS templates.

Topics

Resources

License

Packages

No packages published

Languages

You can’t perform that action at this time.