crwlr.software is a collection of open source PHP composer packages that provide the necessary tools for web crawling and scraping tasks. The crawler package contains everything and helps you build crawlers as fast as possible. And there are also parts of it that you can use standalone.
A library for convenient handling of query strings used in HTTP requests.
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
Extension for the crwlr/crawler package containing steps utilizing a headless browser.
Extension configurations for integration of crwlr/crawler-ext-browser into the crwl.io app.
