robots-txt

Polite, slim and concurrent web crawler.

advertools - online marketing productivity and analysis tools

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

NuxtJS module for robots.txt

The robots.txt exclusion protocol implementation for Go language

A simple but powerful web crawler library for .NET

A set of reusable Java components that implement functionality common to any web crawler

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers

Ultimate Website Sitemap Parser

NodeJS robots.txt parser with support for wildcard (*) matching.

Gatsby plugin that automatically creates robots.txt for your site

Open-Source Python Based SEO Web Crawler

grobotstxt is a native Go port of Google's robots.txt parser and matcher library.

Php class for robots.txt parse

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

Parser for robots.txt for node.js

Generator robots.txt for node js

Makes it easy to add robots.txt, sitemap and web app manifest during build to your Astro app.

Privacy Web Search Engine (not meta, own crawler)

List of useful links, tools and resources