#

robots-parser

Here are 3 public repositories matching this topic...

nicholasbergesen / RobotsParser

Parse robots.txt and traverse sitemaps.

sitemap spider crawl robots-txt robots useragent robots-parser useragentparser robotstxt sitemap-url

Updated Nov 19, 2023
C#

drmathias / robots

Parse robots.txt and sitemaps using dotnet

dotnet nuget robots-parser robots-exclusion-protocol sitemap-parser rfc-9309

Updated Sep 2, 2023
C#

toimik / RobotsProtocol

Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag

robots-txt robots meta-tags robots-parser robots-exclusion-standard robotstxt robots-tag robots-exclusion-protocol x-robots-tag

Updated May 2, 2024
C#

Improve this page

Add a description, image, and links to the robots-parser topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the robots-parser topic, visit your repo's landing page and select "manage topics."