A simple but powerful web crawler library for .NET
-
Updated
Dec 15, 2023 - C#
#
robots-txt
Here are 10 public repositories matching this topic...
.Net Core Plugin Manager, extend web applications using plugin technology enabling true SOLID and DRY principles when developing applications
-
Updated
Oct 14, 2024 - C#
A "robots.txt" parsing and querying library for .NET
-
Updated
Sep 3, 2024 - C#
This package helps you to add meta-tags, sitemap.xml and robots.txt into your project easily.
-
Updated
May 26, 2023 - C#
A friendly tool for creating dynamic robots.txt files in Umbraco
-
Updated
Feb 27, 2022 - C#
An admin extension for Optimizely CMS 12+ for managing robots.txt on a per site basis.
-
Updated
Oct 8, 2024 - C#
Parse robots.txt and traverse sitemaps.
sitemap
spider
crawl
robots-txt
robots
useragent
robots-parser
useragentparser
robotstxt
sitemap-url
-
Updated
Nov 19, 2023 - C#
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
robots-txt
robots
meta-tags
robots-parser
robots-exclusion-standard
robotstxt
robots-tag
robots-exclusion-protocol
x-robots-tag
-
Updated
May 2, 2024 - C#
PowerShell module for reading robots.txt files
-
Updated
May 18, 2022 - C#
Improve this page
Add a description, image, and links to the robots-txt topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the robots-txt topic, visit your repo's landing page and select "manage topics."