Robots.txt parser / generator
-
Updated
Sep 18, 2018 - TypeScript
Robots.txt parser / generator
Super lightweight plain TypeScript parser for robots.txt with 0 dependencies.
A lightweight crawler frontier implementation in TypeScript using Redis.
"Page Auditor for Technical SEO" is an open source Google Chrome Extension created by Franco Folini. Once you added Page Auditor to your browser it will let you explore and analyze Structured Data, JavaScript scripts, Meta-Tags, Robots.txt and Sitemap.xml files from any webpage. All these elements are critical to improve the on-page SEO.
A lightweight and simple robots.txt parser in node
Makes it easy to add robots.txt, sitemap and web app manifest during build to your Astro app.
RFC 5234 spec compliant robots.txt builder and parser. 🦾
The right robots.txt file for your project
Collection of SEO utilities like sitemap, robots.txt, etc. for a Remix application. Forked from https://github.com/balavishnuvj/remix-seo
An extension for Visual Studio Code that enables support for robots.txt files. 🤖
Website of Alp Ozkan: Product Manager living in Istanbul, passionate about UX, Software Development and Web3.
Tame the robots crawling and indexing your Nuxt site.
Add a description, image, and links to the robots-txt topic page so that developers can more easily learn about it.
To associate your repository with the robots-txt topic, visit your repo's landing page and select "manage topics."