nodejs web crawler
-
Updated
Feb 24, 2017 - JavaScript
nodejs web crawler
🤖 Robots.txt generator done right.
A simple script to open all the pages in a website's robots.txt files
🤖 Handle and parse a site's robots.txt file and extract actionable information
A robots.txt generating Express Middleware
A robots.txt script for Lambda Edge
A tool for debugging robots.txt
A Webpack 3 plugin for generating robots.txt file
🤖 Browser extension to check for and preview a site's robots.txt in a new tab (if it exists)
Chrome extension which blocks urls based on robots.txt (compatible to Chrome 41)
Generates a robots.txt
Typescript robots.txt parser with support for wildcard (*) matching.
Robots.js is a tool used to generate robots.txt according to your rules. Adapted from FastGitORG/SpiderFucker & Kinetix-Lee/spiderfucker-python.
Front-end workflow to start a new project with Eleventy and Webpack.
Generate sitemap and robots.txt for NextJS used web hook from STRAPI
Parser for robots.txt for node.js
🧑🏻👩🏻 "We are people, not machines" - An initiative to know the creators of a website. Contains the information about humans to the web building - A Nuxt Module to statically integrate and generate a humans.txt author file - Based on the HumansTxt Project.
Higher order Next.js config to generate sitemap.xml and robots.txt
Fully native robots.txt parsing component without any dependencies.
Sharp SEO Tools is collection of free web tools completely written in Javascript (19 tools available), feel free to use
Add a description, image, and links to the robots-txt topic page so that developers can more easily learn about it.
To associate your repository with the robots-txt topic, visit your repo's landing page and select "manage topics."