Ussfull tool for Hackers

Data Export API Demo(Java version) http://www.octoparse.com/Tutorial/API

Fake server that produces the specified amount of pages in directories to test web crawlers

Pluggable meetup crawler

A NAVER data crawler based in PHP

A robots.txt file is a file at the root of your site that indicates those parts of your site you don’t want accessed by search engine crawlers.

crawlers

This web crawler uses Scrapy py to crawl Wikipedia. It prints the page title, total word count, and page category (using openpyxl) to an Excel workbook, in order to analyze the verbosity of articles by category.

This is a code base for crawlers in python, and welcome to the conversation.

Little practice with selenium

Web scraping and automation using python

Crawlee - A framework for web scraping

htcap is a web application scanner able to crawl single page application (SPA) in a recursive manner by intercepting ajax calls and DOM changes.

Article of WeChat to RSS

An open source web crawling platform

Performs a reverse lookup for a certain ip to check if it belongs to Google crawlers

Nodejs web crawler functional for a single domain call