aNinja

🗡️ aNinja is a project in order to help your spiders deal with anti-crawling mechanism.

Feature

Core Part

• Headers forgery
• Cookies management between different clients
• Captcha catcher & solver
• An Agent wraps request/JS clients.

Spiders Part

• Proxy pool spider
• Login implementation for specific website

Auxiliary Part

• Exporter for prometheus
• Templates such as Scrapy Middleware
• Notes for learning web-crawling