Skip to content
This is a scrapy project in which I have implemented several crawlers for different torrent and direct link websites.
Branch: master
Clone or download
Pull request Compare This branch is 1 commit ahead of yasoob:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
piratebay
LICENSE
README.md

README.md

Torrents-Crawler

This is a scrapy project in which I have implemented several crawlers for different torrent and direct link websites. This project crawls for the title and the url only right now because this was my only requirement.

###Dependency:

  • Scrapy pip install scrapy

###Websites currently supported:

  • Piratebay
  • Kickass
  • 1337x
  • SkidrowCrack
  • TorrentDownloads
  • SumoTorrent
  • BitSnoop

###How to run a crawler: If you want to run a crawler then goto the top level directory which is piratebay and issue this command: $ scrapy crawl piratebay (see TODO) This command will run the piratebay crawler. Names for all of the crawlers are:

  • piratebay
  • leetx
  • bitsnoop
  • sumotorrent
  • torrentdownloads
  • kickass
  • skidrowcrack

###TODO: Currently I have not added any pipeline so right now if we want to save the crawled data we will have to issue one of the following commands:

  • scrapy crawl piratebay -o items.json -t json
  • scrapy crawl piratebay -o items.csv -t csv
  • scrapy crawl piratebay -o items.xml -t xml
  • scrapy crawl piratebay -o items.json -t jsonlines
    If anyone is willing to write an ItemPipeline then feel free to submit a PR

###How to add support: If you want to add further support for some more websites then feel free to edit this crawler. Just go to piratebay/piratebay/spiders/piratebay_spider.py and edit that file. Copy one of the class and edit it to adapt for another website. Basically you will have to edit the xpaths, link to .txt file and the name of the crawler. After that copy all of the links which you want to crawl and paste them in a new text file in piratebay/links/. Thats it. After that just run your crawler.

###Author:

###Support: Official scrapy docs If there is an issue with the scripts present here do open an issue or if you want me to add support for a website do tell me.

###Contribute: You can contribute literally anything. Whether a new crawler or some small optimisations. Just do a PR and make this project grow.

Note: This project is not maintained anymore.

You can’t perform that action at this time.