An extensible crawler for downloading Android applications in third-party markets.
Python Shell
Latest commit 77a88c3 Dec 17, 2015 @mssun Update XPATH to hiapk.com
Permalink
Failed to load latest commit information.
crawler Update XPATH to hiapk.com Dec 17, 2015
downloader Fixing proxy setup Oct 15, 2014
repo update README in repo May 27, 2014
.gitignore updated .gitignore Mar 11, 2014
README.md Update README.md May 20, 2014

README.md

Android Apps Crawler

Overview

Android Apps Crawler is an extensible crawler for downloading Android applications in the third-party markets. It can crawl the download url addresses of applications and automatically download applications into repository.

Requirements

  • Python 2.6 or up
  • Scrapy 0.22 or up: http://scrapy.org (didn't have a full test in lower version.)
  • Works on Linux, Windows, Mac OSX, BSD
  • Currently, downloader cannot work on Windows.
  • For Ubuntu users, "Don't use the python-scrapy package provided by Ubuntu, they are typically too old and slow to catch up with latest Scrapy. Instead, use the official Ubuntu Packages."

Usage

  • Set the third-party markets you want to crawl in settings.py
  • Set the proxy if you have
  • Start crawler:
./crawl.sh <market name>
  • Start downloader:
./downloader.py <database file path> <output directory>

Settings

You can set proxy, user-agen, database name, etc in crawler/android_apps_crawler/settings.py file.

Supported Third-party Markets (market names used in crawl.sh)

More Android Markets

See: https://github.com/mssun/android-markets-list

TODO

  • Windows support for downloader.
  • Crawl apps from shared cloud storage link (e.g, pan.baidu.com, dbank.com).