Skip to content
This repository has been archived by the owner on Aug 16, 2018. It is now read-only.


Repository files navigation

Armslist scraper

  • What is this?
  • [Assumptions] (#assumptions)
  • [Installation] (#installation)
  • [Run the project] (#run)
  • [What to expect] (#what-to-expect)

What is this?

Armslist is a tool for aggregating listings from, a site used as a marketplace for buying and selling guns.


The following things are assumed to be true in this documentation.

  • You are running OSX.
  • You are using Python 2.7. (Probably the version that came OSX.)
  • You have virtualenv and virtualenvwrapper installed and working.
  • You have GNU Parallel installed -- a shell tool to execute multiple commands from standard input simultaneously.

For more details on the technology stack used with the app-template, see our development environment blog post.

This code should work fine in most recent versions of Linux, but package installation and system dependencies may vary.


If you don’t already have GNU Parallel, get it like this:

brew install parallel

Then clone the project:

git clone
cd armslist

Run the project

Next, create a virtual environment and install the requirements:

mkvirtualenv armslist
pip install –r requirements.txt

Finally, run the script:


What to expect

The scraper will output the listings into a csv format.

Running script will make two main csv files as well as some extra files that may be handy for testing:

  • cache/index.csv is a master index of listings with the URL, state and date. This file is used to scrape each page individually.
  • cache/listings.csv is the bulk of the data. Each row represents a listing and the associated details.
  • cache/state/1/<statename> includes the log messages in a file called stderr and scraped index page data in a file called stdout on a per state basis.

The script could take a long time to run. We recommend using a fairly powerful Amazon EC2 instance in the Virginia/us-east-1 region to run the scraper.


A simple scraper for listings







No releases published


No packages published