No description, website, or topics provided.
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
app
bin
common
scraper
spec
.gitignore
.nvmrc
.travis.yml
package-lock.json
package.json
readme.md
to-csso.txt
to-uglify.txt

readme.md

Alabama Legals Scraper

Build Status

Setup

  1. Install electron system requirements (required for nightmare.js)
apt-get install -y libgtk2.0-0 libgconf-2-4 \
    libasound2 libxtst6 libxss1 libnss3 xvfb
  1. Run npm install

  2. Setup the database

$ bin/setup_db.sh
$ mysql -u scraper alabamalegals -p < bin/create_tables.sql
  1. Set environment variables
export AL_GOOGLE_CLIENT_SECRET=supersecretclientsecret
export NODE_ENV=dev
export SCRAPERPASS=thedbpassword
  1. Start the app with npm start

Usage

Allow access to data interface on port 3000 npm run start

Run scraper (takes a couple of minutes) npm run scraper

Production

forever start app/index.js

Debugging

Electron and nightmare have a lot of dependencies and are sometimes buggy. To debug the installation of these requirements, use the debug flag.

DEBUG=nightmare* node scraper/index.js

If node is throwing weird errors, check the version and ensure the correct one is in use.

node --version
nvm use

About

This is basically a project for my mom, and likely not useful to anyone else on the internet, except as a case study in writing frontend scrapers for shitty websites.