Skip to content
A web scraper for wegmans.com receipt data.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci
.idea
infrastructure
src
.editorconfig
.env.example
.gitignore
.nvmrc
Dockerfile
README.md
docker-compose.yml
package-lock.json
package.json
tsconfig.json
tslint.json

README.md

Wegmans.com Web Scraper

Codacy Badge CircleCI Coverage Status

A tool to scrape the wegmans.com website and download receipt information for a given user.

Prerequisites

  • NodeJS >= 10.4.0
  • NPM >= 6.1.0
  • Docker (including Swarm, and Docker-Compose)

Getting Setup for Development

  1. Clone this repository

    git clone git@github.com:lapinski/web-scraper.wegmans.git
  2. Install Dependencies

    npm install
  3. Start Postgres DB Image (or use any other Postgres Server)

    docker stack deploy -c docker-compose.db.yml
  4. Run DB Migrations (see note on TypeORM & Typescript)

    npm run db:migrate

Generating Migrations

(see note on TypeORM & Typescript) Run the following command:

npm run db:generate -- "-n <EntityName>"

Where <EntityName> is the exported class name of an entity to generate a new migration from.

Note on TypeORM CLI & Typescript

Typeorm loads JS files, so we need to build the TS files first, and the paths for entities,migrations,etc used by the CLI must target the dist/ folder.

The NPM scripts in this package are setup to build first, so just execute the db:* commands included.

Infrastructure

This project uses Google Cloud Platform and Terraform.

  1. Create project in GCP

  2. Update project id in main.tf

  3. Create GCP Service User In order to run the included terraform scripts, you must first setup the service user and then download the associated credentials (json key file).

  4. Download keyfile

    mv ~/Downloads/{{keyfile}}.json ~/.gcp/
  5. Export ENV Variable with keyfile location

    export GOOGLE_CLOUD_KEYFILE_JSON={{path}}
  6. Run terraform

    terraform apply
You can’t perform that action at this time.