Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
src
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Wegmans.com Web Scraper

Codacy Badge CircleCI Coverage Status

A tool to scrape the wegmans.com website and download receipt information for a given user.

Prerequisites

  • NodeJS >= 10.4.0
  • NPM >= 6.1.0
  • Docker (including Swarm, and Docker-Compose)

Getting Setup for Development

  1. Clone this repository

    git clone git@github.com:lapinski/web-scraper.wegmans.git
  2. Install Dependencies

    npm install
  3. Start Postgres DB Image (or use any other Postgres Server)

    docker stack deploy -c docker-compose.db.yml
  4. Run DB Migrations (see note on TypeORM & Typescript)

    npm run db:migrate

Generating Migrations

(see note on TypeORM & Typescript) Run the following command:

npm run db:generate -- "-n <EntityName>"

Where <EntityName> is the exported class name of an entity to generate a new migration from.

Note on TypeORM CLI & Typescript

Typeorm loads JS files, so we need to build the TS files first, and the paths for entities,migrations,etc used by the CLI must target the dist/ folder.

The NPM scripts in this package are setup to build first, so just execute the db:* commands included.

Infrastructure

This project uses Google Cloud Platform and Terraform.

  1. Create project in GCP

  2. Update project id in main.tf

  3. Create GCP Service User In order to run the included terraform scripts, you must first setup the service user and then download the associated credentials (json key file).

  4. Download keyfile

    mv ~/Downloads/{{keyfile}}.json ~/.gcp/
  5. Export ENV Variable with keyfile location

    export GOOGLE_CLOUD_KEYFILE_JSON={{path}}
  6. Run terraform

    terraform apply

About

A web scraper for wegmans.com receipt data.

Topics

Resources

Releases

No releases published

Packages

No packages published

Languages