Skip to content
This API scrapes credit card websites to pull down the monthly rewards and return them as a json object. Uses: Python, Selenium, Falcon, Gunicorn, Venv
Python PLpgSQL
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.vscode
falcon_api
.gitignore
database.db
database.sql
fetch_categories.py
readme.md

readme.md

Credit Card Rewards Scraper API

This api scrapes credit card websites to pull down the monthly rewards and return them as a json object.

Technologies Used

This project uses

  • Selenium: for web scraping (grab the text from websites)
  • Falcon: the lightest weight python api framework I could find
  • Venv: linux virtual environment to maintain easily deployable container
  • Gunicorn: run the falcon api locally
  • Sqlite: storing the categories

How to use

fetch_categories.py should be ran every week with a cronjob to update the sqlite database (this gives us a faster response because it prevents the selenium test from running on each request as the test takes a few seconds to complete).

Run python app on webserver, then make a get request to /rewards

http.get('yourwebserver/rewards').response(success=>{
    console.log(success);
}.error=>{
    console.log(error);
}

//output
[
    [1, "Discover", "it", "Grocery Stores"],
    [2, "Citi", "Simplicity", "Home Furnishing Stores"]
]

App Requirements

I've saved the dependencies to requirements.txt, so quickly install all python requirements with pip install -r requirements.txt

Developed using the following versions:

  • Python: 3.6
  • Pip: 10.0.1
  • Firefox: 60.0.2
  • Gunicorn: 19.8.1
  • Sqlite: 3

Install guide

$ apt-get install virtualenv
$ git clone https://github.com/kaburkett/API-CreditCardRewardsScraper.git
$ cd API-CreditCardRewardsScraper/falcon_api
$ virtualenv venv -p /user/bin/python3
$ . ven/bin/activate
(venv) $ apt-get install build-essentials python3-dev
(venv) $ pip install cython
(venv) $ pip install --no-binary :all: falcon
(venv) $ pip install gunicorn
(venv) $ gunicorn -b 127.0.0.1:8080 main:app 

Now the api should be running on localhost:8080!

Additional Info

Great write up on running falcon apps with gunicorn: https://www.digitalocean.com/community/tutorials/how-to-deploy-falcon-web-applications-with-gunicorn-and-nginx-on-ubuntu-16-04

You can’t perform that action at this time.