Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 

Credit Card Rewards Scraper API

This api scrapes credit card websites to pull down the monthly rewards and return them as a json object.

Technologies Used

This project uses

  • Selenium: for web scraping (grab the text from websites)
  • Falcon: the lightest weight python api framework I could find
  • Venv: linux virtual environment to maintain easily deployable container
  • Gunicorn: run the falcon api locally
  • Sqlite: storing the categories

How to use

fetch_categories.py should be ran every week with a cronjob to update the sqlite database (this gives us a faster response because it prevents the selenium test from running on each request as the test takes a few seconds to complete).

Run python app on webserver, then make a get request to /rewards

http.get('yourwebserver/rewards').response(success=>{
    console.log(success);
}.error=>{
    console.log(error);
}

//output
[
    [1, "Discover", "it", "Grocery Stores"],
    [2, "Citi", "Simplicity", "Home Furnishing Stores"]
]

App Requirements

I've saved the dependencies to requirements.txt, so quickly install all python requirements with pip install -r requirements.txt

Developed using the following versions:

  • Python: 3.6
  • Pip: 10.0.1
  • Firefox: 60.0.2
  • Gunicorn: 19.8.1
  • Sqlite: 3

Install guide

$ apt-get install virtualenv
$ git clone https://github.com/kaburkett/API-CreditCardRewardsScraper.git
$ cd API-CreditCardRewardsScraper/falcon_api
$ virtualenv venv -p /user/bin/python3
$ . ven/bin/activate
(venv) $ apt-get install build-essentials python3-dev
(venv) $ pip install cython
(venv) $ pip install --no-binary :all: falcon
(venv) $ pip install gunicorn
(venv) $ gunicorn -b 127.0.0.1:8080 main:app 

Now the api should be running on localhost:8080!

Additional Info

Great write up on running falcon apps with gunicorn: https://www.digitalocean.com/community/tutorials/how-to-deploy-falcon-web-applications-with-gunicorn-and-nginx-on-ubuntu-16-04

About

This API scrapes credit card websites to pull down the monthly rewards and return them as a json object. Uses: Python, Selenium, Falcon, Gunicorn, Venv

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published