Skip to content

ian-whitestone/crypto-data

master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
src
 
 
 
 
 
 
 
 

crypto-data

This is a quick repo I pulled together for fetching and storing historical cryptocurrency data from various sources. Each source is programmed as an individual class in CryptoSources.py. Currently, I have Coindesk and Poloniex as two different data sources.

Usage

First, create a table in a Postgres database.

CREATE TABLE hist_prices (
  snap_time timestamp,
  ticker VARCHAR(10),
  data_source VARCHAR(30),
  high DECIMAL,
  low DECIMAL,
  open DECIMAL,
  close DECIMAL,
  weighted_avg DECIMAL,
  base_volume DECIMAL,
  quote_volume DECIMAL
)

Then install postgrez and setup your database connection parameters in the ~/.postgrez yaml config file. Once setup, you can run the following:

$ python main.py --source poloniex --ticker BTC_ETH --start 2017-07-01 --end 2017-09-01 --period 5

to retrieve and store pricing data for the Bitcoin/Ethereum pair at 5-minute intervals.

Behind the scenes

This project implements a simple ETL pipeline. After each source's API is hit, the retrieved data is sent through a set of cleaning functions, as specified by the config.yaml file. These functions do things like check if the supplied fields are integers, floats, dates, or epoch dates and convert where necessary.

The config file looks like this:

coindesk:
  fields:
    timestamp:
      cleaning_func: check_epoch
      mapped_name: snap_time
    price:
      cleaning_func: check_float
      mapped_name: close

poloniex:

For each field returned in the Coindesk API response (i.e. timestamp & price), it specifies the cleaning function to pass each value through, and the associated column name in the database. The latter is done since each source won't necessarily have all the fields in hist_prices. With coindesk, they only have a price and a timestamp.

About

A quick repo for pulling historical cryptocurrency pricing data

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published