A set of Pushpop jobs that collect data for anyone to use
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
jobs
Gemfile
Gemfile.lock
LICENSE
Procfile
README.md

README.md

Open Data Collectors

A set of Pushpop jobs that collect data from various APIs over time and push data into Keen IO.

There are 2 ways to use this project:

  • Use the Keen IO Project ID and Read Key specified in each collector's documentation to use the data.
  • Capture the data yourself by cloning and deploying this repository pointing to your own project ID.

Note: Install keen-cli to run example CLI queries.

Collectors

Bitcoin Prices

Uses the BitcoinAverage API to capture Bitcoin price trends over time.

  • Code – jobs/bitcoin_prices.rb
  • Interval - Every minute
  • Contact Info - @dzello
  • Runs On – Heroku
  • Example Event - Extract One
  • Keen IO Data Set Information
    • Project ID: 53f3a8687d8cb95095000001
    • Read Key: 99a06e48fd7fb1279bc40995160eb0b61a9e0efaab8b651b029f0d895f77c0a804ba089282eff28bf8ad07f337422441d0542b7feaac9fea1e92fc153ee7efc51afad3276bda8d7754a338b349d540bfb402cba0dfdc82498c217054efd8abd0f47a0c0bc963bbdf0dc938c91b17d9f2
    • Collection Name: bitcoin-prices

Example CLI Queries

Note: Make sure to set environment variables in .env or specify --project and --read-key parameters.

# price last 30 minutes piped to spark
$ keen average -c bitcoin-prices -y "USD.averages.last" -t last_30_minutes -i minutely --spark | spark
▄███▄▄▄▄▄▁▄▄▄▄▄▄▄▁▁▁▄▄▄▄██▄

# highest USD price today
$ keen maximum -c bitcoin-prices -y "USD.averages.last"
481.77

# total USD volume last 24 hours
$ keen extraction -c bitcoin-prices --property-names "USD.averages.total_vol" --latest 1
[
  {
    "USD": {
      "averages": {
        "total_vol": 38276.39
      }
    }
  }
]

Contributing

More jobs! More open data sets!

If you add a job please fill out all the details (use Bitcoin Prices as a guide). Keen IO project and credentials are not required if you expect someone to capture the data on their own and not use a shared project.