A library which allows to push data to the Santiment data processing pipelines
Branch: master
Clone or download
Latest commit 1191297 Dec 6, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
examples Add keyed data Dec 5, 2018
.dockerignore
.gitignore Initial commit Jun 21, 2018
Dockerfile
LICENSE Initial commit Jun 21, 2018
README.md
docker-compose.yml
index.js
package-lock.json Switch to using librdkafka with lz4 compression Jun 21, 2018
package.json

README.md

San Exporter

A small library which allows to push data to Santiment data processing pipelines.

Installing

You can install the library by adding it to your package.json with npm:

$ npm install --save https://github.com/santiment/san-exporter

If you are on OS X you might need to set some additional ENV variables for the compilation to pass: https://github.com/Blizzard/node-rdkafka#mac-os-high-sierra

Writing exporters

When writing data exporter you need to make sure the following things:

  • All the logging should be on the stdout. Do not create any temp files, as they will most probably disappear in an event of a restart
  • All the config should come from ENV variables
  • An exporter should continue from where it was interrupted in case of a restart. You can save the current position using the savePosition(position) API.
  • Encode the data as JSON

See examples/send_dates.js for an example how to build an exporter that sends the current date every 10 sec. You can run the example with docker-compose:

$ docker-compose up --build

This is going to ensure that all dependent services are created.

API

  • connect - establish connection to the dependent services. Returns a Promise
  • getLastPosition - fetch the last saved position. Returns a Promise
  • savePosition - update the last position of the exporter. Returns a Promise
  • sendData - push an array of events. Returns a Promise

Getting Help

If you have an idea for a specific data source, which you would like to export and include in the Santiment platform, don't hesitate to contact us on Discord: https://santiment.net/discord

We are willing to reward developers that contribute data feed to our platform. For more details around the rewards, ping us on Discord.