Extract, Transform and Load for Nodejs
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.gitignore
Dockerfile
LICENSE
README.md
couchdb-container.sh
honeybadger-container.sh
honeybadger.js
package.json

README.md

Honey Badger

This project seeks to provide a simple set of tools libraries for expressing and orchestrating Extract, Transform and Load solutions in Node.js.

The honeybadger package is itself split into 3 packages:

  • honeybadger-etl, independent libraries for implementing ETL in Node.js Streams
  • honeybadger-service, core Node.js services, workers and scheduling for automated ETL solutions
  • honeybadger-admin, a browser-based Solution Designer, System Admin and Reporting for Honey Badger services.

Documentation

Dependencies

  • CouchDB
    • configuration for services and admin
    • metadata caching
  • Docker
    • running production stack
  • VirtualBox (OSX)
    • required to run docker-machine
  • Docker Machine (OSX)
    • running docker on OSX hosts

Usage

Whole Stack from Docker

For convenience you can run the HoneyBadger stack locally or in production via Docker. Work is being done to run/scale in Kubernetes for large scale production deployments.

docker run honeybadger/honeybadger:latest

Note: See Getting Started - this will not run CouchDB

In Your Own Services Stack

You can also run a headless version of the HoneyBadger Service engine within your existing stack. The Service engine provides the scheduled running of ETL solutions you have configured via either the admin or directly through JSON configs stored in CouchDB.

See honeybadger-service, core Node.js services, workers and scheduling for automated ETL solutions.

In Your Own Application

HoneyBadger ETL libraries are nicely packaged for doing some low-level ETL within your own applications. You can directly consume the npm packages to provide simple ETL workflows to your application with robust support for various sources, transforms.

honeybadger-etl, independent libraries for implementing ETL in Node.js Streams.

Getting Started

If you are wanting to run the complete solution to start your ETL workloads, you'll want to use the docker container approach. To make this extremely simple, this meta-package provides a method for automatically running CouchDB in a docker container, and firing up the HoneyBadger stack in another container. From this package, simply run:

npm start

*Note you must have docker installed properly.

Examples

Contributing

Contributions are welcomed from the greater community via GitHub pull request. For questions or help contributing please create an issue.

Dependencies

You must have CouchDB installed and running locally. This is pretty easy to do. Any local development or locally running stack will require a local CouchDB.

Running Stack for Local Development

This package is a meta-package and is not where actual development is done. However, you can fire up the stack locally from this package to see it running before you start cloning the other repos.

npm test

You should be able to browse to [http://localhost:8090/admin] and see the HoneyBadger Admin Console.

Maintainer(s)

@dermidgen

Versioning

Honey Badger is maintained under the Semantic Versioning guidelines.

Related Projects

License

Code released under the MIT license. Docs released under Creative Commons.