Welcome to the BipIO API Server (Arya 0.2)
BipIO is Billion Instructions Per I/O - For People and Robots.
Imagine you can send a single standard payload and have a limitless host of API's orchestrate at your command.
That's what BipIO does.
BipIO is a highly parallel nodejs based API integration framework (iPaas). It uses graph based pipelines or 'Bips' to create ephemeral endpoints, complex workflows and message hubs with 3rd party API's and RPC's. Bips come in a variety of flavors for performing useful work - WebHooks/Sockets, Email, and Event Triggers.
It's a RESTful JSON API that supports account level namespacing and multiple domains (fqdn) per account. Clients authenticate over HTTP Basic.
BipIO can be installed alongside your existing open source app or prototype for out-of-band message transformation, feed aggregation, queuing, social network fanout or whatever you like, even on your Rasberry Pi.
Follow @bipioapp on Twitter for regular news and updates, or /join #bipio on Freenode IRC.
For a technical deep-dive, don't be afraid to use The Wiki - it's open to contributors
If you're familiar with Yahoo Pipes, IFTTT, Zapier, Mulesoft - the concept is a little similar. BipIO is a message transformation hub for connecting API's and creating an internet of things that matter to you. Over time and as more people use the system it builds a corpus of transformation data that serve as 'hints' for use by other community members in their own integrations. Complete configurations can also be shared openly by different users and systems allowing for the instant installation of common workflows.
Bipio is dynamic, flexible, fast, modular, opinionless and gplv3 open source.
Bips are graph structures which transform messages between adjacent Channels and chain outputs to inputs indefinitely across disparate 'cloud' services. The structures also contain metadata defining the flavor, lifespan and overall characteristics of the endpoint or trigger.
Some of their characteristics include :
- dynamic or automatically derived naming
- pausing or self-destructing after a certain time or impressions volume
- binding to connecting clients with soft ACLs over the course of their 'life'
- able to be reconfigured dynamically without changing a client implementation
- infinitely extensible, from any channel to any other channel.
- can serve (render) protected channel content while inheriting all of the above characteristics
It's a fairly large topic, find out more in the wiki.
Channels are pointers to discrete actions provided by 3rd party API's and services. They are reusable entities which perform a discrete unit of work and emit a predictable result.
The collection of channels you create becomes something like a swatch from which you can orchestrate complex API messaging patterns. When dropped onto a Bip's graph, a channels export becomes the next adjacent channels transformed import, which can be chained indefinitely.
Channels are instantiated from service containers called Pods. Pods only concern is providing a set of possible actions, and doing that well.
Channels can store, track, serve or transform content and messages as part of a pipeline or in autonomous isolation.
The server ships with a few handy 'Pods' which you can use right away - Email, Text/HTML/Markdown Templating, Flow Control, Syndication, Web Hooks, Time. Extra Pods can be found in the GitHub Repository - to install it's just :
npm install bip-pod-{pod-name}
./tools/pod-install.js -a {pod-name}
And follow the instructions, or feel free to craft your own.
Here's a quick example. Lets say I have a private email address that I want to protect or obfuscate - I could use an SMTP Bip to create a temporary relay which will forward emails for 1 day only.
Here's how :
Create an SMTP Forwarder Channel to email me with any messages it receives :
POST /rest/channel
{
action : "email.smtp_forward",
name : "Helo FuBa"
config : {
rcpt_to : "foo@bar.net"
}
}
RESPONSE
{
id : "206fe27f-5c98-11e3-8ad3-c860002bd1a4"
}
... I can then build the relay with a SMTP Bip having a single edge pointing to the the generated Channel ID :
POST /rest/bip
{
type : "smtp",
hub : {
"source" : {
edges : [ "206fe27f-5c98-11e3-8ad3-c860002bd1a4" ],
transforms : {
"206fe27f-5c98-11e3-8ad3-c860002bd1a4" : {
"subject" : "[%source#subject%]",
"body_html" : "[%source#body_html%]",
"body_text" : "[%source#body_text%]",
"reply_to" : "[%source#reply_to%]",
}
},
_note : "^^ Transforms aren't mandatory, but here for illustration - you only need an edge"
}
},
end_life : {
imp : 0,
time : '+1d'
},
note : "No name, no problem. Let the system generate something short and unique"
}
RESPONSE
{
name : "lcasKQosWire22"
_repr : "lcasKQosWire22@yourdomain.net"
}
And thats it. There's actually a little chrome extension which does just this for web based email forms.
For an extra credit example, I could store attachments arriving on that email address straight to dropbox by just adding an edge - check out how in the cookbook
- Node.js >= 0.10.15 API and graph resolver
- MongoDB Server data store
- RabbitMQ message broker
SMTP Bips are available out of the box with a Haraka plugin. Configs under bipio-contrib/haraka.
npm install -g bipio
bipio
git clone git@github.com:bipio-server/bipio.git
cd bipio
make install
node . (or `npm start`)
When setting BipIO up for the first time, the install process will enter interactive mode, saving to the path of NODE_CONFIG_DIR environment variable,if set (otherwise, just config/{environment.json}.
export NODE_CONFIG_DIR=<path_to_your_config_directory>
Be sure to have a MongoDB server and Rabbit broker ready and available before install. Otherwise, follow the prompts during the install process to get a basically sane server running that you can play with.
For Ubuntu users, a sample upstart script is supplied in config/upstart_bip.conf which should be copied to /etc/init and reconfigured to suit your environment.
If you have a more complex deployment environment and the packaged sparse config doesn't suit, don't worry! Set the environment variable BIPIO_SPARSE_CONFIG to the path of your preferred config file, and it will use that instead.
For a non-interactive setup (ie: make install without any user interaction) - set environment variable HEADLESS=true
Updating BipIO via npm will resolve any new dependencies for you, however if you're checking out from the repository
directly with git pull you may need to manually run npm install to download any new dependencies (bundled pods, for example).
If you're going the git pull route and want to save this step, create a git 'post merge' hook by copying it from ./tools like so :
mkdir -p .git/hooks
cp ./tools/post-merge .git/hooks
chmod ug+x .git/hooks/post-merge
This will automatically install any missing dependencies every time you git pull
Periodic tasks will run from the server master instance automatically, you can find the config
in the config/{environment}.json file, keyed by 'cron'.
- stats - network chord stats, every hour
- triggers - trigger channels, every 15 minutes
- expirer - bip expirer, every hour
To disable a cron, either remove it from config or set an empty string.
To have these crons handled by your system scheduler rather than the bipio server, disable the crons
in config as described. Wrapper scripts can be found in ./tools for each of stats (tools/generate-hub-stats.js),
triggers (tools/bip-trigger.js) and expirer (tools/bip-expire.js).
Here's some example wrappers.
Cron: */15 * * * * {username} /path/to/bipio/tools/trigger-runner.sh
trigger-runner.sh :
#!/bin/bash
# trigger-runner.sh
export NODE_ENV=production
export HOME="/path/to/bipio"
cd $HOME (date && node ./tools/bip-trigger.js ) 2>&1 >> /path/to/bipio/logs/trigger.log
Cron: 0 * * * * {username} /path/to/bipio/tools/expire-runner.sh
expire-runner.sh :
#!/bin/bash
# expire-runner.sh
export NODE_ENV=production
export HOME="/path/to/bipio"
cd $HOME (date && node ./tools/bip-expire.js ) 2>&1 >> /path/to/bipio/logs/cron_server.log
Cron: */15 * * * * {username} /path/to/bipio/tools/stats-runner.sh
stats-runner.sh :
#!/bin/bash
# stats-runner.sh
export NODE_ENV=production
export HOME="/path/to/bipio"
cd $HOME (date && node ./tools/generate-hub-stats.js ) 2>&1 >> /path/to/bipio/logs/stats.log
/etc/monit/config.d/bipio.conf
#!monit
set logfile /var/log/monit.log
check process node with pidfile "/var/run/bip.pid"
start program = "/sbin/start bipio"
stop program = "/sbin/stop bipio"
if failed port 5000 protocol HTTP
request /
with timeout 10 seconds
then restart
The BipIO server software is the basic framework for processing bips and their delivery graphs and is currently distributed headless. For visual tools, sign in to bipio to mount your local install from your browser under My Account > Mounts > Create Mount.
The BipIO website is not a first class citizen or tightly coupled to one particular endpoint, so you can mount your local install(s) even if behind a firewall.
By itself, Bipio does not provide SSL termination or any load balancing beyond node-cluster. If you need SSL termination this should be delegated to a forward proxy such as NginX, Apache, HAProxy etc.
A healthy contributor community is great for everyone! Take a look at the Contribution Document to see how to get your changes merged in.
Please log issues to the repository issue tracker on GitHub.
Our open source license is the appropriate option if you are creating an open source application under a license compatible with the GNU GPLv3.
If you'd like to integrate BipIO with your proprietary system, GPLv3 is likely incompatible. To secure a Commercial OEM License for Bipio, please reach us