README.md

What

Megapis is a node.js server application that runs workers. A worker is an npm module, and can do anything you can write: fetch data, send email, etc. It includes a utility module for workers to read and write data to a Redis backend.

Why

here's lots of data out there, but it's often not in a format you can use. Megapis is a framework to help get the info you want, how you want it. I tried IFTTT, but found it too limited -- can only get content from their list of sources (including RSS but not web pages). I tried Huginn, but the abstraction didn't seem quite right -- it's hard to express logic in JSON. What I really wanted was an easy way to take advantage of the vast resources just an npm install away.

Who

Megapis is for someone who wants fine-grained control over what info to get, how to process it, and what to do with it. There is no user interface; you'll need to install and run servers, and write workers in JavaScript.

This repository contains:

server - runs workers described by config file (default config/global.json)

worker_util - utilities for workers to load, save, and compare values

workers - sample workers

Run a Megapis server

see running a server

Writing a Megapis worker

A Megapis worker is a just a node module that exports a run method and (optionally) a requiredConfigKeys array. Require megapis-worker for functions that make it easier to work with the Redis persistence store.

Before running the worker, the server merges its global config with the worker config, and passes the config to the worker's run method.

To bootstrap a worker with all the files it needs:

Set environment variables:

MEGAPIS_GIT_URL - repository URL for use in worker's package.json MEGAPIS_AUTHOR - author name for use in worker's package.json

From megapis/worker_util, run new_Worker.js *worker_name* *worker_path*. This will generate a set of files in worker_path/worker_nameand runs npm install` to get dependencies. The files are:

• index.js - code to get information and save it to the store
• package.json - starter file with log4js and megapis-worker dependencies
• README.md - describe your worker
• workerConfig.json - sample configuration file

To have the Megapis server run your module without publishing it, use npm link to make it available to your server:

npm-link
cd ../*server-directory*
npm link *worker-name*

In index.js:

Specify required configuration keys:

module.exports.requiredConfigKeys = ["location", "storageKeys.self", "storageKeys.output"];

Create a run method that gets a configration object:

module.exports.run = function(callback) {
    ... do something ...
    callback(err, results);

Connect to store and config values redis.port, redis.host, and redis.options:

var client = workerUtil.store.createClient(config);

Save results with key storageKeys.self in config, and back up previously saved value:

var tidesKey = config.storageKeys.self;
client.save(tidesKey, tides);

Get values that weren't seen in the previous run, save to storageKeys.output, and close connection:

client.getDiffJson(tidesKey, function(err, unseen) {
    if (unseen && unseen.length > 0) {
        client.add(config.storageKeys.output, unseen, 'Low tide');
    }
    client.quit();
    callback();
});

Testing a worker

Use megapis/server/run-worker.js to run a single worker.

First, set up your worker:

Add the worker to the workers hash in server/config/global.json. For example,

"workers": {
    "workerId": {
        "name": "Half Moon Bay Low Tide",
        "module": "megapis-low-tide"
    }
},

Copy the worker's workerConfig.json to server/config/workerId.json, and edit as needed.

Run the worker with run_worker.js *worker_id*. For example,

./run_worker.js lowTide