Elliptics server side applications tutorial

Introduction

This tutorial is intended to cover the gap in the reverbrain elliptics documentation in server side data processing topic which is outdated and incomplete.

To proceed with this tutorial you should have elliptics installed, and get some idea how to configure and operate it.
You also will need following packages installed:

• elliptics
• elliptics-client
• cocaine-tools
• libcocaine-plugin-elliptics
• cocaine-framework-python
• cocaine-framework-native

Overview [TODO: consider extending description]

Elliptics has a mechanism to run apps on it's nodes intended for processing data put into storage, app engine is event driven and provide mechanism to run data processing apps locally to the data. App engine is a layer on top of Cocaine, and provides own protocol for managing apps and messages. Attempts to mix Cocaine and elliptics app management lead to Elliptics crashes. [TODO: check that this is true] Each app[TODO: or even app instance?] uses its own Cocaine instance, so you won't get any Cocaine balansing capabilities.

Configuring elliptics

Already configured elliptics should have one additional parameter in it's configuration file
"srw_config": "/etc/elliptics/srw.json"
this is a link to the Cocaine config. Click for full sample config file. Note that all directories mentioned in the config must be created before launchig Elliptics.
To run elliptics with this config files put them to /etc/elliptics and run command
dnet_ioserv -c /etc/elliptics/ioserv_srw.json[TODO: I totally omit if it should run as root or own user, have to investigate]

Configuring cocaine

Full sample config is under the link, lets go though it in some details.

Services section defines what services will be used in Cocaine layer
"services": { "logging": { "type": "logging", "args": { "port": 12501 } }, "storage": { "type": "elliptics" } }
logging service and storage service are mandatory, storage should be set to elliptics, uploaded apps would be stored there[TODO: how change to file storage affect app distribution to nodes in case of elliptics managed Cocaine? in standalone Cocaine app will be upploaded to current node only].

[TODO: residual from experiments, probably should be thrown away] "node": { "type": "node", "args": { "runlist": "default" } }

Storages section sets parameters for elliptics storage, list of nodes to connect, set of groups and logging verbosity "storages": { "core": { "type": "elliptics", "args": { "nodes": ["localhost:1025:2"], "groups": [1], "verbosity": 2[TODO: is it still a number or word should be used?] } } },

I prepared the mighty script that gets all done as a reference

questions: how apps and profiles are matched when elliptics server side processing is used?
how to write responses in python?
how to create event handlers that can be called from both cocaine interface and through elliptics messaging?