Consumer microservice that performs simple count, sum and stats operations on incoming data.
Switch branches/tags
Nothing to show
Clone or download
Pull request Compare This branch is 7 commits ahead of glynnbird:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
lib
plugins
.gitignore
README.md
app.js
package.json

README.md

Simple Logging Service - Aggregation Microservice

This is a microservice that subscribes to a queue or pubsub channel hosted on

  • Redis
  • RabbitMQ
  • Apache Kafka

and aggregates the data using one of the built-in functions:

  • count - count incoming data messages
  • sum - sum a single element of the message
  • stats - calculate stats on a single element of the message
  • countdistinct - count numbers of distinct values of a single element of the message
  • customstats - calculate stats on a value returned by a supplied function

This service is designed to be paired with the Simple Logging Service which sends web traffic data to a Redis, RabbitMQ or Kafka queue or pubsub channel. Other Microservices can listen to the data arriving on those channels, services such as the Simple Logging Storage Microservice which can store data in Cloudant, MongoDB or ElasticSearch.

The story is told in our blog here with a step-by-step guide to deploying on Bluemix with Compose services here.

Running Locally

Clone this repository and then run

npm install

Set your environment variables and start up the app e.g.

export QUEUE_TYPE=redis_pubsub
node app.js

API

This Microservice is controlled by a simple API

GET /configure

Allows the choice of aggregator to be defined together with a choice of which part of the incoming JSON is to be analysed.

Parameters

  • mode - one of count, sum, stats or null
  • selector - choose which part of the incoming Javascript object to aggregate (sum and stats only)

e.g.

  • GET /configure?mode=count
  • GET /configure?mode=sum&selector=a
  • GET /configure?mode=stats&selector=c
  • GET /configure?mode=countdistinct&selector=d
  • GET /configure?mode=customstats&selector=function(doc)%7B%20return%20(doc.c>20000)%3F%20doc.c%20:%20null%7D

Reply:

{
	"ok": true,
	"mode": "stats",
	"selector": "c"
}

or

{
	"ok": true,
	"mode": "customstats",
	"selector": "function(doc){ return (doc.c>20000)? doc.c : null}"
}

GET /query

Returns the current state of the aggregator

Parameters: None

e.g. GET /query

Reply:

// stats
{
	"ok": true,
	"err": null,
	"data": {
		"sum": 3769312,
		"count": 220,
		"min": 70,
		"max": 32735,
		"sumsqr": 140920641
	}
}

or

// countdistinct
{
	"ok": true,
	"err": null,
	"data": {
		"rat": 75,
		"gerbil": 58,
		"dog": 65,
		"robin": 61,
		"squirrel": 70,
		"donkey": 54,
		"cat": 74,
		"wolf": 65,
		"crab": 55,
		"cow": 64,
		"ant": 67,
		"fox": 68,
		"chicken": 51
	}
}

GET /reset

Rest the aggregator's data.

Parameters: none

e.g. GET /reset

Reply:

Reply:

{
	"ok": true
}

Environment variables

QUEUE_TYPE

One of

  • redis_queue - A Redis list data structure
  • redis_pubsub - A Redis PubSub channel
  • rabbit_queue - A RabbitMQ PUSH/WORKER queue
  • rabbit_pubsub - A RabbitMQ PUBLISH/SUBSCRIBE channel
  • kafka - An Apache Kafka topic
  • null - default (does nothing)

QUEUE_NAME

The name of the queue/channel that is subscribed to. If omitted, it takes the following values for each of the queue types:

  1. stdout - n/a
  2. redis_queue - mcqueue
  3. redis_pubsub - mcpubsub
  4. rabbit_queue - mcqueue
  5. rabbit_pubsub - mcpubsub
  6. kafka - mcqueue

VCAP_SERVICES

VCAP_SERVICES is created for you by the Bluemix Cloud Foundry service. It defines the credentials of the attached services that this app can connect to.