Skip to content

Analytics

Jump to bottom Edit New page
Luke Taylor edited this page Jun 11, 2013 · 11 revisions

Introdution

Spring XD Analytics provides support for real-time analysis of data using metrics such as counters and gauges. Spring XD intends to support a wide range of these metrics and analytical data structures as a general purpose class library that works with several backend storage technologies.

We’ll look at the following metrics

An in memory implementation and a Redis implementation are provided in M1. Other metrics that will be provided in a future release are Aggregate Counters, Rate Counters, and Histograms.

Metrics can be used directly in place of a sink just as if you were creating any other stream, but you can also analyse data from an existing stream using a tap. We’ll look at some examples of of using metrics with taps in the following sections. As a prerequisite start the XD Container as instructed in the Getting Started page.

Counter

A counter is a Metric that associates a unique name with a long value. It is primarily used for counting events triggered by incoming messages on a target stream. You create a counter with a unique name and optionally an initial value then set its value in response to incoming messages. The most straightforward use for counter is simply to count messages coming into the target stream. That is, its value is incremented on every message. This is exactly what the counter module provided by Spring XD does.

Here’s an example:

Start by creating a data ingestion stream. Something like:

$ curl -d "twittersearch --consumerKey=<your_key> --consumerSecret=<your_secret> --query=spring | file --directory=/tweets/" http://localhost:8080/streams/springtweets

Next, create a tap on the springtweets stream that sets a message counter named tweetcount

$ curl -d "tap @ springtweets | counter --name=tweetcount" http://localhost:8080/streams/tweettap

The results are written to redis under the key counter.${name}. To retrieve the count:

$ redis-cli
redis 127.0.0.1:6379> get counters.tweetcount

Field Value Counter

A field value counter is a Metric used for counting occurrences of unique values for a named field in a message payload. XD Supports the following payload types out of the box:

  • POJO (Java bean)

  • Tuple

  • JSON String

For example suppose a message source produces a payload with a field named user :

class Foo {
   String user;
   public Foo(String user) {
       this.user = user;
   }
}

If the stream source produces messages with the following objects:

   new Foo("fred")
   new Foo("sue")
   new Foo("dave")
   new Foo("sue")

The field value counter on the field user will contain:

fred:1, sue:2, dave:1

Multi-value fields are also supported. For example, if a field contains a list, each value will be counted once:

users:["dave","fred","sue"]
users:["sue","jon"]

The field value counter on the field users will contain:

dave:1, fred:1, sue:2, jon:1

field_value_counter has the following options:

fieldName

The name of the field for which values are counted (required)

counterName

A key used to access the counter values. (default: ${fieldName})

To try this out, create a stream to ingest twitter feeds containing the word spring and output to a file:

curl -d "twittersearch --consumerKey=<your_key> --consumerSecret=<your_secret> --query=spring | file" http://localhost:8080/streams/springtweets

Now create a tap for a field value counter:

curl -d "tap@springtweets | field-value-counter --fieldName=fromUser" http://localhost:8080/streams/tweettap

The twittersearch source produces JSON strings which contain the user id of the tweeter in the fromUser field. The field_value_counter sink parses the tweet and updates a field value counter named fromUser in Redis. To view the counts:

$ redis-cli
redis 127.0.0.1:6379>zrange fieldvaluecounters.fromUser 0 -1 withscores

Gauge

A guage is a Metric, similar to a counter in that it holds a single long value associated with a unique name. In this case the value can represent any numeric value defined by the application.

The gauge sink provided with XD stores expects a numeric value as a payload, typically this would be a decimal formatted string, and stores its values in Redis. The gauge includes the following attributes:

name

The name for the gauge (default: <streamname>)

Here is an example of creating a tap for a gauge:

Simple Tap Example

Create an ingest stream

$ curl -d "http --port=9090 | file" http://localhost:8080/streams/test

Next create the tap:

$ curl -d "tap@test | gauge" http://localhost:8080/streams/simplegauge

Now Post a message to the ingest stream:

$ curl -d "10" http://localhost:9090

Check the gauge:

$ redis-cli
redis 127.0.0.1:6379> get gauges.simplegauge
"10"

Rich Gauge

A rich guage is a Metric that holds a double value associated with a unique name. In addition to the value, the rich guage keeps a running average, along with the minimum and maximum values and the sample count.

The richgauge sink provided with XD expects a numeric value as a payload, typically this would be a decimal formatted string, and stores its values in Redis. The richgauge includes the following attributes:

name

The name for the gauge (default: <streamname>)

The values are stored in Redis as a space delimited string, formatted as value mean max min count

Here are some examples of creating a tap for a rich gauge:

Simple Tap Example

Create an ingest stream

$ curl -d "http --port=9090 | file" http://localhost:8080/streams/test

Next create the tap:

$ curl -d "tap@test | richgauge" http://localhost:8080/streams/testgauge

Now Post some messages to the ingest stream:

$ curl -d "10" http://localhost:9090
$ curl -d "13" http://localhost:9090
$ curl -d "16" http://localhost:9090

Check the gauge:

$ redis-cli
redis 127.0.0.1:6379> get richgauges.testgauge
"16.0 13.0 16.0 10.0 3"

Stock Price Example

In this example, we will track stock prices, which is a more practical example. The data is ingested as JSON strings like

{"symbol":"VMW","price":72.04}

Create an ingest stream

$ curl -d "http --port=9090 | file" http://localhost:8080/streams/stocks

Next create the tap, using the json-field-extractor to extract the stock price from the payload:

$ curl -d "tap@stocks | json-field-extractor --fieldName=price | richgauge" http://localhost:8080/streams/stockprice

Now Post some messages to the ingest stream:

$ curl -d "{\"symbol\":\"VMW\",\"price\":72.04}" http://localhost:9000
$ curl -d "{\"symbol\":\"VMW\",\"price\":72.06}" http://localhost:9000
$ curl -d "{\"symbol\":\"VMW\",\"price\":72.08}" http://localhost:9000

Check the gauge:

$ redis-cli
redis 127.0.0.1:6379> get richgauges.stockprice
"72.08 72.04 72.08 72.02 3"

Improved Stock Price Example

In this example, we will track stock prices for selected stocks. The data is ingested as JSON strings like

{"symbol":"VMW","price":72.04}
{"symbol":"EMC","price":24.92}

The previous example would feed these prices to a single gauge. What we really want is to create a separate tap for each ticker symbol in which we are interested:

Create an ingest stream

$ curl -d "http --port=9090 | file" http://localhost:8080/streams/stocks

Next create the taps, using the json-field-extractor to extract the stock price from the payload:

$ curl -d "tap@stocks |json-field-value-filter --fieldName=symbol --fieldValue=VMW| json-field-extractor --fieldName=price | richgauge" http://localhost:8080/streams/vmwprice
$ curl -d "tap@stocks |json-field-value-filter --fieldName=symbol --fieldValue=EMC| json-field-extractor --fieldName=price | richgauge" http://localhost:8080/streams/emcprice

Now Post some messages to the ingest stream:

$ curl -d "{\"symbol\":\"VMW\",\"price\":72.04}" http://localhost:9000
$ curl -d "{\"symbol\":\"VMW\",\"price\":72.06}" http://localhost:9000
$ curl -d "{\"symbol\":\"VMW\",\"price\":72.08}" http://localhost:9000
$ curl -d "{\"symbol\":\"EMC\",\"price\":24.92}" http://localhost:9000
$ curl -d "{\"symbol\":\"EMC\",\"price\":24.90}" http://localhost:9000
$ curl -d "{\"symbol\":\"EMC\",\"price\":24.96}" http://localhost:9000

Check the gauge:

$ redis-cli
redis 127.0.0.1:6379> get richgauges.emcprice
"24.96 24.926666666666666 24.96 24.9 3"
redis 127.0.0.1:6379> get richgauges.vmwprice
"72.08 72.04 72.08 72.02 3"

Clone this wiki locally