(Fig. 1: General architecture. This project only provides the Real-time analysis box; the rest is part of the Rage Analytics platform)
This is the default analysis that gets executed for analytics traces from games. The Analytics Back-end can use any other analysis that can read from the input kafka-queue in the required format. This documentation is therefore useful both to understand the Analytics Realtime code, and to build your own analysis from scratch. For a detailed information about the data flow check out Understanding RAGE Analytics Traces Flow.
Your analysis must mimic the signature of the RealTime class, by providing methods to return a suitable StormTopology as a response to a suitable config. As can be seen in Figure 1, the analysis will run within Apache Trident.
The project can be tested outside this architecture by using the built-in tests (via mvn test
, assuming you have Maven correctly installed).
Incoming tuples will be of the form versionId, Map<String, Object>
. versionId
s track particular instances of games being played; for example, all students in a class could share the same versionId
, but if the game were to be played later, the teacher would typically generate another versionId
. Map keys are generally of the form derived by the Analytics Back-end in several steps:
- From xAPI to simplified JSON, which is then sent to a Kafka queue. The queue provides a buffer to prevent the loss of traces if the analysis cannot keep up with a spike in trace activity. For a full understanding of the input format of the data from Kafka check out Understanding RAGE Analytics Traces Flow - Step 2 - Collector (Backend) to Kibana then Storm Realtime
- From Kafka into your analysis: Extraction from Kafka, and conversion into final format. Note that you could choose to reimplement these differently, as they are both part of this module. For a full understanding of how to connect with Kafka check out Understanding RAGE Analytics Traces Flow - Step 3 - Realtime, transforming data from Kafka to ElasticSearch
Sample processed traces (with columns representing versionId
, gameplayId
, event
, target
, and response
, respectively) could be the following:
23,14,preferred,menu_start,tutorial_mode
23,14,skipped,introvideo1
This example would indicate a player 14 has selected menu_start
(type alternative
), and skipped var introvideo1 cutscene
. Also note that in the second trace, response is not set.
xAPI traces sent by games should comply with the xAPI for serious games specification.
The analysis will include details on how to connect to the ElasticSearch back-end. The information obtained from the analysis is stored in ElasticSearch for its use in visualizations (via Kibana). For a full understanding about the analysis output check out Storm Trident Computation.
mvn clean install
: run tests, check correct headers and generaterealtime-jar-with-dependencies.jar
file inside/target
foldermvn clean test
: run tests checking topology outputmvn license:check
: verify if some files miss license header. This goal is attached to the verify phase if declared in your pom.xml like above.mvn license:format
: add the license header when missing. If a header is existing, it is updated to the new one.mvn license:remove
: remove existing license header
This is a list of considerations before implementing a new RAGE Analytics Realtime file.
- The analysis package must follow the restrictions described here.
- Starting method of our topology class can be found here.
- Example connecting to Kafka to pull data here.
- Converting String data from Kafka to JSON objects here.
- ElasticSearch indices names definition here, methods
getTracesIndex(String sessionId)
andgetResultsIndex(String sessionId)
. - Topology definition (Storm Trident API) here.
- Sanitizing traces and defining the analysis stream and persisting it to
sessionId
ElasticSearch index (traces index). - Realtime GamePlay State analysis definition and persisting to
results-sessionId
ElasticSearch index (results index).
- Sanitizing traces and defining the analysis stream and persisting it to
- Topology tests example.
- Project dependencies assembled here.