Persistence plugin PoC completed #243

Open
wants to merge 2 commits into
from

Projects

None yet

2 participants

@P3trur0
Contributor
P3trur0 commented Dec 8, 2016

This commit introduces a first version of a Persistence plugin to store serialized events on a Cassandra (version 3.9) instance.

The events are:

  • serialized using AVRO (ver. 1.8.1)
  • persisted on Cassandra together with their own serialization schema

Later, when required, the serialized events can be retrieved from Cassandra to be processed and analyzed elsewhere (see here for a testing example to retrieve and deserialize events).

Suggestions and improvements are welcome!


Please note that the plugin comes with a bundled Integration Test configuration, so it will be possible to easily verify the behavior of Cassandra persistence.

These are the Maven goals relevant to the plugin:

mvn compile generates the Avro Event object starting from the Avro schema (however this Java class is not used in the plugin so far);

mvn verify runs the integration tests: it starts a 32 bit Docker instance of Cassandra 3.9 and performs a series of write/read operations to verify if events are properly serialized and persisted.

@P3trur0
Contributor
P3trur0 commented Dec 14, 2016

@mcicolella I've fixed a couple of errors that did not allow to start the plugin properly. Now on my freedomotic local instance the Persistence plugin is properly loaded. It operates with a local Cassandra Docker image!

@P3trur0
Contributor
P3trur0 commented Jan 14, 2017

@mcicolella finally here I am with a proper edit of this PR. Currently this plugin should be able to persist any kind of events, independently from their actually properties set.
Basically, each time an event is triggered, it is serialized with Avro and saved, together with the serialization schema, in a Cassandra table.

This serialized version of the event contains an array field, named "properties", where are stored all the properties as [key,value] couples.
I've tested it in my local Freedomotic environment and it seems to work properly now.

Please, let me know if you have any problem while executing it.

Meanwhile, have a nice evening,

@P3trur0

P.s.: to speed up testing in my environment, I've prepared a dummy script that:

  1. Build the plugin
  2. Build freedomotic
  3. copies the built plugin artifacts in the proper freedomotic plugin location
  4. runs freedomotic

it is available here.

@mcicolella
Contributor

@P3trur0 Great work! I'm testing it.
For now there is a "reaction" that should be removed or modified because the command "Save data on the persistence layer" doesn't exist.

@P3trur0
Contributor
P3trur0 commented Jan 15, 2017

@mcicolella where can I perform this removal? Could you please provide me a reference?
Thank you!

@mcicolella
Contributor

You can simply remove the .xrea file into data/rea folder.

@P3trur0
Contributor
P3trur0 commented Jan 15, 2017

Here you go, .xrea removed properly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment