Incompatible with Akka 2.4-M2 persistence API

[zackangelo]

Looks like some breaking changes were introduced in the 2.4-M2 persistence API.

I have an updated branch that is compatible with 2.4-M2, but it will break binary compatibility with all the versions before that. How would you want to handle it? Publish a separate artifact?

[okumin]

I'm going to publish an updated version that is compatible with Akka 2.4 when 2.4 becomes stable.

[gavares]

@zackangelo is your branch public? I', interested in reviewing the code and doing some testing.

[zackangelo]

@gavares switched to using the 2.4 branch by @okumin in our own testing: https://github.com/okumin/akka-persistence-sql-async/tree/akka-2.4

[okumin]

@gavares @zackangelo I'm working on master branch now and have a plan to release the new version as soon as Akka 2.4 is published.
Notice that the new akka-persistence-sql-async may have no backward compatibility, especially DB schema compatibility.

[gavares]

Thanks @okumin. Have you started any of the work required to support Persistence Query? If not, I'm happy to help out via PRs etc.

[okumin]

@gavares I have not yet started, but I will support it.
I welcome your PRs to persistence-query project.

[gavares]

@okumin - I've forked your repo and started work

Have you given any thought as to how you'd like to store tags?

[okumin]

@gavares I'm looking forward to your commits.

Tags of persistence-query has the following requirements.

1. One journaled message can have many tags
2. Journaled messages can be searched by the specified tag

For the first requirement, we have to create a table to store relational record, between journaled messages and tags.
The second, we need the index to select journaled messages by a tag.
And I think that tags should be normalized on RDBMS because that minimizes index sizes and may provide usable APIs such as the list of all tags.

However akka-persistence-query is still an experimental project, so please feel free to define interfaces.

LevelDB's APIs may helpful to us.

https://github.com/akka/akka/blob/master/akka-persistence/src/main/scala/akka/persistence/journal/Tagged.scala#L18
https://github.com/akka/akka/blob/master/akka-persistence/src/main/scala/akka/persistence/journal/leveldb/LeveldbStore.scala#L52-L86
https://github.com/akka/akka/tree/master/akka-persistence-query/src/main/scala/akka/persistence/query/journal/leveldb

[gavares]

@okumin - Agree with your points above. I've taken most of my code from the LevelDb example and adapted it where needed for sql-async.

As for the tags table, I've thought through several options and the one I'm leaning toward at the moment would look something like:

CREATE TABLE IF NOT EXISTS tags (
  id BIGINT AUTOINCREMENT NOT NULL,
  tag VARCHAR(64) NOT NULL, -- could be made longer, just picked 64 at random
  sequence_nr BIGINT NOT NULL
)

And bridge table:

CREATE TABLE IF NOT EXISTS tags_journal_bridge (
  tag_id BIGINT NOT NULL,
  persistence_id VARCHAR(255) NOT NULL,
  jrnl_seq_nr BIGINT NOT NULL,
  UNIQUE KEY tps_key (tag_id, persistence_id, jrnl_seq_nr)
);

I'm thinking that the number of tags T, the number of journal entries J and the number of journal entry rows with tags R will probably have a relationship like: T << R < J. Given that, I think it makes sense to introduce a bridge table rather than just a single tags table.

It probably makes sense to create a covering or compound index over all three columns or two indexes on tag_id and persistence_id. I'm not a database expert so without further reading and testing I'm not sure which is the right approach. I suppose that's also the sort of thing we could leave to the users so that they could tailor it to their specific workload.

[okumin]

That looks good but I have overlooked a point.

The document for LevelDB says that deleteMessages does not delete events from tagged stream.
If we imitate that design, all journaled messages must not be deleted from the journal table.

I have not yet understand akka-persistence-query, so I'll read the document and the implementation of LevelDB's journal.

[okumin]

@gavares And I created a issue for this topic.
#8

[gavares]

I did notice that events are not deleted from the tagged stream as you mentioned above. Their implementation seems to store tagged entries as just another event stream with a special prefix. Essentially, all tagged entries are written to the journal twice. Once with the original persistence_id and once with a special persistence_id for the tag. Deleting the original entry leaves the tagged version of the entry in place which is why they can achieve that behavior. This is certainly one possible implementation for sql-async and would actually have fewer moving parts in the end.

Interestingly, the other query types don't have the same behavior. I'm not sure I understand why the behavior should be different for the different query types.