Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cassandra Projection #21

Closed
5 tasks done
octonato opened this issue Apr 1, 2020 · 0 comments
Closed
5 tasks done

Cassandra Projection #21

octonato opened this issue Apr 1, 2020 · 0 comments
Assignees
Milestone

Comments

@octonato
Copy link
Member

octonato commented Apr 1, 2020

The goal is to provide a projection that is backed by a Cassandra offset storage that can be used to track offsets.

The offset storage should support different kinds of offsets: Long, akka.persistence.Sequence, akka.persistence.TimeBasedUUID.

This projection won't support transactions and therefore can be offer delayed commits (commit every 100 events for example).

Use cases:

  • read from any source and track the offset on a Cassandra table (non-transactional)
  • read from Akka Persistence journal (Cassandra or JDBC) and track the offset on a Cassandra table (non-transactional). Specialization of the above but since very common we can provide a boilerplate free variant.

Specs:

  • Offset is managed in a Cassandra table
  • needs an offset extraction function to extract the offset from the envelope
  • requires a Flow that commits to a predefined Cassandra table (non-transactional), supports at-least-once and at-most-once
    since non-transactional, event handler can be a function E => Future[Done] where E is any envelope
  • Provide a Cassandra Offset Store based on Long. When reading Kafka, the offset will be a Long. Other offset typed may be supported and can be added when needed (or implemented by user)
  • Convenience API when reading from Akka Persistence Journal.
    We have a privileged situation when reading from Akka Persistence Journal, as we know how the envelope looks like, we know how to extract the offset (no need for extractor). However, we need a special offset store that knows how to write and read akka.persistence.Offset

Tasks:

@patriknw patriknw added this to the 0.1.0 milestone Apr 2, 2020
patriknw added a commit that referenced this issue Apr 20, 2020
* CassandraProjection
  * only added atLeastOnce to start with, will add atMostOnce and grouped in separate PR
* CassandraOffsetStore
  * offset stored as string and manifest, similar to Slick OffsetStore
* tests very similar to Slick tests, credit to @renatocaval
@patriknw patriknw self-assigned this Apr 21, 2020
patriknw added a commit that referenced this issue Apr 21, 2020
* CassandraProjection
  * only added atLeastOnce to start with, will add atMostOnce and grouped in separate PR
* CassandraOffsetStore
  * offset stored as string and manifest, similar to Slick OffsetStore
* tests very similar to Slick tests, credit to @renatocaval
patriknw added a commit that referenced this issue Apr 22, 2020
* CassandraProjection
  * only added atLeastOnce to start with, will add atMostOnce and grouped in separate PR
* CassandraOffsetStore
  * offset stored as string and manifest, similar to Slick OffsetStore
* tests very similar to Slick tests, credit to @renatocaval
patriknw added a commit that referenced this issue Apr 22, 2020
* CassandraProjection
  * only added atLeastOnce to start with, will add atMostOnce and grouped in separate PR
* CassandraOffsetStore
  * offset stored as string and manifest, similar to Slick OffsetStore
* tests very similar to Slick tests, credit to @renatocaval
* test saveOffsetAfterElements and saveOffsetAfterDuration
* Docker Cassandra in Travis
patriknw added a commit that referenced this issue Apr 22, 2020
patriknw added a commit that referenced this issue Apr 22, 2020
* no need for user to use it directly
* rename slick.OffsetStore to SlickOffsetStore
@patriknw patriknw closed this as completed May 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants