Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support a pre-existing JetStream stream for events #8

Open
autodidaddict opened this issue Dec 9, 2021 · 1 comment
Open

Support a pre-existing JetStream stream for events #8

autodidaddict opened this issue Dec 9, 2021 · 1 comment
Labels
enhancement New feature or request

Comments

@autodidaddict
Copy link
Member

  • The following new options should be made available for starting a lattice observer:
    • Stream name - The name of a pre-existing stream, observer should assume this contains cloudevents from a wasmbus event stream.
    • JS domain - JetStream domain of the pre-existing stream
    • Snapshot every N - Indicates the observer should emit a snapshot event (which will purge pre-existing events from the stream) after every n events.
    • Snapshot every Mins - Indicates the observer should emit a snapshot event after every n minutes. This option is mutually exclusive with the previous option.
  • When an observer starts, it will:
    • Create an ephemeral consumer for the indicated stream (if one is provided)
    • Read stored events until NATS indicates the pending event count in the stream is 0.
    • Stop the replay subscription when the event pending count from NATS is 0.
    • Ignore all snapshot instructions during replay

The net result of these changes will make it so that, in the presence of a JetStream event stream for wasmbus.evt.* events, an observer can start at any time and automatically catch up, generating accurate state. It can also short-cut the amount of events it needs to read upon replay through the addition of a new cloud event, the lattice_snapshot_created event.

@autodidaddict autodidaddict added the enhancement New feature or request label Dec 9, 2021
@autodidaddict
Copy link
Member Author

Some JS reference - using the reply-to field on the messages we can tokenize the subject and obtain the pending count: https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-15.md#jsack

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant