New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Event Metadata #780

Open
jeremydmiller opened this Issue May 30, 2017 · 9 comments

Comments

Projects
None yet
5 participants
@jeremydmiller
Contributor

jeremydmiller commented May 30, 2017

Had a couple folks ask lately if there was any way to embed metadata into the event store. The usual examples are things like customer id or region names.

We could handle this by saying "just use base classes for the common information." From there we'd possibly allow some aggregated querying against the event store by the base type.

Other ideas:

  1. Support an extra HStore column for user supplied key/value header information
  2. Support something like duplicated fields for the mt_events and/or mt_streams table

The question though becomes how do we expose that information and allow users to search on it? Sticking a Dictionary<string, object> property on Event is no big deal.

I guess my question is, what would you use this for? Where would you want to consume this information?

Also really unsure how we'd go about capturing this information in the IEventStore API

@jeremydmiller jeremydmiller added this to the 2.0 milestone May 30, 2017

@jeremydmiller jeremydmiller modified the milestone: 2.0 Jun 7, 2017

@wastaz

This comment has been minimized.

Contributor

wastaz commented Jun 9, 2017

For me personally I would like to use event metadata for the following cases:

  • Store command id of the command responsible for causing the event.
  • Store event "tags" in order to be able to read up partial streams.

In both of these cases the only thing I would really require is an api similar to
EventStore.GetEventsMatchingMetadata(stream_id, metadata_key, metadata_value)
This api is of course quite ugly and I'm in no way proposing that this should be the api. But if Im looking at the tiniest thing I would need on the query side in order to solve my current use cases then this would be it.

Im always a bit wary of introducing Dictionary<string, object> as an offical api. Maybe it could be possible to add a strongly typed event metadata object?

Im thinking maybe something like

class AccountOpenedMetadata {
    [Duplicate]
    public Guid CustomerId { get; set; }
    public string Tag { get; set; }
    public Guid CommandId { get; set; }
}
class AccountOpened {
    Guid AccountId { get; set; }
    Guid OpenedAt { get; set; }
}

session.Events.AppendEvent(new AccountOpened(), new AccountOpenedMetadata());

At this point maybe we could wrap the AccountOpenedMetadata in a wrapper class that adds a link back to the original event, and store the metadata as a normal marten document (which could make attributes such as Duplicate work as well) and we could query the metadata objects like normal marten docs?

Not at all sure if this is a good idea, or how doable it would be but I figured Id throw the idea out there and see what you think.

@jeffdoolittle

This comment has been minimized.

Contributor

jeffdoolittle commented Jun 9, 2017

@wastaz perhaps I'm misunderstanding but it sounds like everything you described is already doable in Marten. If you want to store changes to events as well as a "metadata" document, you can do that in one DocumentSession and accomplish what you describe. Sure there isn't an official method for this on DocumentSession but it would be pretty easy to spin up your own general use abstraction for this.

Am I understanding you or am I missing something?

@wastaz

This comment has been minimized.

Contributor

wastaz commented Jun 12, 2017

@jeffdoolittle Well, yes and no. It's certainly possible today, but there's a bit of heavy lifting involved and it's not really possible to get it done in "one query" without writing direct sql queries against the tables. Basically my point is that if we are considering event metadata as a feature and we already have the docstore stuff in marten then why not try to use it as much as possible? :)

@jeffdoolittle

This comment has been minimized.

Contributor

jeffdoolittle commented Jun 12, 2017

I don't recall off the top of my head, but can you do a batch query that includes documents and events? Might help if your concern is trips to the database.

To be clear, I'm not opposed to Event Metadata, just trying to understand how people would want to use it so we can come up with a good abstraction and api for it that will stand the test of time.

@aprooks

This comment has been minimized.

aprooks commented Mar 30, 2018

I'm evaluating Marten and stumbled on this issue almost instantly. Metadata is important part
"stream-entry". It's not an event data, but some infrastructure concerns that are written along side an event.

Some of metadata fields I usually use are: user Id and scopes, correlation/causation ids (especially useful when debugging process managers). Your aggregate might have zero knowledge of these but they should be written along side business data - event.

Seems like it can be achieved by using data envelope like:

class Envelope<T> {
Data T {get;set;}
Dictionary<string,string> Metadata {get;set;}
}

But it feels like it'll make code especially aggregates uglier.

@eouw0o83hf

This comment has been minimized.

Contributor

eouw0o83hf commented Aug 13, 2018

Also evaluating Marten and lack of metadata capability is a big deal. Other frameworks provide this readily, and an enterprise event store needs to be able to handle metadata/headers.

@aprooks , I actually pursued that route and there's some deserialization bug lurking within Marten that actually renders that route infeasible for now. See #1069 for details.

@jeremydmiller

This comment has been minimized.

Contributor

jeremydmiller commented Aug 17, 2018

@eouw0o83hf This got left out of Marten 2.0 because it's a lot of work and there wasn't much demand for it at the time. Since it is an awful lot of work to support this and there's very little concrete definition about what it means or how it'd be used, can you add some concrete examples of what you want here? And as I commented in #1069, you could easily effect this yourself with a base class

@eouw0o83hf

This comment has been minimized.

Contributor

eouw0o83hf commented Aug 17, 2018

@jeremydmiller definitely doable with a base class (and absolutely understand that "not enough of a priority" reasoning), but I've used NEventStore in the past and the ability to persist headers gave a lot of extra strength to an event-sourcing framework when called upon as an audit log (for either security/"who did this" purposes or debugging/"what did the user do" purposes).

Top few things I've tossed into headers/metadata that were super useful:

  • ExecutingUserId: Who performed the action which resulted in this event?
  • AuthenticatedUserId: In a system which supports impersonation, what administrator took this action in the role of another user?
  • RequestId: Unique identifier for the web request which resulted in this event, useful for cross-correlating with Splunk/logging for debugging
  • CorrelationId: For a distributed transaction or chain of business events, what do we need to tie this event to (again useful for debugging logs)
@eouw0o83hf

This comment has been minimized.

Contributor

eouw0o83hf commented Aug 17, 2018

Oh and also in direct response to the original question, I think a straightforward Dictionary<string, object> property on Event would solve this simply and beautifully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment