QUESTION: How do you support regulatory requirements to purge persisted Personally Data (e.g. GDPR)?

[thewoolleyman]

Hello,

This is a somewhat generic question about event sourcing, so feel free to refer me to a more appropriate place to ask it.

However, I'm sure that at Jet you have had to deal with this in real-life scenarios, so I thought you might have some insight.

QUESTION:

1. If my event stream encodes some data that may be considered personal data that is subject to purging under regulatory requirements, for example, Personal Data under GDPR,
2. And then I have a request to purge that data...

...how is this managed, when the data is already persisted in the immutable event stream?

I would expect the answer is:

1. Enable snapshotting for the stream
2. Enable the stream to roll over regularly and delete old events
3. Add events to the stream to delete the data
4. Force (or wait for) a snapshot and the previous events to roll over and disappear.

However, I'm curious what (other) solutions there are for this in practice.

Thanks!
-- Chad

[bartelink]

Jet was US only, which brought CCPA-driven requirements. I've not personally been involved in those, and Equinox doesn't present any specific features and/or have stances that alter the overall equation on this. In terms of active systems using Equinox, those by design and/or nature of their business function don't tend to maintain personal data.

I'd venture this is best asked on the DDD-CQRS-ES forum - this is a regular question, and there are experienced folk there.

As general advice from someone who has not traversed the issue IRL properly: talking to lawyers and product managers is going to be the first port of call - storing less data, deciding high level strategies about how to segregate where personal data lives, considering things like encrypting sensitive data and then discarding the keys in the event of a request to be forgotten.

For Equinox.EventStore: How one technically gets rid of data in either an ESDB in-stream snapshot is going to be tied up in the exercise/discussion of if/how you're going to get rid of the event that entered it into the state underlying the snapshot in the first instance.

For Equinox.Cosmos, the pruner and archiver mechanisms may or may not be useful

For Equinox.SqlStreamStore, I'd ask the question as a SqlStreamStore question

[thewoolleyman]

Thanks for the response @bartelink .

I found two existing threads in the DDD/CQRS list:

1. https://groups.google.com/g/dddcqrs/c/diLl2l7E_0c/m/GTXwsxoDAgAJ
2. https://groups.google.com/g/dddcqrs/c/tH5NwF1M8YA/m/mh2hHaOfBQAJ

One approach mentioned was, as I described, having some support for ensuring the original event with the personal data is removed from the stream (by purging, ,recreating, etc).

The other interesting alternative is to encrypt personal data on the stream, then "forget" the encryption key if the data is requested to be purged. Apparently this has been tested as a valid legal approach with GDPR.

In any case, I think this adequately answers my question for now. I'll close this issue.

Thanks!

[bartelink]

Thanks for the followup; hopefully you're adequately covered now.

For avoidance of doubt I was referring to https://github.com/ddd-cqrs-es/slack-community - while the fundamental approaches are as described in the linked threads on the email group, the bulk of discussion now takes place on the Slack (there are no archives there though), the place to get a contextual and complete answer to such a question especially if you can provide some details of the actual nature of your system and its specific constraints is probably there.

I also forget to mention earlier that:

1. there is a purge API in Equinox.Cosmos for removing events from a stream (proPruner uses it, but not for for GDPR-drive data removal)
2. For EventStore or SqlStreamStore, you'd use the APIs to do the removal of the events; Equinox doesn't expose/wrap a specific API (though it'll cope with missing events etc)

[thewoolleyman]

Also, the obvious other alternate solution is - don't store personal data in your event streams ;)