[Spike] Datastore level ExecutionLog groupings (Timebox: 3 day)

[seanpreston]

Right now the execution logs only express collection level groupings of logs. What would it take to report execution logs at a datastore level?

For example, if the UI wanted to show all execution logs on a specific datastore, when they started and stopped.

How will we know when the datastore has stopped processing (when we've hit all nodes for that datastore in the traversal)

[pattisdr]

Re: How will we know when the datastore has stopped processing?

From a high level, we have the information to deduce whether a datastore is complete, provided the graph hasn't changed in the interim. We temporarily store a representation of the entire access graph, as well as which collections have been visited for an access graph, and which collections have been erased in the erasure "graph".

While Fidesops has a current definition of when it has decided a datastore is "done", it is a little complicated when new collections are added. A datastore may be "done", and then the request fails on a collection in a different datastore, and in the interim, a collection is added to the datastore, making it no longer "done".

The definition of "done" may change as a result of #1036 too.