RFD 0171: Database session playback

[gabrielcorado]

Related to #5799 and #9019.

Rendered version

[greedy52]

First quick around. Thanks for drafting this!

[greedy52]

what mode is preferred in case of mode conflict?

For recording_only, I assume the "start" and "end" events will still emit? Should a toggle on audit-events event be part of record_session?

[gabrielcorado]

what mode is preferred in case of mode conflict?

Most verbose one. full > recording_only > on > off

I assume the "start" and "end" events will still emit?

They will be included, like in the SSH sessions (I'll update the text to clarify that).

Should a toggle on audit-events event be part of record_session?

I'm still unsure about this too, because it doesn't look like a recording option. But moving this around (for example, to a cluster-level config) would reduce the ability to more dynamic configurations (for example, per protocol).

[Tener]

Configuring this at role level leads to some awkwardness. The fact that user has multiple roles forces you to perform merging on this, and few people can track the end result. You end up with a situation where more finicky configurations are unstable (people will be surprised that the recording was made/was not made), and ultimately disable or enable it for everyone.

Instead of roles, how about we scope it around individual databases? The default values may be put in a global settings object, but individual database objects can contain overrides. For auto-discovered databases we can use labels to guide these.

Ultimately, I think this setting is less about the people accessing the data and more about the nature of data itself.

Moving on, do we need these many levels for the configuration? I'd propose we keep the current capabilities without any changes and only treat the full session recording as an optional add-on. That collapses the full / recording / on / off enum down to session_recording_enable: true|false.

However, I do think we could use a more complex configuration, perhaps something like this:

database_session_recording:
    enable: true | false
    response_max_rows: 100 # only meaningful for row-based protocols
    response_max_bytes: 2M # meaningful for all protocols, including row-based
    anonymise: disabled | ellipsis | hash | stats_only (default)

With a strong default anonymisation we can make the feature enabled by default, which both prevents surprise capture of sensitive data as well as promotes the feature among the users.

[gabrielcorado]

I've removed the full (as we're no longer recording the data returned) and recording_only (considered out of this RFD scope as it is not directly related to session recording) options. We're left only with on|off.

[greedy52]

What does the player show when there is no data result but only queries?

[greedy52]

Will older db sessions before this feature become playable? What happens when the db sessions are not playable (e.g. protocol opensearch)?

[gabrielcorado]

Will older db sessions before this feature become playable?

That's the initial ideal.

What happens when the db sessions are not playable (e.g. protocol opensearch)?

Protocol-specific events will require additional implementation (to convert them into SessionPrint). So, the recording will not be available if all the events can be converted. We could always try to render them based on the generic events, but some protocols might never emit them (opensearch that you mentioned is one of them).

I'll investigate better how we can improve this UX-wise so we don't show the play button for sessions that we cannot convert properly.

(I'll add all this info to the RFD)

[Tener]

I think it would be nice to add non-textual playback option too. The database responses are inherently more structured than SSH terminal events. We can exploit that, for example to show the query response as a actual HTML table (sortable and with client-side filters!) or properly convey the request-response nature of some protocols, like opensearch.

Obviously this would require some UI work, but in the end I think it would be a great alternative to purely text-based reply, which has numerous inherent limitations.

[gabrielcorado]

I think it would be nice to add non-textual playback option too.

This can be implemented outside the session player (currently focused on text-based sessions). All the information will still be present on the recordings.

[greedy52]

I'll investigate better how we can improve this UX-wise so we don't show the play button for sessions that we cannot convert properly.

(I'll add all this info to the RFD)

Did I miss this? I don't find a section on this.

[gabrielcorado]

I've added a section describing the general necessary changes on the WebUI. I haven't included the full details, as we might discuss them on the implementation PR.

[greedy52]

I've added a section describing the general necessary changes on the WebUI. I haven't included the full details, as we might discuss them on the implementation PR.

I am more looking for where users can find the recordings. Before this feature, users have to grab an ID from an audit event. So it's safe to assume that now we can see the database sessions in "Session Recordings" tab and tsh recordings ls right?

[gabrielcorado]

Right, we'll bring the database recordings to those listings.

[Tener]

We may want to rename it to DatabseResultStatus or similar. What will be the contents of the status?

[gabrielcorado]

It is a message already present on our events package. I've copied it to the RFD to make it easier to understand.

[Tener]

An optional error_code might be nice: these are often present for both relational databases as well as for those based on HTTP requests.

[Tener]

Configuring this at role level leads to some awkwardness. The fact that user has multiple roles forces you to perform merging on this, and few people can track the end result. You end up with a situation where more finicky configurations are unstable (people will be surprised that the recording was made/was not made), and ultimately disable or enable it for everyone.

Instead of roles, how about we scope it around individual databases? The default values may be put in a global settings object, but individual database objects can contain overrides. For auto-discovered databases we can use labels to guide these.

Ultimately, I think this setting is less about the people accessing the data and more about the nature of data itself.

Moving on, do we need these many levels for the configuration? I'd propose we keep the current capabilities without any changes and only treat the full session recording as an optional add-on. That collapses the full / recording / on / off enum down to session_recording_enable: true|false.

However, I do think we could use a more complex configuration, perhaps something like this:

database_session_recording:
    enable: true | false
    response_max_rows: 100 # only meaningful for row-based protocols
    response_max_bytes: 2M # meaningful for all protocols, including row-based
    anonymise: disabled | ellipsis | hash | stats_only (default)

With a strong default anonymisation we can make the feature enabled by default, which both prevents surprise capture of sensitive data as well as promotes the feature among the users.

[Tener]

I think it would be nice to add non-textual playback option too. The database responses are inherently more structured than SSH terminal events. We can exploit that, for example to show the query response as a actual HTML table (sortable and with client-side filters!) or properly convey the request-response nature of some protocols, like opensearch.

Obviously this would require some UI work, but in the end I think it would be a great alternative to purely text-based reply, which has numerous inherent limitations.

[gabrielcorado]

For now, we're removing the data recording from this RFD scope. I've updated all the sections and will resolve some related comments.

[greedy52]

Could you also add some info on how the user can find these recordings? like how recordings are listed, in both Web UI and tsh.

[greedy52]

The recording mode is optional for the initial MVP. And it might depend on how we proceed with #40170

[greedy52]

The DatabaseSessionCommandResult only captures a number affected_records. How does the player known how to render Returned xxx vs xxx inserted? If the logic is based on the query, how reliable is our logic?

[gabrielcorado]

It is the player translator's job to generate those descriptions. For example, tt can be implemented similarly to our PoC as a state machine. So, it knows the type of the command that generated the result. Databases that generate concurrent executions could be used; in that case, we would need a field to make the relationship of command -> result.

[greedy52]

I'll investigate better how we can improve this UX-wise so we don't show the play button for sessions that we cannot convert properly.

(I'll add all this info to the RFD)

Did I miss this? I don't find a section on this.

[r0mant]

As far as I recall, all database access events currently go to both audit log and session recording. So we're just adding an ability to replay them (and an extra event for query result). Is that correct? Will off option basically mean that we stop emitting events to session recordings?

[gabrielcorado]

As far as I recall, all database access events currently go to both audit log and session recording. So we're just adding an ability to replay them (and an extra event for query result). Is that correct?

Yes.

Will off option basically mean that we stop emitting events to session recordings?

Yes. This change will not affect audit events (similar to SSH enhanced session recordings). Currently, there is no option to do this other than completely disabling the session recordings on the cluster level (auth_server.session_recording: "off").