[EBT] Apply PII filters to the events

[afharo]

Our developers are very conscious about the data that we send and care about not sending any PII.

Especially, since we need to declare the schemas, it provides us with an additional read-proof understanding of the data to identify these cases. On top of that, whenever we identify a leak, we can remotely disable the offending event type.

However, this system is far from perfect and we should implement a piece of logic to apply some schema-agnostic PII filtering.

We may choose to implement this on the receiving end to avoid any performance overload on the Kibana end. However, the sooner we apply the filters, the better for the end-user. Also, if implemented on the client-side, we'll prove that we care about PII because it'll show redacted in the audit logs #132256

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgayvallet]

What exact filtering mechanism are we thinking about here?

Usually, sensible data filtering is done by redacting a predetermined subset of a list of fields or data structures, e.g

kibana/src/core/server/http/logging/get_response_log.ts

Lines 18 to 19 in 755950a

	const FORBIDDEN_HEADERS = ['authorization', 'cookie', 'set-cookie'];
	const REDACTED_HEADER_TEXT = '[REDACTED]';

Given that, for our EBT events, the source of each event is fully in control of the event's 'owner', such per-field filtering does not make sense, given we already know precisely which fields are going to be sent, doesn't it?

So I'm assuming we're talking about value-based and not key-based filtering here? In that case, which data are we thinking of trying to identify as being sensible, and more importantly, how?

[afharo]

So I'm assuming we're talking about value-based and not key-based filtering here?

Your assumption is correct! I tried to claim this intent in schema-agnostic PII filtering in the description 😇

In that case, which data are we thinking of trying to identify as being sensible, and more importantly, how?

I guess that's to be defined? Probably the @elastic/platform-analytics and @pauldotpower can help out with the list.