Enrich payloads after PII scrubbing

[jjbayer]

We run event normalization before PII scrubbing, and for good reason: The user expects that their scrubbing rules are evaluated on the event in its canonical form.

However, what we currently call "normalization" includes enriching the event with information derived from the event itself, for example in span.sentry_tags. These derived fields should be populated after PII scrubbing, for two reasons:

1. It's very easy to miss that the derived field needs to be scrubbed. Currently we potentially have to update our default scrubbers for every derived field we add in normalization. See fix(pii): Scrub user fields in span.sentry_tags #4364, sentry_user not PII scrubbed #3917.
2. If users define a custom rule on a derived field, they might miss the fact that they need to scrub the original field as well. Running the "enrichment" before PII scrubbing forces the user to define the rule on the correct field.

Proposal: Introduce a new processing step named "enrichment" or similar that runs after PII scrubbing. Gradually move derived fields from normalization to enrichment.

ref: INC-984

[jjbayer]

ref: getsentry/security-advisories#16

[jjbayer]

Con: Doing enrichments after PII scrubbing also means that selectors for inbound filters / dynamic sampling might not be available. So we should initially do it only for sentry_tags.

[jjbayer]

Q: Could on-demand metric extraction leak PII?