Custom Metric Rollout Strategy

[mitsuhiko]

We need a strategy to send metrics from Sentry to relay in production without negatively impacting the pipeline. The risk here is that if the volume is too high it can take down parts of the system until that scales up. Assumed bottlenecks are the relay parser and then later the consumer.

For now the risks that need management are:

• need a dynamic sampling rate on metrics emission to scale up/down quickly
• Ability to send to Relay without sending to storage relay#2462
• playbook for increasing sampling rate and opting in more metrics

[mitsuhiko]

My proposal for how to roll this out is the following:

The goal for us is to scale a given number of metrics to 100%. We do not need to target all metrics, but metrics we feel are valuable for later driving our product experience. So in some sense what we want is to be able to replicate dashboards we care about based on our own metrics backend.

Given the recent changes on Relay’s aggregator I’m no longer particularly worried about traffic hitting relay which puts most of the concern now onto the consumers and the client side SDK aggregation.

What can go wrong?

SDK side

Distributions are quite memory intensive even on the SDK side as we are waiting for a flush interval to actually get them out. That means that high volume metrics can create memory issues on the client. We also use a global aggregator on the SDK which shares a large lock. Since Python is generally not very contended due to the GIL I do not expect the lock to be contended in itself, but I do expect that the cost of acquiring the lock to be meaningful.

Another point that however is unlikely to be an issue at the moment is that if we were to have high cardinality metrics, we might be running into very frequent flushes/metric bucket re-creations if they are running into complexity limiter. This would then greatly increase the number out outgoing envelopes.

Ways to re-disk the SDK side:

• make force flush intervals reconfigurable (force a flush with a different complexity than the hardcoded one)
• add a killswitch (complete, remove a particular metric)

Relay Side

I currently do not expect Relay to be a significant concern in this rollout. However as Relay is the component that ultimately forwards metrics to the storage layer we need to have ways to disable the outflow there. The goal here is initially to observe the number and complexity of buckets created on the relay side on the way to storage. For that purpose we want to be able to route custom metrics to a separate kafka topic for either measurement or emergency reasons.

Ways to de-risk:

• Allow custom metrics to go to a custom kafka topic to avoid overwhealming generic metrics platform with bad data

Consumer Side

The consumer on the snuba side is unlikely to work in the current form with the volume we expect. As this consumer is shared with other metrics without further changes we are running into the risk of disrupting other projects. Both the number of data points from distributions and generally the number of items coming from the JSON payload are likely concerns for the current consumer.

Ways to de-risk:

• With the data collected on the custom kafka topic it should be possible to stress test the consumer with actual real world data
• Consider a separate deployment to read from the custom kafka topic until the consumer is validated to be safe and stable

Clickhouse

I currently do not believe our use of clickhouse to require modifications. The reason here is that we are largely increasing the volume of data points but not the cardinality of the data we observe. This should play quite well into the current structure of the data for both clickhouse and even our indexer design.

[jan-auer]

Executed.