Issues Sampling (like traces_sampler)

[saippuakauppias]

Problem Statement

I ran into the same problem as was voiced in: getsentry/sentry#28792 (comment), getsentry/sentry#46670, getsentry/sentry#27884.
In short: there is a service that periodically restarts due to heavy load on it, we can't do anything about it. At the moment of restarting the service - there is a large (hundreds of thousands) number of sending logged messages to Sentry. It is impossible to disable messages at all, because we need to control the situation when the service has stopped responding at all.

There is a way to set rate limit for tracing in the documentation (and it is mentioned in getsentry/sentry#46670 (comment)). But there is no way to set rate limit by message text from the logger.

We are using Django, and if we try to do according to the example in the documentation, there is no useful information about the message itself in sampling_context. There's just the transaction, no error, no logger message text, no traceback.

Solution Brainstorm

Please make issues_sampler so that it would be possible to filter a errors before sending it to Sentry.

[antonpirker]

Hey @saippuakauppias !
Thanks for bringing this up. This is good idea and I think on another platform we support we already have something like you suggested. I put this on our todo list, but it will take some time because we are all working on something right now.

[sentrivana]

@saippuakauppias This sounds like something that could be solved by defining a custom before_send. All events but transactions will pass through it if you define one and you can decide to drop an event based on whatever criteria you need (at that point you have the ready-to-be-sent event available). Would that solve the issue?

[saippuakauppias]

@sentrivana, thanks for the idea, I'll try this, even though it looks like a hack

[sentrivana]

@saippuakauppias It's a perfectly valid before_send usecase though -- it's currently the intended way to go about dropping events before they leave the SDK. It's also more powerful (though arguably less convenient, if all you need is to define a sample rate) than an issues sampler since you have full control over the criteria.

It does seem like our messaging around it is not ideal though if it sounds like a hack (and in general many folks don't seem to know about it). So we could certainly improve our docs around this.