Sentry delays execution of scheduled commands when offline

[ZacharyDuBois]

How do you use Sentry?

Sentry SaaS (sentry.io)

SDK version

4.5.1

Steps to reproduce

Currently, Sentry is having either ingestion delays or other service issues. All of my cron checks have been marked as failed even though they are working.

1. Create a cron monitor like this (something that would run otherwise fast):

Schedule::command('horizon:snapshot')
    ->everyFiveMinutes()
    ->onOneServer()
    ->withoutOverlapping()
    ->sentryMonitor('horizon_snapshot');

2. Run the command without a DSN set. Command completes almost instantly and will move on to the next task immediately.
3. Run again with DSN set while Sentry is having issues. Command runs successfully but delays the next roughly 10s (essentially whatever Sentry has for a timeout).
4. If multiple scheduled tasks are needing to be run, they are all delayed resulting in a backlog.

Expected result

Since Sentry is only a monitor, it should not impact the rest of the application's operations. If possible, I would think Sentry would either wait for all scheduled commands to complete so a backlog doesn't develop when they have an issue on their end.

Actual result

Causes a backlog of scheduled commands to run which also can result in two schedule:run's overlapping (thank you Laravel for that withoutOverlapping() flag!). Also impacts internal health checks within the Laravel app itself. Like Spatie's health package for monitoring queue/schedule/heartbeat.

[cleptric]

Could you let me know the time frame in which you saw these longer response times?

One way to make your application less reliant on wait times due to PHP's synchronous nature is to use a self-hosted relay.
It's a small proxy server you put somewhere next to your application. The server then accepts all events and forwards them to Sentry.

[ZacharyDuBois]

Roughly today at 09:30 ET. It was right after a deployment so it made me concern we pushed something broken to production. HA!

Yeah, definitely understand the synchronous part and how that can be a pain. Is the default timeout not 5s anymore? According to the Sentry Laravel docs, the default timeout should be 5s. My scheduled jobs were taking 10s each to run (when they are normally <1s). The only thing I can think of is the Sentry package was attempting two requests for that transaction?

Also, I only briefly looked at Relay. Might end up being something that needs to happen sooner than later. Thanks for the recommendation!

[cleptric]

We do set a connection timeout of 2s and a timeout of 5s. These values can also be customized in your config/sentry.php file, as http_connect_timeout and http_timeout.

We also recently made some changes to our ingestion time outs server side, but what you are reporting concerns me that these changes do not have the effect we were hoping to see.

[stayallive]

I'm wondering if the doubled timeout you are seeing is because for scheduled commands we do a ping at the start of the command and one at the end. Which is why it's double of the expected 5s timeout although 5s might still be on the long side for a timeout.