New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log delays in delivery of toots #5225

Closed
wants to merge 5 commits into
base: master
from

Conversation

Projects
None yet
2 participants
@zunda
Collaborator

zunda commented Oct 5, 2017

After Japan national day of load testing, aka televising of "Castle in the Sky" on which many internet users post "balse" to SNSs, I noticed Mastodon doesn't have metrics that indicate how it is communicating to the fedeverse.

This change is to add logs like below that will show how quickly Mastodon is delivering toots to remote instances as a good measure of how quickly Mastodon is working on queued tasks.

source=Pubsubhubbub::DeliveryWorker destination="https://example.com/api/subscriptions/12345" measure#delivery.delay=65sec count#delivered=1
source=ActivityPub::DeliveryWorker destination="https://example.com/inbox" measure#delivery.delay=66sec count#delivered=1

The additional metrics can be visualized with, e.g., Librato like below.

screen shot 2017-10-04 at 9 56 30 pm

zunda added some commits Oct 3, 2017

Log delays in delivery of toots
After Japan national day of load testing, aka televising of "Castle
in the Sky" on which many internet users post "balse" to SNSs, I
noticed Mastodon doesn't have metrics that indicate how it is
communicating to the fedeverse.

This change is to add logs like below that will show how quickly
Mastodon is delivering toots to remote instances as a good measure
of how quickly Mastodon is working on queued tasks.

source=Pubsubhubbub::DeliveryWorker destination="https://example.com/api/subscriptions/12345" measure#delivery.delay=65sec count#delivered=1

source=ActivityPub::DeliveryWorker destination="https://example.com/inbox" measure#delivery.delay=66sec count#delivered=1
Avoid parsing nil as time
Many specs doesn't set published time.
@Gargron

This comment has been minimized.

Show comment
Hide comment
@Gargron

Gargron Oct 5, 2017

Member

Have you tried hooking up Mastodon to statsd? When configured, Mastodon sends render times, query times, cache hits, and sidekiq processing times to statsd, which can output them into a time series database. Perhaps you could make this be a statsd invocation too, instead of a log entry?

Member

Gargron commented Oct 5, 2017

Have you tried hooking up Mastodon to statsd? When configured, Mastodon sends render times, query times, cache hits, and sidekiq processing times to statsd, which can output them into a time series database. Perhaps you could make this be a statsd invocation too, instead of a log entry?

@zunda

This comment has been minimized.

Show comment
Hide comment
@zunda

zunda Oct 5, 2017

Collaborator

Ok, I'll see how it runs on Heroku.

Collaborator

zunda commented Oct 5, 2017

Ok, I'll see how it runs on Heroku.

@zunda

This comment has been minimized.

Show comment
Hide comment
@zunda

zunda Nov 13, 2017

Collaborator

I'm withdrawing this pull request.

Hooking up statsd+Datadog and viewing some code, I found that some of Sidekiq queue are already reporting latency. As for Mastodon, Mastodon.production.sidekiq.queues.push.latency seems to provide similar information as measure#delivery.delay proposed here:

statsd+Datadog

screen shot 2017-11-12 at 3 47 19 pm

this pull request+Librato

screen shot 2017-11-12 at 3 47 37 pm

Collaborator

zunda commented Nov 13, 2017

I'm withdrawing this pull request.

Hooking up statsd+Datadog and viewing some code, I found that some of Sidekiq queue are already reporting latency. As for Mastodon, Mastodon.production.sidekiq.queues.push.latency seems to provide similar information as measure#delivery.delay proposed here:

statsd+Datadog

screen shot 2017-11-12 at 3 47 19 pm

this pull request+Librato

screen shot 2017-11-12 at 3 47 37 pm

@zunda zunda closed this Nov 13, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment