Webfinger timeout seems too short : increasing sidekiq error rate caused by PR #13836

[ponapalt]

Expected behaviour

Acceptable sidekiq error rate.

Actual behaviour

Sidekiq retry and dead job queue growing more than before.

Below is error message sample:

HTTP::TimeoutError
execution expired
/usr/home/mastodon/live/app/helpers/webfinger_helper.rb:8:in initialize' /usr/home/mastodon/live/app/helpers/webfinger_helper.rb:8:in open'
/usr/home/mastodon/live/app/helpers/webfinger_helper.rb:8:in `block in connect'

Steps to reproduce the problem

Upgrade mastodon to master branch d890abf (PR 13836) or newer.

Specifications

This problem is caused by PR #13836

Amendment Proposal

I think connection timeout is too short for busy servers.
It's difficult to find optimal value of timeout, but, based on my investigation, It seems 120sec. is good start for us.

line 29 of app/helpers/webfinger_helper.rb:

timeout_options: {
 write_timeout: 10,
 connect_timeout: 120,  # <== increase timeout value
 read_timeout: 10,
},

Note

This problem seems to occur while processing ActivityPub Announce activity (a.k,a, Boost, BT).
I haven't checked in detail, but there may be performance issues in processing Announce activity.

[Gargron]

I believe raised error rates is a natural effect of tightened timeout limits. Previously those requests were holding up the Sidekiq queue by possibly minutes. The idea is to fail fast and retry later because it allows more successful requests to be processed instead of waiting for just one thing.

[ponapalt]

Thanks for your comment.
I thought twice about shortening the timeout, and it's also a convincing strategy for saving resources.

It seems I was too concerned about missing toots due to too short timeout.
I'd like to close this issue after waiting a while for any other opinions.

[Gargron]

I apologize for dismissing this bug report. I believe there was something amiss after all.

I noticed that my dead job queue has many entries for HTTP::TimeoutError. When I check the domain from a browser, it opens instantly. When I retry the job, it raises HTTP::TimeoutError. I was able to narrow it down to the goldfinger gem and vanilla HTTP.rb that it uses. Something around its default DNS resolution and timeout handling is not correct. Therefore, I am submitting a fix that removes the goldfinger gem and instead re-implements simplified webfinger resolution using our own, more advanced Request class.