fetcher

Bruno Renié edited this page Oct 18, 2016 · 4 revisions
Clone this wiki locally

If you've arrived here, it's probably because you're seeing the FeedHQ User-Agent in you logs.

What is FeedHQ?

FeedHQ is an open-source web-based feed reader. It is built with readability and mobility in mind.

What kind of traffic does FeedHQ generate?

FeedHQ does a couple of things:

  • It polls your Atom or RSS feeds for updates
  • It tries to fetch your site's favicon

You should see four kinds of User-Agents in your logs. The feed fetcher itself:

FeedHQ/X.Y.Z (https://github.org/feedhq/feedhq; 1 subscriber; https://github.com/feedhq/feedhq/wiki/fetcher; like FeedFetcher-Google)

The link checker, which makes HEAD requests to detect broken links:

FeedHQ/X.Y.Z (https://github.org/feedhq/feedhq; ping; https://github.com/feedhq/feedhq/wiki/fetcher; like FeedFetcher-Google)

The favicon fetcher (you shouldn't see it often):

FeedHQ/X.Y.Z (https://github.com/feedhq/feedhq; favicon fetcher; https://github.com/feedhq/feedhq/wiki/fetcher; like FeedFetcher-Google)

The feed checker (when someone subscribes to a feed, FeedHQ checks that the given URL is an actual feed):

FeedHQ/X.Y.Z (https://github.com/feedhq/feedhq; checking feed; https://github.com/feedhq/feedhq/wiki/fetcher; like FeedFetcher-Google)

The feed fetcher has support for exponential backoff. This means if your feed returns an error-ish HTTP status code (401, 403, 404, 502, 503, etc.) its update period increases. From 1 hour for working feeds, this goes gradually up to about a day for failing feeds. Once the 24-hour period is reached it doesn't increase anymore. If your feed starts working again the period automatically resets to the default 1-hour period.

While you're here, we recommend that you implement PubSubHubbub for your feeds if it's not already the case. It'll propagate your updates faster to your readers and FeedHQ will poll less often. There are many plugins for different content publication systems.

Thanks for reading this!