Observations of audiobookshelf's feed update behavior

[tisba]

Hey all 👋

First, some context
I'm running a small proxy service which generates RSS podcast feeds from a GraphQL API provided by ARD Sounds¹. The idea is that you give it a URL for a show from https://ardsounds.de and I'll give you a URL you can subscribe to with your podcast client, like audiobookshelf. You can find the service at https://ardsounds.tisba.de (German only, sorry 😅).

The API by ARD is public and not particularly fast. I try to play nice and do apply quite aggressive rate limiting, currently based on client IP.

I'm not a user of audiobookshelf myself, nor do I know how it works exactly. But a few of my users use audiobookshelf to subscribe to multiple shows using my proxy, which is great!

My Observations

A few seem to use audiobookshelf with this service and I noticed a few things:

• refreshing of feeds seem to happen -exactly- every 60 minutes (within 1-2 seconds)
  • per installation, my service gets hammered, exactly every 60 minutes :-/
  • it seems that all feed updates are scheduled in the same second (or maybe done in concurrently?). Is this intentional?
  • It would be great to limit feed updates at least per host, or spread out the update interval a bit. Or at least retry 429 after a backoff time and/or utilize ratelimit-* response headers (see below).
• No support for conditional requests
  • The services responds with Last-Modified headers, but audiobookshelf does not seem to utilize them on subsequent requests as If-Modified-Since request headers.
  • Does audiobookshelf have the information when a feed was last updated? This would make it trivial to implement conditional requests! This would make it MUCH easier to reduce load and save a lot of bandwidth, especially because podcasts don't tend to be updated once per hour ;)
• there seems to be no support to handle rate-limited responses or retries on HTTP errors (at least 429)
  • when a client gets rate limited, my service responds with HTTP 429 and ratelimit-limit, ratelimit-remaining and ratelimit-reset headers. audiobookshelf does not seem to retry on 429 responses, or utilize the ratelimit-* headers.
  • Would it be possible to do retries on 429, ideally with some (exponential) backoff? I'm sure this would improve user experience as some users run reproducible into my rate limiting with the same feeds to be updated.

Footnotes

1. For those unfamiliar with it, ARD is a joint organisation of Germany's regional public-service broadcasters. ARD Sounds is their offering of various audio formats, which they call "podcasts", but don't offer feeds 🙄 ↩

[Vito0912]

refreshing of feeds seem to happen -exactly- every 60 minutes (within 1-2 seconds)

Users do schedule it themself. ABS is a self-hosted media server. Probably one user if it's really just every 60 minutes and no outsiders. Some ABS users tend to feel to update very often.

there seems to be no support to handle rate-limited responses or retries on HTTP errors (at least 429)

Because of above. Idk (and I am not a maintainer) if that would be considered a bug. I do not think so, because at the time the user set, the podcast was unavailable. It tries again when the user scheduled it.

But in general it's probably just easier to have the users host this. It seems very fresh and if you already have problems, it probably won't get better :). ARD is a shitty "company" for that matter and it does not make sense that they provide no rss feed. But I guess the "subscription" prices are not high enough haha for that. I had/have a bridge myself. Maybe check the HAL+JSON endpoints. I never tried the GraphQL ones, but these never had any problems with speed

[nichwall]

Like Vito said, the default is to check every 60 minutes, but this is a setting exposed to the user on a per-podcast basis so users can change the query frequency and offset to match their podcast release schedules or personal preference.

I agree with Vito that a rate limit request not being respected is not really a bug, if the request from ABS being rate limited by the provider with the default of every 60 minutes, realistically it will still be rate limited if it's retried immediately/shortly afterwards.

[tisba]

I should have put more focus on the conditional requests part :)

Do you know, if audiobookshelf remembers when it last updated something? It would be trivial to implement in this case and this would make probably the biggest difference for me and basically all other sources of feeds as well. I don't need to rate limit requests leading to 304 Not Modified responses, as they are extremely cheap (cachable, no upstream request needed, no response body).

[tisba]

refreshing of feeds seem to happen -exactly- every 60 minutes (within 1-2 seconds)

Users do schedule it themself. ABS is a self-hosted media server. Probably one user if it's really just every 60 minutes and no outsiders. Some ABS users tend to feel to update very often.

I think there is a misunderstanding. If the update frequency is 60 minutes, the system could add some jitter to avoid synchronization. The user experience does not change in this case, it's just nicer behavior (especially if there are many feeds fetched from the same server).

Re: No retries on HTTP errors and rate limiting: From my logs it looks like that some users get a bunch of their ~10-15 feeds only updated every 5-6 hours, even though they try to do it every 60 minutes. For me this doesn't matter, but a backoff+retry on errors would help to make the outcome closer to what the user intended I guess.