Newsboat too aggresive when pulling in a feed?

[mboelen]

Newsboat version (copy the output of newsboat -v or the first line of git show): 2.35.0

Looks like Newsboat is doing multiple requests (in 1 second time) to retrieve a feed. Our server returned a HTTP 304 response, so Newsboat should now that nothing has changed.

We rate-limit clients that are too aggressive, and we see multiple IP addresses getting rate-limited that use Newsboat.

2024-04-07T21:19:04+00:00 304 a.b.c.d "GET /feed/ HTTP/2.0" 0 "-" "Newsboat/2.35.0 (Linux x86_64)" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .
2024-04-07T21:19:04+00:00 304 a.b.c.d "GET /feed/ HTTP/2.0" 0 "-" "Newsboat/2.35.0 (Linux x86_64)" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .
2024-04-07T21:19:04+00:00 304 a.b.c.d "GET /feed/ HTTP/2.0" 0 "-" "Newsboat/2.35.0 (Linux x86_64)" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .
2024-04-07T21:19:04+00:00 304 a.b.c.d "GET /feed/ HTTP/2.0" 0 "-" "Newsboat/2.35.0 (Linux x86_64)" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .
2024-04-07T21:19:04+00:00 429 a.b.c.d "GET /feed/ HTTP/2.0" 162 "-" "Newsboat/2.35.0 (Linux x86_64)" TLSv1.3/TLS_AES_256_GCM_SHA384 0.000 .

Good to know: I don't use Newsboat myself. Just reporting this, to make sure that the client is behaving correctly and reducing resources (on the client and server).

Above a part of the log, with IP address masked (but it was the same one, 5 times). As you can see, the 5th request is gets the rate-limit response (HTTP 429).

[dennisschagt]

That is definitely unexpected, thanks for reporting it!
Can you share the full feed url? (If that is not sensitive data)

[mboelen]

Sure! It's /feed/ on the blog Linux Audit, which is an alias for the RSS Atom link at the bottom of the page.

Let me know if you need anything else. So far I have seen multiple of these requests (different IP addresses, but all with the same outcome: multiple requests, getting rate-limited by our configuration).

[dennisschagt]

Thanks, I can reproduce this by configuring download-retries 5 in Newsboat's config (download-retries doc).

Testing with git bisect points towards commit e464c83.

[mboelen]

Perfect. Looks like when this is resolved, it will save a lot of bytes :)

Every web server implements a different set of headers, so maybe others might be useful to reduce the number of queries during a day.

Ideas:

• Use the Expires header as an additional hint to requery?
• Maybe If-Modified-Since might also be useful?
• Have a look at REDbot, as they parse the the headers and give some good hints about the implemented headers (Example). Might be useful to test against a few other feeds.

[mboelen]

I encountered a few more issues with RSS feed readers, so created a blog post to report those to the different projects and monitor the outcome. Might be good input to learn from other software as well. I have acknowledged your quick response to this issue, which is much appreciated 👍

[jarv]

I have also reproduced this bug with download-retries which resulted in my client being blocked due to excessive traffic.