Occasional 429 errors due to limits introduced to 11.X

[ArturAkh]

Dear dCache developers,

After the switch to dCache 11.2.3, we at KIT started to see HTTP 429 errors more often on our WebDAV doors and on our Web Frontends.

According to our investigations, this seems to be related to rate limiters introduced to 11.X (commits: 3f6d72a and 8e7c5b6).

While we understand now, that we have to adjust related rate limits:

Property	Meaning
.limits.rate.overall	Maximum overall request rate accepted by the service
.limits.rate.per-client.fractions	Maximum share of the overall request budget one client may use
.limits.error.max-allowed	Number of auth or permission failures allowed before temporary blocking
.limits.error.block.window.time	How long a client is blocked after too many auth or permission failures
.limits.rate.per-client.block.window.time	How long a client is blocked after exceeding its per-client request rate
.limits.blocked-clients.idle-time	How long blocked-client state is retained while idle
.limits.max-blocked-clients	Upper bound for tracked blocked-client entries

We wonder about which numbers to use, since the current defaults in dCache are too small for our operations. In addition, we can't increase those incrementaly and then observe for a few days, since a change would require a service restart of the door and/or the frontend, which means usually a downtime if being strict.

So far, we found, that logging of particular rejections due to having too many requests is only complete on DEBUG log level, such that we have to modify the logging of the domain at runtime by an appropriate admin interface command:

log set stdout org.dcache.util.jetty.RateLimitedHandlerList DEBUG

We wonder therefore, whether it is on purpose, that some of the logged messages are only at DEBUG level by default.

All in all, we would like to discuss with you the current state of dCache in that context (logging, conditions for rejection, corresponding workflow, etc.), and ask for some guidance on appropriate limit numbers.

For documentation purposes, I'm attaching at the end of this issue also the AI-based investigations which helped us understand the situation a bit:

• Study on what might be leading to HTTP 429 errors on WebDAV door and Frontend: http-429-analysis.md
• Study on whether and where the rate limiting was introduced: http-429-version-comparison-10.2.16-vs-11.2.3.md
• Study on which possibilities there are to monitor and identify events with rejected requests due to rate limits:
  http-429-live-monitoring.md

Thank you very much for your help in advance!

Best,

Artur on behalf of dCache admin team at KIT

[DmitryLitvintsev]

@ArturAkh yeah. this is a tradition to use defaults thats are not good.

Practical advice: set all these limits to ininity. Again, unfortunately, you can’t set thet to “infinity” like many other settings (and “iunfinity” will converted intio Integer.MAX_VALUE or Long.MAX_VALUE

But you probably can set all of these to an arbitrary “large number” say, 1000000000.

In the spirit of “backward compatibility” dCache should not have had any of these new limits defined (that is to be “unlimited” by default). This way you do not need to trial and error for correct limits from the bottom up.

I think an action item for dCache team is to make these defaulted to “infinity”

Dmitry

[XMol]

Hi @DmitryLitvintsev,

I would not be that harsh of a critic about the new defaults. Its just that some default are needed - whether those are infinite or not - and without any operational experience on the subject, those are chosen tentatively conservative. What I would comment on however is, that this new limiter should have been highlighted on the change log. But lets not cry over spilled milk.

Setting a large limit helps only such that we'll likely never reach it. But that then undermines the purpose of the feature, no?
As a short-term solution, this approach might be necessary just to facilitate high service quality again. Long term however, we should be able to gather data to somehow get to a sensible limit. With a sky-high limit, we'll never get feedback on what sensible limits could be, since the sole feedback we can get is, that one of the limits was reached.

Ciao,
Xavier.

[kofemann]

First of all, I am quite happy that the rate limiter works. An as KIT is only site that completed, de default are not that bad 😎 Nevertheless, not mentioning it nor in release notes neither in migration guide is a big mistake on our side.

…

-kofemann /** caffeinated mutations of the core personality */

On Thu, Apr 30, 2026, 08:59 Xavier Mol ***@***.***> wrote: *XMol* left a comment (dCache/dcache#8102) <#8102 (comment)> Hi @DmitryLitvintsev <https://github.com/DmitryLitvintsev>, I would not be that harsh of a critic about the new defaults. Its just that some default are needed - whether those are infinite or not - and without any operational experience on the subject, those are chosen tentatively conservative. What I would comment on however is, that this new limiter should have been highlighted on the change log. But lets not cry over spilled milk. Setting a large limit helps only such that we'll likely never reach it. But that then undermines the purpose of the feature, no? As a short-term solution, this approach might be necessary just to facilitate high service quality again. Long term however, we should be able to gather data to somehow get to a sensible limit. With a sky-high limit, we'll never get feedback on what sensible limits could be, since the sole feedback we can get is, that one of the limits was reached. Ciao, Xavier. — Reply to this email directly, view it on GitHub <#8102 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEMTXNXQ2WIGJVIPQKJDOD4YL2WRAVCNFSM6AAAAACYLBRLR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DGNJQGMZDSMJZGY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ArturAkh]

Thanks for your assessment on that topic, @DmitryLitvintsev, @XMol, @kofemann!

As Xavier, I'd be also in favor of having sensible limits rather than putting the settings to infinite :)

We may discuss that in more detail during the dCache Workshop. I'll try to collect some logging and more observations in the meantime.

[kofemann]

The release notes and upgrade guide are updated.

https://github.com/dCache/upgrade-guide/blob/master/UPGRADE112.md
dCache/release-notes@1460ea5

[DmitryLitvintsev]

Hi @DmitryLitvintsev,

I would not be that harsh of a critic about the new defaults. Its just that some default are needed - whether those are infinite or not - and without any operational experience on the subject, those are chosen tentatively conservative. What I would comment on however is, that this new limiter should have been highlighted on the change log. But lets not cry over spilled milk.

Setting a large limit helps only such that we'll likely never reach it. But that then undermines the purpose of the feature, no? As a short-term solution, this approach might be necessary just to facilitate high service quality again. Long term however, we should be able to gather data to somehow get to a sensible limit. With a sky-high limit, we'll never get feedback on what sensible limits could be, since the sole feedback we can get is, that one of the limits was reached.

Ciao, Xavier.

Setting the limits high is just a starting point to approach the problem from the other end (from high to low as opposed from low to high). Ostensibly if you had no limit and your system started to struggle at certaint point you would know the limits corresponding to the failure and adjust the inifnite to reasonable for your system. (like say you monitor number of connections and if it fails at N you set limit to N-1 .... just an example. Whereas if you set low limit to M, then you don’t know what to do if you hit the limit w/o system being busy. Double it? Do a bisect on limit optimization?)

[XMol]

Sure @DmitryLitvintsev, I understand that. But, the big problem here is, that we have no clue what N is, when the issues arise. 😉

[kofemann]

This is, as @ArturAkh said, something to discuss at the workshop. Looking at the local deployments, we see 4-5 429 errors per day. This is such an invisible number of hits (> 0.0001%) that can be ignored, but other sites might have a different access profile.

[ArturAkh]

After enabling the log level DEBUG for the RateLimiter on the doors today, we see "Too many requests" error in case of CMS for a few FTS transfers, and also for the Belle 2 experiment. Example:

[root@dccmsdoor-disk-e ~]# grep "Blocking.*2001.*" /var/log/dcache/webdav-dccmsdoor-disk-eDomain.log
30 Apr 2026 11:54:46 (WebDAV-dccmsdoor-disk-e) [] Blocking client with too many requests [2001:1458:301:cd:0:0:100:30e]
[...]
[root@dccmsdoor-disk-e ~]# host 2001:1458:301:cd:0:0:100:30e
e.0.3.0.0.0.1.0.0.0.0.0.0.0.0.0.d.c.0.0.1.0.3.0.8.5.4.1.1.0.0.2.ip6.arpa domain name pointer fts-cms-003.cern.ch.
[root@dccmsdoor-disk-e ~]# grep "Blocking.*2001.*" /var/log/dcache/webdav-dccmsdoor-disk-eDomain.log | wc -l
34

While that might be of smaller fraction of overall number of transfers, I would not say, that we can ignore that, since the behavior of dCache has changed, and - in particular for the case of FTS being a production activity - it is catching "false positives".

So at least some tuning for our dCache setup at KIT is required - even with the higher numbers we have currently deployed already.

[kofemann]

We can follow @DmitryLitvintsev suggestion: bump the limits and decrease them when it hurts.

[ArturAkh]

Sure we can do that once we have the next occasion for a restart.

Maybe it would be worth elevating those messages to "WARN" or "INFO" level by default on dCache software side such that we see that directly, and don't have to analyse the code and the changes since 10.2 as we have done for that issue...

That wouldn't be more verbose than the messages discussed in #8081 and the "Blocking client..." messages on INFO level would be very useful for us to debug or to assess on whether it is a problem or not - at least on my opinion.

[calestyo]

I ran into the rate limits of frontend for the Prometheus exporter I'm writing for dCache... (like when i concurrently get /pools/*/usage for all pools).

I think it would be nice if there was a way to completely disable any rate limiting... either per frontend, so that people can run dedicated one which is just internally accessible... and/or perhaps per IP or authenticated user?