XrootD throttling improvements (conn limit based per user)

[juztas]

We had this email exchange with @bbockelm some time ago (1 year ago) [EMAILCOPY]. We have been hit now several times with an excessive xrootd usage and a single user filled our uplink with transfers to WAN (other sites).

So it would be nice to have throttling control based on the user.

[EMAILCOPY]

Do we have a way to limit specific users and max connections they can open over xrootd? I like this feature in GridFTP [1] and would be nice to have the same in xrootd. I see some users are abusing the system (dataset is only at Caltech and they run via xrootd opening 20k-30k connections from all around the world) - which overall affects storage and many other things make it inefficient... 

Maybe it is not the right place to ask, but is there plans to put some logic in CRAB that can ignore whitelist and what can not? 

[1]
export GRIDFTP_TRANSFER_LIMIT="80"
export GRIDFTP_DEFAULT_USER_TRANSFER_LIMIT="50"
export GRIDFTP_<UNIX USERNAME>_USER_TRANSFER_LIMIT="40"

From Brian:
I wrote the GridFTP throttling code originally (as well as the Xrootd throttling code) ... I don't think the per-user features have been ported from the GridFTP code base to the XRootD one.  There's only a process-wide concurrency limit.

Good idea though!  Sounds like we're saving up a number of items for an "Xrootd Creature Comforts Hackathon"!

[bbockelm]

Hi @juztas -

I think this remains an excellent idea.

@djw8605 - I know you're currently looking at the overhaul of the multiuser plugin. Any chance you could put this on your plate after that?

Brian

[jmarra13]

Hi Guys,

Facing same issue here at T2_BR_SPRACE, there was any improvement on this?

Jadir

[abh3]

We have reviewed this and, frankly, we don't see any way of doing a native implementation that would satisfy all concerned parties short of implementing a rather complex set of rules, even then it would be problematic. So, the best we would be able to do is provide a post-authentication connection management plugin. The unfortunate part is that wouldn't work with SciTokens because SciTokens totally subverts the authentication path rendering any plug-in that relies on authentication useless. So, if you plan on using SciTokens (and I think you may have to whether you want to or not) anything we do along these lines will not work. So, please understand the conundrum. While we would like to address this, the landscape is changing faster than any solution we can devise. Every time we make forward progress a new external requirement negates it. Sad to say, the only short-term option is to identify the offending user and tell them to stop. Not a scalable solution but from what I understand it's not like everyone is doing this (yet). I wish I could be more positive.

[abh3]

Getting back to this. If it is acceptable to limit per connection not username; then this may be doable. As I noted the "user" is a fluid concept and while we can grasp it if you have a mapping function, we can't do it without one.