cmsd: Have ability to quarantine blacklisted hosts to a different cmsd cluster

[bbockelm]

This came up during the CMS Offline & Computing Week -

We'd like to have the ability to blacklist certain sites from our central redirectors. Instead of disabling them within the current cmsd cluster, we'd like to be able to redirect the incoming cmsd into a separate testing cluster.

[abh3]

I think we agreed at the CMS meeting that this would get done. I have been looking on how to do it and it's not exactly straight forward because redirectors may be replicated and so you have to handle the general m-to-n redirection case, sigh.

Also, there was some question as to whether you also want a white-list capability. If so, there are a couple of ways to do it. However, perhaps the easiest way (though not exactly mentally obvious) is to extend the blacklist file format to allow for white listing hosts as well. Feedback would be appreciated.

[abh3]

I'd like to know if it's acceptable that unless a cmsd supports being redirected because of being blacklisted it is simply blacklisted (i.e. no redirected). The problem here is that current cmsd's have no conception of being redirected outside of their tree (i.e. cluster). While they would accept being redirected that way the logic would still assume they were in fact in the original tree but just placed elsewhere. Invariably, this would produce odd side effects; none critical but very confusing that would just burn up people's time. The solution is to not perform such a redirect unless the cmsd is aware that it can be redirected that way (i.e. it's simply blacklisted without a redirect). This, however, means that people would have to upgrade if they wanted to have that feature; It's likely they would because blacklisting with a redirect keeps them functional and they really are blacklisted anyway.

[bbockelm]

Hi Andy,

I sent your questions to the internal CMS list. Will hopefully have an answer tomorrow.

Brian

[abh3]

OK, here is what I am testing (as I didn't get any feedback yet).

cms.blacklist [check ] [invert] []

The only new thing here is "invert" that makes the blacklist work like a whitelist.

The file rules are extended, and follows

[redirect <target[+]:]
Either blacklists or whitelists the matching host unless redirect is specified. In which case the match will simply redirect the client to expanded list of hosts. I will likely extend this to allow multiple targets on the line. Currently, the only way you can have multiple targets is by DNS alias expansion via the "+" (same as in the config file). The redirect happens whether or not the blacklist is inverted.

A redirect to a server applies to all ,manager connections regardless of which manager sent it (the first one wins). This is to prevent inconsistent state when two replicated managers have conflicting redirects.

The above poses a large problem when you want a server to join two different clusters or federations. There is no good automatic solution to this. So, the idea is to expand the all.manager directive, as follows:

all.manager [ meta | peer | proxy ] [ all | any ] host[+]{:port | port} [site ] [if ...]

The addition of the site name groups together all connections to managers tagged with that site. This allows redirects to work independently at the site level. So far as I can tell no one has done multiple site federations which is good because it won't work past 64 servers due to a limitation on how redirects are handled in the current code (they work just fine for single site federations). You need not specify "site" if you have a single site federation.

Finally, the beta code still only applies blacklist redirects to servers that support it properly (i.e. old servers will simply be blacklisted but not redirected).

Finally, if there is no blacklist file then no black or white list applies and everyone can join just like today (subject to the allow directive). I am also pushing in some new code to make sure that blacklisting only gets turned on if you specified the cms.blacklist directive.

All comments welcomed.

[abh3]

I have not gotten any feedback on this so it's a bit difficult to continue moving forward. What I do have appears to work though I have some more testing to do. Feedback anyone or don't you really want this anymore?

[bbockelm]

Hi,

Had the first AAA meeting of the year and Ken kindly reminded me I let this one drop.

The consensus is the proposed change (in terms of the semantics of "only redirect for newer cmsd; otherwise blacklists" and the config file format) would work for us. Full steam ahead, I suppose!

Brian

[abh3]

Thanks for the feedback. You will see a commit implementing this in a week or so.

[abh3]

The feature has been added to git head and will appear in 4.2.