Security per Listener

[mrocklin]

In #3288 we allowed the scheduler to listen to multiple Listeners at the same time. This allowed for a single scheduler to accept connections from two different networks at the same time, such as a set of local inproc:// workers, and a remote set of tcp:// workers. This seems to have been a relatively painless change, and works smoothly.

However, when we start rolling security into this, we run into issues because we now accept only one security object, which gets uniformly applied to all of our listeners. This is troublesome in the inproc/tls case, and in theory you could imagine having a tls/tls case that would also be unpleasant.

So what is the right way to handle this? We could accept a list of Security objects (including None in that list for no-security) and apply those objects appropriately to each of the listeners. Even if we did that, it becomes unclear which security object we should use when making new connections. For example if we have workers on two different tls:// networks and we're asked to make a connection to tls://alice, which security object should we use?

In this case should we have two ConnectionPools each with its own security and some sort of function to help dispatch between the two? Or should we have a single ConnectionPool with a set of Security objects that it tries, one after the other?

cc @jcrist @sodre @mariusvniekerk

[mrocklin]

It would probably be simple to have a security per protocol

Scheduler(..., security={"tls://": my_security, "inproc://": None})

But I'm not sure how to handle the multiple tls case.

I may also be thinking about this situation wrong. I think that everyone cc'ed on this issue knows more about security than I do.

[sodre]

For the use-case I have in mind, it would be sufficient/necessary to have one security object per listener, just like there is support for multiple ports.

Here is an example of how we could call it:

python -m distributed.cli.dask_spec  --spec \
  '{"cls": "dask.distributed.Scheduler",
    "opts": {"port": [8785, 8786],
                  "protocol": ["tls", "gssapi"],
                  "security": ["distributed.security.TLSSecurity", 
                               "dask_gssapi.security.GSSAPISecurity}}'  

[mrocklin]

When the scheduler makes an outgoing connection which security object should it use?

[sodre]

I see the issue better now. Some additional here, we can assume that "gssapi" is used for talking to clients only, and the "tls" is used for talking to the workers.

[mrocklin]

Right, so in this case the scheduler need only ever use the TLS Security object when making outgoing connections. I'm not sure how general we want to be here.

…

On Sun, Dec 22, 2019 at 8:03 PM Patrick Sodré ***@***.***> wrote: I see the issue better now. Some additional here, we can assume that "gssapi" is used for talking to clients only, and the "tls" is used for talking to the workers. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3339?email_source=notifications&email_token=AACKZTAQVHITYMEPKZ4IRR3Q2A2BJA5CNFSM4J6PGR52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHQEPEY#issuecomment-568346515>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTCEU37345KYXXRX6GLQ2A2BJANCNFSM4J6PGR5Q> .

[mrocklin]

Some small cleanup here: #3340

[sodre]

The idea of a SecurityContext is not new and maybe referring to the notes on the python-gssapi SecurityContext implementation might help shed some light.

The key thing I would like to point out is the first sentence in that section:

Security contexts represent active sessions between two different entities.

In dask's terms instead of creating a Security object per role, we would create/define a security context for each pair of comms, client-scheduler, scheduler-worker, scheduler-nanny, worker-worker, etc...