Unexpected kwargs error

[cicdw]

It appears that #3117 introduced a check for whether certain kwargs were used when initializing a Client.

Previously, this sort of pattern was allowed:

from distributed import Client

c = Client(processes=True, silence_logs=True)

with the expectation that these kwargs were passed down to the LocalCluster object which is created when a scheduler address is not provided. Now this raises:

ValueError: Unexpected keyword arguments: ['processes', 'silence_logs']

• is this an unsupported use of the API with the release of 2.6.0?
• or should Dask only be raising these kwarg errors whenever a scheduler_address is actually passed?

[cicdw]

Update: I'm using a utility function to create Clients, in which I always pass those two keyword arguments even when I'm connecting to a known scheduler address, so I suppose this is on me to change how I do this.

[mrocklin]

Correct. Previously we were silently ignoring those keywords. Now we raise with an informative message. Should we change this or do you think that this should work for you.

…

On Wed, Oct 16, 2019, 9:23 AM Chris White ***@***.***> wrote: Update: I'm using a utility function to create Clients, in which I always pass those two keyword arguments even when I'm connecting to a known scheduler address, so I suppose this is on me to change how I do this. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3151?email_source=notifications&email_token=AACKZTAKWMBGQBBD6PAILIDQO4PWNA5CNFSM4JBMHPCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBMVURI#issuecomment-542726725>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTEJFA537B5JE6YV3M3QO4PWNANCNFSM4JBMHPCA> .

[cicdw]

That works for me! I had forgotten that these clients were created with this utility which force-fed the kwargs when I originally opened the issue - I've already fixed it on my side 👍

[SOUMYASHUKLA]

@mrocklin I am trying to get rid of contineous rolling of worker warnings on the console.
I tried Unexpected keyword arguments: client(..., silence_logs=30) but it does not work. Also checked in the distributed.yaml file. I did not find any particular setting for this.
How can I do this with the new version of dask?

[fjetter]

@SOUMYASHUKLA this exception is only raised if you are providing an address. Either you provide an address to an existing cluster or we'll use the extra kwargs to initialize a new LocalCluster. Both is not possible. If you are still experiencing issues, may I ask you to open a new issue with your specific exception and traceback?

[SOUMYASHUKLA]

@fjetter Thank you for your response.

I have setup the dask cluster in docker environment to run my computations inside docker container. As per reference, I used the following syntax,

from dask.distributed import Client dask_client = Client('dask_scheduler:8786', silence_logs=True)

There are a few things I am having trouble with as follows:

1. Stop worker logs from printing on console every second. Maybe a better way would be to print the warning message once and append a counter to 'number of occurrences'
   ex: dask_worker_2 | distributed.worker - WARNING... msg (50)

2. I want to tweak worker configurations to allow out of memory spill and I tried below 2 approaches, but looks like both of them do not work:
   2.1 updated parameters in distributed.yaml file (inside docker). These updates are lost with everydocker-compose down command.
   2.2 I tried the following from python code,
   dask.config.set({'worker.memory.target': False}) dask.config.set({'worker.memory.pause': False}) dask.config.set({'worker.memory.spill': 0.70}) dask.config.set({'worker.memory.terminate': False})

I can open a new issue for docker issues if you suggest. Thanks again!

[bryanwweber]

@SOUMYASHUKLA I suspect this would be a better topic for the Discourse group, here: https://dask.discourse.group/ since it is about using Dask/distributed, and not so much about a bug report. Please let me know if you have any questions 😄

[SOUMYASHUKLA]

@bryanwweber Thank you. This issue was reported already at discourse discussion

[bryanwweber]

@SOUMYASHUKLA Great, then let's avoid duplicating discussion here.

[kchawla-pi]

@SOUMYASHUKLA this exception is only raised if you are providing an address. Either you provide an address to an existing cluster or we'll use the extra kwargs to initialize a new LocalCluster. Both is not possible. If you are still experiencing issues, may I ask you to open a new issue with your specific exception and traceback?

This should be added in the documentation. If it has been, in a more prominent place; I perused the documentation and never saw this.