Error when using client for DaskXGBClassifier

[mariaangelesjs]

Hi,
I've ran into a problem when I was trying to use XGBClassifier for dask within a Docker image see here also.

Summary

Using:
client = Client(n_workers=4, threads_per_worker=1, processes=False, memory_limit='16GB', scheduler_port=0, silence_logs=False, diagnostics_port=0)
And after fitting with:
model.fit(X_t, y_t, eval_set=(X_e, y_e))
I get:
model.fit( File "/usr/local/lib/python3.8/dist-packages/xgboost/dask.py", line 1817, in fit return self._client_sync(self._fit_async, **args) File "/usr/local/lib/python3.8/dist-packages/xgboost/dask.py", line 1617, in _client_sync with distributed.worker_client() as client: File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/usr/local/lib/python3.8/dist-packages/distributed/worker_client.py", line 55, in worker_client duration = time() - thread_state.start_time AttributeError: '_thread._local' object has no attribute 'start_time' distributed.worker - INFO - Stopping worker at inproc://172.17.0.2/1/5 distributed.worker - INFO - Stopping worker at inproc://172.17.0.2/1/4 distributed.worker - INFO - Stopping worker at inproc://172.17.0.2/1/6 distributed.worker - INFO - Stopping worker at inproc://172.17.0.2/1/7 distributed.scheduler - INFO - Remove worker <WorkerState 'inproc://172.17.0.2/1/5', name: 0, status: closing, memory: 0, processing: 0> distributed.core - INFO - Removing comms to inproc://172.17.0.2/1/5 distributed.scheduler - INFO - Remove worker <WorkerState 'inproc://172.17.0.2/1/4', name: 1, status: closing, memory: 0, processing: 0> distributed.core - INFO - Removing comms to inproc://172.17.0.2/1/4 distributed.scheduler - INFO - Remove worker <WorkerState 'inproc://172.17.0.2/1/6', name: 2, status: closing, memory: 0, processing: 0> distributed.core - INFO - Removing comms to inproc://172.17.0.2/1/6 distributed.scheduler - INFO - Remove worker <WorkerState 'inproc://172.17.0.2/1/7', name: 3, status: closing, memory: 0, processing: 0> distributed.core - INFO - Removing comms to inproc://172.17.0.2/1/7 distributed.scheduler - INFO - Lost all workers distributed.scheduler - INFO - Scheduler closing... distributed.scheduler - INFO - Scheduler closing all comms

Environment: docker image - ubuntu-latest

• Dask version: 2021.12.0
• Python version: 3.9.9
• Operating System: MAC OS Big Sur 11.6.1
• Install method (conda, pip, source): pip

[quasiben]

Can you build a minimal reproducer for the error ?

[mariaangelesjs]

The use of dask with XGBClassifier was not optimal and using the same conditions with a regular pandas dataframe gave no errors. I can discuss with my team how minimal we can do the example, but we are working with 4,4 million records and data that is GDPR sensitive. So, we tend to avoid descriptive code unless strictly necessary.
Will update soon and sorry for the delay.

[paulbauriegel]

@quasiben I have the same error. Here is a minimal code example:

from dask.distributed import Client

# On a Azure Standard_D13_v2
client = Client(processes=False, threads_per_worker=2,
                n_workers=6, memory_limit='50GB')

import dask.array as da
from dask_ml.preprocessing import StandardScaler
from xgboost.dask import DaskXGBClassifier

X = da.random.random((360448, 20000), chunks={0: 2816, 1: 20000})
y = da.random.random((360448), chunks=2816)

scaler = StandardScaler(with_mean=False).fit(X)
X_scaled = scaler.transform(X)

X_scaled_rechunk = X_scaled.rechunk({0: 2816, 1: 20000})
y_train_rechunk = y.rechunk(2816)

model = DaskXGBClassifier().fit(X_scaled_rechunk, y_train_rechunk)

Here is the conda environment: conda list | grep dask

dask                      2022.7.1                 pypi_0    pypi
dask-core                 2022.2.1           pyhd3eb1b0_0  
dask-glm                  0.2.0                    pypi_0    pypi
dask-ml                   2022.5.27                pypi_0    pypi
dask-sql                  2022.6.0                 pypi_0    pypi
dask-xgboost              0.2.0                    pypi_0    pypi

[paulbauriegel]

After I changed the Client to processes=True the error disappeared.

client = Client(processes=True, threads_per_worker=2, # 3
                n_workers=8, memory_limit='45GB')

[mariaangelesjs]

Will also test that then. Thank you very much! ons. 17. aug. 2022 kl. 16:06 skrev Paul Bauriegel ***@***.***

…

: After I changed the Client to processes=True the error disappeared. client = Client(processes=True, threads_per_worker=2, # 3 n_workers=8, memory_limit='45GB') — Reply to this email directly, view it on GitHub <#5645 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHG445LY3MZNLMAYA2XET23VZTWX3ANCNFSM5LSJUTZA> . You are receiving this because you authored the thread.Message ID: ***@***.***>