Massive memory (100GB) used by dask-scheduler

[pseudotensor]

Using rapids 0.14 with dask 2.17.0 python 3.6, conda, Ubuntu 16.04

I'm running xgboost using dask on GPUs. I do:

1. convert in-memory numpy frame -> dask distributed frame using from_array()
2. chunk the frames sufficiently for every worker (here 3 nodes, 2 GPUs/node each) has data as required so xgboost does not hang
3. Run dataset like 5M rows x 10 columns of airlines data

Notes:

1. Every time 1-3 is done it is in an isolate fork that dies at end of the fit. So whatever instances of client etc. are destroyed. Nothing remains on GPU, nothing remains in a process since it's gone. So I don't believe I need a client.close() call.
2. Even though these forks are gone, within the code I'm always using client as a context manager within a with statement. So again shouldn't need a client.close() call or something like that.

I see my application use reasonable amount of memory based upon that dataset. I see workers using not much memory at all, like 2-3%.

However, the dask-scheduler is using 70% of my 128GB system! I don't understand how/why since the scheduler shouldn't hold data as far as I understand. Perhaps the above sequence of sending dask frame to xgboost is a problem, but it would be odd that task graph is forced to hold data.

Even if a single graph held data, which is already a problem, there's no way 90GB are needed to hold the data involved, so it's like there is repeatedly old data being stored.

I don't have code to share to repro since it's not easy to extract, but I'm hoping still for ideas. I will work on a repro, but any fixes/ideas would be good.

[pseudotensor]

maps.txt

Here is output of sudo cat /proc/32193/maps > maps.txt

0x7f3b29ace000 0x7f52ac000000 0x1782532000 0x0 is about 94GB

[pseudotensor]

Apart from being all odd and a serious issue (can't use dask like this), is there a way to ask scheduler for its memory usage?

[pseudotensor]

I'm guessing I'm misunderstanding a user's responsibility when using dask.

1. I assume that if I did not persist the data, then when a process closes that data is gone

2. I assume data is distributed, not sitting on a single node's scheduler. That would defeat the purpose of distribution.

3. I assume from_array() is taking the numpy array and distributing that on the cluster, not just storing it within the scheduler. That would defeat the purpose of distribution as well.

[pseudotensor]

Here's a code fragment, but it's obvious from my description:

X is 2D numpy array, y 1D numpy array.

with client:
    import xgboost as xgb

    import dask.dataframe as dd
    chunksize=500000
    X = dd.from_array(X, columns=columns, chunksize=chunksize)
    y = dd.from_array(y, chunksize=chunksize)
    model = xgb.dask.DaskXGBClassifier()
    model.fit(X, y)

This is done in a fork that dies every fit on about 5M rows x 10 columns.

I expect the dask-scheduler memory should never be large or hold data. And certainly memory should not keep growing.

[pseudotensor]

FYI before posting I googled and search around and didn't find exactly the same scale of problem. Mostly stack overflow stuff was misuses or misinterpretations AFAIK.

I'm also not clear if I should post to dask or dask/distributed.

[rjzamora]

This question may be a better fit for dask/distributed, but I can try to offer some thoughts...

I assume that if I did not persist the data, then when a process closes that data is gone

I assume that it is definitely better to explicitly close your client (and maybe even cancel futures). Although your client process is dying, I don't think the relevent data will be immediately deleted from the cluster.

I assume data is distributed, not sitting on a single node's scheduler. That would defeat the purpose of distribution.

The data will be distributed for processing. However, since your data is being generated with numpy, the initial data has to originate on a single process.

I assume from_array() is taking the numpy array and distributing that on the cluster, not just storing it within the scheduler. That would defeat the purpose of distribution as well.

Same answer as above - To perform a truely "data-distributed" workflow, you will need a parallel read, or to use from_delayed to allow the data to be generated in parallel. When you are using a single numpy array as the source data, it is indeed likely that the entirity of your data will be stored in the graph.

[pseudotensor]

This question may be a better fit for dask/distributed, but I can try to offer some thoughts...

I assume that if I did not persist the data, then when a process closes that data is gone

I assume that it is definitely better to explicitly close your client (and maybe even cancel futures). Although your client process is dying, I don't think the relevent data will be immediately deleted from the cluster.

I assume data is distributed, not sitting on a single node's scheduler. That would defeat the purpose of distribution.

The data will be distributed for processing. However, since your data is being generated with numpy, the initial data has to originate on a single process.

I assume from_array() is taking the numpy array and distributing that on the cluster, not just storing it within the scheduler. That would defeat the purpose of distribution as well.

Same answer as above - To perform a truely "data-distributed" workflow, you will need a parallel read, or to use from_delayed to allow the data to be generated in parallel. When you are using a single numpy array as the source data, it is indeed likely that the entirity of your data will be stored in the graph.

I understand that the initial numpy data is on the single server, but this is not on the dask-scheduler. Also, it's relatively small amount of data, only 200MB of numpy data.

I understand to get a fully distributed experience I need to start from get-go with dask, but I'm far from that if even simple uses are showing massive use by the scheduler.

I'm confused why the data needs to be stored in the graph. Do I need to force distribution before passing the data to xgboost? If I run .compute I just manifest the data locally, so I assume I need to run .scatter or some equivalent? .persist()? Not sure what will force data to be distributed and the graph completed so scheduler is done with data.

Is there a command that will take numpy or pandas data and bypass the single-node scheduler and immediately distribute the data?

Also, why does the data keep accumulating? Every fit adds 200MB to scheduler. Why are (I guess) the old graphs still there after xgboost is done and all forks related to the client are gone?

[rjzamora]

Right - I understand that your data is small, and I agree that it shouldn't be taking up a lot of space in the graph. When I say "the entirity of your data will be stored in the graph", I am just saying that the very first task in the graph will include the entire array as input. If you call X.persist() after the from_array call, your data will become distributed (you are correct that compute is not what you want).

Also, why does the data keep accumulating? Every fit adds 200MB to scheduler. Why are (I guess) the old graphs still there after xgboost is done and all forks related to the client are gone?

This is where someone with better distributed knowledge my have more intuition than me (cc @quasiben ). Perhaps you can try to be explicit about cleaning up collections before you let the client die (e.g. client.cancel(X))?

[pseudotensor]

Will use of dd.from_array(X).persist() bypass the scheduler or must it go through it?

It seems a bit odd of a design if all in-memory dask distributed data operations have to go from the client process -> scheduler -> workers. Then the scheduler always ends up holding all the data meant for the cluster of workers for from_pandas, from_array, etc. How can one just get the data in memory on the cluster?

I understand read_csv or other url/file operations can lazily slice data and use it to distribute, and there the lazy thing stored is just the file name or link. Of course, read_csv does not work unless disk is shared. And connectors that read data from cloud require special other infrastructure.

The simplest mode should be in-memory and be optimal by itself IMHO. As-is, it's at least an excessive copy of the full data frame on the scheduler. Factors of 2 are important! :) In reality, the numpy frame is converted into a dask frame that I think has more overhead like pandas, but maybe not. It seems scheduler should just remember the client has memory and use it when performing any task in the task graph, not the actual data.

At worst, for whatever reason, the frames never go away. I'll try the .persist and client.cancel. Thanks.

[rjzamora]

It seems a bit odd of a design if all in-memory dask distributed data operations have to go from the client process -> scheduler -> workers. Then the scheduler always ends up holding all the data meant for the cluster of workers for from_pandas, from_array, etc. How can one just get the data in memory on the cluster?

I see - If you are starting with a literal numpy array or pandas DataFrame, you are correct that the data will flow client process -> scheduler -> workers. There is really no way around this, since the starting point is an in-memory data structure. If you don't want to read from disk, you may be looking for something like dask.delayed for the generation of your data (where you can supply a user-defined function to generate in in-memory numpy array for each initial chunk).

[pseudotensor]

But why can't the scheduler and task just keep track of the client having the memory? Of course, one can't del the object provided as that would mess up things, but one can't rm the file for read_csv anyways. So it's the same issue, no new issue.

Actually del isn't a problem. As long as the client/dask stuff keeps track of memory and holds a reference, del won't matter. So it's even easier to manage than file issues.

[pseudotensor]

Looking at https://distributed.dask.org/en/latest/memory.html I see no reason why there is accumulation. The fork goes away, the future held should be deleted. I would think I don't have to cancel the future myself.

However, perhaps I'm uncleanly shutting down the fork. Maybe then the future isn't entirely removed? That is, while in-memory stuff is clearly gone after a fork is gone, any distributed off-memory things are not.

[pseudotensor]

I added more details and repros at: dmlc/xgboost#6388 (comment)

There are some things mentioned in the issue that seem to be xgboost's fault, like excessive memory usage on CPU before dask is even used.

However, the fact that the scheduler and worker both keep growing in memory use every completed client python script, despite having client.cancel(X) and client.cancel(y) even (and a clean exit of the script) seems to be a problem with dask.

In the context I originally posted about, only the scheduler seems to grow in memory use, but excessively so. cancels didn't help there either.

[pseudotensor]

Eventually the worker gets killed and restarts due to the OOM accumulated.

distributed.nanny - INFO - Worker process 29371 was killed by signal 9
distributed.nanny - WARNING - Restarting worker

This keeps happening over and over.

Everytime this happens, the new worker starts again with less memory but again accumulating after each iteration in the loop. Again it gets killed, etc.

Meanwhile the scheduler continues itself to grow in memory use.