Massive memory usage by parallel RandomForestClassifier

[jni]

I think this will be hard to fix without swapping out joblib (or maybe even the GIL ;), but basically the amount of memory used by RandomForestClassifier is exorbitant for n_jobs > 1. In my case, I have a dataset of about 1GB (300,000 samples by 415 features by 64-bit float), but doing fit() on a RandomForestClassifier having n_jobs=16 results in 45GB of memory being used.

Does anyone have any ideas or is this hopeless without moving everything to C?

[glouppe]

The problem is that two copies of X (X and X_argsorted) are made for each job.

You cant circumvent that by putting X into shared memory. I did that in one my branches:
https://github.com/glouppe/scikit-learn/blob/cytomine/sklearn/ensemble/forest.py#L274

I used that module: https://bitbucket.org/cleemesser/numpy-sharedmem/issue/2/sharedmemory_sysvso-not-added-correctly-to

It was not put into master because of this additional depedency though.

[ogrisel]

To use shared memory you can memory map your input set with joblib:

from sklearn.externals import joblib

filename = '/tmp/dataset.joblib'
joblib.dump(np.asfortranarray(X), filename)
X = joblib.load(filename, mmap_mode='c')

IIRC the random forest model need a fortran layout data to work efficiently hence the call to np.asfortranarray before serialization on the disk.

[ogrisel]

BTW @glouppe is the above strategy works as expected it would be great to make the RandomForest/ExtraTrees* classes able to do it automatically using a mmap_folder=None parameter. If provided, memmapping is used of n_jobs > 1.

[ogrisel]

@jni any news on this? Have you tried any of the afore-mentionned solutions? If that work for you we should devise a way to make it simpler to implement or at least better documented.

[jni]

Haven't tried it, busy weekend — I'll do it today! Thanks!

[jni]

Ok, two failures to report.

First, I tried to combine @glouppe's code with @ogrisel's joblib modification. This crashed and burned and anyway didn't seem to much affect memory usage: it was up to 20GB before it crashed. I've made a gist with the diff with scikit-learn 0.11.X and the error for njobs=2.

I then tried @glouppe's cytomine branch directly, after installing sharedmem, but this also failed for some unknown reason.

... Any ideas?

[amueller]

@jni have you tried playing with "min_density"? That can really affect memory usage (also cpu usage though in a non-linear way).

[jni]

@amueller, I just tried min_density=0.001 and it still goes to 45GB on my dataset. By my understanding, the original data gets replicated to each process before any parameters like min_density take effect, so those parameters will only minimally affect the total memory footprint, which is dominated by njobs * (dataset_size) * c, with c in [2, 3]. (2 according to the above comments, but closer to 3 empirically. ;)

[jni]

I stand corrected: usage is >50GB with njobs=8 and min_density=1.0. But it looks like I can't make it any lower.

[amueller]

hm ok. was just an idea. don't know where the additional copy comes from (2 instead of 3). You did take care of the memory layout, right?

[amueller]

I meant if you use fortran or c ordering. IIRC the forests want fortran ordering, so if you provide c-ordered arrays, they'll make a copy.

[jni]

I thought that's what you might have meant, so I ran rf.fit(np.asfortranarray(...)) instead, but same result (about 3x). =\

[glouppe]

For my branch to work, you need

1. to use bootstrap=False in your forests

2. Increase SHMMAX if you intend to share a large block of memory.

• On OSX:
  sudo sysctl -w kern.sysv.shmmax=YOUR_VALUE
  sudo sysctl -w kern.sysv.shmall=YOUR_VALUE
• On Linux
  sudo sysctl -w kernel.shmmax=YOUR_VALUE
  sudo sysctl -w kernel.shmall=YOUR_VALUE
  sudo sysctl -p /etc/sysctl.conf

Hope this helps!

[jni]

Thanks, @glouppe! This'll help me but I'm disappointed it's of limited use if it won't make it into the scikit proper... It seems to me that if I can get it to work, this can be an optional dependency... I often use the following pattern:

try:
    import sharedmem as shm
    shm_available = True
except ImportError:
    logging.warning('sharedmem library is not available')
    shm_available = False

Otherwise, I would still be interested in getting joblib persistence to work...

Secondly, this may explain the close-to-3x memory usage when my data is copied, since it's not float32. It's probably a good idea to coerce the data within BaseForest before it is copied by multiprocessing, only to be coerced inside each tree. This'll bring the memory usage way down, and speed things up too I'm sure.

I'll run these experiments this afternoon and report back. Thanks everyone!

[ogrisel]

@jni could you please report the error message you get with memory mapped file solution, or better tell me if this is the same as the following? https://gist.github.com/3084146

If so, I will try to give it a deeper look by trying to reproduce it in a pure joblib context, outside of scikit-learn. Maybe @GaelVaroquaux has an idea on the cause of the problem.