Facing exception while processing 1 Million data..

[aswinjose89]

Refer the exception below

A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

The exit codes of the workers are {SIGKILL(-9), SIGKILL(-9)}

@atif-hassan

[atif-hassan]

If n_jobs > 1 then PyImpetus internally makes that many copies of the data. You can try setting n_jobs to 1 in order to stop PyImpetus from making multiple copies of the data. This will make the algorithm slower but you shouldn't run into above said error.

[aswinjose89]

Thanks for the quick response..

[atif-hassan]

Please note that "n_jobs>1" case also includes "n_jobs=-1"

[aswinjose89]

Noted..I have tried n_jobs=70(cores) and n_jobs=-1(To consume all the cores).Both cases were throwing similar exception..

[atif-hassan]

Yes, the reason being that with any value other than 1 for n_jobs will result in internal copies of the data which will definitely lead to memory error for such a large dataset.

[aswinjose89]

Thanks for the inputs