Sklearn/XGBoost n threads is hardcoded

[svandenhoek]

Describe the bug

Number of parallel threads used by XGBoost is hardcoded instead of checking n threads available on system:

• https://github.com/molgenis/capice/blob/master/src/molgenis/capice/main_train.py#L146 (refers to https://github.com/molgenis/capice/blob/master/src/molgenis/capice/main_train.py#L39)
• https://github.com/molgenis/capice/blob/master/src/molgenis/capice/main_train.py#L160

Expected behavior

Either:
A) Value is based on available threads on system. Should be fixable with n_jobs=-1 (except perhaps on cluster, see below).
B) Set through a command line argument (with value 1 or option A as default).

Additional context

Pre-1.3.0 versions of XGBoost have a bug and should use something like this instead.

XGBoost uses cgroup to determine if CPU limit should be set lower than actual system resources (when using n_jobs=-1). Not sure if this works nicely together with Slurm. Something like this might be needed for that instead. But either way would require some testing on the cluster no matter what solution is chosen.

[dennishendriksen]

I prefer B) with 1 as default