83_parallelism.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Parallelism, Resources, Configuration](https://scikit-learn.org/stable/computing/parallelism.html)\n",
    "\n",
    "- Some Scikit-Learn estimators can run jobs on multiple CPUs in parallel thanks to [joblib](https://joblib.readthedocs.io/en/latest/) and the `n_jobs` parameter, or via `openMP`.\n",
    "\n",
    "- Some internal NumPy-based methods can be parallelized if NumPy is installed with numerical analysis libraries such as MKL, OpenBLAS or BLAS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Joblib-based parallelism\n",
    "\n",
    "- When the underlying code uses `joblib`, the number of workers (threads or processes) running in parallel is controlled via `n_jobs'.\n",
    "\n",
    "- Joblib supports multiprocessing & multithreading - the choice depends on the backend choice.\n",
    "\n",
    "    - Scikit-Learn usually relies on `loky` (Joblib's default), which is for multiprocessing. Joblib creates a [memory map](https://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html) that is shared by all processes - when data size is >1MB.\n",
    "    \n",
    "    - In some case, Scikit-Learn will tell Joblib that multithreading is preferable.\n",
    "    \n",
    "\n",
    "- You can control the backend:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done\n"
     ]
    }
   ],
   "source": [
    "from joblib import parallel_backend\n",
    "with parallel_backend('threading', n_jobs=2):\n",
    "    print('done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OpenMP-based parallelism\n",
    "\n",
    "- OpenMP parallelizes code written in Cython or C. It relies exclusively on *multithreading* and will try to use (by default) as many threads as possible.\n",
    "\n",
    "- You can control thread count via an environmental variable:\n",
    "\n",
    "`$OMP_NUM_THREADS=4 python my_script.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NumPy Accelerator-based parallelism\n",
    "\n",
    "- NumPy & SciPy rely on multithreaded linear algebra libraries such as MKL, OpenBLAS or BLIS. The number of threads used by the libraries can be set via `MKL_NUM_THREADS`, `OPENBLAS_NUM_THREADS` or `BLIS_NUM_THREADS` environmental variables.\n",
    "\n",
    "- NumPy & SciPy distributed on pypi.org and conda-forge are linked to OpenBLAS.\n",
    "\n",
    "- conda packages on Anaconda's \"defaults\" channel are linked by default to MKL."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Oversubscription (too many threads)\n",
    "\n",
    "- Defined as running more threads than available CPUs.\n",
    "- Consider a case on an 8-CPU machine with `GridSearchCV` (on Joblib) running with `n_jobs=8`, plus a `HistGradientBoostingClassifier` (on OpenMP). Each instance of the Classifier will spawn 8 threads (one for each CPU). That's 8*8=64 threads, which will cause too much scheduling overhead.\n",
    "\n",
    "- Starting with `joblib>=0.14` with the `loky` backend, joblib limits child processes' thread counts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Config Switches using [sklearn.set_config](https://scikit-learn.org/stable/modules/generated/sklearn.set_config.html#sklearn.set_config)\n",
    "\n",
    "- (for Python runtime):\n",
    "- `assume_finite`: skip-validation flag (for faster computations)\n",
    "- `working_memory`: size of temp arrays\n",
    "\n",
    "- (environmental variables, before importing sklearn):\n",
    "- `SKLEARN_SITE_JOBLIB` - if nonzero, sklearn uses site joblib instead of a vendored verson.\n",
    "- `SKLEARN_ASSUME_FINITE` - default for `assume_finite`\n",
    "- `SKLEARN_WORKING_MEMORY` - default for `working_memory`\n",
    "- `SKLEARN_SEED` - sets global random generator seed\n",
    "- `SKLEARN_SKIP_NETWORK_TESTS`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'assume_finite': False, 'working_memory': 1024, 'print_changed_only': True, 'display': 'text'}\n"
     ]
    }
   ],
   "source": [
    "import sklearn\n",
    "print(sklearn.get_config())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:working] *",
   "language": "python",
   "name": "conda-env-working-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}