5_cython_sol.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cython\n",
    "\n",
    "Dans cette section, nous allons voir comment **améliorer les performances** d'un code Python et comment **interfacer du code C à Python** en utilisant [**Cython**](http://cython.org/).\n",
    "\n",
    "Cette section est inspirée du [tutorial](https://www.youtube.com/watch?v=gMvkiQ-gOW8) donné par Kurt Smith à la conférence SciPy de 2015 et de la documentation Cython.\n",
    "\n",
    "Cython est un **langage de programmation** très proche de Python (\"Cython is Python with C data types\"). Il permet deux choses d'obtenir:  \n",
    "* du **code compilé** C à partir de code Python.  \n",
    "L'idée est de commencer avec du code Python, ajouter des informations sur les types, et obtenir du code compilé plus performant.  \n",
    "* un **wrapper pour des fonctions C/C++**. \n",
    "L'idée est de commencer avec du code compilé (C/C++) et l'exposer à Python avec Cython. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Les étapes sont les suivantes:\n",
    "1. écrire un fichier cython (`.pyx`)  \n",
    "2. exécuter le compiler Cython pour générer du code C  \n",
    "3. exécuter un compiler C pour générer une librairie compilé\n",
    "4. exécuter l'interpréteur Python et demander d'importer le module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comment intéragir avec Cython?\n",
    "\n",
    "On va prendre comme exemple une fonction qui renvoie une combinaison linéaire de deux entiers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%writefile exo_cython/python_add.py\n",
    "def pyadd(a, b):\n",
    "    return 3.1415926 * a + 2.718281828 * b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. compiler avec distutils\n",
    "\n",
    "On commence par d'abord écrire un fichier `.pyx`, en gros du code Python en ajoutant des informations sur le type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%writefile exo_cython/distutils/cython_add.pyx\n",
    "def add(int a, int b):\n",
    "    return 3.1415926 * a + 2.718281828 * b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!cp exo_cython/python_add.py exo_cython/distutils/python_add.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Il faut ensuite créer un fichier `setup.py` qui est une sorte de Makefile Python et qui fait appel à `distutils` et `cython`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%writefile exo_cython/distutils/setup.py\n",
    "from distutils.core import setup\n",
    "from Cython.Build import cythonize\n",
    "\n",
    "setup(name=\"cython_add\",\n",
    "      ext_modules = cythonize(\"cython_add.pyx\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut maintenant utiliser ce fichier pour compiler le fichier Cython avec la commande:  \n",
    "`$ python setup.py buil_ext --inplace`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cd exo_cython/distutils; python setup.py build_ext --inplace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "On obtient alors un fichier `cython_add.so` sous unix (ou `cython_add.pyd` sous Windows) que l'on peut utiliser en l'important dans un interpréteur Python comme n'importe quel autre module Python: `import cython_add`.\n",
    "\n",
    "Comparons les performances avec la version Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%writefile exo_cython/distutils/compare_add.py\n",
    "from timeit import timeit\n",
    "from python_add import pyadd\n",
    "from cython_add import add\n",
    "\n",
    "print('Pure Python version: %fs' % timeit('pyadd(1, 2)', 'from python_add import pyadd'))\n",
    "print('Cython version: %fs' % timeit('add(1, 2)', 'from cython_add import add'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cd exo_cython/distutils; python compare_add.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. pyximport\n",
    "\n",
    "pyximport est un package qui permet d'import un fichier Cython comme si c'était un module Python. Il détecte s'il y a eu des changements dans le fichier Cython, recompile au besoin (ou charge le module du cache). C'est pratique pour les cas simples.\n",
    "\n",
    "On commence par écrire notre fichier `pyx`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%writefile exo_cython/pyximport/cython_add.pyx\n",
    "def add(int a, int b):\n",
    "    return 3.1415926 * a + 2.718281828 * b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pour utiliser directement le fichier, on utilise:\n",
    "<code>\n",
    "import pyximport; pyximport.install()\n",
    "import cython_add\n",
    "print(cython_add.add(1,2))\n",
    "<code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!cp exo_cython/python_add.py exo_cython/pyximport/python_add.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comparons les performances avec la version Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%writefile exo_cython/pyximport/compare_add.py\n",
    "from timeit import timeit\n",
    "from python_add import pyadd\n",
    "import pyximport; pyximport.install()\n",
    "from cython_add import add\n",
    "\n",
    "print('Pure Python version: %fs' % timeit('pyadd(1, 2)', 'from python_add import pyadd'))\n",
    "print('Cython version: %fs' % timeit('add(1, 2)', 'from cython_add import add'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cd exo_cython/pyximport; python compare_add.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. IPython\n",
    "\n",
    "Il existe une commande magique IPython: \n",
    "<code>\n",
    "%load_ext Cython\n",
    "%%cython\n",
    "</code>\n",
    "Comme avec `pyximport`, la compilation est faite pour nous et c'est une méthode recommandée seulement pour des cas simples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load_ext Cython"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%cython\n",
    "def add(int a, int b):\n",
    "    return 3.1415926 * a + 2.718281828459045 * b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pyadd(a, b):\n",
    "    return 3.1415926 * a + 2.718281828459045 * b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%timeit add(2, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%timeit pyadd(2, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Le langage Cython\n",
    "\n",
    "### 1.1 cdef\n",
    "\n",
    "`cdef` est la porte d'entrée de C en Python. Il permet de déclarer des variables locales, des fonctions C et des extension types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%cython\n",
    "def add(int a, int b):\n",
    "    cdef:\n",
    "        float c\n",
    "        float d = 2.718281828459045\n",
    "    c = 3.1415926  \n",
    "    return c * a + d * b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%timeit add(2, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut aussir définir une fonction C:\n",
    "<code>\n",
    "cdef float def add(int a, int b):\n",
    "    cdef:\n",
    "        float c = 3.1415926, d = 2.718281828459045\n",
    "    return c * a + d * b\n",
    "<code>\n",
    "\n",
    "Ou même une extension type (classe bas niveau):  \n",
    "<code>\n",
    "cdef class Particle(object): \n",
    "    cdef float...\n",
    "</code>\n",
    "\n",
    "D'autres declarations cdef:  \n",
    "\n",
    "cdef      | signification  \n",
    "--------- | -------------  \n",
    "cdef int i, j, k | déclare des entiers C   \n",
    "cdef char *s | déclare une string C  \n",
    "cdef float x = 0.0 | déclare et initialise un float C  \n",
    "cdef list names | déclare une liste Python statique  \n",
    "cdef dict dd = {} | déclare et initialise un dictionnaire Python  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Comment déclarer une fonction en Cython?\n",
    "\n",
    "Il y a trois manières de déclarer une fonction en Cython:  \n",
    "* les fonctions `def`: disponibles en Python et Cython. Elles prennent et retournent un objet Python.  \n",
    "* les fonctions `cdef`: fonction C. On ne peut pas y accéder depuis Python.  \n",
    "* les fonctions `cpdef`: prennent le meilleur des deux mondes. Deux fonctions sont générées: une fonction C et un wrapper Python autour de cette fonction. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%writefile exo_cython/inc1.pyx\n",
    "def inc(int num, int offset):\n",
    "    return num + offset\n",
    "\n",
    "def inc_seq(seq, offset):\n",
    "    result = []\n",
    "    for val in seq:\n",
    "        res = inc(val, offset)\n",
    "        result.append(res)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%writefile exo_cython/setup1.py\n",
    "from distutils.core import setup\n",
    "from Cython.Build import cythonize\n",
    "\n",
    "setup(name=\"inc1\",\n",
    "      ext_modules = cythonize(\"inc1.pyx\")\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cd exo_cython; python setup1.py build_ext --inplace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%writefile exo_cython/inc2.pyx\n",
    "cdef int fast_inc(int num, int offset):\n",
    "    return num + offset\n",
    "\n",
    "def fast_inc_seq(seq, offset):\n",
    "    result = []\n",
    "    for val in seq:\n",
    "        res = fast_inc(val, offset)\n",
    "        result.append(res)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%writefile exo_cython/setup2.py\n",
    "from distutils.core import setup\n",
    "from Cython.Build import cythonize\n",
    "\n",
    "setup(name=\"inc2\",\n",
    "      ext_modules = cythonize(\"inc2.pyx\")\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cd exo_cython; python setup2.py build_ext --inplace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cd exo_cython; python -c 'import inc1; print(inc1.inc(1, 3))'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cd exo_cython; python -c 'import inc1; a = range(4); print(inc1.inc_seq(a, 3))'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cd exo_cython; python -c 'import inc2; print(inc2.fast_inc(1, 3))'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!cd exo_cython; python -c 'import inc2; a = range(4); print(inc2.fast_inc_seq(a, 3))'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%writefile exo_cython/inc3.pyx\n",
    "cpdef int fast_inc(int num, int offset):\n",
    "    return num + offset\n",
    "\n",
    "def inc_seq(seq, offset):\n",
    "    result = []\n",
    "    for val in seq:\n",
    "        res = fast_inc(val, offset)\n",
    "        result.append(res)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%writefile exo_cython/setup3.py\n",
    "from distutils.core import setup\n",
    "from Cython.Build import cythonize\n",
    "\n",
    "setup(name=\"inc3\",\n",
    "      ext_modules = cythonize(\"inc3.pyx\")\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cd exo_cython; python setup3.py build_ext --inplace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cd exo_cython; python -c 'import inc3; print(inc3.fast_inc(1, 3))'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cd exo_cython; python -c 'import inc3; a = range(4); print(inc3.inc_seq(a, 3))'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Exercice de typage\n",
    "\n",
    "Nous allons nous entraîner à ajouter des informations sur les type.  \n",
    "La fonction `hamming.py` a deux fonctions simples qui calculent la distance de Hamming entre deux strings: `hamming_sum()` et `hamming_loop()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%writefile exo_cython/typing/hamming.py\n",
    "def hamming_sum(s0, s1):\n",
    "    if len(s0) != len(s1):\n",
    "        raise ValueError()\n",
    "    return sum(c0 != c1 for (c0, c1) in zip(s0, s1))\n",
    "\n",
    "def hamming_loop(s0, s1):\n",
    "    if len(s0) != len(s1):\n",
    "        raise ValueError()\n",
    "    count = 0\n",
    "    for i in range(len(s0)):\n",
    "        count += (s0[i] != s1[i])\n",
    "    return count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ecrire une version Cython `exo_cython/typing/hamming_cython.pyx` en ajoutant des informations sur les types pour agmentant la performance du code. Vous pouvez utiliser `exo_cython/typing/setup_hamming.py` pour compiler votre code et `exo_cython/typing/test_hamming.py` pour comparer les performances."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Comment interfacer des fonctions C en Python?\n",
    "\n",
    "**Remarque:** Cython n'est pas la seule méthode pour interfacer du code C ([plus de détails ici](http://www.scipy-lectures.org/advanced/interfacing_with_c/interfacing_with_c.html)), toutefois Cython est la plus avancée et celle à considérer en premier. \n",
    "\n",
    "On va prendre comme exemple l'interfaçage de la fonction C `sin`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%writefile exo_cython/sin_module.pyx\n",
    "# first include the header file you need\n",
    "cdef extern from \"math.h\":\n",
    "    # describe the interface for the functions used\n",
    "    double sin(double x)\n",
    "    \n",
    "def get_sin(double x):\n",
    "    # sin can now be used from Cython code (but not python)\n",
    "    return sin(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cython ne lit pas le fichier C, on a donc besoin de l'informer des déclarations que l'on va utiliser.  \n",
    "On doit aussi créer une `def` fonction pour que la fonction soit accessible depuis Python. \n",
    "\n",
    "On doit ensuite écrire un `setup`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%writefile exo_cython/setup_sin.py\n",
    "from distutils.core import setup, Extension\n",
    "from Cython.Build import cythonize\n",
    "\n",
    "s = Extension(name=\"sin_module\",\n",
    "                sources=[\"sin_module.pyx\"])\n",
    "setup(ext_modules=cythonize([s]))\n",
    "# setup(name=\"sin_module\", ext_modules=cythonize(\"sin_module.pyx\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cd exo_cython; python setup_sin.py build_ext --inplace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!ls exo_cython/sin*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cd exo_cython; python -c 'import sin_module; print(sin_module.sin)'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!cd exo_cython; python -c 'import sin_module; print(sin_module.get_sin)'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercice: \n",
    "\n",
    "Dans le dossier `exo_cython` se trouve la fonction `levenshtein.c` à interfacer. Pour cela, vous devez compléter `levenshtein_cython.pyx` que vous pourrez compiler avec `setup.py` et dont vous pourrez comparer les résultats avec la solution en utilisant `test_levenstein.py`. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}