languages/julia/julia-tutorial/source-notebooks/Julia-Calling-Python-C-Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f316e657",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Calling Python and C from Julia\n",
    "\n",
    "1. Python -- the PyCall package\n",
    "2. C (and Fortran) Libraries -- the native Julia functions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27ee9119",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Calling Python\n",
    "\n",
    "We can use the `PyCall` package to call Python code from Julia."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f2f35d5",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "using PyCall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6e32445",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# The following makes it so that print statements in python will appear in this notebook\n",
    "# This is not necessary when using PyCall in a terminal based Julia instance\n",
    "pyimport(\"sys\").\"stdout\" = PyTextIO(stdout)\n",
    "pyimport(\"sys\").\"stderr\" = PyTextIO(stderr);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4e7e190",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can execute arbitrary Python code with the special Julia strings `py\"...\"` and `py\"\"\"...\"\"\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a60ddef1",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "py\"\"\"\n",
    "import math\n",
    "class Point:\n",
    "    def __init__(self, x,y):\n",
    "        self.x = x\n",
    "        self.y = y\n",
    "    def distance(self, p):\n",
    "        return math.sqrt((self.x - p.x)**2 + (self.y - p.y)**2)\n",
    "\"\"\"\n",
    "\n",
    "p = py\"Point(1.0, 2.0)\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9510370e",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Calling Python\n",
    "\n",
    "We can even use Julia's string interpolation to give values to the Python code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a7895d0",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "x = rand()\n",
    "q = py\"Point($(x), $rand())\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05c02918",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Attributes are directly accessible through the standard dot syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3530772",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "@show p.x\n",
    "@show p.distance(q);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95db18bc",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Calling Python\n",
    "\n",
    "But say we have a module in Python that we want to call from Julia.  We can do that too (otherwise this wouldn't be much use would it?). The `pyimport` function returns an object that gives us access to that modules functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "463cb4ba",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "np = pyimport(\"numpy\")\n",
    "A = rand(3,3)\n",
    "b = rand(3)\n",
    "x = np.linalg.solve(A, b)\n",
    "@show maximum(abs.(A * x - b));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ab5e377",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Calling Python\n",
    "\n",
    "In the previous slide `A` and `b` are created by Julia while `x` is created by Python but we are using them interchangeably. We can do this because PyCall handles most type conversions automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df78e3ba",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "for x in [5.0, 2, [\"a\", \"b\"], Dict(\"a\"=>rand(), \"b\"=>rand()), A]\n",
    "    @show typeof(x)\n",
    "    py\"\"\"print(type($x))\"\"\"\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "198a5fc6",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Note that the matrix is converted to a numpy array if numpy is installed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44fb52e3",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Calling Python\n",
    "\n",
    "The same is true going from Python to Julia."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0c06832",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "py\"\"\"\n",
    "objs = [{'a':1,'b':2}, [1, 'a', 3.0], 2.0+3j]\n",
    "for k in range(len(objs)):\n",
    "    $println($typeof(objs[k]))\n",
    "    print(type(objs[k]))\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e96a8087",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Calling Python\n",
    "\n",
    "We do need to be a little careful with some of Julia's less common types especially if we give it to python and bring it back:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41acb527",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "a = Int32(5)\n",
    "@show typeof(a)\n",
    "@show typeof(py\"$a\");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d118cceb",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In these cases, we may want to handle the conversion ourselves. One option is getting the raw `PyObject` back by using the `py\"...\"o` syntax and then calling an appropriate `convert` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7ccf797",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "@show typeof(a)\n",
    "@show typeof(py\"$a\"o)\n",
    "@show typeof(convert(Int32, py\"$a\"o));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cee85801",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Calling Python\n",
    "\n",
    "Another way of handling (or preventing) type conversions is to use the `pycall` function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "425d0d01",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "pycall(np.random.normal, PyObject, size=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a2bf548",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Here we specified to leave the object as a raw PyObject (i.e. no type conversion at all)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06b63ef7",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can also give it a Julia type to convert to"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2dc77f1d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "pycall(np.random.normal, Vector{ComplexF32}, size=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccf014e3",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Here we forced the type conversion to complex numbers with 32-bit precision for the real and imaginary parts."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "167150f3",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Calling Python\n",
    "\n",
    "But what if we need to call a Python function that requires a callback? Not a problem. PyCall will automatically convert Julia functions to Python callable objects!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8e80362",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "si = pyimport(\"scipy.integrate\")\n",
    "tk = 0.0:1e-2:10.0\n",
    "function my_ode(t::Float64, y::Vector{Float64})::Vector{Float64}\n",
    "    dy = zeros(length(y))\n",
    "    dy[1] = 5.0*y[1] - 5.0*y[1]*y[2]\n",
    "    dy[2] = y[1]*y[2] - y[2]\n",
    "    return dy\n",
    "end\n",
    "soln = si.solve_ivp(my_ode, (0.0, 10.0), [5.0, 1.0], t_eval=tk);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de969766",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "using Plots\n",
    "plot(soln[\"t\"], soln[\"y\"]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c84e5075",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "plot(soln[\"y\"][1,:], soln[\"y\"][2,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1063bd78",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Calling Python\n",
    "\n",
    "For more details, see the [PyCall github repo](https://github.com/JuliaPy/PyCall.jl.git)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4c84e39",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Calling C/Fortran Library\n",
    "\n",
    "Here we will discuss how to call a C library function from within Julia."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23e96058",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Calling a Fortran library function is the same except that Fortran compilers \"mangle\" the function names. This means that they are not precisely the same names as in the source code and you need to know what compiler was used to compile the Fortran library so you can determine the mangling scheme."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79ff3293",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Note that the library we are calling must be compiled as a shared library."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5df7eea1",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Calling C Library\n",
    "\n",
    "As an example we will use the \"silly\" library that was written just for this.\n",
    "\n",
    "Here are the functions available in the silly library:\n",
    "```C\n",
    "void fill_zeros(double *to_fill, int size);\n",
    "void fill_value(double *to_fill, int size, double value);\n",
    "void fill_cb(double *to_fill, int size, double (*func)(int));\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0ada1c4",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Calling C Library\n",
    "\n",
    "To call one of these functions, we will use the builtin Julia function `ccall`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0487fd49",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "N = 4\n",
    "my_vector = Vector{Float64}(undef, N)\n",
    "@show my_vector\n",
    "ccall((:fill_zeros,\"fake-lib/libsilly\"), # function and library\n",
    "    Cvoid, # return type\n",
    "    (Ref{Float64}, Cint), # argument types\n",
    "    my_vector, N # arguments\n",
    ")\n",
    "@show my_vector\n",
    "ccall((:fill_value,\"fake-lib/libsilly\"),\n",
    "    Cvoid,\n",
    "    (Ref{Float64}, Cint, Cdouble),\n",
    "    my_vector, N, pi\n",
    ")\n",
    "@show my_vector;"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "026b4131",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Calling C Library\n",
    "\n",
    "What if we want to use a function that requires a callback (so one of its arguments is a function pointer)? We can create a pointer to a Julia function with the `@cfunction` macro."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e72bd09",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "function my_filler(index::Int)::Float64\n",
    "    return index / 10.0\n",
    "end\n",
    "cfunc = @cfunction(my_filler, Float64, (Int,))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ea69208",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Now we call the C function with `ccall` as before. The type of the function pointer is `Ptr{Cvoid}`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af3fbc67",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "ccall((:fill_cb, \"fake-lib/libsilly\"),\n",
    "    Cvoid,\n",
    "    (Ref{Float64}, Cint, Ptr{Cvoid}),\n",
    "    my_vector, N, cfunc)\n",
    "@show my_vector;"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "763edfe5",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Calling C Library\n",
    "\n",
    "For more details, see the [Calling C and Fortran Code](https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code/) section of the Julia documentation. (If the link does not work, just google \"julia call c library\".)\n",
    "\n",
    "A more complex example is provided by [Ipopt.jl](https://github.com/jump-dev/Ipopt.jl). You may also wish to look at the Ipopt library C API. The easiest way to do this is actually to just look at the header file at src/Interfaces/IpStdCInterface.h which is viewable at the [Ipopt github repo](https://github.com/coin-or/Ipopt)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a8f1b36",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Other Interfaces\n",
    "\n",
    "Interested in calling a function/library written in something other than Python, C or Fortran? Checkout the [Julia Interop](https://github.com/JuliaInterop) group on GitHub. Interfaces already exist for C++, MATLAB, Mathematica and R to name a few."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Julia 1.8.1",
   "language": "julia",
   "name": "julia-1.8"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}