GP02OCIMandOMPA.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nitrogenlab/GP15_watermassanalysis/blob/main/GP02OCIMandOMPA.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FUPY0WuqddQe"
      },
      "source": [
        "This notebook first uses fluxes from a pre-defined OCIM (Ocean Circulation Inverse Model) to explain the composition of each gridbox in the ocean corresponding to the GP02 cruise transect in terms of the fractional contributions from user-defined `end-members'. These results are then used to refine the solution produced by pyompa (a software for conducting OMP analysis) to choose an OMP solution that most resembles the OCIM solution while maintaining residuals comparable to the best OMP solution.\n",
        "\n",
        "This notebook was adapted from a notebook of Avanti Shrikumar (avanti@cs.stanford.edu)'s notebooks for GP02 by Rian Lawrence in the Casciotti lab at Stanford (https://nitrogen.stanford.edu/)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UbNwmEGdjDOg"
      },
      "source": [
        "# Use OCIM to perform water mass analysis for the GP02 transect\n",
        "\n",
        "Our goal will be to explain observations gathered from the GP02 cruise in terms of user-specified end-members using the fluxes from OCIM2."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oQ0qMZfvePzW"
      },
      "source": [
        "## Install and import needed software\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wuti9hpWTC85"
      },
      "source": [
        "### Install miniconda"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "resources used--\n",
        "https://saturncloud.io/blog/conda-environment-in-google-colab-a-guide-for-software-engineers/ and https://saturncloud.io/blog/how-to-install-conda-package-to-google-colab/ and https://www.linkedin.com/pulse/how-install-run-conda-google-colab-ambu-vijayan/"
      ],
      "metadata": {
        "id": "o5BUZTghdpR2"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p_bDHiWp2VtB"
      },
      "source": [
        "setting up"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "149k-NqdQga0",
        "outputId": "bd738109-22de-413f-d6d0-903abaea5a85"
      },
      "source": [
        "%env PYTHONPATH=\n",
        "#CLEARED PYTHON PATH"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "env: PYTHONPATH=\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!wget -O mini.sh https://repo.anaconda.com/miniconda/Miniconda3-py38_4.8.2-Linux-x86_64.sh\n",
        "!chmod +x mini.sh\n",
        "!bash ./mini.sh -b -f -p /usr/local\n",
        "!conda install -q -y jupyter\n",
        "!conda install -q -y google-colab -c conda-forge\n",
        "!python -m ipykernel install --name \"py38\" --user"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ure6oL0aU-yz",
        "outputId": "fb4d2554-5a48-4faa-8dea-f52ad344638f"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-08-28 22:46:50--  https://repo.anaconda.com/miniconda/Miniconda3-py38_4.8.2-Linux-x86_64.sh\n",
            "Resolving repo.anaconda.com (repo.anaconda.com)... 104.16.130.3, 104.16.131.3, 2606:4700::6810:8303, ...\n",
            "Connecting to repo.anaconda.com (repo.anaconda.com)|104.16.130.3|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 89817099 (86M) [application/x-sh]\n",
            "Saving to: ‘mini.sh’\n",
            "\n",
            "mini.sh             100%[===================>]  85.66M   162MB/s    in 0.5s    \n",
            "\n",
            "2023-08-28 22:46:50 (162 MB/s) - ‘mini.sh’ saved [89817099/89817099]\n",
            "\n",
            "PREFIX=/usr/local\n",
            "Unpacking payload ...\n",
            "Collecting package metadata (current_repodata.json): - \b\b\\ \b\b| \b\bdone\n",
            "Solving environment: - \b\b\\ \b\b| \b\b/ \b\b- \b\bdone\n",
            "\n",
            "## Package Plan ##\n",
            "\n",
            "  environment location: /usr/local\n",
            "\n",
            "  added / updated specs:\n",
            "    - _libgcc_mutex==0.1=main\n",
            "    - asn1crypto==1.3.0=py38_0\n",
            "    - ca-certificates==2020.1.1=0\n",
            "    - certifi==2019.11.28=py38_0\n",
            "    - cffi==1.14.0=py38h2e261b9_0\n",
            "    - chardet==3.0.4=py38_1003\n",
            "    - conda-package-handling==1.6.0=py38h7b6447c_0\n",
            "    - conda==4.8.2=py38_0\n",
            "    - cryptography==2.8=py38h1ba5d50_0\n",
            "    - idna==2.8=py38_1000\n",
            "    - ld_impl_linux-64==2.33.1=h53a641e_7\n",
            "    - libedit==3.1.20181209=hc058e9b_0\n",
            "    - libffi==3.2.1=hd88cf55_4\n",
            "    - libgcc-ng==9.1.0=hdf63c60_0\n",
            "    - libstdcxx-ng==9.1.0=hdf63c60_0\n",
            "    - ncurses==6.2=he6710b0_0\n",
            "    - openssl==1.1.1d=h7b6447c_4\n",
            "    - pip==20.0.2=py38_1\n",
            "    - pycosat==0.6.3=py38h7b6447c_0\n",
            "    - pycparser==2.19=py_0\n",
            "    - pyopenssl==19.1.0=py38_0\n",
            "    - pysocks==1.7.1=py38_0\n",
            "    - python==3.8.1=h0371630_1\n",
            "    - readline==7.0=h7b6447c_5\n",
            "    - requests==2.22.0=py38_1\n",
            "    - ruamel_yaml==0.15.87=py38h7b6447c_0\n",
            "    - setuptools==45.2.0=py38_0\n",
            "    - six==1.14.0=py38_0\n",
            "    - sqlite==3.31.1=h7b6447c_0\n",
            "    - tk==8.6.8=hbc83047_0\n",
            "    - tqdm==4.42.1=py_0\n",
            "    - urllib3==1.25.8=py38_0\n",
            "    - wheel==0.34.2=py38_0\n",
            "    - xz==5.2.4=h14c3975_4\n",
            "    - yaml==0.1.7=had09818_2\n",
            "    - zlib==1.2.11=h7b6447c_3\n",
            "\n",
            "\n",
            "The following NEW packages will be INSTALLED:\n",
            "\n",
            "  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main\n",
            "  asn1crypto         pkgs/main/linux-64::asn1crypto-1.3.0-py38_0\n",
            "  ca-certificates    pkgs/main/linux-64::ca-certificates-2020.1.1-0\n",
            "  certifi            pkgs/main/linux-64::certifi-2019.11.28-py38_0\n",
            "  cffi               pkgs/main/linux-64::cffi-1.14.0-py38h2e261b9_0\n",
            "  chardet            pkgs/main/linux-64::chardet-3.0.4-py38_1003\n",
            "  conda              pkgs/main/linux-64::conda-4.8.2-py38_0\n",
            "  conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.6.0-py38h7b6447c_0\n",
            "  cryptography       pkgs/main/linux-64::cryptography-2.8-py38h1ba5d50_0\n",
            "  idna               pkgs/main/linux-64::idna-2.8-py38_1000\n",
            "  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7\n",
            "  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0\n",
            "  libffi             pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4\n",
            "  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0\n",
            "  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0\n",
            "  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_0\n",
            "  openssl            pkgs/main/linux-64::openssl-1.1.1d-h7b6447c_4\n",
            "  pip                pkgs/main/linux-64::pip-20.0.2-py38_1\n",
            "  pycosat            pkgs/main/linux-64::pycosat-0.6.3-py38h7b6447c_0\n",
            "  pycparser          pkgs/main/noarch::pycparser-2.19-py_0\n",
            "  pyopenssl          pkgs/main/linux-64::pyopenssl-19.1.0-py38_0\n",
            "  pysocks            pkgs/main/linux-64::pysocks-1.7.1-py38_0\n",
            "  python             pkgs/main/linux-64::python-3.8.1-h0371630_1\n",
            "  readline           pkgs/main/linux-64::readline-7.0-h7b6447c_5\n",
            "  requests           pkgs/main/linux-64::requests-2.22.0-py38_1\n",
            "  ruamel_yaml        pkgs/main/linux-64::ruamel_yaml-0.15.87-py38h7b6447c_0\n",
            "  setuptools         pkgs/main/linux-64::setuptools-45.2.0-py38_0\n",
            "  six                pkgs/main/linux-64::six-1.14.0-py38_0\n",
            "  sqlite             pkgs/main/linux-64::sqlite-3.31.1-h7b6447c_0\n",
            "  tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0\n",
            "  tqdm               pkgs/main/noarch::tqdm-4.42.1-py_0\n",
            "  urllib3            pkgs/main/linux-64::urllib3-1.25.8-py38_0\n",
            "  wheel              pkgs/main/linux-64::wheel-0.34.2-py38_0\n",
            "  xz                 pkgs/main/linux-64::xz-5.2.4-h14c3975_4\n",
            "  yaml               pkgs/main/linux-64::yaml-0.1.7-had09818_2\n",
            "  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3\n",
            "\n",
            "\n",
            "Preparing transaction: | \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\bdone\n",
            "Executing transaction: / \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\b- \b\b\\ \b\b| \b\b/ \b\bdone\n",
            "installation finished.\n",
            "Collecting package metadata (current_repodata.json): ...working... done\n",
            "Solving environment: ...working... done\n",
            "\n",
            "## Package Plan ##\n",
            "\n",
            "  environment location: /usr/local\n",
            "\n",
            "  added / updated specs:\n",
            "    - jupyter\n",
            "\n",
            "\n",
            "The following packages will be downloaded:\n",
            "\n",
            "    package                    |            build\n",
            "    ---------------------------|-----------------\n",
            "    _openmp_mutex-5.1          |            1_gnu          21 KB\n",
            "    anyio-3.5.0                |   py38h06a4308_0         165 KB\n",
            "    argon2-cffi-21.3.0         |     pyhd3eb1b0_0          15 KB\n",
            "    argon2-cffi-bindings-21.2.0|   py38h7f8727e_0          33 KB\n",
            "    asttokens-2.0.5            |     pyhd3eb1b0_0          20 KB\n",
            "    attrs-22.1.0               |   py38h06a4308_0          85 KB\n",
            "    babel-2.11.0               |   py38h06a4308_0         6.8 MB\n",
            "    backcall-0.2.0             |     pyhd3eb1b0_0          13 KB\n",
            "    beautifulsoup4-4.12.2      |   py38h06a4308_0         209 KB\n",
            "    bleach-4.1.0               |     pyhd3eb1b0_0         123 KB\n",
            "    ca-certificates-2023.05.30 |       h06a4308_0         120 KB\n",
            "    certifi-2023.7.22          |   py38h06a4308_0         153 KB\n",
            "    comm-0.1.2                 |   py38h06a4308_0          13 KB\n",
            "    cyrus-sasl-2.1.28          |       h9c0eb46_1         237 KB\n",
            "    dbus-1.13.18               |       hb2f20db_0         504 KB\n",
            "    debugpy-1.6.7              |   py38h6a678d5_0         2.0 MB\n",
            "    decorator-5.1.1            |     pyhd3eb1b0_0          12 KB\n",
            "    defusedxml-0.7.1           |     pyhd3eb1b0_0          23 KB\n",
            "    entrypoints-0.4            |   py38h06a4308_0          16 KB\n",
            "    executing-0.8.3            |     pyhd3eb1b0_0          18 KB\n",
            "    expat-2.4.9                |       h6a678d5_0         156 KB\n",
            "    fontconfig-2.14.1          |       h4c34cd2_2         281 KB\n",
            "    freetype-2.12.1            |       h4a9f257_0         626 KB\n",
            "    giflib-5.2.1               |       h5eee18b_3          80 KB\n",
            "    glib-2.69.1                |       he621ea3_2         1.9 MB\n",
            "    gst-plugins-base-1.14.1    |       h6a678d5_1         2.2 MB\n",
            "    gstreamer-1.14.1           |       h5eee18b_1         1.7 MB\n",
            "    icu-58.2                   |       he6710b0_3        10.5 MB\n",
            "    importlib_resources-5.2.0  |     pyhd3eb1b0_1          21 KB\n",
            "    ipykernel-6.25.0           |   py38h2f386ee_0         228 KB\n",
            "    ipython-8.12.2             |   py38h06a4308_0         1.1 MB\n",
            "    ipython_genutils-0.2.0     |     pyhd3eb1b0_1          27 KB\n",
            "    ipywidgets-8.0.4           |   py38h06a4308_0         194 KB\n",
            "    jedi-0.18.1                |   py38h06a4308_1         982 KB\n",
            "    jinja2-3.1.2               |   py38h06a4308_0         211 KB\n",
            "    jpeg-9e                    |       h5eee18b_1         262 KB\n",
            "    json5-0.9.6                |     pyhd3eb1b0_0          21 KB\n",
            "    jsonschema-4.17.3          |   py38h06a4308_0         140 KB\n",
            "    jupyter-1.0.0              |   py38h06a4308_8           7 KB\n",
            "    jupyter_client-7.4.9       |   py38h06a4308_0         205 KB\n",
            "    jupyter_console-6.6.3      |   py38h06a4308_0          45 KB\n",
            "    jupyter_core-5.3.0         |   py38h06a4308_0          89 KB\n",
            "    jupyter_server-1.23.4      |   py38h06a4308_0         382 KB\n",
            "    jupyterlab-3.3.2           |     pyhd3eb1b0_0         3.6 MB\n",
            "    jupyterlab_pygments-0.1.2  |             py_0           8 KB\n",
            "    jupyterlab_server-2.10.3   |     pyhd3eb1b0_1          48 KB\n",
            "    jupyterlab_widgets-3.0.5   |   py38h06a4308_0         178 KB\n",
            "    krb5-1.20.1                |       h568e23c_1         1.3 MB\n",
            "    lerc-3.0                   |       h295c915_0         196 KB\n",
            "    libclang-14.0.6            |default_hc6dbbc7_1         137 KB\n",
            "    libclang13-14.0.6          |default_he11475f_1         9.8 MB\n",
            "    libcups-2.4.2              |       ha637b67_0         4.4 MB\n",
            "    libdeflate-1.17            |       h5eee18b_0          69 KB\n",
            "    libedit-3.1.20221030       |       h5eee18b_0         181 KB\n",
            "    libevent-2.1.12            |       h8f2d780_0         425 KB\n",
            "    libffi-3.4.4               |       h6a678d5_0         142 KB\n",
            "    libgcc-ng-11.2.0           |       h1234567_1         5.3 MB\n",
            "    libgomp-11.2.0             |       h1234567_1         474 KB\n",
            "    libllvm14-14.0.6           |       hdb19cb5_3        33.4 MB\n",
            "    libpng-1.6.39              |       h5eee18b_0         304 KB\n",
            "    libpq-12.15                |       h37d81fd_1         2.4 MB\n",
            "    libsodium-1.0.18           |       h7b6447c_0         244 KB\n",
            "    libstdcxx-ng-11.2.0        |       h1234567_1         4.7 MB\n",
            "    libtiff-4.5.0              |       h6a678d5_2         479 KB\n",
            "    libuuid-1.41.5             |       h5eee18b_0          27 KB\n",
            "    libwebp-1.2.4              |       h11a3e52_1          86 KB\n",
            "    libwebp-base-1.2.4         |       h5eee18b_1         376 KB\n",
            "    libxcb-1.15                |       h7f8727e_0         505 KB\n",
            "    libxkbcommon-1.0.1         |       h5eee18b_1         590 KB\n",
            "    libxml2-2.10.4             |       hcbfbd50_0         755 KB\n",
            "    libxslt-1.1.37             |       h2085143_0         266 KB\n",
            "    lxml-4.9.2                 |   py38h5eee18b_0         1.5 MB\n",
            "    lz4-c-1.9.4                |       h6a678d5_0         154 KB\n",
            "    markupsafe-2.1.1           |   py38h7f8727e_0          21 KB\n",
            "    matplotlib-inline-0.1.6    |   py38h06a4308_0          16 KB\n",
            "    mistune-0.8.4              |py38h7b6447c_1000          55 KB\n",
            "    mysql-5.7.24               |       he378463_2        60.0 MB\n",
            "    nbclassic-0.5.5            |   py38h06a4308_0         6.1 MB\n",
            "    nbclient-0.5.13            |   py38h06a4308_0          91 KB\n",
            "    nbconvert-6.5.4            |   py38h06a4308_0         513 KB\n",
            "    nbformat-5.7.0             |   py38h06a4308_0         133 KB\n",
            "    ncurses-6.4                |       h6a678d5_0         914 KB\n",
            "    nest-asyncio-1.5.6         |   py38h06a4308_0          14 KB\n",
            "    notebook-6.5.4             |   py38h06a4308_1         532 KB\n",
            "    notebook-shim-0.2.2        |   py38h06a4308_0          22 KB\n",
            "    nspr-4.35                  |       h6a678d5_0         244 KB\n",
            "    nss-3.89.1                 |       h6a678d5_0         2.1 MB\n",
            "    openssl-1.1.1v             |       h7f8727e_0         3.7 MB\n",
            "    packaging-23.0             |   py38h06a4308_0          68 KB\n",
            "    pandocfilters-1.5.0        |     pyhd3eb1b0_0          11 KB\n",
            "    parso-0.8.3                |     pyhd3eb1b0_0          70 KB\n",
            "    pcre-8.45                  |       h295c915_0         207 KB\n",
            "    pexpect-4.8.0              |     pyhd3eb1b0_3          53 KB\n",
            "    pickleshare-0.7.5          |  pyhd3eb1b0_1003          13 KB\n",
            "    pkgutil-resolve-name-1.3.10|   py38h06a4308_0           9 KB\n",
            "    platformdirs-3.10.0        |   py38h06a4308_0          33 KB\n",
            "    ply-3.11                   |           py38_0          81 KB\n",
            "    prometheus_client-0.14.1   |   py38h06a4308_0          90 KB\n",
            "    prompt-toolkit-3.0.36      |   py38h06a4308_0         574 KB\n",
            "    prompt_toolkit-3.0.36      |       hd3eb1b0_0           5 KB\n",
            "    psutil-5.9.0               |   py38h5eee18b_0         330 KB\n",
            "    ptyprocess-0.7.0           |     pyhd3eb1b0_2          17 KB\n",
            "    pure_eval-0.2.2            |     pyhd3eb1b0_0          14 KB\n",
            "    pygments-2.15.1            |   py38h06a4308_1         1.8 MB\n",
            "    pyqt-5.15.7                |   py38h6a678d5_1         5.1 MB\n",
            "    pyqt5-sip-12.11.0          |   py38h6a678d5_1          87 KB\n",
            "    pyrsistent-0.18.0          |   py38heee7806_0          94 KB\n",
            "    python-3.8.17              |       h7a1cb2a_0        25.3 MB\n",
            "    python-dateutil-2.8.2      |     pyhd3eb1b0_0         233 KB\n",
            "    python-fastjsonschema-2.16.2|   py38h06a4308_0         230 KB\n",
            "    pytz-2022.7                |   py38h06a4308_0         209 KB\n",
            "    pyzmq-23.2.0               |   py38h6a678d5_0         448 KB\n",
            "    qt-main-5.15.2             |       h5b8104b_9        53.7 MB\n",
            "    qt-webengine-5.15.9        |       h9ab4d14_7        53.8 MB\n",
            "    qtconsole-5.4.2            |   py38h06a4308_0         191 KB\n",
            "    qtpy-2.2.0                 |   py38h06a4308_0          84 KB\n",
            "    qtwebkit-5.212             |       h3fafdc1_5        16.2 MB\n",
            "    readline-8.2               |       h5eee18b_0         357 KB\n",
            "    send2trash-1.8.0           |     pyhd3eb1b0_1          19 KB\n",
            "    sip-6.6.2                  |   py38h6a678d5_0         425 KB\n",
            "    sniffio-1.2.0              |   py38h06a4308_1          15 KB\n",
            "    soupsieve-2.4              |   py38h06a4308_0          69 KB\n",
            "    sqlite-3.41.2              |       h5eee18b_0         1.2 MB\n",
            "    stack_data-0.2.0           |     pyhd3eb1b0_0          22 KB\n",
            "    terminado-0.17.1           |   py38h06a4308_0          31 KB\n",
            "    tinycss2-1.2.1             |   py38h06a4308_0          40 KB\n",
            "    tk-8.6.12                  |       h1ccaba5_0         3.0 MB\n",
            "    toml-0.10.2                |     pyhd3eb1b0_0          20 KB\n",
            "    tornado-6.3.2              |   py38h5eee18b_0         644 KB\n",
            "    traitlets-5.7.1            |   py38h06a4308_0         200 KB\n",
            "    typing-extensions-4.7.1    |   py38h06a4308_0           9 KB\n",
            "    typing_extensions-4.7.1    |   py38h06a4308_0          55 KB\n",
            "    wcwidth-0.2.5              |     pyhd3eb1b0_0          26 KB\n",
            "    webencodings-0.5.1         |           py38_1          20 KB\n",
            "    websocket-client-0.58.0    |   py38h06a4308_4          66 KB\n",
            "    widgetsnbextension-4.0.5   |   py38h06a4308_0         875 KB\n",
            "    xz-5.4.2                   |       h5eee18b_0         642 KB\n",
            "    zeromq-4.3.4               |       h2531618_0         331 KB\n",
            "    zipp-3.11.0                |   py38h06a4308_0          19 KB\n",
            "    zlib-1.2.13                |       h5eee18b_0         103 KB\n",
            "    zstd-1.5.5                 |       hc292b87_0         647 KB\n",
            "    ------------------------------------------------------------\n",
            "                                           Total:       345.7 MB\n",
            "\n",
            "The following NEW packages will be INSTALLED:\n",
            "\n",
            "  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu\n",
            "  anyio              pkgs/main/linux-64::anyio-3.5.0-py38h06a4308_0\n",
            "  argon2-cffi        pkgs/main/noarch::argon2-cffi-21.3.0-pyhd3eb1b0_0\n",
            "  argon2-cffi-bindi~ pkgs/main/linux-64::argon2-cffi-bindings-21.2.0-py38h7f8727e_0\n",
            "  asttokens          pkgs/main/noarch::asttokens-2.0.5-pyhd3eb1b0_0\n",
            "  attrs              pkgs/main/linux-64::attrs-22.1.0-py38h06a4308_0\n",
            "  babel              pkgs/main/linux-64::babel-2.11.0-py38h06a4308_0\n",
            "  backcall           pkgs/main/noarch::backcall-0.2.0-pyhd3eb1b0_0\n",
            "  beautifulsoup4     pkgs/main/linux-64::beautifulsoup4-4.12.2-py38h06a4308_0\n",
            "  bleach             pkgs/main/noarch::bleach-4.1.0-pyhd3eb1b0_0\n",
            "  comm               pkgs/main/linux-64::comm-0.1.2-py38h06a4308_0\n",
            "  cyrus-sasl         pkgs/main/linux-64::cyrus-sasl-2.1.28-h9c0eb46_1\n",
            "  dbus               pkgs/main/linux-64::dbus-1.13.18-hb2f20db_0\n",
            "  debugpy            pkgs/main/linux-64::debugpy-1.6.7-py38h6a678d5_0\n",
            "  decorator          pkgs/main/noarch::decorator-5.1.1-pyhd3eb1b0_0\n",
            "  defusedxml         pkgs/main/noarch::defusedxml-0.7.1-pyhd3eb1b0_0\n",
            "  entrypoints        pkgs/main/linux-64::entrypoints-0.4-py38h06a4308_0\n",
            "  executing          pkgs/main/noarch::executing-0.8.3-pyhd3eb1b0_0\n",
            "  expat              pkgs/main/linux-64::expat-2.4.9-h6a678d5_0\n",
            "  fontconfig         pkgs/main/linux-64::fontconfig-2.14.1-h4c34cd2_2\n",
            "  freetype           pkgs/main/linux-64::freetype-2.12.1-h4a9f257_0\n",
            "  giflib             pkgs/main/linux-64::giflib-5.2.1-h5eee18b_3\n",
            "  glib               pkgs/main/linux-64::glib-2.69.1-he621ea3_2\n",
            "  gst-plugins-base   pkgs/main/linux-64::gst-plugins-base-1.14.1-h6a678d5_1\n",
            "  gstreamer          pkgs/main/linux-64::gstreamer-1.14.1-h5eee18b_1\n",
            "  icu                pkgs/main/linux-64::icu-58.2-he6710b0_3\n",
            "  importlib_resourc~ pkgs/main/noarch::importlib_resources-5.2.0-pyhd3eb1b0_1\n",
            "  ipykernel          pkgs/main/linux-64::ipykernel-6.25.0-py38h2f386ee_0\n",
            "  ipython            pkgs/main/linux-64::ipython-8.12.2-py38h06a4308_0\n",
            "  ipython_genutils   pkgs/main/noarch::ipython_genutils-0.2.0-pyhd3eb1b0_1\n",
            "  ipywidgets         pkgs/main/linux-64::ipywidgets-8.0.4-py38h06a4308_0\n",
            "  jedi               pkgs/main/linux-64::jedi-0.18.1-py38h06a4308_1\n",
            "  jinja2             pkgs/main/linux-64::jinja2-3.1.2-py38h06a4308_0\n",
            "  jpeg               pkgs/main/linux-64::jpeg-9e-h5eee18b_1\n",
            "  json5              pkgs/main/noarch::json5-0.9.6-pyhd3eb1b0_0\n",
            "  jsonschema         pkgs/main/linux-64::jsonschema-4.17.3-py38h06a4308_0\n",
            "  jupyter            pkgs/main/linux-64::jupyter-1.0.0-py38h06a4308_8\n",
            "  jupyter_client     pkgs/main/linux-64::jupyter_client-7.4.9-py38h06a4308_0\n",
            "  jupyter_console    pkgs/main/linux-64::jupyter_console-6.6.3-py38h06a4308_0\n",
            "  jupyter_core       pkgs/main/linux-64::jupyter_core-5.3.0-py38h06a4308_0\n",
            "  jupyter_server     pkgs/main/linux-64::jupyter_server-1.23.4-py38h06a4308_0\n",
            "  jupyterlab         pkgs/main/noarch::jupyterlab-3.3.2-pyhd3eb1b0_0\n",
            "  jupyterlab_pygmen~ pkgs/main/noarch::jupyterlab_pygments-0.1.2-py_0\n",
            "  jupyterlab_server  pkgs/main/noarch::jupyterlab_server-2.10.3-pyhd3eb1b0_1\n",
            "  jupyterlab_widgets pkgs/main/linux-64::jupyterlab_widgets-3.0.5-py38h06a4308_0\n",
            "  krb5               pkgs/main/linux-64::krb5-1.20.1-h568e23c_1\n",
            "  lerc               pkgs/main/linux-64::lerc-3.0-h295c915_0\n",
            "  libclang           pkgs/main/linux-64::libclang-14.0.6-default_hc6dbbc7_1\n",
            "  libclang13         pkgs/main/linux-64::libclang13-14.0.6-default_he11475f_1\n",
            "  libcups            pkgs/main/linux-64::libcups-2.4.2-ha637b67_0\n",
            "  libdeflate         pkgs/main/linux-64::libdeflate-1.17-h5eee18b_0\n",
            "  libevent           pkgs/main/linux-64::libevent-2.1.12-h8f2d780_0\n",
            "  libgomp            pkgs/main/linux-64::libgomp-11.2.0-h1234567_1\n",
            "  libllvm14          pkgs/main/linux-64::libllvm14-14.0.6-hdb19cb5_3\n",
            "  libpng             pkgs/main/linux-64::libpng-1.6.39-h5eee18b_0\n",
            "  libpq              pkgs/main/linux-64::libpq-12.15-h37d81fd_1\n",
            "  libsodium          pkgs/main/linux-64::libsodium-1.0.18-h7b6447c_0\n",
            "  libtiff            pkgs/main/linux-64::libtiff-4.5.0-h6a678d5_2\n",
            "  libuuid            pkgs/main/linux-64::libuuid-1.41.5-h5eee18b_0\n",
            "  libwebp            pkgs/main/linux-64::libwebp-1.2.4-h11a3e52_1\n",
            "  libwebp-base       pkgs/main/linux-64::libwebp-base-1.2.4-h5eee18b_1\n",
            "  libxcb             pkgs/main/linux-64::libxcb-1.15-h7f8727e_0\n",
            "  libxkbcommon       pkgs/main/linux-64::libxkbcommon-1.0.1-h5eee18b_1\n",
            "  libxml2            pkgs/main/linux-64::libxml2-2.10.4-hcbfbd50_0\n",
            "  libxslt            pkgs/main/linux-64::libxslt-1.1.37-h2085143_0\n",
            "  lxml               pkgs/main/linux-64::lxml-4.9.2-py38h5eee18b_0\n",
            "  lz4-c              pkgs/main/linux-64::lz4-c-1.9.4-h6a678d5_0\n",
            "  markupsafe         pkgs/main/linux-64::markupsafe-2.1.1-py38h7f8727e_0\n",
            "  matplotlib-inline  pkgs/main/linux-64::matplotlib-inline-0.1.6-py38h06a4308_0\n",
            "  mistune            pkgs/main/linux-64::mistune-0.8.4-py38h7b6447c_1000\n",
            "  mysql              pkgs/main/linux-64::mysql-5.7.24-he378463_2\n",
            "  nbclassic          pkgs/main/linux-64::nbclassic-0.5.5-py38h06a4308_0\n",
            "  nbclient           pkgs/main/linux-64::nbclient-0.5.13-py38h06a4308_0\n",
            "  nbconvert          pkgs/main/linux-64::nbconvert-6.5.4-py38h06a4308_0\n",
            "  nbformat           pkgs/main/linux-64::nbformat-5.7.0-py38h06a4308_0\n",
            "  nest-asyncio       pkgs/main/linux-64::nest-asyncio-1.5.6-py38h06a4308_0\n",
            "  notebook           pkgs/main/linux-64::notebook-6.5.4-py38h06a4308_1\n",
            "  notebook-shim      pkgs/main/linux-64::notebook-shim-0.2.2-py38h06a4308_0\n",
            "  nspr               pkgs/main/linux-64::nspr-4.35-h6a678d5_0\n",
            "  nss                pkgs/main/linux-64::nss-3.89.1-h6a678d5_0\n",
            "  packaging          pkgs/main/linux-64::packaging-23.0-py38h06a4308_0\n",
            "  pandocfilters      pkgs/main/noarch::pandocfilters-1.5.0-pyhd3eb1b0_0\n",
            "  parso              pkgs/main/noarch::parso-0.8.3-pyhd3eb1b0_0\n",
            "  pcre               pkgs/main/linux-64::pcre-8.45-h295c915_0\n",
            "  pexpect            pkgs/main/noarch::pexpect-4.8.0-pyhd3eb1b0_3\n",
            "  pickleshare        pkgs/main/noarch::pickleshare-0.7.5-pyhd3eb1b0_1003\n",
            "  pkgutil-resolve-n~ pkgs/main/linux-64::pkgutil-resolve-name-1.3.10-py38h06a4308_0\n",
            "  platformdirs       pkgs/main/linux-64::platformdirs-3.10.0-py38h06a4308_0\n",
            "  ply                pkgs/main/linux-64::ply-3.11-py38_0\n",
            "  prometheus_client  pkgs/main/linux-64::prometheus_client-0.14.1-py38h06a4308_0\n",
            "  prompt-toolkit     pkgs/main/linux-64::prompt-toolkit-3.0.36-py38h06a4308_0\n",
            "  prompt_toolkit     pkgs/main/noarch::prompt_toolkit-3.0.36-hd3eb1b0_0\n",
            "  psutil             pkgs/main/linux-64::psutil-5.9.0-py38h5eee18b_0\n",
            "  ptyprocess         pkgs/main/noarch::ptyprocess-0.7.0-pyhd3eb1b0_2\n",
            "  pure_eval          pkgs/main/noarch::pure_eval-0.2.2-pyhd3eb1b0_0\n",
            "  pygments           pkgs/main/linux-64::pygments-2.15.1-py38h06a4308_1\n",
            "  pyqt               pkgs/main/linux-64::pyqt-5.15.7-py38h6a678d5_1\n",
            "  pyqt5-sip          pkgs/main/linux-64::pyqt5-sip-12.11.0-py38h6a678d5_1\n",
            "  pyrsistent         pkgs/main/linux-64::pyrsistent-0.18.0-py38heee7806_0\n",
            "  python-dateutil    pkgs/main/noarch::python-dateutil-2.8.2-pyhd3eb1b0_0\n",
            "  python-fastjsonsc~ pkgs/main/linux-64::python-fastjsonschema-2.16.2-py38h06a4308_0\n",
            "  pytz               pkgs/main/linux-64::pytz-2022.7-py38h06a4308_0\n",
            "  pyzmq              pkgs/main/linux-64::pyzmq-23.2.0-py38h6a678d5_0\n",
            "  qt-main            pkgs/main/linux-64::qt-main-5.15.2-h5b8104b_9\n",
            "  qt-webengine       pkgs/main/linux-64::qt-webengine-5.15.9-h9ab4d14_7\n",
            "  qtconsole          pkgs/main/linux-64::qtconsole-5.4.2-py38h06a4308_0\n",
            "  qtpy               pkgs/main/linux-64::qtpy-2.2.0-py38h06a4308_0\n",
            "  qtwebkit           pkgs/main/linux-64::qtwebkit-5.212-h3fafdc1_5\n",
            "  send2trash         pkgs/main/noarch::send2trash-1.8.0-pyhd3eb1b0_1\n",
            "  sip                pkgs/main/linux-64::sip-6.6.2-py38h6a678d5_0\n",
            "  sniffio            pkgs/main/linux-64::sniffio-1.2.0-py38h06a4308_1\n",
            "  soupsieve          pkgs/main/linux-64::soupsieve-2.4-py38h06a4308_0\n",
            "  stack_data         pkgs/main/noarch::stack_data-0.2.0-pyhd3eb1b0_0\n",
            "  terminado          pkgs/main/linux-64::terminado-0.17.1-py38h06a4308_0\n",
            "  tinycss2           pkgs/main/linux-64::tinycss2-1.2.1-py38h06a4308_0\n",
            "  toml               pkgs/main/noarch::toml-0.10.2-pyhd3eb1b0_0\n",
            "  tornado            pkgs/main/linux-64::tornado-6.3.2-py38h5eee18b_0\n",
            "  traitlets          pkgs/main/linux-64::traitlets-5.7.1-py38h06a4308_0\n",
            "  typing-extensions  pkgs/main/linux-64::typing-extensions-4.7.1-py38h06a4308_0\n",
            "  typing_extensions  pkgs/main/linux-64::typing_extensions-4.7.1-py38h06a4308_0\n",
            "  wcwidth            pkgs/main/noarch::wcwidth-0.2.5-pyhd3eb1b0_0\n",
            "  webencodings       pkgs/main/linux-64::webencodings-0.5.1-py38_1\n",
            "  websocket-client   pkgs/main/linux-64::websocket-client-0.58.0-py38h06a4308_4\n",
            "  widgetsnbextension pkgs/main/linux-64::widgetsnbextension-4.0.5-py38h06a4308_0\n",
            "  zeromq             pkgs/main/linux-64::zeromq-4.3.4-h2531618_0\n",
            "  zipp               pkgs/main/linux-64::zipp-3.11.0-py38h06a4308_0\n",
            "  zstd               pkgs/main/linux-64::zstd-1.5.5-hc292b87_0\n",
            "\n",
            "The following packages will be UPDATED:\n",
            "\n",
            "  ca-certificates                                2020.1.1-0 --> 2023.05.30-h06a4308_0\n",
            "  certifi                                 2019.11.28-py38_0 --> 2023.7.22-py38h06a4308_0\n",
            "  libedit                           3.1.20181209-hc058e9b_0 --> 3.1.20221030-h5eee18b_0\n",
            "  libffi                                   3.2.1-hd88cf55_4 --> 3.4.4-h6a678d5_0\n",
            "  libgcc-ng                                9.1.0-hdf63c60_0 --> 11.2.0-h1234567_1\n",
            "  libstdcxx-ng                             9.1.0-hdf63c60_0 --> 11.2.0-h1234567_1\n",
            "  ncurses                                    6.2-he6710b0_0 --> 6.4-h6a678d5_0\n",
            "  openssl                                 1.1.1d-h7b6447c_4 --> 1.1.1v-h7f8727e_0\n",
            "  python                                   3.8.1-h0371630_1 --> 3.8.17-h7a1cb2a_0\n",
            "  readline                                   7.0-h7b6447c_5 --> 8.2-h5eee18b_0\n",
            "  sqlite                                  3.31.1-h7b6447c_0 --> 3.41.2-h5eee18b_0\n",
            "  tk                                       8.6.8-hbc83047_0 --> 8.6.12-h1ccaba5_0\n",
            "  xz                                       5.2.4-h14c3975_4 --> 5.4.2-h5eee18b_0\n",
            "  zlib                                    1.2.11-h7b6447c_3 --> 1.2.13-h5eee18b_0\n",
            "\n",
            "\n",
            "Preparing transaction: ...working... done\n",
            "Verifying transaction: ...working... done\n",
            "Executing transaction: ...working... done\n",
            "Collecting package metadata (current_repodata.json): ...working... done\n",
            "Solving environment: ...working... \n",
            "The environment is inconsistent, please check the package plan carefully\n",
            "The following packages are causing the inconsistency:\n",
            "\n",
            "  - defaults/linux-64::terminado==0.17.1=py38h06a4308_0\n",
            "  - defaults/linux-64::libwebp==1.2.4=h11a3e52_1\n",
            "  - defaults/linux-64::zlib==1.2.13=h5eee18b_0\n",
            "  - defaults/linux-64::prometheus_client==0.14.1=py38h06a4308_0\n",
            "  - defaults/linux-64::prompt-toolkit==3.0.36=py38h06a4308_0\n",
            "  - defaults/noarch::json5==0.9.6=pyhd3eb1b0_0\n",
            "  - defaults/linux-64::openssl==1.1.1v=h7f8727e_0\n",
            "  - defaults/linux-64::requests==2.22.0=py38_1\n",
            "  - defaults/linux-64::pyqt==5.15.7=py38h6a678d5_1\n",
            "  - defaults/linux-64::jupyter_client==7.4.9=py38h06a4308_0\n",
            "  - defaults/linux-64::qt-main==5.15.2=h5b8104b_9\n",
            "  - defaults/linux-64::nss==3.89.1=h6a678d5_0\n",
            "  - defaults/noarch::importlib_resources==5.2.0=pyhd3eb1b0_1\n",
            "  - defaults/linux-64::pkgutil-resolve-name==1.3.10=py38h06a4308_0\n",
            "  - defaults/linux-64::soupsieve==2.4=py38h06a4308_0\n",
            "  - defaults/linux-64::libedit==3.1.20221030=h5eee18b_0\n",
            "  - defaults/linux-64::entrypoints==0.4=py38h06a4308_0\n",
            "  - defaults/linux-64::notebook-shim==0.2.2=py38h06a4308_0\n",
            "  - defaults/linux-64::pytz==2022.7=py38h06a4308_0\n",
            "  - defaults/noarch::bleach==4.1.0=pyhd3eb1b0_0\n",
            "  - defaults/linux-64::freetype==2.12.1=h4a9f257_0\n",
            "  - defaults/linux-64::cffi==1.14.0=py38h2e261b9_0\n",
            "  - defaults/linux-64::giflib==5.2.1=h5eee18b_3\n",
            "  - defaults/linux-64::libclang==14.0.6=default_hc6dbbc7_1\n",
            "  - defaults/noarch::argon2-cffi==21.3.0=pyhd3eb1b0_0\n",
            "  - defaults/linux-64::anyio==3.5.0=py38h06a4308_0\n",
            "  - defaults/noarch::send2trash==1.8.0=pyhd3eb1b0_1\n",
            "  - defaults/linux-64::qtpy==2.2.0=py38h06a4308_0\n",
            "  - defaults/linux-64::libwebp-base==1.2.4=h5eee18b_1\n",
            "  - defaults/linux-64::lerc==3.0=h295c915_0\n",
            "  - defaults/noarch::pure_eval==0.2.2=pyhd3eb1b0_0\n",
            "  - defaults/linux-64::libxkbcommon==1.0.1=h5eee18b_1\n",
            "  - defaults/linux-64::chardet==3.0.4=py38_1003\n",
            "  - defaults/linux-64::ncurses==6.4=h6a678d5_0\n",
            "  - defaults/linux-64::glib==2.69.1=he621ea3_2\n",
            "  - defaults/linux-64::pyrsistent==0.18.0=py38heee7806_0\n",
            "  - defaults/linux-64::libpng==1.6.39=h5eee18b_0\n",
            "  - defaults/linux-64::ipykernel==6.25.0=py38h2f386ee_0\n",
            "  - defaults/linux-64::libllvm14==14.0.6=hdb19cb5_3\n",
            "  - defaults/linux-64::argon2-cffi-bindings==21.2.0=py38h7f8727e_0\n",
            "  - defaults/linux-64::notebook==6.5.4=py38h06a4308_1\n",
            "  - defaults/linux-64::jupyter==1.0.0=py38h06a4308_8\n",
            "  - defaults/linux-64::webencodings==0.5.1=py38_1\n",
            "  - defaults/linux-64::pygments==2.15.1=py38h06a4308_1\n",
            "  - defaults/linux-64::idna==2.8=py38_1000\n",
            "  - defaults/linux-64::websocket-client==0.58.0=py38h06a4308_4\n",
            "  - defaults/linux-64::zstd==1.5.5=hc292b87_0\n",
            "  - defaults/linux-64::comm==0.1.2=py38h06a4308_0\n",
            "  - defaults/linux-64::jpeg==9e=h5eee18b_1\n",
            "  - defaults/linux-64::attrs==22.1.0=py38h06a4308_0\n",
            "  - defaults/noarch::jupyterlab==3.3.2=pyhd3eb1b0_0\n",
            "  - defaults/linux-64::expat==2.4.9=h6a678d5_0\n",
            "  - defaults/linux-64::ipython==8.12.2=py38h06a4308_0\n",
            "  - defaults/linux-64::libstdcxx-ng==11.2.0=h1234567_1\n",
            "  - defaults/noarch::wcwidth==0.2.5=pyhd3eb1b0_0\n",
            "  - defaults/noarch::pexpect==4.8.0=pyhd3eb1b0_3\n",
            "  - defaults/linux-64::six==1.14.0=py38_0\n",
            "  - defaults/noarch::executing==0.8.3=pyhd3eb1b0_0\n",
            "  - defaults/linux-64::conda-package-handling==1.6.0=py38h7b6447c_0\n",
            "  - defaults/linux-64::lz4-c==1.9.4=h6a678d5_0\n",
            "  - defaults/linux-64::typing_extensions==4.7.1=py38h06a4308_0\n",
            "  - defaults/linux-64::libpq==12.15=h37d81fd_1\n",
            "  - defaults/linux-64::jsonschema==4.17.3=py38h06a4308_0\n",
            "  - defaults/noarch::ipython_genutils==0.2.0=pyhd3eb1b0_1\n",
            "  - defaults/linux-64::setuptools==45.2.0=py38_0\n",
            "  - defaults/noarch::pycparser==2.19=py_0\n",
            "  - defaults/noarch::python-dateutil==2.8.2=pyhd3eb1b0_0\n",
            "  - defaults/linux-64::conda==4.8.2=py38_0\n",
            "  - defaults/linux-64::pip==20.0.2=py38_1\n",
            "  - defaults/linux-64::xz==5.4.2=h5eee18b_0\n",
            "  - defaults/linux-64::libdeflate==1.17=h5eee18b_0\n",
            "  - defaults/noarch::ptyprocess==0.7.0=pyhd3eb1b0_2\n",
            "  - defaults/linux-64::qtconsole==5.4.2=py38h06a4308_0\n",
            "  - defaults/noarch::backcall==0.2.0=pyhd3eb1b0_0\n",
            "  - defaults/linux-64::pysocks==1.7.1=py38_0\n",
            "  - defaults/linux-64::qt-webengine==5.15.9=h9ab4d14_7\n",
            "  - defaults/linux-64::ipywidgets==8.0.4=py38h06a4308_0\n",
            "  - defaults/linux-64::zipp==3.11.0=py38h06a4308_0\n",
            "  - defaults/linux-64::jupyter_core==5.3.0=py38h06a4308_0\n",
            "  - defaults/linux-64::ply==3.11=py38_0\n",
            "  - defaults/linux-64::debugpy==1.6.7=py38h6a678d5_0\n",
            "  - defaults/linux-64::jedi==0.18.1=py38h06a4308_1\n",
            "  - defaults/linux-64::tinycss2==1.2.1=py38h06a4308_0\n",
            "  - defaults/linux-64::libtiff==4.5.0=h6a678d5_2\n",
            "  - defaults/linux-64::mysql==5.7.24=he378463_2\n",
            "  - defaults/linux-64::zeromq==4.3.4=h2531618_0\n",
            "  - defaults/noarch::stack_data==0.2.0=pyhd3eb1b0_0\n",
            "  - defaults/linux-64::pyopenssl==19.1.0=py38_0\n",
            "  - defaults/linux-64::traitlets==5.7.1=py38h06a4308_0\n",
            "  - defaults/linux-64::packaging==23.0=py38h06a4308_0\n",
            "  - defaults/noarch::jupyterlab_server==2.10.3=pyhd3eb1b0_1\n",
            "  - defaults/linux-64::jupyter_server==1.23.4=py38h06a4308_0\n",
            "  - defaults/linux-64::widgetsnbextension==4.0.5=py38h06a4308_0\n",
            "  - defaults/linux-64::ruamel_yaml==0.15.87=py38h7b6447c_0\n",
            "  - defaults/linux-64::libxslt==1.1.37=h2085143_0\n",
            "  - defaults/linux-64::markupsafe==2.1.1=py38h7f8727e_0\n",
            "  - defaults/linux-64::libuuid==1.41.5=h5eee18b_0\n",
            "  - defaults/linux-64::python-fastjsonschema==2.16.2=py38h06a4308_0\n",
            "  - defaults/noarch::tqdm==4.42.1=py_0\n",
            "  - defaults/linux-64::asn1crypto==1.3.0=py38_0\n",
            "  - defaults/linux-64::nspr==4.35=h6a678d5_0\n",
            "  - defaults/linux-64::sniffio==1.2.0=py38h06a4308_1\n",
            "  - defaults/linux-64::babel==2.11.0=py38h06a4308_0\n",
            "  - defaults/linux-64::readline==8.2=h5eee18b_0\n",
            "  - defaults/noarch::toml==0.10.2=pyhd3eb1b0_0\n",
            "  - defaults/linux-64::icu==58.2=he6710b0_3\n",
            "  - defaults/linux-64::pyzmq==23.2.0=py38h6a678d5_0\n",
            "  - defaults/linux-64::krb5==1.20.1=h568e23c_1\n",
            "  - defaults/noarch::parso==0.8.3=pyhd3eb1b0_0\n",
            "  - defaults/linux-64::libffi==3.4.4=h6a678d5_0\n",
            "  - defaults/linux-64::dbus==1.13.18=hb2f20db_0\n",
            "  - defaults/linux-64::psutil==5.9.0=py38h5eee18b_0\n",
            "  - defaults/noarch::pandocfilters==1.5.0=pyhd3eb1b0_0\n",
            "  - defaults/linux-64::beautifulsoup4==4.12.2=py38h06a4308_0\n",
            "  - defaults/linux-64::pcre==8.45=h295c915_0\n",
            "  - defaults/linux-64::mistune==0.8.4=py38h7b6447c_1000\n",
            "  - defaults/linux-64::matplotlib-inline==0.1.6=py38h06a4308_0\n",
            "  - defaults/linux-64::sqlite==3.41.2=h5eee18b_0\n",
            "  - defaults/noarch::asttokens==2.0.5=pyhd3eb1b0_0\n",
            "  - defaults/linux-64::jinja2==3.1.2=py38h06a4308_0\n",
            "  - defaults/linux-64::python==3.8.17=h7a1cb2a_0\n",
            "  - defaults/linux-64::libclang13==14.0.6=default_he11475f_1\n",
            "  - defaults/linux-64::yaml==0.1.7=had09818_2\n",
            "  - defaults/noarch::prompt_toolkit==3.0.36=hd3eb1b0_0\n",
            "  - defaults/linux-64::qtwebkit==5.212=h3fafdc1_5\n",
            "  - defaults/linux-64::typing-extensions==4.7.1=py38h06a4308_0\n",
            "  - defaults/linux-64::pycosat==0.6.3=py38h7b6447c_0\n",
            "  - defaults/linux-64::cryptography==2.8=py38h1ba5d50_0\n",
            "  - defaults/linux-64::sip==6.6.2=py38h6a678d5_0\n",
            "  - defaults/linux-64::urllib3==1.25.8=py38_0\n",
            "  - defaults/linux-64::platformdirs==3.10.0=py38h06a4308_0\n",
            "  - defaults/linux-64::nbclient==0.5.13=py38h06a4308_0\n",
            "  - defaults/noarch::jupyterlab_pygments==0.1.2=py_0\n",
            "  - defaults/linux-64::lxml==4.9.2=py38h5eee18b_0\n",
            "  - defaults/linux-64::nbconvert==6.5.4=py38h06a4308_0\n",
            "  - defaults/linux-64::cyrus-sasl==2.1.28=h9c0eb46_1\n",
            "  - defaults/linux-64::tk==8.6.12=h1ccaba5_0\n",
            "  - defaults/linux-64::wheel==0.34.2=py38_0\n",
            "  - defaults/linux-64::libxml2==2.10.4=hcbfbd50_0\n",
            "  - defaults/linux-64::libsodium==1.0.18=h7b6447c_0\n",
            "  - defaults/linux-64::nbclassic==0.5.5=py38h06a4308_0\n",
            "  - defaults/linux-64::pyqt5-sip==12.11.0=py38h6a678d5_1\n",
            "  - defaults/noarch::pickleshare==0.7.5=pyhd3eb1b0_1003\n",
            "  - defaults/linux-64::libxcb==1.15=h7f8727e_0\n",
            "  - defaults/linux-64::libcups==2.4.2=ha637b67_0\n",
            "  - defaults/linux-64::libgcc-ng==11.2.0=h1234567_1\n",
            "  - defaults/noarch::defusedxml==0.7.1=pyhd3eb1b0_0\n",
            "  - defaults/linux-64::jupyter_console==6.6.3=py38h06a4308_0\n",
            "  - defaults/noarch::decorator==5.1.1=pyhd3eb1b0_0\n",
            "  - defaults/linux-64::libevent==2.1.12=h8f2d780_0\n",
            "  - defaults/linux-64::tornado==6.3.2=py38h5eee18b_0\n",
            "  - defaults/linux-64::gstreamer==1.14.1=h5eee18b_1\n",
            "  - defaults/linux-64::jupyterlab_widgets==3.0.5=py38h06a4308_0\n",
            "  - defaults/linux-64::nbformat==5.7.0=py38h06a4308_0\n",
            "  - defaults/linux-64::fontconfig==2.14.1=h4c34cd2_2\n",
            "  - defaults/linux-64::gst-plugins-base==1.14.1=h6a678d5_1\n",
            "  - defaults/linux-64::certifi==2023.7.22=py38h06a4308_0\n",
            "  - defaults/linux-64::nest-asyncio==1.5.6=py38h06a4308_0\n",
            "done\n",
            "\n",
            "## Package Plan ##\n",
            "\n",
            "  environment location: /usr/local\n",
            "\n",
            "  added / updated specs:\n",
            "    - google-colab\n",
            "\n",
            "\n",
            "The following packages will be downloaded:\n",
            "\n",
            "    package                    |            build\n",
            "    ---------------------------|-----------------\n",
            "    aiohttp-3.8.1              |   py38h0a891b7_1         574 KB  conda-forge\n",
            "    aiosignal-1.3.1            |     pyhd8ed1ab_0          12 KB  conda-forge\n",
            "    async-timeout-4.0.3        |     pyhd8ed1ab_0          11 KB  conda-forge\n",
            "    ca-certificates-2023.7.22  |       hbcca054_0         146 KB  conda-forge\n",
            "    cachetools-5.3.1           |     pyhd8ed1ab_0          14 KB  conda-forge\n",
            "    charset-normalizer-2.0.4   |     pyhd3eb1b0_0          35 KB\n",
            "    cryptography-3.4.8         |   py38h3e25421_1         1.1 MB  conda-forge\n",
            "    frozenlist-1.3.3           |   py38h5eee18b_0          45 KB\n",
            "    google-auth-2.22.0         |     pyh1a96a4e_0         100 KB  conda-forge\n",
            "    google-colab-1.0.0         |     pyh44b312d_0          77 KB  conda-forge\n",
            "    libblas-3.9.0              |15_linux64_openblas          12 KB  conda-forge\n",
            "    libcblas-3.9.0             |15_linux64_openblas          12 KB  conda-forge\n",
            "    libgfortran-ng-13.1.0      |       h69a702a_0          23 KB  conda-forge\n",
            "    libgfortran5-13.1.0        |       h15d22d2_0         1.4 MB  conda-forge\n",
            "    liblapack-3.9.0            |15_linux64_openblas          12 KB  conda-forge\n",
            "    libopenblas-0.3.20         |pthreads_h78a6416_0        10.1 MB  conda-forge\n",
            "    multidict-6.0.2            |   py38h5eee18b_0          49 KB\n",
            "    numpy-1.22.3               |   py38h99721a1_2         6.8 MB  conda-forge\n",
            "    pandas-1.3.5               |   py38h43a58ef_0        13.0 MB  conda-forge\n",
            "    portpicker-1.6.0           |     pyhd8ed1ab_0          20 KB  conda-forge\n",
            "    pyasn1-0.4.8               |             py_0          53 KB  conda-forge\n",
            "    pyasn1-modules-0.2.7       |             py_0          60 KB  conda-forge\n",
            "    pyopenssl-20.0.1           |     pyhd8ed1ab_0          48 KB  conda-forge\n",
            "    python_abi-3.8             |           2_cp38           4 KB  conda-forge\n",
            "    pyu2f-0.1.5                |     pyhd8ed1ab_0          31 KB  conda-forge\n",
            "    rsa-4.9                    |     pyhd8ed1ab_0          29 KB  conda-forge\n",
            "    yarl-1.7.2                 |   py38h0a891b7_2         133 KB  conda-forge\n",
            "    ------------------------------------------------------------\n",
            "                                           Total:        33.8 MB\n",
            "\n",
            "The following NEW packages will be INSTALLED:\n",
            "\n",
            "  aiohttp            conda-forge/linux-64::aiohttp-3.8.1-py38h0a891b7_1\n",
            "  aiosignal          conda-forge/noarch::aiosignal-1.3.1-pyhd8ed1ab_0\n",
            "  async-timeout      conda-forge/noarch::async-timeout-4.0.3-pyhd8ed1ab_0\n",
            "  cachetools         conda-forge/noarch::cachetools-5.3.1-pyhd8ed1ab_0\n",
            "  charset-normalizer pkgs/main/noarch::charset-normalizer-2.0.4-pyhd3eb1b0_0\n",
            "  frozenlist         pkgs/main/linux-64::frozenlist-1.3.3-py38h5eee18b_0\n",
            "  google-auth        conda-forge/noarch::google-auth-2.22.0-pyh1a96a4e_0\n",
            "  google-colab       conda-forge/noarch::google-colab-1.0.0-pyh44b312d_0\n",
            "  libblas            conda-forge/linux-64::libblas-3.9.0-15_linux64_openblas\n",
            "  libcblas           conda-forge/linux-64::libcblas-3.9.0-15_linux64_openblas\n",
            "  libgfortran-ng     conda-forge/linux-64::libgfortran-ng-13.1.0-h69a702a_0\n",
            "  libgfortran5       conda-forge/linux-64::libgfortran5-13.1.0-h15d22d2_0\n",
            "  liblapack          conda-forge/linux-64::liblapack-3.9.0-15_linux64_openblas\n",
            "  libopenblas        conda-forge/linux-64::libopenblas-0.3.20-pthreads_h78a6416_0\n",
            "  multidict          pkgs/main/linux-64::multidict-6.0.2-py38h5eee18b_0\n",
            "  numpy              conda-forge/linux-64::numpy-1.22.3-py38h99721a1_2\n",
            "  pandas             conda-forge/linux-64::pandas-1.3.5-py38h43a58ef_0\n",
            "  portpicker         conda-forge/noarch::portpicker-1.6.0-pyhd8ed1ab_0\n",
            "  pyasn1             conda-forge/noarch::pyasn1-0.4.8-py_0\n",
            "  pyasn1-modules     conda-forge/noarch::pyasn1-modules-0.2.7-py_0\n",
            "  python_abi         conda-forge/linux-64::python_abi-3.8-2_cp38\n",
            "  pyu2f              conda-forge/noarch::pyu2f-0.1.5-pyhd8ed1ab_0\n",
            "  rsa                conda-forge/noarch::rsa-4.9-pyhd8ed1ab_0\n",
            "  yarl               conda-forge/linux-64::yarl-1.7.2-py38h0a891b7_2\n",
            "\n",
            "The following packages will be UPDATED:\n",
            "\n",
            "  ca-certificates    pkgs/main::ca-certificates-2023.05.30~ --> conda-forge::ca-certificates-2023.7.22-hbcca054_0\n",
            "  cryptography       pkgs/main::cryptography-2.8-py38h1ba5~ --> conda-forge::cryptography-3.4.8-py38h3e25421_1\n",
            "  pyopenssl          pkgs/main/linux-64::pyopenssl-19.1.0-~ --> conda-forge/noarch::pyopenssl-20.0.1-pyhd8ed1ab_0\n",
            "\n",
            "\n",
            "Preparing transaction: ...working... done\n",
            "Verifying transaction: ...working... done\n",
            "Executing transaction: ...working... done\n",
            "Installed kernelspec py38 in /root/.local/share/jupyter/kernels/py38\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Don't forget to reload the browser page after you installed the kernel (as explained in the original post). https://stackoverflow.com/questions/60775160/install-python-3-8-kernel-in-google-colaboratory/71511943#71511943"
      ],
      "metadata": {
        "id": "l3hQKb6cVZH5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "CHECK VERSION OF PYTHON"
      ],
      "metadata": {
        "id": "_5VAcf8TtRFb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!which python\n",
        "!python --version\n",
        "!echo $PYTHONPATH"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GHzVBAfciwlY",
        "outputId": "eeca2485-c586-4b63-94f5-81103fa69883"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/usr/local/bin/python\n",
            "Python 3.8.17\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3gF3iB8MU2RH",
        "outputId": "e74d1fb9-b8a3-462f-a5e4-289f26c96e9d"
      },
      "source": [
        "#! wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\n",
        "#! chmod +x Miniconda3-latest-Linux-x86_64.sh\n",
        "#! bash ./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local/\n",
        "#!wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\n",
        "#!bash Miniconda3-latest-Linux-x86_64.sh -bfp /usr/local\n",
        "\n",
        "#above doesn't work because scikits.umfpack does not work with later versions\n",
        "#of python and conda"
      ],
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-08-27 20:35:37--  https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\n",
            "Resolving repo.anaconda.com (repo.anaconda.com)... 104.16.130.3, 104.16.131.3, 2606:4700::6810:8303, ...\n",
            "Connecting to repo.anaconda.com (repo.anaconda.com)|104.16.130.3|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 103219356 (98M) [application/x-sh]\n",
            "Saving to: ‘Miniconda3-latest-Linux-x86_64.sh.2’\n",
            "\n",
            "Miniconda3-latest-L 100%[===================>]  98.44M   148MB/s    in 0.7s    \n",
            "\n",
            "2023-08-27 20:35:38 (148 MB/s) - ‘Miniconda3-latest-Linux-x86_64.sh.2’ saved [103219356/103219356]\n",
            "\n",
            "PREFIX=/usr/local\n",
            "Unpacking payload ...\n",
            "                                                                                   \n",
            "Installing base environment...\n",
            "\n",
            "\n",
            "Downloading and Extracting Packages\n",
            "\n",
            "Preparing transaction: - \b\bdone\n",
            "Executing transaction: | \b\bdone\n",
            "installation finished.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! conda install -c conda-forge scikit-umfpack --yes\n",
        "import sys\n",
        "sys.path.append(\"/usr/local/lib/python3.11/site-packages\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PKE9ImFOwxiI",
        "outputId": "2f3329c3-bb92-4cbd-b8d0-3c440c9e0150"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<\n",
            "\n",
            "    Traceback (most recent call last):\n",
            "      File \"/usr/local/lib/python3.11/site-packages/requests/compat.py\", line 11, in <module>\n",
            "        import chardet\n",
            "    ModuleNotFoundError: No module named 'chardet'\n",
            "    \n",
            "    During handling of the above exception, another exception occurred:\n",
            "    \n",
            "    Traceback (most recent call last):\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/exception_handler.py\", line 16, in __call__\n",
            "        return func(*args, **kwargs)\n",
            "               ^^^^^^^^^^^^^^^^^^^^^\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/cli/main.py\", line 84, in main_subshell\n",
            "        exit_code = do_call(args, p)\n",
            "                    ^^^^^^^^^^^^^^^^\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/cli/conda_argparse.py\", line 124, in do_call\n",
            "        module = import_module(relative_mod, __name__.rsplit(\".\", 1)[0])\n",
            "                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "      File \"/usr/local/lib/python3.11/importlib/__init__.py\", line 126, in import_module\n",
            "        return _bootstrap._gcd_import(name[level:], package, level)\n",
            "               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "      File \"<frozen importlib._bootstrap>\", line 1204, in _gcd_import\n",
            "      File \"<frozen importlib._bootstrap>\", line 1176, in _find_and_load\n",
            "      File \"<frozen importlib._bootstrap>\", line 1147, in _find_and_load_unlocked\n",
            "      File \"<frozen importlib._bootstrap>\", line 690, in _load_unlocked\n",
            "      File \"<frozen importlib._bootstrap_external>\", line 940, in exec_module\n",
            "      File \"<frozen importlib._bootstrap>\", line 241, in _call_with_frames_removed\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/cli/main_install.py\", line 6, in <module>\n",
            "        from ..notices import notices\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/notices/__init__.py\", line 3, in <module>\n",
            "        from .core import notices  # noqa: F401\n",
            "        ^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/notices/core.py\", line 13, in <module>\n",
            "        from . import cache, fetch, views\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/notices/fetch.py\", line 9, in <module>\n",
            "        import requests\n",
            "      File \"/usr/local/lib/python3.11/site-packages/requests/__init__.py\", line 45, in <module>\n",
            "        from .exceptions import RequestsDependencyWarning\n",
            "      File \"/usr/local/lib/python3.11/site-packages/requests/exceptions.py\", line 9, in <module>\n",
            "        from .compat import JSONDecodeError as CompatJSONDecodeError\n",
            "      File \"/usr/local/lib/python3.11/site-packages/requests/compat.py\", line 13, in <module>\n",
            "        import charset_normalizer as chardet\n",
            "    ModuleNotFoundError: No module named 'charset_normalizer'\n",
            "\n",
            "`$ /usr/local/bin/conda install -c conda-forge scikit-umfpack --yes`\n",
            "\n",
            "  environment variables:\n",
            "                 CIO_TEST=<not set>\n",
            "COLAB_DEBUG_ADAPTER_MUX_PATH=/usr/local/bin/dap_multiplexer\n",
            "COLAB_LANGUAGE_SERVER_PROXY=<set>\n",
            "               CONDA_ROOT=/usr/local\n",
            "           CURL_CA_BUNDLE=<not set>\n",
            "          LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64\n",
            "               LD_PRELOAD=<not set>\n",
            "             LIBRARY_PATH=/usr/local/cuda/lib64/stubs\n",
            "                     PATH=/opt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/us\n",
            "                          r/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/googl\n",
            "                          e-cloud-sdk/bin\n",
            "               PYTHONPATH=\n",
            "           PYTHONWARNINGS=ignore:::pip._internal.cli.base_command\n",
            "       REQUESTS_CA_BUNDLE=<not set>\n",
            "            SSL_CERT_FILE=<not set>\n",
            "               TCLLIBPATH=/usr/share/tcltk/tcllib1.20\n",
            "\n",
            "     active environment : None\n",
            "       user config file : /root/.condarc\n",
            " populated config files : \n",
            "          conda version : 23.5.2\n",
            "    conda-build version : not installed\n",
            "         python version : 3.11.4.final.0\n",
            "       virtual packages : __archspec=1=x86_64\n",
            "                          __glibc=2.35=0\n",
            "                          __linux=5.15.109=0\n",
            "                          __unix=0=0\n",
            "       base environment : /usr/local  (writable)\n",
            "      conda av data dir : /usr/local/etc/conda\n",
            "  conda av metadata url : None\n",
            "           channel URLs : https://conda.anaconda.org/conda-forge/linux-64\n",
            "                          https://conda.anaconda.org/conda-forge/noarch\n",
            "                          https://repo.anaconda.com/pkgs/main/linux-64\n",
            "                          https://repo.anaconda.com/pkgs/main/noarch\n",
            "                          https://repo.anaconda.com/pkgs/r/linux-64\n",
            "                          https://repo.anaconda.com/pkgs/r/noarch\n",
            "          package cache : /usr/local/pkgs\n",
            "                          /root/.conda/pkgs\n",
            "       envs directories : /usr/local/envs\n",
            "                          /root/.conda/envs\n",
            "               platform : linux-64\n",
            "             user-agent : conda/23.5.2 requests/2.28.2 CPython/3.11.4 Linux/5.15.109+ ubuntu/22.04.2 glibc/2.35\n",
            "                UID:GID : 0:0\n",
            "             netrc file : None\n",
            "           offline mode : False\n",
            "\n",
            "\n",
            "An unexpected error has occurred. Conda has prepared the above report.\n",
            "\n",
            "Upload did not complete.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import scikits.umfpack as umfpack"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 322
        },
        "id": "1vJ0g7Jpw8uH",
        "outputId": "774674b0-c486-4448-ed37-0077f165da54"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "error",
          "ename": "ModuleNotFoundError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-9-ecfa3cd98e61>\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mscikits\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mumfpack\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mumfpack\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'scikits'",
            "",
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0;32m\nNOTE: If your import is failing due to a missing package, you can\nmanually install dependencies using either !pip or !apt.\n\nTo view examples of installing some common dependencies, click the\n\"Open Examples\" button below.\n\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n"
          ],
          "errorDetails": {
            "actions": [
              {
                "action": "open_url",
                "actionText": "Open Examples",
                "url": "/notebooks/snippets/importing_libraries.ipynb"
              }
            ]
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "check to get version number. if no version number somethign went wrong"
      ],
      "metadata": {
        "id": "X5VXy9ZRd8oZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!conda --version\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SHP1lj3pd8V0",
        "outputId": "91de7174-aed1-4de2-9b9c-b4f12100ab8d"
      },
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "conda 23.5.2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!which conda"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uEnEx0_-hkUy",
        "outputId": "91e587f0-e3a9-4bc6-d3fa-8fda8c63bf08"
      },
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/usr/local/bin/conda\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uHqbOP7W2jLE"
      },
      "source": [
        "# Appending to the sys.path"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import sys\n",
        "sys.path"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MbOxJEJweYyi",
        "outputId": "08c19b11-bdeb-4e72-8fe2-be883ac7d58f"
      },
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['/content',\n",
              " '/env/python',\n",
              " '/usr/lib/python310.zip',\n",
              " '/usr/lib/python3.10',\n",
              " '/usr/lib/python3.10/lib-dynload',\n",
              " '',\n",
              " '/usr/local/lib/python3.10/dist-packages',\n",
              " '/usr/lib/python3/dist-packages',\n",
              " '/usr/local/lib/python3.10/dist-packages/IPython/extensions',\n",
              " '/root/.ipython']"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!ls /usr/local/lib/python3.10/dist-packages"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SQe5SnT9iXuC",
        "outputId": "1fb20d1b-4bc9-45b8-f70e-c7f882123f27"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "absl\n",
            "absl_py-1.4.0.dist-info\n",
            "aiohttp\n",
            "aiohttp-3.8.5.dist-info\n",
            "aiosignal\n",
            "aiosignal-1.3.1.dist-info\n",
            "alabaster\n",
            "alabaster-0.7.13.dist-info\n",
            "albumentations\n",
            "albumentations-1.3.1.dist-info\n",
            "altair\n",
            "altair-4.2.2.dist-info\n",
            "annotated_types\n",
            "annotated_types-0.5.0.dist-info\n",
            "anyio\n",
            "anyio-3.7.1.dist-info\n",
            "apiclient\n",
            "appdirs-1.4.4.dist-info\n",
            "appdirs.py\n",
            "apt\n",
            "apt_inst.cpython-310-x86_64-linux-gnu.so\n",
            "apt_inst-stubs\n",
            "apt_pkg.cpython-310-x86_64-linux-gnu.so\n",
            "apt_pkg-stubs\n",
            "aptsources\n",
            "argon2\n",
            "argon2_cffi-23.1.0.dist-info\n",
            "_argon2_cffi_bindings\n",
            "argon2_cffi_bindings-21.2.0.dist-info\n",
            "array_record\n",
            "array_record-0.4.1.dist-info\n",
            "arviz\n",
            "arviz-0.15.1.dist-info\n",
            "astropy\n",
            "astropy-5.3.2.dist-info\n",
            "astunparse\n",
            "astunparse-1.6.3.dist-info\n",
            "async_timeout\n",
            "async_timeout-4.0.3.dist-info\n",
            "attr\n",
            "attrs\n",
            "attrs-23.1.0.dist-info\n",
            "audioread\n",
            "audioread-3.0.0.dist-info\n",
            "AUTHORS.rst\n",
            "autograd\n",
            "autograd-1.6.2.dist-info\n",
            "babel\n",
            "Babel-2.12.1.dist-info\n",
            "backcall\n",
            "backcall-0.2.0.dist-info\n",
            "beautifulsoup4-4.11.2.dist-info\n",
            "bin\n",
            "bleach\n",
            "bleach-6.0.0.dist-info\n",
            "blis\n",
            "blis-0.7.10.dist-info\n",
            "blosc2\n",
            "blosc2-2.0.0.dist-info\n",
            "bokeh\n",
            "bokeh-3.2.2.dist-info\n",
            "branca\n",
            "branca-0.6.0.dist-info\n",
            "bs4\n",
            "build\n",
            "build-0.10.0.dist-info\n",
            "cachecontrol\n",
            "cachecontrol-0.13.1.dist-info\n",
            "cachetools\n",
            "cachetools-5.3.1.dist-info\n",
            "catalogue\n",
            "catalogue-2.0.9.dist-info\n",
            "certifi\n",
            "certifi-2023.7.22.dist-info\n",
            "cffi\n",
            "cffi-1.15.1.dist-info\n",
            "_cffi_backend.cpython-310-x86_64-linux-gnu.so\n",
            "chardet\n",
            "chardet-5.2.0.dist-info\n",
            "charset_normalizer\n",
            "charset_normalizer-3.2.0.dist-info\n",
            "chex\n",
            "chex-0.1.7.dist-info\n",
            "clang\n",
            "click\n",
            "click-8.1.7.dist-info\n",
            "click_plugins\n",
            "click_plugins-1.1.1.dist-info\n",
            "cligj\n",
            "cligj-0.7.2.dist-info\n",
            "cloudpickle\n",
            "cloudpickle-2.2.1.dist-info\n",
            "cmake\n",
            "cmake-3.27.2.dist-info\n",
            "cmdstanpy\n",
            "cmdstanpy-1.1.0.dist-info\n",
            "colorcet\n",
            "colorcet-3.0.1.dist-info\n",
            "colorlover\n",
            "colorlover-0.3.0.dist-info\n",
            "community\n",
            "community-1.0.0b1.dist-info\n",
            "confection\n",
            "confection-0.1.1.dist-info\n",
            "cons\n",
            "cons-0.4.6.dist-info\n",
            "contextlib2\n",
            "contextlib2-21.6.0.dist-info\n",
            "contourpy\n",
            "contourpy-1.1.0.dist-info\n",
            "convertdate\n",
            "convertdate-2.4.0.dist-info\n",
            "cpuinfo\n",
            "cryptography\n",
            "cryptography-41.0.3.dist-info\n",
            "cufflinks\n",
            "cufflinks-0.17.3.dist-info\n",
            "cv2\n",
            "_cvxcore.cpython-310-x86_64-linux-gnu.so\n",
            "cvxopt\n",
            "cvxopt-1.3.2.dist-info\n",
            "cvxopt.libs\n",
            "cvxpy\n",
            "cvxpy-1.3.2.dist-info\n",
            "cycler-0.11.0.dist-info\n",
            "cycler.py\n",
            "cymem\n",
            "cymem-2.0.7.dist-info\n",
            "Cython\n",
            "Cython-0.29.36.dist-info\n",
            "cython.py\n",
            "dask\n",
            "dask-2023.8.1.dist-info\n",
            "datascience\n",
            "datascience-0.17.6.dist-info\n",
            "dateutil\n",
            "db_dtypes\n",
            "db_dtypes-1.1.1.dist-info\n",
            "debugpy\n",
            "debugpy-1.6.6.dist-info\n",
            "decorator-4.4.2.dist-info\n",
            "decorator.py\n",
            "defusedxml\n",
            "defusedxml-0.7.1.dist-info\n",
            "distributed\n",
            "distributed-2023.8.1.dist-info\n",
            "_distutils_hack\n",
            "distutils-precedence.pth\n",
            "dlib\n",
            "dlib-19.24.2.dist-info\n",
            "_dlib_pybind11.cpython-310-x86_64-linux-gnu.so\n",
            "dm_tree-0.1.8.dist-info\n",
            "doc\n",
            "docutils\n",
            "docutils-0.18.1.dist-info\n",
            "dopamine\n",
            "dopamine_rl-4.0.6.dist-info\n",
            "dot_parser.py\n",
            "duckdb-0.8.1.dist-info\n",
            "duckdb.cpython-310-x86_64-linux-gnu.so\n",
            "duckdb-stubs\n",
            "earthengine_api-0.1.364.dist-info\n",
            "easydict\n",
            "easydict-1.10.dist-info\n",
            "ecos\n",
            "ecos-2.0.12.dist-info\n",
            "_ecos.cpython-310-x86_64-linux-gnu.so\n",
            "editdistance\n",
            "editdistance-0.6.2.dist-info\n",
            "ee\n",
            "en_core_web_sm\n",
            "en_core_web_sm-3.6.0.dist-info\n",
            "entrypoints-0.4.dist-info\n",
            "entrypoints.py\n",
            "ephem\n",
            "ephem-4.1.4.dist-info\n",
            "erfa\n",
            "etils\n",
            "etils-1.4.1.dist-info\n",
            "etuples\n",
            "etuples-0.3.9.dist-info\n",
            "et_xmlfile\n",
            "et_xmlfile-1.1.0.dist-info\n",
            "exceptiongroup\n",
            "exceptiongroup-1.1.3.dist-info\n",
            "extension\n",
            "fastai\n",
            "fastai-2.7.12.dist-info\n",
            "fastcore\n",
            "fastcore-1.5.29.dist-info\n",
            "fastdownload\n",
            "fastdownload-0.0.7.dist-info\n",
            "fastjsonschema\n",
            "fastjsonschema-2.18.0.dist-info\n",
            "fastprogress\n",
            "fastprogress-1.0.3.dist-info\n",
            "fastrlock\n",
            "fastrlock-0.8.1.dist-info\n",
            "filelock\n",
            "filelock-3.12.2.dist-info\n",
            "fiona\n",
            "Fiona-1.9.4.post1.dist-info\n",
            "Fiona.libs\n",
            "firebase_admin\n",
            "firebase_admin-5.3.0.dist-info\n",
            "flask\n",
            "Flask-2.2.5.dist-info\n",
            "flatbuffers\n",
            "flatbuffers-23.5.26.dist-info\n",
            "flax\n",
            "flax-0.7.2.dist-info\n",
            "folium\n",
            "folium-0.14.0.dist-info\n",
            "fontTools\n",
            "fonttools-4.42.1.dist-info\n",
            "frozendict\n",
            "frozendict-2.3.8.dist-info\n",
            "frozenlist\n",
            "frozenlist-1.4.0.dist-info\n",
            "fsspec\n",
            "fsspec-2023.6.0.dist-info\n",
            "functorch\n",
            "future\n",
            "future-0.18.3.dist-info\n",
            "gast\n",
            "gast-0.4.0.dist-info\n",
            "gcsfs\n",
            "gcsfs-2023.6.0.dist-info\n",
            "GDAL-3.4.3.dist-info\n",
            "gdown\n",
            "gdown-4.6.6.dist-info\n",
            "gensim\n",
            "gensim-4.3.1.dist-info\n",
            "geographiclib\n",
            "geographiclib-2.0.dist-info\n",
            "geopandas\n",
            "geopandas-0.13.2.dist-info\n",
            "geopy\n",
            "geopy-2.3.0.dist-info\n",
            "gin\n",
            "gin_config-0.5.0.dist-info\n",
            "glob2\n",
            "glob2-0.7.dist-info\n",
            "google\n",
            "google-2.0.3.dist-info\n",
            "googleapiclient\n",
            "google_api_core-2.11.1.dist-info\n",
            "google_api_core-2.11.1-py3.9-nspkg.pth\n",
            "google_api_python_client-2.84.0.dist-info\n",
            "googleapis_common_protos-1.60.0.dist-info\n",
            "googleapis_common_protos-1.60.0-py3.9-nspkg.pth\n",
            "google_auth-2.17.3.dist-info\n",
            "google_auth-2.17.3-py3.9-nspkg.pth\n",
            "google_auth_httplib2-0.1.0.dist-info\n",
            "google_auth_httplib2.py\n",
            "google_auth_oauthlib\n",
            "google_auth_oauthlib-1.0.0.dist-info\n",
            "google_cloud_bigquery-3.10.0.dist-info\n",
            "google_cloud_bigquery-3.10.0-py3.9-nspkg.pth\n",
            "google_cloud_bigquery_connection-1.12.1.dist-info\n",
            "google_cloud_bigquery_connection-1.12.1-py3.9-nspkg.pth\n",
            "google_cloud_bigquery_storage-2.22.0.dist-info\n",
            "google_cloud_bigquery_storage-2.22.0-py3.9-nspkg.pth\n",
            "google_cloud_core-2.3.3.dist-info\n",
            "google_cloud_core-2.3.3-py3.9-nspkg.pth\n",
            "google_cloud_datastore-2.15.2.dist-info\n",
            "google_cloud_datastore-2.15.2-py3.9-nspkg.pth\n",
            "google_cloud_firestore-2.11.1.dist-info\n",
            "google_cloud_firestore-2.11.1-py3.9-nspkg.pth\n",
            "google_cloud_functions-1.13.2.dist-info\n",
            "google_cloud_functions-1.13.2-py3.9-nspkg.pth\n",
            "google_cloud_language-2.9.1.dist-info\n",
            "google_cloud_language-2.9.1-py3.9-nspkg.pth\n",
            "google_cloud_storage-2.8.0.dist-info\n",
            "google_cloud_storage-2.8.0-py3.9-nspkg.pth\n",
            "google_cloud_translate-3.11.3.dist-info\n",
            "google_cloud_translate-3.11.3-py3.9-nspkg.pth\n",
            "google_colab-1.0.0.dist-info\n",
            "google_colab-1.0.0-py3.10-nspkg.pth\n",
            "google_crc32c\n",
            "google_crc32c-1.5.0.dist-info\n",
            "google_crc32c.libs\n",
            "google_drive_downloader\n",
            "googledrivedownloader-0.4.dist-info\n",
            "google_pasta-0.2.0.dist-info\n",
            "google_resumable_media-2.5.0.dist-info\n",
            "google_resumable_media-2.5.0-py3.9-nspkg.pth\n",
            "googlesearch\n",
            "graphviz\n",
            "graphviz-0.20.1.dist-info\n",
            "greenlet\n",
            "greenlet-2.0.2.dist-info\n",
            "grpc\n",
            "grpc_google_iam_v1-0.12.6.dist-info\n",
            "grpc_google_iam_v1-0.12.6-py3.9-nspkg.pth\n",
            "grpcio-1.57.0.dist-info\n",
            "grpcio_status-1.48.2.dist-info\n",
            "grpc_status\n",
            "gspread\n",
            "gspread-3.4.2.dist-info\n",
            "gspread_dataframe-3.3.1.dist-info\n",
            "gspread_dataframe.py\n",
            "gym\n",
            "gym-0.25.2.dist-info\n",
            "gym_notices\n",
            "gym_notices-0.0.8.dist-info\n",
            "h5netcdf\n",
            "h5netcdf-1.2.0.dist-info\n",
            "h5py\n",
            "h5py-3.9.0.dist-info\n",
            "h5py.libs\n",
            "helper\n",
            "holidays\n",
            "holidays-0.31.dist-info\n",
            "holoviews\n",
            "holoviews-1.17.1.dist-info\n",
            "html5lib\n",
            "html5lib-1.1.dist-info\n",
            "httpimport-1.3.1.dist-info\n",
            "httpimport.py\n",
            "httplib2\n",
            "httplib2-0.22.0.dist-info\n",
            "humanize\n",
            "humanize-4.7.0.dist-info\n",
            "hyperopt\n",
            "hyperopt-0.2.7.dist-info\n",
            "idna\n",
            "idna-3.4.dist-info\n",
            "imageio\n",
            "imageio-2.31.1.dist-info\n",
            "imageio_ffmpeg\n",
            "imageio_ffmpeg-0.4.8.dist-info\n",
            "imagesize\n",
            "imagesize-1.4.1.dist-info\n",
            "imagesize.py\n",
            "imbalanced_learn-0.10.1.dist-info\n",
            "imblearn\n",
            "imgaug\n",
            "imgaug-0.4.0.dist-info\n",
            "importlib_metadata\n",
            "importlib_metadata-6.8.0.dist-info\n",
            "importlib_resources\n",
            "importlib_resources-6.0.1.dist-info\n",
            "imutils\n",
            "imutils-0.5.4.dist-info\n",
            "inflect\n",
            "inflect-7.0.0.dist-info\n",
            "iniconfig\n",
            "iniconfig-2.0.0.dist-info\n",
            "intel_openmp-2023.2.0.dist-info\n",
            "ipykernel\n",
            "ipykernel-5.5.6.dist-info\n",
            "ipykernel_launcher.py\n",
            "IPython\n",
            "ipython-7.34.0.dist-info\n",
            "ipython_genutils\n",
            "ipython_genutils-0.2.0.dist-info\n",
            "ipython_sql-0.5.0.dist-info\n",
            "ipywidgets\n",
            "ipywidgets-7.7.1.dist-info\n",
            "isympy.py\n",
            "itsdangerous\n",
            "itsdangerous-2.1.2.dist-info\n",
            "jax\n",
            "jax-0.4.14.dist-info\n",
            "jaxlib\n",
            "jaxlib-0.4.14+cuda11.cudnn86.dist-info\n",
            "jieba\n",
            "jieba-0.42.1.dist-info\n",
            "jinja2\n",
            "Jinja2-3.1.2.dist-info\n",
            "joblib\n",
            "joblib-1.3.2.dist-info\n",
            "jsonpickle\n",
            "jsonpickle-3.0.2.dist-info\n",
            "jsonschema\n",
            "jsonschema-4.19.0.dist-info\n",
            "jsonschema_specifications\n",
            "jsonschema_specifications-2023.7.1.dist-info\n",
            "jupyter_client\n",
            "jupyter_client-6.1.12.dist-info\n",
            "jupyter_console\n",
            "jupyter_console-6.1.0.dist-info\n",
            "jupyter_core\n",
            "jupyter_core-5.3.1.dist-info\n",
            "jupyterlab_plotly\n",
            "jupyterlab_pygments\n",
            "jupyterlab_pygments-0.2.2.dist-info\n",
            "jupyterlab_widgets\n",
            "jupyterlab_widgets-3.0.8.dist-info\n",
            "jupyter.py\n",
            "jupyter_server\n",
            "jupyter_server-1.24.0.dist-info\n",
            "kaggle\n",
            "kaggle-1.5.16.dist-info\n",
            "kanren\n",
            "keras\n",
            "keras-2.12.0.dist-info\n",
            "kiwisolver\n",
            "kiwisolver-1.4.4.dist-info\n",
            "langcodes\n",
            "langcodes-3.3.0.dist-info\n",
            "lazy_loader\n",
            "lazy_loader-0.3.dist-info\n",
            "libclang-16.0.6.dist-info\n",
            "libfuturize\n",
            "libpasteurize\n",
            "librosa\n",
            "librosa-0.10.1.dist-info\n",
            "LICENSE\n",
            "lightgbm\n",
            "lightgbm-4.0.0.dist-info\n",
            "linkify_it\n",
            "linkify_it_py-2.0.2.dist-info\n",
            "lit\n",
            "lit-16.0.6.dist-info\n",
            "llvmlite\n",
            "llvmlite-0.39.1.dist-info\n",
            "locket\n",
            "locket-1.0.0.dist-info\n",
            "logical_unification-0.4.6.dist-info\n",
            "lunarcalendar\n",
            "LunarCalendar-0.0.9.dist-info\n",
            "lxml\n",
            "lxml-4.9.3.dist-info\n",
            "markdown\n",
            "Markdown-3.4.4.dist-info\n",
            "markdown_it\n",
            "markdown_it_py-3.0.0.dist-info\n",
            "markupsafe\n",
            "MarkupSafe-2.1.3.dist-info\n",
            "matplotlib\n",
            "matplotlib-3.7.1.dist-info\n",
            "matplotlib-3.7.1-py3.10-nspkg.pth\n",
            "matplotlib_inline\n",
            "matplotlib_inline-0.1.6.dist-info\n",
            "matplotlib_venn\n",
            "matplotlib_venn-0.11.9.dist-info\n",
            "mdit_py_plugins\n",
            "mdit_py_plugins-0.4.0.dist-info\n",
            "mdurl\n",
            "mdurl-0.1.2.dist-info\n",
            "miniKanren-1.0.3.dist-info\n",
            "missingno\n",
            "missingno-0.5.2.dist-info\n",
            "mistune-0.8.4.dist-info\n",
            "mistune.py\n",
            "mizani\n",
            "mizani-0.9.2.dist-info\n",
            "mkl-2023.2.0.dist-info\n",
            "ml_dtypes\n",
            "ml_dtypes-0.2.0.dist-info\n",
            "mlxtend\n",
            "mlxtend-0.22.0.dist-info\n",
            "more_itertools\n",
            "more_itertools-10.1.0.dist-info\n",
            "moviepy\n",
            "moviepy-1.0.3.dist-info\n",
            "mpl_toolkits\n",
            "mpmath\n",
            "mpmath-1.3.0.dist-info\n",
            "msgpack\n",
            "msgpack-1.0.5.dist-info\n",
            "multidict\n",
            "multidict-6.0.4.dist-info\n",
            "multipledispatch\n",
            "multipledispatch-1.0.0.dist-info\n",
            "multitasking\n",
            "multitasking-0.0.11.dist-info\n",
            "murmurhash\n",
            "murmurhash-1.0.9.dist-info\n",
            "music21\n",
            "music21-9.1.0.dist-info\n",
            "natsort\n",
            "natsort-8.4.0.dist-info\n",
            "nbclassic\n",
            "nbclassic-1.0.0.dist-info\n",
            "nbclient\n",
            "nbclient-0.8.0.dist-info\n",
            "nbconvert\n",
            "nbconvert-6.5.4.dist-info\n",
            "nbformat\n",
            "nbformat-5.9.2.dist-info\n",
            "nest_asyncio-1.5.7.dist-info\n",
            "nest_asyncio.py\n",
            "networkx\n",
            "networkx-3.1.dist-info\n",
            "nibabel\n",
            "nibabel-4.0.2.dist-info\n",
            "nisext\n",
            "nltk\n",
            "nltk-3.8.1.dist-info\n",
            "notebook\n",
            "notebook-6.5.5.dist-info\n",
            "notebook_shim\n",
            "notebook_shim-0.2.3.dist-info\n",
            "numba\n",
            "numba-0.56.4.dist-info\n",
            "numbergen\n",
            "numexpr\n",
            "numexpr-2.8.5.dist-info\n",
            "numpy\n",
            "numpy-1.23.5.dist-info\n",
            "numpy.libs\n",
            "nvfuser\n",
            "oauth2client\n",
            "oauth2client-4.1.3.dist-info\n",
            "oauthlib\n",
            "oauthlib-3.2.2.dist-info\n",
            "opencv_contrib_python-4.8.0.76.dist-info\n",
            "opencv_contrib_python.libs\n",
            "opencv_python-4.8.0.76.dist-info\n",
            "opencv_python_headless-4.8.0.76.dist-info\n",
            "opencv_python_headless.libs\n",
            "opencv_python.libs\n",
            "OpenGL\n",
            "openpyxl\n",
            "openpyxl-3.1.2.dist-info\n",
            "OpenSSL\n",
            "optax\n",
            "optax-0.1.7.dist-info\n",
            "opt_einsum\n",
            "opt_einsum-3.3.0.dist-info\n",
            "orbax\n",
            "orbax_checkpoint-0.3.5.dist-info\n",
            "osgeo\n",
            "osgeo_utils\n",
            "osqp\n",
            "osqp-0.6.2.post8.dist-info\n",
            "osqppurepy\n",
            "packaging\n",
            "packaging-23.1.dist-info\n",
            "pandas\n",
            "pandas-1.5.3.dist-info\n",
            "pandas_datareader\n",
            "pandas_datareader-0.10.0.dist-info\n",
            "pandas_gbq\n",
            "pandas_gbq-0.17.9.dist-info\n",
            "pandocfilters-1.5.0.dist-info\n",
            "pandocfilters.py\n",
            "panel\n",
            "panel-1.2.1.dist-info\n",
            "param\n",
            "param-1.13.0.dist-info\n",
            "parso\n",
            "parso-0.8.3.dist-info\n",
            "partd\n",
            "partd-1.4.0.dist-info\n",
            "past\n",
            "pasta\n",
            "pathlib-1.0.1.dist-info\n",
            "pathlib.py\n",
            "pathy\n",
            "pathy-0.10.2.dist-info\n",
            "patsy\n",
            "patsy-0.5.3.dist-info\n",
            "pexpect\n",
            "pexpect-4.8.0.dist-info\n",
            "pickleshare-0.7.5.dist-info\n",
            "pickleshare.py\n",
            "PIL\n",
            "Pillow-9.4.0.dist-info\n",
            "Pillow.libs\n",
            "pip\n",
            "pip-23.1.2.dist-info\n",
            "piptools\n",
            "pip_tools-6.13.0.dist-info\n",
            "pkg_resources\n",
            "pkg_resources-stubs\n",
            "platformdirs\n",
            "platformdirs-3.10.0.dist-info\n",
            "plotly\n",
            "plotly-5.15.0.dist-info\n",
            "_plotly_future_\n",
            "_plotly_utils\n",
            "plotnine\n",
            "plotnine-0.12.2.dist-info\n",
            "pluggy\n",
            "pluggy-1.2.0.dist-info\n",
            "polars\n",
            "polars-0.17.3.dist-info\n",
            "pooch\n",
            "pooch-1.7.0.dist-info\n",
            "portpicker-1.5.2.dist-info\n",
            "portpicker.py\n",
            "prefetch_generator\n",
            "prefetch_generator-1.0.3.dist-info\n",
            "preshed\n",
            "preshed-3.0.8.dist-info\n",
            "prettytable\n",
            "prettytable-3.8.0.dist-info\n",
            "proglog\n",
            "proglog-0.1.10.dist-info\n",
            "progressbar\n",
            "progressbar2-4.2.0.dist-info\n",
            "prometheus_client\n",
            "prometheus_client-0.17.1.dist-info\n",
            "promise\n",
            "promise-2.3.dist-info\n",
            "prompt_toolkit\n",
            "prompt_toolkit-3.0.39.dist-info\n",
            "prophet\n",
            "prophet-1.1.4.dist-info\n",
            "prophet.libs\n",
            "proto\n",
            "protobuf-3.20.3.dist-info\n",
            "protobuf-3.20.3-py3.10-nspkg.pth\n",
            "proto_plus-1.22.3.dist-info\n",
            "psutil\n",
            "psutil-5.9.5.dist-info\n",
            "psycopg2\n",
            "psycopg2-2.9.7.dist-info\n",
            "ptyprocess\n",
            "ptyprocess-0.7.0.dist-info\n",
            "py4j\n",
            "py4j-0.10.9.7.dist-info\n",
            "pyarrow\n",
            "pyarrow-9.0.0.dist-info\n",
            "pyasn1\n",
            "pyasn1-0.5.0.dist-info\n",
            "pyasn1_modules\n",
            "pyasn1_modules-0.3.0.dist-info\n",
            "__pycache__\n",
            "pycocotools\n",
            "pycocotools-2.0.7.dist-info\n",
            "pycparser\n",
            "pycparser-2.21.dist-info\n",
            "py_cpuinfo-9.0.0.dist-info\n",
            "pyct\n",
            "pyct-0.5.0.dist-info\n",
            "pydantic\n",
            "pydantic-2.2.1.dist-info\n",
            "pydantic_core\n",
            "pydantic_core-2.6.1.dist-info\n",
            "pydata_google_auth\n",
            "pydata_google_auth-1.8.2.dist-info\n",
            "pydot-1.4.2.dist-info\n",
            "pydot_ng\n",
            "pydot_ng-2.0.0.dist-info\n",
            "pydotplus\n",
            "pydotplus-2.0.2.dist-info\n",
            "pydot.py\n",
            "pydrive\n",
            "PyDrive-1.3.1.dist-info\n",
            "pydrive2\n",
            "PyDrive2-1.6.3.dist-info\n",
            "pyduckdb\n",
            "pyerfa-2.0.0.3.dist-info\n",
            "pygame\n",
            "pygame-2.5.1.dist-info\n",
            "pygame.libs\n",
            "pygments\n",
            "Pygments-2.16.1.dist-info\n",
            "pylab.py\n",
            "pymc\n",
            "pymc-5.7.2.dist-info\n",
            "pymeeus\n",
            "PyMeeus-0.5.12.dist-info\n",
            "pymystem3\n",
            "pymystem3-0.2.0.dist-info\n",
            "PyOpenGL-3.1.7.dist-info\n",
            "pyOpenSSL-23.2.0.dist-info\n",
            "pyparsing\n",
            "pyparsing-3.1.1.dist-info\n",
            "pyproj\n",
            "pyproj-3.6.0.dist-info\n",
            "pyproject_hooks\n",
            "pyproject_hooks-1.0.0.dist-info\n",
            "pyproj.libs\n",
            "py.py\n",
            "PySocks-1.7.1.dist-info\n",
            "pytensor\n",
            "pytensor-2.14.2.dist-info\n",
            "_pytest\n",
            "pytest\n",
            "pytest-7.4.0.dist-info\n",
            "python_apt-0.0.0-py3.10.egg-info\n",
            "python_dateutil-2.8.2.dist-info\n",
            "python_louvain-0.16.dist-info\n",
            "python_slugify-8.0.1.dist-info\n",
            "python_utils\n",
            "python_utils-3.7.0.dist-info\n",
            "pytz\n",
            "pytz-2023.3.dist-info\n",
            "pyviz_comms\n",
            "pyviz_comms-3.0.0.dist-info\n",
            "PyWavelets-1.4.1.dist-info\n",
            "pywt\n",
            "pyximport\n",
            "PyYAML-6.0.1.dist-info\n",
            "pyzmq-23.2.1.dist-info\n",
            "pyzmq.libs\n",
            "qdldl-0.1.7.post0.dist-info\n",
            "qdldl.cpython-310-x86_64-linux-gnu.so\n",
            "qudida\n",
            "qudida-0.0.4.dist-info\n",
            "README.md\n",
            "referencing\n",
            "referencing-0.30.2.dist-info\n",
            "regex\n",
            "regex-2023.6.3.dist-info\n",
            "requests\n",
            "requests-2.31.0.dist-info\n",
            "requests_oauthlib\n",
            "requests_oauthlib-1.3.1.dist-info\n",
            "requirements\n",
            "requirements_parser-0.5.0.dist-info\n",
            "rich\n",
            "rich-13.5.2.dist-info\n",
            "_rinterface_cffi_abi.py\n",
            "_rinterface_cffi_api.abi3.so\n",
            "rpds\n",
            "rpds_py-0.9.2.dist-info\n",
            "rpy2\n",
            "rpy2-3.4.2.dist-info\n",
            "rsa\n",
            "rsa-4.9.dist-info\n",
            "scikit_image-0.19.3.dist-info\n",
            "scikit_image.libs\n",
            "scikit_learn-1.2.2.dist-info\n",
            "scikit_learn.libs\n",
            "scipy\n",
            "scipy-1.10.1.dist-info\n",
            "scipy.libs\n",
            "scs\n",
            "scs-3.2.3.dist-info\n",
            "_scs_direct.cpython-310-x86_64-linux-gnu.so\n",
            "_scs_indirect.cpython-310-x86_64-linux-gnu.so\n",
            "scs.libs\n",
            "seaborn\n",
            "seaborn-0.12.2.dist-info\n",
            "send2trash\n",
            "Send2Trash-1.8.2.dist-info\n",
            "setuptools\n",
            "setuptools-67.7.2.dist-info\n",
            "setuptools-stubs\n",
            "shapely\n",
            "shapely-2.0.1.dist-info\n",
            "shapely.libs\n",
            "six-1.16.0.dist-info\n",
            "six.py\n",
            "skimage\n",
            "sklearn\n",
            "sklearn_pandas\n",
            "sklearn_pandas-2.2.0.dist-info\n",
            "slugify\n",
            "smart_open\n",
            "smart_open-6.3.0.dist-info\n",
            "sniffio\n",
            "sniffio-1.3.0.dist-info\n",
            "snowballstemmer\n",
            "snowballstemmer-2.2.0.dist-info\n",
            "sockshandler.py\n",
            "socks.py\n",
            "sortedcontainers\n",
            "sortedcontainers-2.4.0.dist-info\n",
            "soundfile-0.12.1.dist-info\n",
            "_soundfile_data\n",
            "_soundfile.py\n",
            "soundfile.py\n",
            "soupsieve\n",
            "soupsieve-2.4.1.dist-info\n",
            "soxr\n",
            "soxr-0.3.6.dist-info\n",
            "spacy\n",
            "spacy-3.6.1.dist-info\n",
            "spacy_legacy\n",
            "spacy_legacy-3.0.12.dist-info\n",
            "spacy_loggers\n",
            "spacy_loggers-1.0.4.dist-info\n",
            "sphinx\n",
            "Sphinx-5.0.2.dist-info\n",
            "sphinxcontrib\n",
            "sphinxcontrib_applehelp-1.0.7.dist-info\n",
            "sphinxcontrib_devhelp-1.0.5.dist-info\n",
            "sphinxcontrib_htmlhelp-2.0.4.dist-info\n",
            "sphinxcontrib_jsmath-1.0.1.dist-info\n",
            "sphinxcontrib_jsmath-1.0.1-py3.7-nspkg.pth\n",
            "sphinxcontrib_qthelp-1.0.6.dist-info\n",
            "sphinxcontrib_serializinghtml-1.1.9.dist-info\n",
            "sql\n",
            "sqlalchemy\n",
            "SQLAlchemy-2.0.20.dist-info\n",
            "sqlparse\n",
            "sqlparse-0.4.4.dist-info\n",
            "srsly\n",
            "srsly-2.4.7.dist-info\n",
            "statsmodels\n",
            "statsmodels-0.14.0.dist-info\n",
            "sympy\n",
            "sympy-1.12.dist-info\n",
            "tables\n",
            "tables-3.8.0.dist-info\n",
            "tables.libs\n",
            "tabulate\n",
            "tabulate-0.9.0.dist-info\n",
            "tbb-2021.10.0.dist-info\n",
            "tblib\n",
            "tblib-2.0.0.dist-info\n",
            "tenacity\n",
            "tenacity-8.2.3.dist-info\n",
            "tensorboard\n",
            "tensorboard-2.12.3.dist-info\n",
            "tensorboard_data_server\n",
            "tensorboard_data_server-0.7.1.dist-info\n",
            "tensorflow\n",
            "tensorflow-2.12.0.dist-info\n",
            "tensorflow_datasets\n",
            "tensorflow_datasets-4.9.2.dist-info\n",
            "tensorflow_estimator\n",
            "tensorflow_estimator-2.12.0.dist-info\n",
            "tensorflow_gcs_config\n",
            "tensorflow_gcs_config-2.12.0.dist-info\n",
            "tensorflow_hub\n",
            "tensorflow_hub-0.14.0.dist-info\n",
            "tensorflow_io_gcs_filesystem\n",
            "tensorflow_io_gcs_filesystem-0.33.0.dist-info\n",
            "tensorflow_metadata\n",
            "tensorflow_metadata-1.14.0.dist-info\n",
            "tensorflow_probability\n",
            "tensorflow_probability-0.20.1.dist-info\n",
            "tensorstore\n",
            "tensorstore-0.1.41.dist-info\n",
            "termcolor\n",
            "termcolor-2.3.0.dist-info\n",
            "terminado\n",
            "terminado-0.17.1.dist-info\n",
            "textblob\n",
            "textblob-0.17.1.dist-info\n",
            "text_unidecode\n",
            "text_unidecode-1.3.dist-info\n",
            "tf_slim\n",
            "tf_slim-1.1.0.dist-info\n",
            "thinc\n",
            "thinc-8.1.12.dist-info\n",
            "third_party\n",
            "threadpoolctl-3.2.0.dist-info\n",
            "threadpoolctl.py\n",
            "tifffile\n",
            "tifffile-2023.8.12.dist-info\n",
            "tinycss2\n",
            "tinycss2-1.2.1.dist-info\n",
            "tlz\n",
            "toml\n",
            "toml-0.10.2.dist-info\n",
            "tomli\n",
            "tomli-2.0.1.dist-info\n",
            "toolz\n",
            "toolz-0.12.0.dist-info\n",
            "torch\n",
            "torch-2.0.1+cu118.dist-info\n",
            "torchaudio\n",
            "torchaudio-2.0.2+cu118.dist-info\n",
            "torchdata\n",
            "torchdata-0.6.1.dist-info\n",
            "torchgen\n",
            "torchsummary\n",
            "torchsummary-1.5.1.dist-info\n",
            "torchtext\n",
            "torchtext-0.15.2.dist-info\n",
            "torchvision\n",
            "torchvision-0.15.2+cu118.dist-info\n",
            "torchvision.libs\n",
            "tornado\n",
            "tornado-6.3.2.dist-info\n",
            "tqdm\n",
            "tqdm-4.66.1.dist-info\n",
            "traitlets\n",
            "traitlets-5.7.1.dist-info\n",
            "tree\n",
            "triton\n",
            "triton-2.0.0.dist-info\n",
            "tweepy\n",
            "tweepy-4.13.0.dist-info\n",
            "typer\n",
            "typer-0.9.0.dist-info\n",
            "types_setuptools-68.1.0.0.dist-info\n",
            "typing_extensions-4.7.1.dist-info\n",
            "typing_extensions.py\n",
            "tzlocal\n",
            "tzlocal-5.0.1.dist-info\n",
            "uc_micro\n",
            "uc_micro_py-1.0.2.dist-info\n",
            "unification\n",
            "uritemplate\n",
            "uritemplate-4.1.1.dist-info\n",
            "urllib3\n",
            "urllib3-2.0.4.dist-info\n",
            "vega_datasets\n",
            "vega_datasets-0.9.0.dist-info\n",
            "wasabi\n",
            "wasabi-1.1.2.dist-info\n",
            "wcwidth\n",
            "wcwidth-0.2.6.dist-info\n",
            "webcolors\n",
            "webcolors-1.13.dist-info\n",
            "webencodings\n",
            "webencodings-0.5.1.dist-info\n",
            "websocket\n",
            "websocket_client-1.6.2.dist-info\n",
            "werkzeug\n",
            "werkzeug-2.3.7.dist-info\n",
            "wheel\n",
            "wheel-0.41.2.dist-info\n",
            "widgetsnbextension\n",
            "widgetsnbextension-3.6.5.dist-info\n",
            "wordcloud\n",
            "wordcloud-1.9.2.dist-info\n",
            "wrapt\n",
            "wrapt-1.14.1.dist-info\n",
            "xarray\n",
            "xarray-2023.7.0.dist-info\n",
            "xarray_einstats\n",
            "xarray_einstats-0.6.0.dist-info\n",
            "xgboost\n",
            "xgboost-1.7.6.dist-info\n",
            "xgboost.libs\n",
            "xlrd\n",
            "xlrd-2.0.1.dist-info\n",
            "xyzservices\n",
            "xyzservices-2023.7.0.dist-info\n",
            "_yaml\n",
            "yaml\n",
            "yarl\n",
            "yarl-1.9.2.dist-info\n",
            "yellowbrick\n",
            "yellowbrick-1.5.dist-info\n",
            "yfinance\n",
            "yfinance-0.2.28.dist-info\n",
            "zict\n",
            "zict-3.0.0.dist-info\n",
            "zipp\n",
            "zipp-3.16.2.dist-info\n",
            "zmq\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import sys\n",
        "sys.path.append(\"/usr/local/lib/python3.10/site-packages\")"
      ],
      "metadata": {
        "id": "mMC6F4VOl_bM"
      },
      "execution_count": 55,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9mNCmszn11DF"
      },
      "source": [
        "### Install packages"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q condacolab\n",
        "import condacolab\n",
        "condacolab.install()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 395
        },
        "id": "2VPdGrc9sDQ9",
        "outputId": "118bab43-6202-4b4c-ceb8-1b39a0bfed08"
      },
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m"
          ]
        },
        {
          "output_type": "error",
          "ename": "ModuleNotFoundError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-71-dfa229587fce>\u001b[0m in \u001b[0;36m<cell line: 2>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msystem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'pip install -q condacolab'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mcondacolab\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mcondacolab\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minstall\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'condacolab'",
            "",
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0;32m\nNOTE: If your import is failing due to a missing package, you can\nmanually install dependencies using either !pip or !apt.\n\nTo view examples of installing some common dependencies, click the\n\"Open Examples\" button below.\n\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n"
          ],
          "errorDetails": {
            "actions": [
              {
                "action": "open_url",
                "actionText": "Open Examples",
                "url": "/notebooks/snippets/importing_libraries.ipynb"
              }
            ]
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N48wGB_QTMbw"
      },
      "source": [
        "Install scikit-umfpack for faster solutions of sparse linear systems"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OO3QiSdbTL0V",
        "outputId": "b0437259-484d-4cce-cf4e-aecb08aecbda"
      },
      "source": [
        "! conda install scikit-umfpack --yes\n",
        "#! conda install -q -y --prefix /usr/local python=3.10 scikit-umfpack"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<\n",
            "\n",
            "    Traceback (most recent call last):\n",
            "      File \"/usr/local/lib/python3.11/site-packages/requests/compat.py\", line 11, in <module>\n",
            "        import chardet\n",
            "    ModuleNotFoundError: No module named 'chardet'\n",
            "    \n",
            "    During handling of the above exception, another exception occurred:\n",
            "    \n",
            "    Traceback (most recent call last):\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/exception_handler.py\", line 16, in __call__\n",
            "        return func(*args, **kwargs)\n",
            "               ^^^^^^^^^^^^^^^^^^^^^\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/cli/main.py\", line 84, in main_subshell\n",
            "        exit_code = do_call(args, p)\n",
            "                    ^^^^^^^^^^^^^^^^\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/cli/conda_argparse.py\", line 124, in do_call\n",
            "        module = import_module(relative_mod, __name__.rsplit(\".\", 1)[0])\n",
            "                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "      File \"/usr/local/lib/python3.11/importlib/__init__.py\", line 126, in import_module\n",
            "        return _bootstrap._gcd_import(name[level:], package, level)\n",
            "               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "      File \"<frozen importlib._bootstrap>\", line 1204, in _gcd_import\n",
            "      File \"<frozen importlib._bootstrap>\", line 1176, in _find_and_load\n",
            "      File \"<frozen importlib._bootstrap>\", line 1147, in _find_and_load_unlocked\n",
            "      File \"<frozen importlib._bootstrap>\", line 690, in _load_unlocked\n",
            "      File \"<frozen importlib._bootstrap_external>\", line 940, in exec_module\n",
            "      File \"<frozen importlib._bootstrap>\", line 241, in _call_with_frames_removed\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/cli/main_install.py\", line 6, in <module>\n",
            "        from ..notices import notices\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/notices/__init__.py\", line 3, in <module>\n",
            "        from .core import notices  # noqa: F401\n",
            "        ^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/notices/core.py\", line 13, in <module>\n",
            "        from . import cache, fetch, views\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/notices/fetch.py\", line 9, in <module>\n",
            "        import requests\n",
            "      File \"/usr/local/lib/python3.11/site-packages/requests/__init__.py\", line 45, in <module>\n",
            "        from .exceptions import RequestsDependencyWarning\n",
            "      File \"/usr/local/lib/python3.11/site-packages/requests/exceptions.py\", line 9, in <module>\n",
            "        from .compat import JSONDecodeError as CompatJSONDecodeError\n",
            "      File \"/usr/local/lib/python3.11/site-packages/requests/compat.py\", line 13, in <module>\n",
            "        import charset_normalizer as chardet\n",
            "    ModuleNotFoundError: No module named 'charset_normalizer'\n",
            "\n",
            "`$ /usr/local/bin/conda install scikit-umfpack --yes`\n",
            "\n",
            "  environment variables:\n",
            "                 CIO_TEST=<not set>\n",
            "COLAB_DEBUG_ADAPTER_MUX_PATH=/usr/local/bin/dap_multiplexer\n",
            "COLAB_LANGUAGE_SERVER_PROXY=<set>\n",
            "               CONDA_ROOT=/usr/local\n",
            "           CURL_CA_BUNDLE=<not set>\n",
            "          LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64\n",
            "               LD_PRELOAD=<not set>\n",
            "             LIBRARY_PATH=/usr/local/cuda/lib64/stubs\n",
            "                     PATH=/opt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/us\n",
            "                          r/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/googl\n",
            "                          e-cloud-sdk/bin\n",
            "               PYTHONPATH=\n",
            "           PYTHONWARNINGS=ignore:::pip._internal.cli.base_command\n",
            "       REQUESTS_CA_BUNDLE=<not set>\n",
            "            SSL_CERT_FILE=<not set>\n",
            "               TCLLIBPATH=/usr/share/tcltk/tcllib1.20\n",
            "\n",
            "     active environment : None\n",
            "       user config file : /root/.condarc\n",
            " populated config files : \n",
            "          conda version : 23.5.2\n",
            "    conda-build version : not installed\n",
            "         python version : 3.11.4.final.0\n",
            "       virtual packages : __archspec=1=x86_64\n",
            "                          __glibc=2.35=0\n",
            "                          __linux=5.15.109=0\n",
            "                          __unix=0=0\n",
            "       base environment : /usr/local  (writable)\n",
            "      conda av data dir : /usr/local/etc/conda\n",
            "  conda av metadata url : None\n",
            "           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64\n",
            "                          https://repo.anaconda.com/pkgs/main/noarch\n",
            "                          https://repo.anaconda.com/pkgs/r/linux-64\n",
            "                          https://repo.anaconda.com/pkgs/r/noarch\n",
            "          package cache : /usr/local/pkgs\n",
            "                          /root/.conda/pkgs\n",
            "       envs directories : /usr/local/envs\n",
            "                          /root/.conda/envs\n",
            "               platform : linux-64\n",
            "             user-agent : conda/23.5.2 requests/2.28.2 CPython/3.11.4 Linux/5.15.109+ ubuntu/22.04.2 glibc/2.35\n",
            "                UID:GID : 0:0\n",
            "             netrc file : None\n",
            "           offline mode : False\n",
            "\n",
            "\n",
            "An unexpected error has occurred. Conda has prepared the above report.\n",
            "\n",
            "Upload did not complete.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kH3aBOF4LyI1"
      },
      "source": [
        "Install METIS for better row-ordering during matrix factorization (greatly reduces memory usage)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qcgGfSFbLzfs",
        "outputId": "603aa850-1d2f-4929-fe91-7dc989973051"
      },
      "source": [
        "! conda install metis --yes\n",
        "#! conda install -q -y --prefix /usr/local python=3.10 metis"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<\n",
            "\n",
            "    Traceback (most recent call last):\n",
            "      File \"/usr/local/lib/python3.11/site-packages/requests/compat.py\", line 11, in <module>\n",
            "        import chardet\n",
            "    ModuleNotFoundError: No module named 'chardet'\n",
            "    \n",
            "    During handling of the above exception, another exception occurred:\n",
            "    \n",
            "    Traceback (most recent call last):\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/exception_handler.py\", line 16, in __call__\n",
            "        return func(*args, **kwargs)\n",
            "               ^^^^^^^^^^^^^^^^^^^^^\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/cli/main.py\", line 84, in main_subshell\n",
            "        exit_code = do_call(args, p)\n",
            "                    ^^^^^^^^^^^^^^^^\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/cli/conda_argparse.py\", line 124, in do_call\n",
            "        module = import_module(relative_mod, __name__.rsplit(\".\", 1)[0])\n",
            "                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "      File \"/usr/local/lib/python3.11/importlib/__init__.py\", line 126, in import_module\n",
            "        return _bootstrap._gcd_import(name[level:], package, level)\n",
            "               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "      File \"<frozen importlib._bootstrap>\", line 1204, in _gcd_import\n",
            "      File \"<frozen importlib._bootstrap>\", line 1176, in _find_and_load\n",
            "      File \"<frozen importlib._bootstrap>\", line 1147, in _find_and_load_unlocked\n",
            "      File \"<frozen importlib._bootstrap>\", line 690, in _load_unlocked\n",
            "      File \"<frozen importlib._bootstrap_external>\", line 940, in exec_module\n",
            "      File \"<frozen importlib._bootstrap>\", line 241, in _call_with_frames_removed\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/cli/main_install.py\", line 6, in <module>\n",
            "        from ..notices import notices\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/notices/__init__.py\", line 3, in <module>\n",
            "        from .core import notices  # noqa: F401\n",
            "        ^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/notices/core.py\", line 13, in <module>\n",
            "        from . import cache, fetch, views\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/notices/fetch.py\", line 9, in <module>\n",
            "        import requests\n",
            "      File \"/usr/local/lib/python3.11/site-packages/requests/__init__.py\", line 45, in <module>\n",
            "        from .exceptions import RequestsDependencyWarning\n",
            "      File \"/usr/local/lib/python3.11/site-packages/requests/exceptions.py\", line 9, in <module>\n",
            "        from .compat import JSONDecodeError as CompatJSONDecodeError\n",
            "      File \"/usr/local/lib/python3.11/site-packages/requests/compat.py\", line 13, in <module>\n",
            "        import charset_normalizer as chardet\n",
            "    ModuleNotFoundError: No module named 'charset_normalizer'\n",
            "\n",
            "`$ /usr/local/bin/conda install metis --yes`\n",
            "\n",
            "  environment variables:\n",
            "                 CIO_TEST=<not set>\n",
            "COLAB_DEBUG_ADAPTER_MUX_PATH=/usr/local/bin/dap_multiplexer\n",
            "COLAB_LANGUAGE_SERVER_PROXY=<set>\n",
            "               CONDA_ROOT=/usr/local\n",
            "           CURL_CA_BUNDLE=<not set>\n",
            "          LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64\n",
            "               LD_PRELOAD=<not set>\n",
            "             LIBRARY_PATH=/usr/local/cuda/lib64/stubs\n",
            "                     PATH=/opt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/us\n",
            "                          r/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/googl\n",
            "                          e-cloud-sdk/bin\n",
            "               PYTHONPATH=\n",
            "           PYTHONWARNINGS=ignore:::pip._internal.cli.base_command\n",
            "       REQUESTS_CA_BUNDLE=<not set>\n",
            "            SSL_CERT_FILE=<not set>\n",
            "               TCLLIBPATH=/usr/share/tcltk/tcllib1.20\n",
            "\n",
            "     active environment : None\n",
            "       user config file : /root/.condarc\n",
            " populated config files : \n",
            "          conda version : 23.5.2\n",
            "    conda-build version : not installed\n",
            "         python version : 3.11.4.final.0\n",
            "       virtual packages : __archspec=1=x86_64\n",
            "                          __glibc=2.35=0\n",
            "                          __linux=5.15.109=0\n",
            "                          __unix=0=0\n",
            "       base environment : /usr/local  (writable)\n",
            "      conda av data dir : /usr/local/etc/conda\n",
            "  conda av metadata url : None\n",
            "           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64\n",
            "                          https://repo.anaconda.com/pkgs/main/noarch\n",
            "                          https://repo.anaconda.com/pkgs/r/linux-64\n",
            "                          https://repo.anaconda.com/pkgs/r/noarch\n",
            "          package cache : /usr/local/pkgs\n",
            "                          /root/.conda/pkgs\n",
            "       envs directories : /usr/local/envs\n",
            "                          /root/.conda/envs\n",
            "               platform : linux-64\n",
            "             user-agent : conda/23.5.2 requests/2.28.2 CPython/3.11.4 Linux/5.15.109+ ubuntu/22.04.2 glibc/2.35\n",
            "                UID:GID : 0:0\n",
            "             netrc file : None\n",
            "           offline mode : False\n",
            "\n",
            "\n",
            "An unexpected error has occurred. Conda has prepared the above report.\n",
            "\n",
            "Upload did not complete.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! conda install -c conda-forge netcdf4 --yes\n",
        "#! conda install -q -y --prefix /usr/local python=3.7 netcdf4"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ssTD42kl-vYC",
        "outputId": "36205005-aabf-4618-dd22-debe4b6aedcd"
      },
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<\n",
            "\n",
            "    Traceback (most recent call last):\n",
            "      File \"/usr/local/lib/python3.11/site-packages/requests/compat.py\", line 11, in <module>\n",
            "        import chardet\n",
            "    ModuleNotFoundError: No module named 'chardet'\n",
            "    \n",
            "    During handling of the above exception, another exception occurred:\n",
            "    \n",
            "    Traceback (most recent call last):\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/exception_handler.py\", line 16, in __call__\n",
            "        return func(*args, **kwargs)\n",
            "               ^^^^^^^^^^^^^^^^^^^^^\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/cli/main.py\", line 84, in main_subshell\n",
            "        exit_code = do_call(args, p)\n",
            "                    ^^^^^^^^^^^^^^^^\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/cli/conda_argparse.py\", line 124, in do_call\n",
            "        module = import_module(relative_mod, __name__.rsplit(\".\", 1)[0])\n",
            "                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "      File \"/usr/local/lib/python3.11/importlib/__init__.py\", line 126, in import_module\n",
            "        return _bootstrap._gcd_import(name[level:], package, level)\n",
            "               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "      File \"<frozen importlib._bootstrap>\", line 1204, in _gcd_import\n",
            "      File \"<frozen importlib._bootstrap>\", line 1176, in _find_and_load\n",
            "      File \"<frozen importlib._bootstrap>\", line 1147, in _find_and_load_unlocked\n",
            "      File \"<frozen importlib._bootstrap>\", line 690, in _load_unlocked\n",
            "      File \"<frozen importlib._bootstrap_external>\", line 940, in exec_module\n",
            "      File \"<frozen importlib._bootstrap>\", line 241, in _call_with_frames_removed\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/cli/main_install.py\", line 6, in <module>\n",
            "        from ..notices import notices\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/notices/__init__.py\", line 3, in <module>\n",
            "        from .core import notices  # noqa: F401\n",
            "        ^^^^^^^^^^^^^^^^^^^^^^^^^\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/notices/core.py\", line 13, in <module>\n",
            "        from . import cache, fetch, views\n",
            "      File \"/usr/local/lib/python3.11/site-packages/conda/notices/fetch.py\", line 9, in <module>\n",
            "        import requests\n",
            "      File \"/usr/local/lib/python3.11/site-packages/requests/__init__.py\", line 45, in <module>\n",
            "        from .exceptions import RequestsDependencyWarning\n",
            "      File \"/usr/local/lib/python3.11/site-packages/requests/exceptions.py\", line 9, in <module>\n",
            "        from .compat import JSONDecodeError as CompatJSONDecodeError\n",
            "      File \"/usr/local/lib/python3.11/site-packages/requests/compat.py\", line 13, in <module>\n",
            "        import charset_normalizer as chardet\n",
            "    ModuleNotFoundError: No module named 'charset_normalizer'\n",
            "\n",
            "`$ /usr/local/bin/conda install -c conda-forge netcdf4 --yes`\n",
            "\n",
            "  environment variables:\n",
            "                 CIO_TEST=<not set>\n",
            "COLAB_DEBUG_ADAPTER_MUX_PATH=/usr/local/bin/dap_multiplexer\n",
            "COLAB_LANGUAGE_SERVER_PROXY=<set>\n",
            "               CONDA_ROOT=/usr/local\n",
            "           CURL_CA_BUNDLE=<not set>\n",
            "          LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64\n",
            "               LD_PRELOAD=<not set>\n",
            "             LIBRARY_PATH=/usr/local/cuda/lib64/stubs\n",
            "                     PATH=/opt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/us\n",
            "                          r/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/googl\n",
            "                          e-cloud-sdk/bin\n",
            "               PYTHONPATH=\n",
            "           PYTHONWARNINGS=ignore:::pip._internal.cli.base_command\n",
            "       REQUESTS_CA_BUNDLE=<not set>\n",
            "            SSL_CERT_FILE=<not set>\n",
            "               TCLLIBPATH=/usr/share/tcltk/tcllib1.20\n",
            "\n",
            "     active environment : None\n",
            "       user config file : /root/.condarc\n",
            " populated config files : \n",
            "          conda version : 23.5.2\n",
            "    conda-build version : not installed\n",
            "         python version : 3.11.4.final.0\n",
            "       virtual packages : __archspec=1=x86_64\n",
            "                          __glibc=2.35=0\n",
            "                          __linux=5.15.109=0\n",
            "                          __unix=0=0\n",
            "       base environment : /usr/local  (writable)\n",
            "      conda av data dir : /usr/local/etc/conda\n",
            "  conda av metadata url : None\n",
            "           channel URLs : https://conda.anaconda.org/conda-forge/linux-64\n",
            "                          https://conda.anaconda.org/conda-forge/noarch\n",
            "                          https://repo.anaconda.com/pkgs/main/linux-64\n",
            "                          https://repo.anaconda.com/pkgs/main/noarch\n",
            "                          https://repo.anaconda.com/pkgs/r/linux-64\n",
            "                          https://repo.anaconda.com/pkgs/r/noarch\n",
            "          package cache : /usr/local/pkgs\n",
            "                          /root/.conda/pkgs\n",
            "       envs directories : /usr/local/envs\n",
            "                          /root/.conda/envs\n",
            "               platform : linux-64\n",
            "             user-agent : conda/23.5.2 requests/2.28.2 CPython/3.11.4 Linux/5.15.109+ ubuntu/22.04.2 glibc/2.35\n",
            "                UID:GID : 0:0\n",
            "             netrc file : None\n",
            "           offline mode : False\n",
            "\n",
            "\n",
            "An unexpected error has occurred. Conda has prepared the above report.\n",
            "\n",
            "Upload did not complete.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#! conda install -c conda-forge gsw\n",
        "#! conda install -q -y --prefix /usr/local python=3.7 gsw"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lKF259S4VojF",
        "outputId": "610ff321-2814-4dbe-866e-b27045da0b2d"
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting package metadata (current_repodata.json): ...working... done\n",
            "Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.\n",
            "Collecting package metadata (repodata.json): ...working... done\n",
            "Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.\n",
            "\n",
            "PackagesNotFoundError: The following packages are not available from current channels:\n",
            "\n",
            "  - gsw\n",
            "\n",
            "Current channels:\n",
            "\n",
            "  - https://repo.anaconda.com/pkgs/main/linux-64\n",
            "  - https://repo.anaconda.com/pkgs/main/noarch\n",
            "  - https://repo.anaconda.com/pkgs/r/linux-64\n",
            "  - https://repo.anaconda.com/pkgs/r/noarch\n",
            "\n",
            "To search for alternate channels that may provide the conda package you're\n",
            "looking for, navigate to\n",
            "\n",
            "    https://anaconda.org\n",
            "\n",
            "and use the search bar at the top of the page.\n",
            "\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IYftq3CqdvzE"
      },
      "source": [
        "Install pyompa for conducting OMP analysis (pyompa was developed by the Casciotti lab and supports several capabilities beyond traditional OMP analysis)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o7MJFZKtvRdV",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "135fdb62-20fc-43c1-9a98-a31e3eb18161"
      },
      "source": [
        "!pip uninstall -y pyompa\n",
        "%cd /content/\n",
        "!rm -rf pyompa\n",
        "!git clone https://github.com/nitrogenlab/pyompa\n",
        "%cd /content/pyompa\n",
        "!git checkout dev #install from the dev branch\n",
        "!git log -1\n",
        "!pip install .\n",
        "%cd /content/"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[33mWARNING: Skipping pyompa as it is not installed.\u001b[0m\u001b[33m\n",
            "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m/content\n",
            "Cloning into 'pyompa'...\n",
            "remote: Enumerating objects: 1408, done.\u001b[K\n",
            "remote: Counting objects: 100% (164/164), done.\u001b[K\n",
            "remote: Compressing objects: 100% (70/70), done.\u001b[K\n",
            "remote: Total 1408 (delta 160), reused 94 (delta 94), pack-reused 1244\u001b[K\n",
            "Receiving objects: 100% (1408/1408), 16.71 MiB | 10.07 MiB/s, done.\n",
            "Resolving deltas: 100% (999/999), done.\n",
            "/content/pyompa\n",
            "Branch 'dev' set up to track remote branch 'dev' from 'origin'.\n",
            "Switched to a new branch 'dev'\n",
            "\u001b[33mcommit e1490ff156ed8db745c73abf8daa9ea4aa3ac3b5\u001b[m\u001b[33m (\u001b[m\u001b[1;36mHEAD -> \u001b[m\u001b[1;32mdev\u001b[m\u001b[33m, \u001b[m\u001b[1;31morigin/dev\u001b[m\u001b[33m)\u001b[m\n",
            "Author: Av Shrikumar <avanti.shrikumar@gmail.com>\n",
            "Date:   Mon Feb 28 08:58:09 2022 -0700\n",
            "\n",
            "    updated with total parameter residual plots\n",
            "Processing /content/pyompa\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting numpy (from pyompa==0.4.2.0)\n",
            "  Downloading numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m18.2/18.2 MB\u001b[0m \u001b[31m22.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pandas (from pyompa==0.4.2.0)\n",
            "  Downloading pandas-2.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.2/12.2 MB\u001b[0m \u001b[31m24.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting cvxpy (from pyompa==0.4.2.0)\n",
            "  Downloading cvxpy-1.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.0/4.0 MB\u001b[0m \u001b[31m23.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting scipy (from pyompa==0.4.2.0)\n",
            "  Downloading scipy-1.11.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m36.3/36.3 MB\u001b[0m \u001b[31m11.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting toml (from pyompa==0.4.2.0)\n",
            "  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)\n",
            "Collecting osqp>=0.4.1 (from cvxpy->pyompa==0.4.2.0)\n",
            "  Downloading osqp-0.6.3-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (299 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m299.6/299.6 kB\u001b[0m \u001b[31m28.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting ecos>=2 (from cvxpy->pyompa==0.4.2.0)\n",
            "  Downloading ecos-2.0.12-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (221 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m221.6/221.6 kB\u001b[0m \u001b[31m21.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting scs>=1.1.6 (from cvxpy->pyompa==0.4.2.0)\n",
            "  Downloading scs-3.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.7/10.7 MB\u001b[0m \u001b[31m94.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: setuptools>65.5.1 in /usr/local/lib/python3.11/site-packages (from cvxpy->pyompa==0.4.2.0) (67.8.0)\n",
            "Collecting python-dateutil>=2.8.2 (from pandas->pyompa==0.4.2.0)\n",
            "  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)\n",
            "Collecting pytz>=2020.1 (from pandas->pyompa==0.4.2.0)\n",
            "  Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)\n",
            "Collecting tzdata>=2022.1 (from pandas->pyompa==0.4.2.0)\n",
            "  Downloading tzdata-2023.3-py2.py3-none-any.whl (341 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m341.8/341.8 kB\u001b[0m \u001b[31m30.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting qdldl (from osqp>=0.4.1->cvxpy->pyompa==0.4.2.0)\n",
            "  Downloading qdldl-0.1.7.post0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m44.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting six>=1.5 (from python-dateutil>=2.8.2->pandas->pyompa==0.4.2.0)\n",
            "  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)\n",
            "Building wheels for collected packages: pyompa\n",
            "  Building wheel for pyompa (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyompa: filename=pyompa-0.4.2.0-py3-none-any.whl size=27924 sha256=b03b47270088c674c8d8f7f93d7a03ef73a62da082be7a6be11bdb57f994485a\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-kf6uczg1/wheels/00/43/7a/11dc5b95e80cdf66fe99b9661c1e139564f963a36893458602\n",
            "Successfully built pyompa\n",
            "Installing collected packages: pytz, tzdata, toml, six, numpy, scipy, python-dateutil, scs, qdldl, pandas, ecos, osqp, cvxpy, pyompa\n",
            "Successfully installed cvxpy-1.3.2 ecos-2.0.12 numpy-1.25.2 osqp-0.6.3 pandas-2.0.3 pyompa-0.4.2.0 python-dateutil-2.8.2 pytz-2023.3 qdldl-0.1.7.post0 scipy-1.11.2 scs-3.2.3 six-1.16.0 toml-0.10.2 tzdata-2023.3\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "dateutil",
                  "six"
                ]
              }
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k7t0-15le5kR"
      },
      "source": [
        "Install and load the gp15wma module, which has configuration settings for the GP15 OMP analysis as well as other handy functions for loading the data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xswiz0DqfBYs",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "458e0fae-0406-487b-8e71-adcde293feb3"
      },
      "source": [
        "!pip uninstall -y GP02wma\n",
        "%cd /content/\n",
        "!rm -rf gp15wmascripts\n",
        "!git clone https://github.com/nitrogenlab/gp15wmascripts\n",
        "%cd /content/gp15wmascripts\n",
        "!git log -1\n",
        "!pip install .\n",
        "%cd /content/"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[33mWARNING: Skipping GP02wma as it is not installed.\u001b[0m\u001b[33m\n",
            "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m/content\n",
            "Cloning into 'gp15wmascripts'...\n",
            "remote: Enumerating objects: 316, done.\u001b[K\n",
            "remote: Counting objects: 100% (90/90), done.\u001b[K\n",
            "remote: Compressing objects: 100% (85/85), done.\u001b[K\n",
            "remote: Total 316 (delta 53), reused 3 (delta 3), pack-reused 226\u001b[K\n",
            "Receiving objects: 100% (316/316), 16.58 MiB | 17.52 MiB/s, done.\n",
            "Resolving deltas: 100% (177/177), done.\n",
            "/content/gp15wmascripts\n",
            "\u001b[33mcommit 1cb48fba3bebcf2234e6bad60ff58880d6563493\u001b[m\u001b[33m (\u001b[m\u001b[1;36mHEAD -> \u001b[m\u001b[1;32mmain\u001b[m\u001b[33m, \u001b[m\u001b[1;31morigin/main\u001b[m\u001b[33m, \u001b[m\u001b[1;31morigin/HEAD\u001b[m\u001b[33m)\u001b[m\n",
            "Author: rml54 <45828236+rml54@users.noreply.github.com>\n",
            "Date:   Sun Aug 13 17:25:37 2023 -0400\n",
            "\n",
            "    Update funcdump.py\n",
            "Processing /content/gp15wmascripts\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting gsw (from gp15wma==0.1.0.0)\n",
            "  Downloading gsw-3.6.17-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.4/2.4 MB\u001b[0m \u001b[31m5.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyompa in /usr/local/lib/python3.11/site-packages (from gp15wma==0.1.0.0) (0.4.2.0)\n",
            "Requirement already satisfied: numpy>=1.21 in /usr/local/lib/python3.11/site-packages (from gsw->gp15wma==0.1.0.0) (1.25.2)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.11/site-packages (from pyompa->gp15wma==0.1.0.0) (2.0.3)\n",
            "Requirement already satisfied: cvxpy in /usr/local/lib/python3.11/site-packages (from pyompa->gp15wma==0.1.0.0) (1.3.2)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.11/site-packages (from pyompa->gp15wma==0.1.0.0) (1.11.2)\n",
            "Requirement already satisfied: toml in /usr/local/lib/python3.11/site-packages (from pyompa->gp15wma==0.1.0.0) (0.10.2)\n",
            "Requirement already satisfied: osqp>=0.4.1 in /usr/local/lib/python3.11/site-packages (from cvxpy->pyompa->gp15wma==0.1.0.0) (0.6.3)\n",
            "Requirement already satisfied: ecos>=2 in /usr/local/lib/python3.11/site-packages (from cvxpy->pyompa->gp15wma==0.1.0.0) (2.0.12)\n",
            "Requirement already satisfied: scs>=1.1.6 in /usr/local/lib/python3.11/site-packages (from cvxpy->pyompa->gp15wma==0.1.0.0) (3.2.3)\n",
            "Requirement already satisfied: setuptools>65.5.1 in /usr/local/lib/python3.11/site-packages (from cvxpy->pyompa->gp15wma==0.1.0.0) (67.8.0)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.11/site-packages (from pandas->pyompa->gp15wma==0.1.0.0) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.11/site-packages (from pandas->pyompa->gp15wma==0.1.0.0) (2023.3)\n",
            "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.11/site-packages (from pandas->pyompa->gp15wma==0.1.0.0) (2023.3)\n",
            "Requirement already satisfied: qdldl in /usr/local/lib/python3.11/site-packages (from osqp>=0.4.1->cvxpy->pyompa->gp15wma==0.1.0.0) (0.1.7.post0)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->pyompa->gp15wma==0.1.0.0) (1.16.0)\n",
            "Building wheels for collected packages: gp15wma\n",
            "  Building wheel for gp15wma (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for gp15wma: filename=gp15wma-0.1.0.0-py3-none-any.whl size=15116 sha256=fce822f87eba06e8b41348179c44a780fc3f4d1c835206229bbbd8ec42989fe3\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-71k8vtie/wheels/76/d6/19/c421f224b7b298b98f2fc8c5bbc3cfb00344ed07be2895b92e\n",
            "Successfully built gp15wma\n",
            "Installing collected packages: gsw, gp15wma\n",
            "Successfully installed gp15wma-0.1.0.0 gsw-3.6.17\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m/content\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OrHIAtrZkW72"
      },
      "source": [
        "### Import all the relevant modules"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip uninstall -y pyompa\n",
        "%cd /content/\n",
        "!rm -rf pyompa\n",
        "!git clone https://github.com/nitrogenlab/pyompa\n",
        "%cd /content/pyompa\n",
        "!git checkout dev #install from the dev branch\n",
        "!git log -1\n",
        "!pip install .\n",
        "%cd /content/"
      ],
      "metadata": {
        "id": "uHiXbIRC1JDi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install scikit-learn"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PWpe4HoH6jGg",
        "outputId": "4fbbaeac-a585-4b74-d4d4-3c4dbaa6246b"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting scikit-learn\n",
            "  Downloading scikit_learn-1.3.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.9/10.9 MB\u001b[0m \u001b[31m14.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.11/site-packages (from scikit-learn) (1.25.2)\n",
            "Requirement already satisfied: scipy>=1.5.0 in /usr/local/lib/python3.11/site-packages (from scikit-learn) (1.11.2)\n",
            "Collecting joblib>=1.1.1 (from scikit-learn)\n",
            "  Downloading joblib-1.3.2-py3-none-any.whl (302 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m302.2/302.2 kB\u001b[0m \u001b[31m18.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting threadpoolctl>=2.0.0 (from scikit-learn)\n",
            "  Downloading threadpoolctl-3.2.0-py3-none-any.whl (15 kB)\n",
            "Installing collected packages: threadpoolctl, joblib, scikit-learn\n",
            "Successfully installed joblib-1.3.2 scikit-learn-1.3.0 threadpoolctl-3.2.0\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install metis"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SG7KA9Kl5djG",
        "outputId": "68eec818-76d0-48c9-c444-57d1c99d0814"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting metis\n",
            "  Downloading metis-0.2a5.tar.gz (10 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Building wheels for collected packages: metis\n",
            "  Building wheel for metis (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for metis: filename=metis-0.2a5-py3-none-any.whl size=11305 sha256=58fd6c823f2780433c6736a414ca1dc6c1986a4d982054a7e139146b1d33292c\n",
            "  Stored in directory: /root/.cache/pip/wheels/20/0c/3a/2fe1d1d85fd5d04cb86520271fa12e529b77ba83737e4738fa\n",
            "Successfully built metis\n",
            "Installing collected packages: metis\n",
            "Successfully installed metis-0.2a5\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install https://github.com/nitrogenlab/GP15_watermassanalysis/blob/main/scikit_learn-0.24.2-cp310-cp310-win32.whl\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7kuCffRa1ncP",
        "outputId": "583b47d3-e265-4e5e-fedd-93d34dfabeff"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[31mERROR: scikit_learn-0.24.2-cp310-cp310-win32.whl is not a supported wheel on this platform.\u001b[0m\u001b[31m\n",
            "\u001b[0m"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install scikit-umfpack\n",
        "\n",
        "! wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ympJzFoxypFo",
        "outputId": "3719e764-e9ac-4c86-996d-3b74afa8efa7"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting scikit-umfpack\n",
            "  Using cached scikit-umfpack-0.3.3.tar.gz (26 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: numpy>=1.25.2 in /usr/local/lib/python3.11/site-packages (from scikit-umfpack) (1.25.2)\n",
            "Requirement already satisfied: scipy>=1.0.0rc1 in /usr/local/lib/python3.11/site-packages (from scikit-umfpack) (1.11.2)\n",
            "Building wheels for collected packages: scikit-umfpack\n",
            "  \u001b[1;31merror\u001b[0m: \u001b[1msubprocess-exited-with-error\u001b[0m\n",
            "  \n",
            "  \u001b[31m×\u001b[0m \u001b[32mpython setup.py bdist_wheel\u001b[0m did not run successfully.\n",
            "  \u001b[31m│\u001b[0m exit code: \u001b[1;36m1\u001b[0m\n",
            "  \u001b[31m╰─>\u001b[0m See above for output.\n",
            "  \n",
            "  \u001b[1;35mnote\u001b[0m: This error originates from a subprocess, and is likely not a problem with pip.\n",
            "  Building wheel for scikit-umfpack (setup.py) ... \u001b[?25lerror\n",
            "\u001b[31m  ERROR: Failed building wheel for scikit-umfpack\u001b[0m\u001b[31m\n",
            "\u001b[0m\u001b[?25h  Running setup.py clean for scikit-umfpack\n",
            "Failed to build scikit-umfpack\n",
            "\u001b[31mERROR: Could not build wheels for scikit-umfpack, which is required to install pyproject.toml-based projects\u001b[0m\u001b[31m\n",
            "\u001b[0m"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "epN_TqvykVyl",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 449
        },
        "outputId": "ac1fad52-3971-4e5d-9c6c-bae0ad5a6e84"
      },
      "source": [
        "import numpy as np\n",
        "from matplotlib import pyplot as plt\n",
        "#import gsw\n",
        "import pandas\n",
        "from collections import OrderedDict, namedtuple\n",
        "from datetime import datetime\n",
        "import json\n",
        "import gc\n",
        "import scipy.io\n",
        "import scipy.sparse\n",
        "import scipy.sparse.linalg\n",
        "from scipy.sparse.linalg import LinearOperator\n",
        "from scipy.sparse import (isspmatrix_csc, isspmatrix_csr, isspmatrix,\n",
        "                          SparseEfficiencyWarning, csc_matrix, csr_matrix)\n",
        "from scipy.sparse.sputils import is_pydata_spmatrix\n",
        "import scikits.umfpack as umfpack\n",
        "import copy\n",
        "import netCDF4\n",
        "import pyompa\n",
        "import GP02wma\n",
        "from GP02wma import gsw"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-10-b73cac3d6059>:15: DeprecationWarning: Please use `is_pydata_spmatrix` from the `scipy.sparse` namespace, the `scipy.sparse.sputils` namespace is deprecated.\n",
            "  from scipy.sparse.sputils import is_pydata_spmatrix\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "ModuleNotFoundError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-10-b73cac3d6059>\u001b[0m in \u001b[0;36m<cell line: 16>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     14\u001b[0m                           SparseEfficiencyWarning, csc_matrix, csr_matrix)\n\u001b[1;32m     15\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mscipy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msparse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msputils\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mis_pydata_spmatrix\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 16\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mscikits\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mumfpack\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mumfpack\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     17\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     18\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mnetCDF4\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'scikits'",
            "",
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0;32m\nNOTE: If your import is failing due to a missing package, you can\nmanually install dependencies using either !pip or !apt.\n\nTo view examples of installing some common dependencies, click the\n\"Open Examples\" button below.\n\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n"
          ],
          "errorDetails": {
            "actions": [
              {
                "action": "open_url",
                "actionText": "Open Examples",
                "url": "/notebooks/snippets/importing_libraries.ipynb"
              }
            ]
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Eoxwlh5j1P1"
      },
      "source": [
        "## Load the OCIM data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LTZDecu1hP18"
      },
      "source": [
        "### Download the OCIM data\n",
        "\n",
        "OCIM data are from https://figshare.com/articles/dataset/OCIM2-48L_base_state_model_output/14802732"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wb-qxf7ddkp6"
      },
      "source": [
        "#Only try to download if the tgz file doesn't exist\n",
        "![[ -e OCIM2_48L_base.tar.gz ]] || wget https://figshare.com/ndownloader/files/28468077 -O OCIM2_48L_base.tar.gz\n",
        "#Unzip the files\n",
        "!tar -xzf OCIM2_48L_base.tar.gz"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tEnbu2CJNJho"
      },
      "source": [
        "### Load the transport matrix\n",
        "\n",
        "We load the original transport matrix"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5CnrwnxWj-HK"
      },
      "source": [
        "base_transport = scipy.io.loadmat(\"OCIM2_48L_base_transport.mat\")['TR']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3vch3k11kkgX"
      },
      "source": [
        "### Load other relevant base data\n",
        "\n",
        "Load latitude, longitude and depth co-ordinates for gridboxes from OCIM2_48L_base_data.nc."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dkjyCmcZns8U"
      },
      "source": [
        "data = netCDF4.Dataset(\"OCIM2_48L_base_data.nc\")\n",
        "#ocnmask is a mask indicating whether a grid cell is in the ocean\n",
        "# the ocean grid cells correspond to the entries in the transport matrix\n",
        "ocnmask = data.variables['ocnmask'][:].data==1.0\n",
        "#ulat, ulon and wz are the lat/lon/depth coordinates corresponding to the fluxes\n",
        "ulat = data.variables['ulon'][:].data #latlon are swapped in the nc file\n",
        "ulon = data.variables['ulat'][:].data #latlon are swapped in the nc file\n",
        "wz = data.variables['wz'][:].data\n",
        "\n",
        "#get the mapping from depth/lon/lan to idx within ocnvec\n",
        "depth_idxs, lon_idxs, lat_idxs = np.indices(ocnmask.shape)\n",
        "ocnvec_mapping = np.zeros_like(ocnmask).astype(\"int\")\n",
        "ocnvec_mapping[:] = np.iinfo(np.int).max\n",
        "ocnvec_mapping[(depth_idxs.ravel()[ocnmask.ravel()],\n",
        "                lon_idxs.ravel()[ocnmask.ravel()],\n",
        "                lat_idxs.ravel()[ocnmask.ravel()])] = np.arange(np.sum(ocnmask))\n",
        "del depth_idxs, lon_idxs, lat_idxs"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1i05tEoxoo7f"
      },
      "source": [
        "As a sanity check, let's do a scatterplot of all the lat/lon coordinates corresponding to the ocean gridpoints. We note that the OCIM longitude co-ordinates go fro 0 to 360."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w2ue4wzroyBq"
      },
      "source": [
        "plt.scatter(ulon[ocnmask], ulat[ocnmask])\n",
        "plt.xlabel(\"Longitude (OCIM co-ordinates)\")\n",
        "plt.ylabel(\"Latitude\")\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PMAS-q1tGD4W"
      },
      "source": [
        "Get the latitude, longitude and depth specifically at ocean gridboxes. Also convert longitudes to have negative values west of the prime meridian, consistent with more standard notation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RBkvdpJYqVu0"
      },
      "source": [
        "#the _ocnvec suffix indicates the vector of properties corresponds to grid\n",
        "# boxes that are in the ocean; it lines up with the axes of the transport matrix\n",
        "ulat_ocnvec = ulat.ravel()[ocnmask.ravel()]\n",
        "ulon_ocnvec = ulon.ravel()[ocnmask.ravel()]\n",
        "#convert ulon to have negative longitudes to the west of the prime meridian,\n",
        "# which makes it more compatible with what other packages expect\n",
        "ulon_ocnvec = (ulon_ocnvec*(ulon_ocnvec <= 180)\n",
        "               + (-(360-ulon_ocnvec)*(ulon_ocnvec > 180)))\n",
        "\n",
        "depth_ocnvec = wz.ravel()[ocnmask.ravel()]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "32HEti34rlSe"
      },
      "source": [
        "As a sanity check, we'll make sure that the converted longitude coordinates correspond to what we want (i.e. negative values west of the prime meridian)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M43qYlyLrrbQ"
      },
      "source": [
        "plt.scatter(ulon_ocnvec, ulat_ocnvec)\n",
        "plt.xlabel(\"Longitude (standard co-ordinates)\")\n",
        "plt.ylabel(\"Latitude\")\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AEQtPFKttY3d"
      },
      "source": [
        "## Load the World Ocean Atlas data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OaXHdVwmho4r"
      },
      "source": [
        "### Download the WOA data\n",
        "\n",
        "Key for reading the WOA data:\n",
        "- s=salinity, t=temperature, i=silicate, o=oxygen  \n",
        "- _an is the objectively analyzed value   \n",
        "- decav is the decadal average  \n",
        "- A5B7 is 2005 to 2017  (A=2000s, B=2010s). \"All\" indicates all time ranges. \"all\" is only available for nutrients and is not available for temp/salinity\n",
        "- The 00 in s00/t00/i00/o00 indicates it's an annual value  \n",
        "- 1.00 indicates it's for a 1-degree-resolution grid, as does the 01 suffix in the file name"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_-bbhDpUwDoS"
      },
      "source": [
        "#WOA18 for A5B7 salinity\n",
        "! [[ -e woa18_A5B7_s00_01.nc ]] || wget https://www.ncei.noaa.gov/data/oceans/woa/WOA18/DATA/salinity/netcdf/A5B7/1.00/woa18_A5B7_s00_01.nc -O woa18_A5B7_s00_01.nc\n",
        "\n",
        "#WOA18 A5B7 temperature\n",
        "! [[ -e woa18_A5B7_t00_01.nc ]] || wget https://www.ncei.noaa.gov/data/oceans/woa/WOA18/DATA/temperature/netcdf/A5B7/1.00/woa18_A5B7_t00_01.nc -O woa18_A5B7_t00_01.nc\n",
        "\n",
        "#WOA18 \"all\" silicate\n",
        "! [[ -e woa18_all_i00_01.nc ]] || wget https://www.ncei.noaa.gov/data/oceans/woa/WOA18/DATA/silicate/netcdf/all/1.00/woa18_all_i00_01.nc -O woa18_all_i00_01.nc\n",
        "\n",
        "#WOA18 \"all\" oxygen\n",
        "! [[ -e woa18_all_o00_01.nc ]] || wget https://www.ncei.noaa.gov/data/oceans/woa/WOA18/DATA/oxygen/netcdf/all/1.00/woa18_all_o00_01.nc -O woa18_all_o00_01.nc\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ts8VzGqPM7rc"
      },
      "source": [
        "### Load WOA into a pandas data frame"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WGHD75c64pJU"
      },
      "source": [
        "def extract_woa_into_df(attr_to_woancdata):\n",
        "    attributes = list(attr_to_woancdata.keys())\n",
        "    lat_ticks = attr_to_woancdata[attributes[0]]['lat'][:].data\n",
        "    lon_ticks = attr_to_woancdata[attributes[0]]['lon'][:].data\n",
        "    depth_ticks = attr_to_woancdata[attributes[0]]['depth'][:].data\n",
        "    #make sure the ticks are the same across all woancdata entries, as a\n",
        "    # sanity check\n",
        "    for attr in attr_to_woancdata:\n",
        "        assert tuple(lat_ticks) == tuple(attr_to_woancdata[attr]['lat'][:].data)\n",
        "        assert tuple(lon_ticks) == tuple(attr_to_woancdata[attr]['lon'][:].data)\n",
        "        assert tuple(depth_ticks) == tuple(\n",
        "                                       attr_to_woancdata[attr]['depth'][:].data)\n",
        "\n",
        "    attr_to_data = dict([(attr, attr_to_woancdata[attr][attr][:].data)\n",
        "                         for attr in attr_to_woancdata])\n",
        "    attr_to_nanfill = dict([(attr, attr_to_woancdata[attr][attr]._FillValue)\n",
        "                          for attr in attr_to_woancdata])\n",
        "\n",
        "    pandas_dict = OrderedDict([\n",
        "        ('depth', []),\n",
        "        ('latitude', []),\n",
        "        ('longitude', [])\n",
        "    ])\n",
        "    for attr in attr_to_woancdata:\n",
        "        pandas_dict[attr] = []\n",
        "\n",
        "    for depthidx in range(len(depth_ticks)):\n",
        "        for latidx in range(len(lat_ticks)):\n",
        "            for lonidx in range(len(lon_ticks)):\n",
        "                #only consider points where data is present\n",
        "                # for some of the readings\n",
        "                data_present = any([(\n",
        "                    attr_to_data[attr][0, depthidx, latidx, lonidx]\n",
        "                    != attr_to_nanfill[attr]) for attr in attributes])\n",
        "                if (data_present):\n",
        "                    pandas_dict['depth'].append(depth_ticks[depthidx])\n",
        "                    pandas_dict['latitude'].append(lat_ticks[latidx])\n",
        "                    pandas_dict['longitude'].append(lon_ticks[lonidx])\n",
        "                    for attr in attr_to_data:\n",
        "                        dataval =\\\n",
        "                            attr_to_data[attr][0, depthidx, latidx, lonidx]\n",
        "                        if (dataval == attr_to_nanfill[attr]):\n",
        "                            dataval = np.nan\n",
        "                        pandas_dict[attr].append(dataval)\n",
        "    return pandas.DataFrame(pandas_dict)\n",
        "\n",
        "woa18_df = extract_woa_into_df(\n",
        "              attr_to_woancdata={\n",
        "                  't_an': netCDF4.Dataset(\"woa18_A5B7_t00_01.nc\"),\n",
        "                  's_an': netCDF4.Dataset(\"woa18_A5B7_s00_01.nc\"),\n",
        "                  'i_an': netCDF4.Dataset(\"woa18_all_i00_01.nc\"),\n",
        "                  'o_an': netCDF4.Dataset(\"woa18_all_o00_01.nc\")\n",
        "                  })"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SIXO7arQXSO_"
      },
      "source": [
        "Add conservative temperature, absolute salinity and potential density to the WOA data frame"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5R6Fz0GqW2mm"
      },
      "source": [
        "def augment_woa_with_conservative_vals(woa_df):\n",
        "    depth = np.array(woa_df[\"depth\"])\n",
        "    lats = np.array(woa_df[\"latitude\"])\n",
        "    lons = np.array(woa_df[\"longitude\"])\n",
        "    #pressure calculation from depth\n",
        "    p = gsw.p_from_z(\n",
        "      z=-depth, #z is expected to be negative in the ocean\n",
        "      lat=lats)\n",
        "    #get the absolute salinity given the practical salinity and pressure\n",
        "    abssal = gsw.SA_from_SP(SP=np.array(woa_df[\"s_an\"]), #practical salinity\n",
        "                            p=p, lon=lons, lat=lats)\n",
        "    #get conservative temp given absolute salinity, temp and pressure\n",
        "    ctemp = gsw.CT_from_t(SA=abssal,\n",
        "                          t=np.array(woa_df[\"t_an\"]), #temperature\n",
        "                          p=p)\n",
        "    #get sigma2/sigma2/sigma4, which will be used for defining the locations of\n",
        "    # endmembers\n",
        "    sig0 = gsw.sigma0(SA=abssal, CT=ctemp)\n",
        "    sig2 = gsw.sigma2(SA=abssal, CT=ctemp)\n",
        "    sig4 = gsw.sigma4(SA=abssal, CT=ctemp)\n",
        "\n",
        "    woa_df[\"pressure\"] = p\n",
        "    woa_df[\"absolute_salinity\"] = abssal\n",
        "    woa_df[\"conservative_temperature\"] = ctemp\n",
        "    woa_df[\"sigma0\"] = sig0\n",
        "    woa_df[\"sigma2\"] = sig2\n",
        "    woa_df[\"sigma4\"] = sig4\n",
        "\n",
        "augment_woa_with_conservative_vals(woa_df=woa18_df)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tLGy5NvdvQBd"
      },
      "source": [
        "### Index WOA data by OCIM gridboxes\n",
        "\n",
        "We organize the measurements into a data structure such that we can easily retrieve the measurements corresponding to particular OCIM gridboxes\n",
        "\n",
        "First, we get the depth, latitude and longitude 'axes tickmarks' corresponding to the OCIM gridboxes"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mzfnsZYUvPdZ"
      },
      "source": [
        "ocim_wz_ticks = tuple(wz[:,0,0])\n",
        "ocim_ulat_ticks = tuple(ulat[0,0,:])\n",
        "ocim_ulon_ticks = tuple(ulon[0,:,0])\n",
        "\n",
        "#also delete obsolete variables\n",
        "del data\n",
        "del ulat\n",
        "del ulon\n",
        "del wz\n",
        "gc.collect()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tXQSDQatxY3w"
      },
      "source": [
        "Next, we define functions that can be used for mapping the measured data into a data structure corresponding to the OCIM gridboxes. Each observation is assigned to the OCIM gridbox that has the closest value of ulat/ulon/depth."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Et0R7v_xfHl"
      },
      "source": [
        "#This function initializes an empty grid\n",
        "def initialize_grid(shape):\n",
        "    grid = []\n",
        "    for depth_idx in range(shape[0]):\n",
        "        grid_depthentry = []\n",
        "        for lon_idx in range(shape[1]):\n",
        "            grid_lonentry = []\n",
        "            for lat_idx in range(shape[2]):\n",
        "                grid_latentry = []\n",
        "                grid_lonentry.append(grid_latentry)\n",
        "            grid_depthentry.append(grid_lonentry)\n",
        "        grid.append(grid_depthentry)\n",
        "    return grid\n",
        "\n",
        "#This function maps observations in a data frame into a grid\n",
        "def prepare_obsgrid_from_df(df, wz_ticks, ulon_ticks, ulat_ticks,\n",
        "                                depth_key, longitude_key, latitude_key,\n",
        "                                keys_to_record, return_idx_mapping=False):\n",
        "    obs_grid = initialize_grid((len(wz_ticks), len(ulon_ticks), len(ulat_ticks)))\n",
        "\n",
        "    ObsEntry = namedtuple('ObsEntry', keys_to_record)\n",
        "\n",
        "    df_lats = np.array(df[latitude_key])\n",
        "    df_lons = np.array(df[longitude_key])\n",
        "    df_depths = np.array(df[depth_key])\n",
        "\n",
        "    #np.searchsorted gives the insertion index in order\n",
        "    # to maintain sort order\n",
        "    #Also need to convert the lons to the 0-360 system used by OCIM\n",
        "    df_lons_converted = (df_lons*(df_lons >= 0.0)\n",
        "                         + (df_lons + 360)*(df_lons < 0.0))\n",
        "    lat_insert_idxs = np.searchsorted(a=ulat_ticks, v=df_lats)\n",
        "    lon_insert_idxs = np.searchsorted(a=ulon_ticks, v=df_lons_converted)\n",
        "    depth_insert_idxs = np.searchsorted(a=wz_ticks, v=df_depths)\n",
        "\n",
        "    #also get the contents associated with keys_to_record, to avoid\n",
        "    # storing unnecessarily large amounts of data\n",
        "    key_to_coldata = dict([\n",
        "        (key, np.array(df[key])) for key in keys_to_record\n",
        "    ])\n",
        "\n",
        "    if (return_idx_mapping):\n",
        "        mapped_idxs = [[], [], []] #depth, lon, lat\n",
        "\n",
        "    num_obs_skipped = 0\n",
        "    for i in range(len(lat_insert_idxs)):\n",
        "        lat_insert_idx = lat_insert_idxs[i]\n",
        "        lon_insert_idx = lon_insert_idxs[i]\n",
        "        depth_insert_idx = depth_insert_idxs[i]\n",
        "\n",
        "        item_lat = df_lats[i]\n",
        "        item_lon = df_lons_converted[i]\n",
        "        item_depth = df_depths[i]\n",
        "\n",
        "        #In an Arakawa-B grid, the velocities are measured in the center of\n",
        "        # the grid. Thus, we want to map the observation to the gridbox\n",
        "        # for which the lat/lon/depth measurement is closer.\n",
        "        lat_idx = (lat_insert_idx\n",
        "              if (abs(ulat_ticks[lat_insert_idx] - item_lat)\n",
        "                  < abs(ulat_ticks[lat_insert_idx-1] - item_lat))\n",
        "              else lat_insert_idx-1)\n",
        "        #Need to do the %360 to account for wrap-around\n",
        "        lon_idx = (lon_insert_idx\n",
        "                  if (abs(ulon_ticks[lon_insert_idx] - item_lon)%360\n",
        "                      < abs(ulon_ticks[lon_insert_idx-1] - item_lon)%360)\n",
        "                   else lon_insert_idx-1)\n",
        "        #If the insertion index is deeper than the tick of the deepest gridbox,\n",
        "        # the index is the one corresponding to the deepest gridbox (0-indexed)\n",
        "        if (depth_insert_idx == len(wz_ticks)):\n",
        "            depth_idx = depth_insert_idx - 1\n",
        "        else:\n",
        "            depth_idx = (depth_insert_idx\n",
        "            if ((abs(wz_ticks[depth_insert_idx] - item_depth)\n",
        "                < abs(wz_ticks[depth_insert_idx-1] - item_depth))\n",
        "                ##avoid going too deep if that gridbox is on land\n",
        "                #and (ocnvec_mapping[depth_insert_idx,lon_idx,lat_idx]\n",
        "                #     != np.iinfo(np.int).max)\n",
        "                )\n",
        "            else depth_insert_idx-1)\n",
        "\n",
        "        #In case the gridbox mapped to corresponds to a land box, then go\n",
        "        # higher at look at adjacent lat/lon until we find a non-land\n",
        "        # gridbox\n",
        "        c_depthidx = depth_idx\n",
        "        c_lonidx = lon_idx\n",
        "        c_latidx = lat_idx\n",
        "\n",
        "        #If we don't have to return the index mapping, then we can\n",
        "        # ignore cases where something maps to a land gridbox\n",
        "        if (return_idx_mapping == False):\n",
        "          if (ocnmask[c_depthidx, c_lonidx, c_latidx]==False):\n",
        "              num_obs_skipped += 1\n",
        "              continue\n",
        "        else: #otherwise, go through the process of finding a match\n",
        "            while (ocnmask[c_depthidx, c_lonidx, c_latidx]==False):\n",
        "                if ocnmask[c_depthidx-1, c_lonidx, c_latidx]:\n",
        "                    c_depthidx = c_depthidx-1 #look higher first\n",
        "                    break\n",
        "                #cycle through lat/lon surroundings\n",
        "                if ocnmask[c_depthidx, c_lonidx-1, c_latidx]:\n",
        "                    c_lonidx = c_lonidx-1\n",
        "                    break\n",
        "                if ocnmask[c_depthidx, (c_lonidx+1) % ocnmask.shape[1], c_latidx]:\n",
        "                    c_lonidx = c_lonidx+1\n",
        "                    break\n",
        "                if ocnmask[c_depthidx, c_lonidx, c_latidx-1]:\n",
        "                    c_latidx = c_latidx-1\n",
        "                    break\n",
        "                if ocnmask[c_depthidx, c_lonidx, (c_latidx+1)  % ocnmask.shape[2]]:\n",
        "                    c_latidx = c_latidx+1\n",
        "                    break\n",
        "                if (c_depthidx==0):\n",
        "                    break #don't keep looking if reached surface\n",
        "                #go higher and do search again\n",
        "                c_depthidx = c_depthidx - 1\n",
        "\n",
        "        if ( (c_depthidx, c_lonidx, c_latidx) != (depth_idx, lon_idx, lat_idx)):\n",
        "            print(\"Warning!\",(item_depth, item_lon, item_lat), \"is closest\"\n",
        "              +\" to land gridbox\",\n",
        "              (wz_ticks[depth_idx], ulon_ticks[lon_idx],ulat_ticks[lat_idx]),\n",
        "              \"so mapping to\",\n",
        "              (wz_ticks[c_depthidx], ulon_ticks[c_lonidx],ulat_ticks[c_latidx]),\n",
        "              \"gridbox instead\")\n",
        "            depth_idx, lon_idx, lat_idx = (c_depthidx, c_lonidx, c_latidx)\n",
        "        if (return_idx_mapping):\n",
        "            #if (ocnvec_mapping[depth_idx,lon_idx,lat_idx]) == np.iinfo(np.int).max:\n",
        "            #    print(depth_idx, lon_idx, lat_idx)\n",
        "            #    print(item_depth, item_lon, item_lat)\n",
        "            #    print(ocnvec_mapping[depth_idx,lon_idx,lat_idx],\n",
        "            #          ocnvec_mapping[depth_idx-1,lon_idx,lat_idx],\n",
        "            #          ocnvec_mapping[depth_idx-2,lon_idx,lat_idx],\n",
        "            #          ocnvec_mapping[depth_idx-3,lon_idx,lat_idx])\n",
        "            mapped_idxs[0].append(depth_idx)\n",
        "            mapped_idxs[1].append(lon_idx)\n",
        "            mapped_idxs[2].append(lat_idx)\n",
        "        obs_grid[depth_idx][lon_idx][lat_idx].append(\n",
        "            ObsEntry(*[key_to_coldata[key][i] for key in keys_to_record ] ))\n",
        "\n",
        "    if (return_idx_mapping == False):\n",
        "      print(\"Skipped\",num_obs_skipped,\"observations out of\",\n",
        "            len(lat_insert_idxs),\"as the closest gridbox was a land gridbox\")\n",
        "\n",
        "    if (return_idx_mapping):\n",
        "        return obs_grid, mapped_idxs\n",
        "    else:\n",
        "        return obs_grid"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2RRQ4Nqnzw0z"
      },
      "source": [
        "Now we run those functions to map the WOA data into gridboxes"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SBkrgyYa8pGu"
      },
      "source": [
        "woa18_obs_grid = prepare_obsgrid_from_df(df=woa18_df,\n",
        "    wz_ticks=ocim_wz_ticks, ulon_ticks=ocim_ulon_ticks,\n",
        "    ulat_ticks=ocim_ulat_ticks,\n",
        "    depth_key=\"depth\", longitude_key=\"longitude\", latitude_key=\"latitude\",\n",
        "    keys_to_record=[\"conservative_temperature\", \"absolute_salinity\",\n",
        "                    \"i_an\", \"o_an\", \"sigma0\", \"sigma2\", \"sigma4\"])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sko2o4JB6HFb"
      },
      "source": [
        "### Compute average WOA data in OCIM gridboxes\n",
        "\n",
        "Compute things like the average silicate concentration, average conservative temperature of the WOA data in each OCIM gridbox."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g67r_r9Y6KgB"
      },
      "source": [
        "#Function that averages the observations in each gridbox,\n",
        "# returning np.nan when there are no observations in a gridbox\n",
        "def get_data_from_obs_grid(func, obs_grid):\n",
        "    to_return = np.zeros(\n",
        "        (len(obs_grid), len(obs_grid[0]),\n",
        "         len(obs_grid[0][0])) )\n",
        "    to_return[:,:,:] = np.nan\n",
        "    for i in range(len(obs_grid)):\n",
        "        for j in range(len(obs_grid[i])):\n",
        "            for k in range(len(obs_grid[i][j])):\n",
        "                if len(obs_grid[i][j][k]) > 0:\n",
        "                    to_return[i,j,k] = func(obs_grid[i][j][k])\n",
        "    return to_return\n",
        "\n",
        "def get_mean_from_obs_grid(attr, obs_grid):\n",
        "    return get_data_from_obs_grid(\n",
        "              func=eval(\"lambda arr: np.mean([y.\"+attr+\" for y in arr])\"),\n",
        "              obs_grid=obs_grid)\n",
        "\n",
        "#Get the mean conservative temperature, absolute salinity,\n",
        "# and silicate in the ocean gridboxes with WOA18 data\n",
        "woa18_mean_ctemp_ocnvec = get_mean_from_obs_grid(\n",
        "    attr=\"conservative_temperature\",\n",
        "    obs_grid=woa18_obs_grid).ravel()[ocnmask.ravel()]\n",
        "woa18_mean_abssal_ocnvec = get_mean_from_obs_grid(\n",
        "    attr=\"absolute_salinity\",\n",
        "    obs_grid=woa18_obs_grid).ravel()[ocnmask.ravel()]\n",
        "woa18_mean_silicate_ocnvec = get_mean_from_obs_grid(\n",
        "    attr=\"i_an\",\n",
        "    obs_grid=woa18_obs_grid).ravel()[ocnmask.ravel()]\n",
        "\n",
        "#also get the potential density and oxygen data as this will be\n",
        "# useful for defining end-member boundaries\n",
        "woa18_mean_sig0_ocnvec = get_mean_from_obs_grid(\n",
        "    attr=\"sigma0\",\n",
        "    obs_grid=woa18_obs_grid).ravel()[ocnmask.ravel()]\n",
        "woa18_mean_sig2_ocnvec = get_mean_from_obs_grid(\n",
        "    attr=\"sigma2\",\n",
        "    obs_grid=woa18_obs_grid).ravel()[ocnmask.ravel()]\n",
        "woa18_mean_sig4_ocnvec = get_mean_from_obs_grid(\n",
        "    attr=\"sigma4\",\n",
        "    obs_grid=woa18_obs_grid).ravel()[ocnmask.ravel()]\n",
        "woa18_mean_oxygen_ocnvec = get_mean_from_obs_grid(\n",
        "    attr=\"o_an\",\n",
        "    obs_grid=woa18_obs_grid).ravel()[ocnmask.ravel()]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P2_lFTcHBTI7"
      },
      "source": [
        "Save memory by deleting the objects corresponding to `woa18_obs_grid` and `woa18_df`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9_cN3jryBZ3r"
      },
      "source": [
        "del woa18_obs_grid\n",
        "del woa18_df\n",
        "\n",
        "gc.collect()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G2fJnsihswha"
      },
      "source": [
        "## Idenitify OCIM gridboxes corresponding to end-members\n",
        "\n",
        "We define endmembers by lat/lon/potential density and figure out the indexes in the vector of ocean gridboxes that correspond to each endmember. We also use WOA data to add in a filter on oxygen."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cZVPChLlevpT"
      },
      "source": [
        "#Function to get the indexes that match a lat/lon/potential density threshold\n",
        "# range. These are indexes into the vector of ocean gridboxes.\n",
        "def get_endmember_idxs(lat_min, lat_max,\n",
        "                      lon_min, lon_max,\n",
        "                      invert_lon=False,\n",
        "                      depth_min=0, depth_max=np.inf,\n",
        "                      sig0_min=0, sig0_max=np.inf,\n",
        "                      sig2_min=0, sig2_max=np.inf,\n",
        "                      sig4_min=0, sig4_max=np.inf,\n",
        "                      ox_min=0, ox_max=np.inf):\n",
        "    nonzero_idxs = np.nonzero(\n",
        "      (ulat_ocnvec >= lat_min)*(ulat_ocnvec <= lat_max)\n",
        "     *((((ulon_ocnvec) >= lon_min)*((ulon_ocnvec) <= lon_max))\n",
        "       ==(False if invert_lon else True))\n",
        "     *(depth_ocnvec >= depth_min)*(depth_ocnvec <= depth_max)\n",
        "     *(woa18_mean_sig0_ocnvec >= sig0_min)*(woa18_mean_sig0_ocnvec <= sig0_max)\n",
        "     *(woa18_mean_sig2_ocnvec >= sig2_min)*(woa18_mean_sig2_ocnvec <= sig2_max)\n",
        "     *(woa18_mean_sig4_ocnvec >= sig4_min)*(woa18_mean_sig4_ocnvec <= sig4_max)\n",
        "     *(woa18_mean_oxygen_ocnvec >= ox_min)\n",
        "     *(woa18_mean_oxygen_ocnvec <= ox_max)\n",
        "    )[0]\n",
        "    return nonzero_idxs\n",
        "\n",
        "endmember_definitions = {\n",
        "    \"AAIW\": {\"lat_min\":-55.0, \"lat_max\":-43.0,\n",
        "             \"lon_min\":-90, \"lon_max\":-80,\n",
        "             \"sig0_min\":27.05, \"sig0_max\":27.15},\n",
        "    \"NPIW\": {\"lat_min\":36.5, \"lat_max\":39.0,\n",
        "             \"lon_min\":148.0, \"lon_max\":154.0,\n",
        "             \"sig0_min\":26.4, \"sig0_max\":26.9},\n",
        "    \"UCDW\": {\"lat_min\":-49.5, \"lat_max\":-44.5,\n",
        "             \"lon_min\":-157.0, \"lon_max\":-147.0,\n",
        "             \"sig0_min\":27.35, \"sig0_max\":27.75},\n",
        "    \"LCDW\": {\"lat_min\":-66.5, \"lat_max\":-61.5,\n",
        "             \"lon_min\":-100.0, \"lon_max\":150.0,\n",
        "             \"invert_lon\":True,\n",
        "             \"sig0_min\":27.79, \"sig0_max\":27.83},\n",
        "    \"AABW\": {\"lat_min\":-66.5, \"lat_max\":-61.5,\n",
        "             \"lon_min\":-100.0, \"lon_max\":150.0,\n",
        "             \"invert_lon\":True,\n",
        "             \"sig4_min\":46.04, \"sig4_max\":200,\n",
        "             \"depth_min\":1500},\n",
        "    \"PDW\": {\"lat_min\":39.0, \"lat_max\":51.0,\n",
        "             \"lon_min\":-170.0, \"lon_max\":-133.0,\n",
        "             \"sig0_min\":27.2, \"sig0_max\":200,\n",
        "             \"sig4_min\":0, \"sig4_max\":45.88},\n",
        "    \"EqIW\": {\"lat_min\":-5, \"lat_max\":5,\n",
        "             \"lon_min\":-90.0, \"lon_max\":-80.0,\n",
        "             \"sig0_min\": 26.86, \"sig0_max\": 27.3},\n",
        "    \"PSUW\": {\"lat_min\":50, \"lat_max\":58,\n",
        "             \"lon_min\":-155, \"lon_max\":-140,\n",
        "             \"sig0_min\": 25.29, \"sig0_max\": 26.5},\n",
        "    \"ENPCW\": {\"lat_min\":16, \"lat_max\":26,\n",
        "             \"lon_min\":-170, \"lon_max\":-140,\n",
        "             \"sig0_min\": 25.29, \"sig0_max\": 26.5},\n",
        "    \"ESSW\": {\"lat_min\":-5, \"lat_max\":5,\n",
        "             \"lon_min\":-90.0, \"lon_max\":-80.0,\n",
        "             \"sig0_min\": 25.29, \"sig0_max\": 26.86},\n",
        "    \"SPCW\": {\"lat_min\":-30, \"lat_max\":-20,\n",
        "             \"lon_min\":-152, \"lon_max\":-130,\n",
        "             \"sig0_min\": 25.29, \"sig0_max\": 26.86},\n",
        "}\n",
        "\n",
        "endmem_to_idxs = dict([(the_key, set(get_endmember_idxs(**val)))\n",
        "                        for the_key,val in endmember_definitions.items()])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w3bc4sW_Ynp6"
      },
      "source": [
        "Delete variables that are no-longer needed"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "08B1tlJxYsi3"
      },
      "source": [
        "del woa18_mean_sig0_ocnvec\n",
        "del woa18_mean_sig2_ocnvec\n",
        "del woa18_mean_sig4_ocnvec\n",
        "del woa18_mean_oxygen_ocnvec\n",
        "\n",
        "gc.collect()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NS2sC1Kn-ViC"
      },
      "source": [
        "As a sanity check, we can visualize the locations on the map for each endmember"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BKaGHMpD-aIb"
      },
      "source": [
        "endmem_names = sorted(endmem_to_idxs.keys())\n",
        "\n",
        "for endmem in endmem_names:\n",
        "    print(endmem)\n",
        "    fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(20,5))\n",
        "\n",
        "    plt.sca(ax[0])\n",
        "    #first make a scatterplot of the water surface\n",
        "    plt.scatter(ulon_ocnvec,\n",
        "            ulat_ocnvec, color=\"blue\")\n",
        "    #now overlay the endmember\n",
        "    #handles.append(\n",
        "    plt.scatter(\n",
        "        ulon_ocnvec[np.array(list(endmem_to_idxs[endmem]))],\n",
        "        ulat_ocnvec[np.array(list(endmem_to_idxs[endmem]))],\n",
        "        color=\"red\")\n",
        "    #)\n",
        "\n",
        "    plt.sca(ax[1])\n",
        "    plt.scatter(\n",
        "        ulat_ocnvec[np.array(list(endmem_to_idxs[endmem]))],\n",
        "        depth_ocnvec[np.array(list(endmem_to_idxs[endmem]))],\n",
        "        color=\"red\")\n",
        "    plt.xlim(-90,90)\n",
        "    plt.ylim(max(depth_ocnvec),0)\n",
        "\n",
        "    plt.sca(ax[2])\n",
        "    plt.scatter(\n",
        "        ulon_ocnvec[np.array(list(endmem_to_idxs[endmem]))],\n",
        "        depth_ocnvec[np.array(list(endmem_to_idxs[endmem]))],\n",
        "        color=\"red\")\n",
        "    plt.xlim(-180,180)\n",
        "    plt.ylim(max(depth_ocnvec),0)\n",
        "    plt.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qCamQJtXMPPH"
      },
      "source": [
        "## Load the GP02 data\n",
        "\n",
        "We load the GP02 data so that we can compare how the predictions from OCIM along the GP02 transect compare to the empirical measurements on the GP02 cruise. We focus on intermediate and deep waters here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vO_CvoBd6nEP"
      },
      "source": [
        "GP02_df, GP02_intermediateanddeep, GP02_thermocline = GP02wma.download_and_load_GP02_data(\n",
        "    station_to_tc_cutoffs_url=\"https://github.com/nitrogenlab/GP15_watermassanalysis/blob/0a64ed0faca01701cf6c84d09365abc706594e2c/GP02_station_to_tc_cutoffs.json\")\n",
        "del GP02_df, GP02_thermocline"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HliP1_n9Vzdj"
      },
      "source": [
        "### Index GP02 data by OCIM gridboxes\n",
        "\n",
        "We can reuse the code from when we read the GLODAP data into OCIM gridboxes"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IJdzk6b1V2w_"
      },
      "source": [
        "gp15_obs_grid, gp15_obsgrid_idxmapping =\\\n",
        "  prepare_obsgrid_from_df(\n",
        "    df=GP02_intermediateanddeep,\n",
        "    wz_ticks=ocim_wz_ticks, ulon_ticks=ocim_ulon_ticks,\n",
        "    ulat_ticks=ocim_ulat_ticks,\n",
        "    depth_key=\"Depth\", longitude_key=\"lon\", latitude_key=\"lat\",\n",
        "    keys_to_record=[\"conservative_temp\", \"absolute_salinity\", \"silicate\"],\n",
        "    return_idx_mapping=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MvbmnpZiYG3z"
      },
      "source": [
        "### Compute average GP02 data in OCIM gridboxes\n",
        "\n",
        "As before, we reuse the code we wrote for doing this on GLODAP data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s7xSgo5_YP7j"
      },
      "source": [
        "#Get the mean conservative temperature, absolute salinity and silicate\n",
        "gp15_mean_ctemp_ocnvec = get_mean_from_obs_grid(\n",
        "    attr=\"conservative_temp\",\n",
        "    obs_grid=gp15_obs_grid).ravel()[ocnmask.ravel()]\n",
        "gp15_mean_asbsal_ocnvec = get_mean_from_obs_grid(\n",
        "    attr=\"absolute_salinity\",\n",
        "    obs_grid=gp15_obs_grid).ravel()[ocnmask.ravel()]\n",
        "gp15_mean_silicate_ocnvec = get_mean_from_obs_grid(\n",
        "    attr=\"silicate\",\n",
        "    obs_grid=gp15_obs_grid).ravel()[ocnmask.ravel()]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KrQNszLPFSf1"
      },
      "source": [
        "Delete objects to save memory"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T7cy8ZHDFWR9"
      },
      "source": [
        "del gp15_obs_grid\n",
        "gc.collect()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fVfVLoCtkvWi"
      },
      "source": [
        "## Compute the distributions at steady-state\n",
        "\n",
        "We compute the distributions of tracers and end-members at steady state, treating the end-members as sources for the rest of the ocean"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tus6p1IeNDW2"
      },
      "source": [
        "### Prepare code to compute steady-state distributions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nVmxWJ58etnN"
      },
      "source": [
        "There are some bugs associated with Google Colab's version of scipy in terms of how it interfaces with umfpack. To fix this, I have ported over an alternative versions of the relevant functions from https://github.com/scipy/scipy/pull/11453.\n",
        "\n",
        "I have also modified the functions to use the METIS ordering for the LU decomposition, since the default umfpack ordering runs out of memory without this."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BHk9a2cibW4e"
      },
      "source": [
        "#see https://github.com/scipy/scipy/pull/11453\n",
        "\n",
        "def _get_umf_family(A):\n",
        "    \"\"\"Get umfpack family string given the sparse matrix dtype.\"\"\"\n",
        "    _families = {\n",
        "        (np.float64, np.int32): 'di',\n",
        "        (np.complex128, np.int32): 'zi',\n",
        "        (np.float64, np.int64): 'dl',\n",
        "        (np.complex128, np.int64): 'zl'\n",
        "    }\n",
        "\n",
        "    f_type = np.sctypeDict[A.dtype.name]\n",
        "    i_type = np.sctypeDict[A.indices.dtype.name]\n",
        "\n",
        "    try:\n",
        "        family = _families[(f_type, i_type)]\n",
        "\n",
        "    except KeyError:\n",
        "        msg = 'only float64 or complex128 matrices with int32 or int64' \\\n",
        "            ' indices are supported! (got: matrix: %s, indices: %s)' \\\n",
        "            % (f_type, i_type)\n",
        "        raise ValueError(msg)\n",
        "\n",
        "    # See gh-8278. Considered converting only if\n",
        "    # A.shape[0]*A.shape[1] > np.iinfo(np.int32).max,\n",
        "    # but that didn't always fix the issue.\n",
        "    family = family[0] + \"l\"\n",
        "    A_new = copy.copy(A)\n",
        "    A_new.indptr = np.array(A.indptr, copy=False, dtype=np.int64)\n",
        "    A_new.indices = np.array(A.indices, copy=False, dtype=np.int64)\n",
        "\n",
        "    return family, A_new\n",
        "\n",
        "\n",
        "def get_umf_context(A):\n",
        "    if is_pydata_spmatrix(A):\n",
        "        A = A.to_scipy_sparse().tocsc()\n",
        "\n",
        "    if not isspmatrix_csc(A):\n",
        "        A = csc_matrix(A)\n",
        "        warn('splu requires CSC matrix format', SparseEfficiencyWarning)\n",
        "\n",
        "    A = A.asfptype()  # upcast to a floating point format\n",
        "\n",
        "    if A.dtype.char not in 'dD':\n",
        "        raise ValueError(\"convert matrix data to double, please, using\"\n",
        "              \" .astype(), or set linsolve.useUmfpack = False\")\n",
        "\n",
        "    umf_family, A = _get_umf_family(A)\n",
        "    umf = umfpack.UmfpackContext(umf_family)\n",
        "    #METIS ordering ends up taking up MUCH less memory!\n",
        "    umf.control[umfpack.UMFPACK_ORDERING] = umfpack.UMFPACK_ORDERING_METIS\n",
        "\n",
        "    #see 'alternative routines' section of the UMFPACK user guide\n",
        "    # page 17 describes the orderings\n",
        "    #Also see documentation for umfpackcontext:\n",
        "    # https://github.com/scikit-umfpack/scikit-umfpack/blob/a2102ef92f4dd060138e72bb5d7c444f8ec49cbc/scikits/umfpack/umfpack.py#L114\n",
        "\n",
        "    return umf, A\n",
        "\n",
        "\n",
        "#creating a fixed version of factorize\n",
        "def fixed_factorize(A):\n",
        "    umf, A = get_umf_context(A)\n",
        "    # Make LU decomposition.\n",
        "    umf.numeric(A)\n",
        "    def solve(b):\n",
        "        return umf.solve(umfpack.UMFPACK_A, A, b, autoTranspose=True)\n",
        "    return solve\n",
        "\n",
        "\n",
        "#create a fixed version of solve\n",
        "def fixed_solve(A, b):\n",
        "    umf, A = get_umf_context(A)\n",
        "\n",
        "    return umf.linsolve(umfpack.UMFPACK_A, A, b,\n",
        "                         autoTranspose=True)\n",
        "\n",
        "    return solve\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qGihXuzle6wM"
      },
      "source": [
        "Now we are ready to define the function to get the steady-state tracer concentrations by solving the linear system."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PBiWwrCXAkuA"
      },
      "source": [
        "\n",
        "class GetSteadyStateTracerConcsViaMatrixSolve(object):\n",
        "\n",
        "    \"\"\"\n",
        "    T: the 2d transport matrix\n",
        "    source_idxs: the set of indexes to designate as source gridboxes\n",
        "    \"\"\"\n",
        "    def __init__(self, T, source_idxs, tau=None):\n",
        "        print(datetime.now(), \"Prepping solver\")\n",
        "        self.tau = tau\n",
        "        to_factorize = self.get_tofactorize(T=T, source_idxs=source_idxs)\n",
        "        #self.find_conditionnumber(mat=to_factorize)\n",
        "        #Don't invert the matrix!\n",
        "        # https://www.johndcook.com/blog/2010/01/19/dont-invert-that-matrix/\n",
        "        self.solver = fixed_factorize(to_factorize)\n",
        "        print(datetime.now(), \"solver prepped\")\n",
        "\n",
        "    @staticmethod\n",
        "    def find_conditionnumber(mat):\n",
        "        print(datetime.now(),\"Computing A A^t\")\n",
        "        AAt = mat @ mat.transpose()\n",
        "        print(datetime.now(),\"Getting largest eigenvalue\")\n",
        "        largest_eig = scipy.sparse.linalg.eigsh(A=AAt, k=1, which='LM',\n",
        "                                                return_eigenvectors=False)\n",
        "        print(datetime.now(), largest_eig)\n",
        "        print(datetime.now(),\"Getting smallest eigenvalue\")\n",
        "        smallest_eig = scipy.sparse.linalg.eigsh(A=AAt, k=1, which='SM',\n",
        "                                                 sigma=0,\n",
        "                                                 return_eigenvectors=False)\n",
        "        print(datetime.now(), smallest_eig)\n",
        "        print(\"Ratio:\", largest_eig/smallest_eig)\n",
        "\n",
        "    #source_vecs should be set to the empirically-determined concentration\n",
        "    # of the tracer for the source gridboxes, and should be 0 for all\n",
        "    # non-source gridboxes.\n",
        "    def __call__(self, source_vecs):\n",
        "        print(datetime.now(), \"Calling solver\")\n",
        "        if (self.tau is not None):\n",
        "            return self.solver(-source_vec/self.tau)\n",
        "        else:\n",
        "            return self.solver(-source_vec)\n",
        "        print(datetime.now(), \"Solver called\")\n",
        "\n",
        "    def get_tofactorize(self, T, source_idxs):\n",
        "        #Return the matrix that needs to be factorized\n",
        "        # [(I - M)T - M] for the tau=None case\n",
        "        # (T - M/tau) for the other case\n",
        "        m_vec = np.zeros(T.shape[0])\n",
        "        m_vec[source_idxs] = 1.0\n",
        "        M = scipy.sparse.diags(m_vec)\n",
        "        if (self.tau is not None):\n",
        "            return csc_matrix(T - M/self.tau)\n",
        "        else:\n",
        "            I_minus_M = scipy.sparse.diags(1 - m_vec)\n",
        "            return csc_matrix(I_minus_M@T - M)\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RBrRcoIZQULL"
      },
      "source": [
        "### Instantiate the solver"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o1xKDpvhORFh"
      },
      "source": [
        "Instantiate a solver that's appropriate for the source indices we are using in this case"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6vixlQ49GA5F"
      },
      "source": [
        "source_idxs = np.array(list(set(np.concatenate(\n",
        "                  [list(x) for x in endmem_to_idxs.values()], axis=0))))\n",
        "\n",
        "solver = GetSteadyStateTracerConcsViaMatrixSolve(\n",
        "            T=base_transport,\n",
        "            source_idxs=source_idxs,\n",
        "            #tau=None,\n",
        "            tau=(1.0/(60*60*24*365))\n",
        "            )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oniPVxPdUFUm"
      },
      "source": [
        "### Compute the steady-state tracer distributions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qMZnl3UdkUt7"
      },
      "source": [
        "With the solver instantiated, we can compute the expected tracer concentrations based on the OCIM fluxes, assuming that the sources for the tracers are the defined end-members"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J0LOXHbnxhUq"
      },
      "source": [
        "tracers = [\n",
        "    (\"absolute salinity\", woa18_mean_abssal_ocnvec, gp15_mean_asbsal_ocnvec),\n",
        "    (\"conservative temp\", woa18_mean_ctemp_ocnvec, gp15_mean_ctemp_ocnvec),\n",
        "    (\"silicate\", woa18_mean_silicate_ocnvec, gp15_mean_silicate_ocnvec)\n",
        "]\n",
        "\n",
        "gp15_idxs = ocnvec_mapping[tuple(gp15_obsgrid_idxmapping)]\n",
        "#gp15_idxs = np.nonzero((np.isnan(gp15_mean_ctemp_ocnvec)==False))\n",
        "\n",
        "fig, ax = plt.subplots(nrows=1, ncols=len(tracers),\n",
        "                       figsize=(len(tracers)*7, 5))\n",
        "\n",
        "tracer_to_ocim_residuals = {}\n",
        "\n",
        "for i,(tracer_name, woa18_tracer_vals, gp15_tracer_vals) in enumerate(tracers):\n",
        "    print(\"On \",tracer_name)\n",
        "    #source vec has to be zero at all entries that are not sources. At\n",
        "    # source entries, it should equal the empirically measured values\n",
        "    source_vec = np.zeros(base_transport.shape[0])\n",
        "    source_vec[source_idxs] = woa18_tracer_vals[source_idxs]\n",
        "    steady_state_soln = solver(source_vec)\n",
        "\n",
        "    plt.sca(ax[i])\n",
        "    #Compute the residual with respect to the GP15 observation\n",
        "    residuals = (steady_state_soln[gp15_idxs] - gp15_tracer_vals[gp15_idxs])\n",
        "    tracer_to_ocim_residuals[tracer_name] = residuals\n",
        "    plt.scatter(\n",
        "        ulat_ocnvec[gp15_idxs],\n",
        "        depth_ocnvec[gp15_idxs],\n",
        "        c=residuals,\n",
        "        vmin=-np.max(np.abs(residuals)), vmax=np.max(np.abs(residuals)),\n",
        "        cmap=\"RdBu\"\n",
        "    )\n",
        "    plt.ylim(plt.ylim()[1], plt.ylim()[0])\n",
        "    plt.ylabel(\"Depth\")\n",
        "    plt.xlabel(\"Latitude\")\n",
        "    plt.title(tracer_name+\" OCIM residuals\\n(compared to GP15 measurement)\")\n",
        "    plt.colorbar()\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZTq8zqnzUdSs"
      },
      "source": [
        "### Compute the steady-state end-member distributions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qWu62nM8Qa44"
      },
      "source": [
        "Eeach end-member is like a tracer that has a value of 1 at all source gridboxes occupied by the end-member, and a value of 0 at all source gridboxes that are not occupied by the end-member. We will also define an end-member called \"_ALL\" that is the union of all endmembers in order to diagnose whether we may have missing end-members (if there are no end-members missing, then the end-member fractions should sum up close to 1 everywhere)\n",
        "\n",
        "We will plot each end-member both on a fixed 0-1 scale (bottom) and on a relative scale that is adjusted to the minimum/maximum concentration of that end-member. The latter allows us to see the end-member distribution more clearly, while the former allows us to compare results for different end-members."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WfIA8ZlXwI4J"
      },
      "source": [
        "#find steady-state distributions\n",
        "\n",
        "endmem_to_idxs[\"_ALL\"] = source_idxs\n",
        "\n",
        "fig, ax = plt.subplots(nrows=2, ncols=len(endmem_to_idxs),\n",
        "                       figsize=(len(endmem_to_idxs)*7, 10))\n",
        "\n",
        "endmemname_to_ocimfracs = {}\n",
        "\n",
        "for i, endmem in enumerate(sorted(endmem_to_idxs.keys())):\n",
        "    print(\"On:\", endmem)\n",
        "    endmem_idxs = np.array(list(endmem_to_idxs[endmem]))\n",
        "    source_vec = np.zeros(base_transport.shape[0])\n",
        "    source_vec[endmem_idxs] = 1.0\n",
        "    steady_state_soln = solver(source_vec)\n",
        "\n",
        "    endmemname_to_ocimfracs[endmem] = steady_state_soln[gp15_idxs]\n",
        "\n",
        "    print(\"Globally:\")\n",
        "    print(\"max val:\",np.max(steady_state_soln))\n",
        "    print(\"min val:\",np.min(steady_state_soln))\n",
        "    print(\"Along GP02:\")\n",
        "    print(\"max val:\",np.max(steady_state_soln[gp15_idxs]))\n",
        "    print(\"min val:\",np.min(steady_state_soln[gp15_idxs]))\n",
        "    #If the transport matrix has no off-diagonal negative entries, then\n",
        "    # numerical errors can lead to frac values less than 0 or more than 1; get\n",
        "    # rid of this\n",
        "    #steady_state_soln = np.maximum(0, np.minimum(steady_state_soln, 1.0))\n",
        "\n",
        "    plt.sca(ax[0,i])\n",
        "    plt.scatter(\n",
        "        ulat_ocnvec[gp15_idxs],\n",
        "        depth_ocnvec[gp15_idxs],\n",
        "        c=steady_state_soln[gp15_idxs]\n",
        "    )\n",
        "    plt.ylim(plt.ylim()[1], plt.ylim()[0])\n",
        "    plt.ylabel(\"Depth\")\n",
        "    plt.xlabel(\"Latitude\")\n",
        "    plt.title(endmem+\" OCIM fractions\")\n",
        "    plt.colorbar()\n",
        "\n",
        "    plt.sca(ax[1,i])\n",
        "    plt.scatter(\n",
        "        ulat_ocnvec[gp15_idxs],\n",
        "        depth_ocnvec[gp15_idxs],\n",
        "        c=steady_state_soln[gp15_idxs],\n",
        "        vmin=0.0, vmax=1.0\n",
        "    )\n",
        "    plt.ylim(plt.ylim()[1], plt.ylim()[0])\n",
        "    plt.ylabel(\"Depth\")\n",
        "    plt.xlabel(\"Latitude\")\n",
        "    plt.title(endmem+\" OCIM fractions (0-1 scale)\")\n",
        "    plt.colorbar()\n",
        "\n",
        "plt.show()\n",
        "\n",
        "#save endmemname_to_ocimfracs for use in other notebooks\n",
        "np.save(\"endmemname_to_ocimfracs\", endmemname_to_ocimfracs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L9y7UQxpOAsn"
      },
      "source": [
        "# Perform OMP analysis for the GP02 transect\n",
        "\n",
        "We will use the pyompa implementation of OMP, as this has several additional features/improvements over traditional OMP analysis."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wanUDeIYOmtM"
      },
      "source": [
        "## Download and load pyompa end-member definitions\n",
        "\n",
        "We focus on intermediate and deep waters here. We will use the gp02wma package."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xr-nk-GXOZg2"
      },
      "source": [
        "interanddeep_endmember_df = GP02wma.load_interanddeep_endmember_df(\n",
        "    df_url=\"https://raw.githubusercontent.com/nitrogenlab/GP15_watermassanalysis/main/GP15_intermediateanddeep_endmemberswithsubtypes.csv\",\n",
        "    df_file_name=\"GP15_intermediateanddeep_endmemberswithsubtypes.csv\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MCHANJ1elJPg"
      },
      "source": [
        "## Run pyompa\n",
        "\n",
        "We use the settings in the GP02wma package"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HzEFoE8OicOZ"
      },
      "source": [
        "pyompa_soln = GP02wma.get_pyompa_soln(\n",
        "    obs_df=gp15_intermediateanddeep,\n",
        "    endmember_df_touse=interanddeep_endmember_df)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BvvAz3jYloAe"
      },
      "source": [
        "## Plot pyompa base solution"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E0sxxs4SaPmp"
      },
      "source": [
        "Plot the residuals"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eXLDKsT6O6-1"
      },
      "source": [
        "from pyompa import plot_ompasoln_residuals\n",
        "plot_ompasoln_residuals(pyompa_soln, xaxis_colname=\"lat\", yaxis_colname=\"Depth\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0aScf86vaR_O"
      },
      "source": [
        "Plot the endmember fractions in the \"base\" solution (this is just one solution, without the uncertainty analysis)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RpqhTep3aXtg"
      },
      "source": [
        "from pyompa import plot_ompasoln_endmember_fractions\n",
        "plot_ompasoln_endmember_fractions(pyompa_soln, xaxis_colname=\"lat\",\n",
        "                                  yaxis_colname=\"Depth\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Iwb_KU5ml8D8"
      },
      "source": [
        "# Use OCIM to refine the OMPA results\n",
        "\n",
        "Our pyompa system is underdetermined, i.e. the solution is ambiguous. We can use OCIM to select a specific solution that achieves similar residuals to the best pyompa solution and also matches the OCIM solution as closely as possible"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-qTUEB8EmZVP"
      },
      "source": [
        "!pip uninstall -y pyompa\n",
        "%cd /content/\n",
        "!rm -rf pyompa\n",
        "!git clone https://github.com/nitrogenlab/pyompa\n",
        "%cd /content/pyompa\n",
        "!git checkout devsmallchanges\n",
        "!git log -1\n",
        "!pip install .\n",
        "%cd /content/\n",
        "\n",
        "%reload_ext autoreload\n",
        "%autoreload 2\n",
        "from importlib import reload\n",
        "import pyompa\n",
        "reload(pyompa.endmemberpenaltyfunc)\n",
        "reload(pyompa)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "piTpaojEJtVH"
      },
      "source": [
        "#set up the matrices needed for enforcing the penalties\n",
        "obj_weights = np.zeros((\n",
        "    len(pyompa_soln.endmembername_to_indices),\n",
        "    len(pyompa_soln.endmember_names) + pyompa_soln.ompa_problem.num_converted_variables))\n",
        "target_endmem_fracs = np.zeros((len(gp15_idxs),\n",
        "                                len(pyompa_soln.endmembername_to_indices)))\n",
        "\n",
        "for endmemberoverallidx, (endmembername, endmember_idxs) in enumerate(pyompa_soln.endmembername_to_indices.items()):\n",
        "    obj_weights[endmemberoverallidx, endmember_idxs] = 1.0\n",
        "    target_endmem_fracs[:, endmemberoverallidx] =\\\n",
        "      endmemname_to_ocimfracs[endmembername]\n",
        "\n",
        "#compute the solution - setting resids to 0 constrains them to be no worse\n",
        "# than what was obtained in the optimal pyompa soln\n",
        "max_resids = 0.0*np.array([0.001, 0.004, 0.4, 0.1, 0.015, 0.2])\n",
        "ocim_pyompa_soln = pyompa.ompacore.OMPASoln.core_quantify_ambiguity_via_residual_limits(\n",
        "    self=pyompa_soln, obj_weights=obj_weights,\n",
        "    max_resids=max_resids, retain_original_penalties=True,\n",
        "    target_endmem_fracs=target_endmem_fracs,\n",
        "    verbose=True, max_iter=1000000)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q76C3sbTJxql"
      },
      "source": [
        "from pyompa import plot_ompasoln_residuals, plot_ompasoln_endmember_fractions\n",
        "plot_ompasoln_residuals(ocim_pyompa_soln, xaxis_colname=\"lat\", yaxis_colname=\"Depth\")\n",
        "plot_ompasoln_endmember_fractions(ocim_pyompa_soln, xaxis_colname=\"lat\", yaxis_colname=\"Depth\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "ZFEFfXDhhhd2"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}