Using_python_poetry_in_Google_Colab.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "E0fVeGsLm2UE"
      ],
      "toc_visible": true,
      "authorship_tag": "ABX9TyO1ADk/gOlAdNcozm1H5Tr4",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/elise-chin-adway/poetry-and-colab/blob/main/Using_python_poetry_in_Google_Colab.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Using `poetry` in Colab\n"
      ],
      "metadata": {
        "id": "bQOhcC-dmuFv"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vHNnaVgjLTi2",
        "outputId": "3f8a508a-3668-4eb9-bf05-9c121a553179"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/gdrive\n"
          ]
        }
      ],
      "source": [
        "from google.colab import drive\n",
        "drive.mount(\"/content/gdrive\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1. Creating your own poetry project\n",
        "---\n",
        "\n"
      ],
      "metadata": {
        "id": "qJcjPHZGFAh4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1.1 Create the test directory on Drive"
      ],
      "metadata": {
        "id": "E0fVeGsLm2UE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Move in your Drive\n",
        "%cd /content/gdrive/MyDrive/"
      ],
      "metadata": {
        "id": "gpS82OFKLzt4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4db31035-8ca2-464e-a9c5-ae97b5e230e5"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/gdrive/MyDrive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Create and move in the new project directory\n",
        "!rm -rf test-poetry\n",
        "!mkdir test-poetry\n",
        "%cd test-poetry"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ac_gJaM0lX1H",
        "outputId": "71293b0c-976c-4a94-f438-b76ea2215067"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/gdrive/MyDrive/test-poetry\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1.2 Install `poetry` "
      ],
      "metadata": {
        "id": "Vuh83_ltGnN4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Install poetry from pip\n",
        "!pip install poetry"
      ],
      "metadata": {
        "id": "mxxpyUXEGmRe"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If you want poetry not to create the virtual environment in the project folder, then run this command `!poetry config virtualenvs.in-project false`, otherwise, run this command `!poetry config virtualenvs.in-project true`.\n",
        "\n",
        "In this tutorial, we will install the packages permanently in Colab, and for convenience, we will create the virtual environment in the project folder.\n",
        "\n"
      ],
      "metadata": {
        "id": "9zx6RYc6XgSR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Configure poetry to create virtual environments in the project folder\n",
        "!poetry config virtualenvs.in-project true"
      ],
      "metadata": {
        "id": "3sNSIT0eKHIY"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Since it is a new project, we need to initialize it with `poetry`. Run `poetry init` and do not specify any dependencies. We will do it later."
      ],
      "metadata": {
        "id": "5qYT3jyCC90M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create the pyproject.toml file\n",
        "!poetry init"
      ],
      "metadata": {
        "id": "rDHrabJ3M-Gk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "de02f623-1253-42da-c7d4-6cc0d03a8d3d"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "This command will guide you through creating your \u001b[34mpyproject.toml\u001b[39m config.\n",
            "\n",
            "\u001b[36mPackage name [\u001b[39m\u001b[32mtest-poetry\u001b[39m\u001b[36m]: \u001b[39m \n",
            "\u001b[36mVersion [\u001b[39m\u001b[32m0.1.0\u001b[39m\u001b[36m]: \u001b[39m \n",
            "\u001b[36mDescription []: \u001b[39m \n",
            "\u001b[36mAuthor [\u001b[39m\u001b[32mNone\u001b[39m\u001b[36m, n to skip]: \u001b[39m Elise Chin\n",
            "\u001b[36mLicense []: \u001b[39m \n",
            "\u001b[36mCompatible Python versions [\u001b[39m\u001b[32m^3.8\u001b[39m\u001b[36m]: \u001b[39m \n",
            "\n",
            "\u001b[36mWould you like to define your main dependencies interactively? (yes/no)\u001b[39m [\u001b[32myes\u001b[39m] no\n",
            "\u001b[36mWould you like to define your development dependencies interactively? (yes/no)\u001b[39m [\u001b[32myes\u001b[39m] no\n",
            "\u001b[34mGenerated file\u001b[39m\n",
            "\n",
            "[tool.poetry]\n",
            "name = \"test-poetry\"\n",
            "version = \"0.1.0\"\n",
            "description = \"\"\n",
            "authors = [\"Elise Chin\"]\n",
            "readme = \"README.md\"\n",
            "packages = [{include = \"test_poetry\"}]\n",
            "\n",
            "[tool.poetry.dependencies]\n",
            "python = \"^3.8\"\n",
            "\n",
            "\n",
            "[build-system]\n",
            "requires = [\"poetry-core\"]\n",
            "build-backend = \"poetry.core.masonry.api\"\n",
            "\n",
            "\n",
            "\u001b[36mDo you confirm generation? (yes/no)\u001b[39m [\u001b[32myes\u001b[39m] yes\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "*pyproject.toml* is created."
      ],
      "metadata": {
        "id": "TJIki3oDDpNU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1.3 Add a new package <a name=\"add-new-package\"></a>\n",
        "\n",
        "Suppose you want to add a new package, for example `torch`.\n",
        "\n",
        "According to poetry's documentation, to install a new package, we need to run `poetry add <package>`. Running this command with modify *pyproject.toml* automatically with the new dependency. However, this only works on your local computer. \n",
        "\n",
        "On Colab, we need add the new dependency by hand in *pyproject.toml*. Still, to know which version of the dependency to install, we run `poetry add <package>`:"
      ],
      "metadata": {
        "id": "ADDCaj1FCnRe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!poetry add torch"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Cf_eYBWEhwP5",
        "outputId": "2cb1a396-e913-4553-9e8d-d6dcff96dd7a"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Creating virtualenv \u001b[36mtest-poetry\u001b[39m in /content/gdrive/MyDrive/test-poetry/.venv\n",
            "Using version \u001b[39;1m^1.13.1\u001b[39;22m for \u001b[36mtorch\u001b[39m\n",
            "\n",
            "\u001b[34mUpdating dependencies\u001b[39m\n",
            "\u001b[2K\u001b[34mResolving dependencies...\u001b[39m \u001b[39;2m(0.5s)\u001b[39;22m\n",
            "\n",
            "\u001b[34mWriting lock file\u001b[39m\n",
            "\n",
            "\u001b[39;1mPackage operations\u001b[39;22m: \u001b[34m6\u001b[39m installs, \u001b[34m1\u001b[39m update, \u001b[34m0\u001b[39m removals\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "3 points:\n",
        "- Since we ran this command for the first time, a virtual environment is created in the project folder. By default, it is called *.venv*;\n",
        "- The version for torch is indicated;\n",
        "- The *poetry.lock* file is created.\n",
        "\n",
        "Let's add `torch = \"^1.13.1\"` in *pyproject.toml* `[tool.poetry.dependencies]` section, after line `python = \"^3.8\"`."
      ],
      "metadata": {
        "id": "WgCw1nxtHEQk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's save the virtual env name and python version in global variables."
      ],
      "metadata": {
        "id": "zv-bkprBHdMN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "VENV_PATH = \"/content/gdrive/MyDrive/test-poetry/.venv/lib/python3.8/site-packages\""
      ],
      "metadata": {
        "id": "Gm7SuPrYLIVG"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's see the packages installed in the virtual environment."
      ],
      "metadata": {
        "id": "PUp-tfOBHjCu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!ls $VENV_PATH"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "akeGX5x0aYWM",
        "outputId": "03188d57-cc80-4feb-9129-519d72673ff7"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "_distutils_hack\t\t  pkg_resources\t\t\t_virtualenv.pth\n",
            "distutils-precedence.pth  __pycache__\t\t\t_virtualenv.py\n",
            "pip\t\t\t  setuptools\t\t\twheel\n",
            "pip-23.0.dist-info\t  setuptools-67.1.0.dist-info\twheel-0.38.4.dist-info\n",
            "pip-23.0.virtualenv\t  setuptools-67.1.0.virtualenv\twheel-0.38.4.virtualenv\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "`torch` was not installed. So let's run `poetry install --no-ansi`."
      ],
      "metadata": {
        "id": "m5mponbGEz-K"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!poetry install --no-ansi"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b1JDo_sR_ti9",
        "outputId": "1e6e1ea8-0534-4d76-80e6-d23a3cd80b6b"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Installing dependencies from lock file\n",
            "\n",
            "Package operations: 6 installs, 1 update, 0 removals\n",
            "\n",
            "  • Updating setuptools (67.1.0 -> 67.3.2)\n",
            "  • Installing nvidia-cublas-cu11 (11.10.3.66)\n",
            "  • Installing nvidia-cuda-nvrtc-cu11 (11.7.99)\n",
            "  • Installing nvidia-cuda-runtime-cu11 (11.7.99)\n",
            "  • Installing nvidia-cudnn-cu11 (8.5.0.96)\n",
            "  • Installing typing-extensions (4.5.0)\n",
            "  • Installing torch (1.13.1)\n",
            "\n",
            "/content/gdrive/MyDrive/test-poetry/test_poetry does not contain any element\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's check if the packages were installed:"
      ],
      "metadata": {
        "id": "dXF4_YoeE54I"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!ls $VENV_PATH"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RKBHF_R4qNoc",
        "outputId": "7118f56a-9392-4ad1-e6b6-ca083ba86fff"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "_distutils_hack\t\t\t\t    setuptools\n",
            "distutils-precedence.pth\t\t    setuptools-67.3.2.dist-info\n",
            "functorch\t\t\t\t    torch\n",
            "nvidia\t\t\t\t\t    torch-1.13.1.dist-info\n",
            "nvidia_cublas_cu11-11.10.3.66.dist-info     torchgen\n",
            "nvidia_cuda_nvrtc_cu11-11.7.99.dist-info    typing_extensions-4.5.0.dist-info\n",
            "nvidia_cuda_runtime_cu11-11.7.99.dist-info  typing_extensions.py\n",
            "nvidia_cudnn_cu11-8.5.0.96.dist-info\t    _virtualenv.pth\n",
            "pip\t\t\t\t\t    _virtualenv.py\n",
            "pip-23.0.dist-info\t\t\t    wheel\n",
            "pip-23.0.virtualenv\t\t\t    wheel-0.38.4.dist-info\n",
            "pkg_resources\t\t\t\t    wheel-0.38.4.virtualenv\n",
            "__pycache__\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1.4 Run project\n",
        "\n",
        "Now, let's add `poetry` virtual environment to python path so that all installed dependencies can be found by the python interpreter."
      ],
      "metadata": {
        "id": "UUvTMpuwFzuu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os, sys\n",
        "\n",
        "LOCAL_VENV_PATH = '/content/venv' # local notebook\n",
        "os.symlink(VENV_PATH, LOCAL_VENV_PATH) # connect to directory in drive\n",
        "sys.path.insert(0, LOCAL_VENV_PATH)"
      ],
      "metadata": {
        "id": "kKMtQjbrcsO4"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "print(torch.__version__)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yZ0VtLNKGNnk",
        "outputId": "a5345eb7-dc5f-4c7d-d204-21f4b8f7a326"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1.13.1+cu117\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Colab has now `torch` installed from `poetry` project."
      ],
      "metadata": {
        "id": "9Cy-DgoBINoy"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2. Cloning a `poetry` project\n",
        "---\n",
        "\n",
        "Suppose you need to clone a Github repo with `poetry` dependencies manager."
      ],
      "metadata": {
        "id": "oZU9d4ETIFwB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Move in your Drive\n",
        "%cd /content/gdrive/MyDrive/"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "I39oqKszII-K",
        "outputId": "8e79bc2d-04ed-4a11-a4b9-f420082906e5"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/gdrive/MyDrive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2.1 Clone the repository"
      ],
      "metadata": {
        "id": "BmLg5JtGMs1x"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Clone this repository\n",
        "!rm -rf poetry-and-colab/\n",
        "!git clone https://github.com/elise-chin-adway/poetry-and-colab.git"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iEVJ0W_NKnDp",
        "outputId": "459d7776-3d9e-4e87-c9f0-f5c5f58dfe45"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'poetry-and-colab'...\n",
            "remote: Enumerating objects: 16, done.\u001b[K\n",
            "remote: Counting objects: 100% (16/16), done.\u001b[K\n",
            "remote: Compressing objects: 100% (15/15), done.\u001b[K\n",
            "remote: Total 16 (delta 5), reused 0 (delta 0), pack-reused 0\u001b[K\n",
            "Unpacking objects: 100% (16/16), 9.92 KiB | 55.00 KiB/s, done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%cd poetry-and-colab"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NBawwY5YMx4A",
        "outputId": "c2ef6f94-b2c0-4c80-c9c3-c4114f7ed121"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/gdrive/MyDrive/poetry-and-colab\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2.2 Install poetry"
      ],
      "metadata": {
        "id": "8_p102quM4pX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Install poetry from pip\n",
        "!pip install poetry"
      ],
      "metadata": {
        "id": "jR3sw95gM_Do"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If you want poetry not to create the virtual environment in the project folder, then run this command `!poetry config virtualenvs.in-project false`, otherwise, run this command `!poetry config virtualenvs.in-project true`.\n",
        "\n",
        "In this tutorial, we will install the packages permanently in Colab, and for convenience, we will create the virtual environment in the project folder.\n",
        "\n"
      ],
      "metadata": {
        "id": "4StyQxbzdo3Y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Configure poetry to create virtual environments in the project folder\n",
        "!poetry config virtualenvs.in-project true"
      ],
      "metadata": {
        "id": "a6UgmndbNB-u"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!poetry install --no-ansi"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KtGPa3pUNH-X",
        "outputId": "87d1e7ea-e228-4cc0-dfc8-23a7204fb90e"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Creating virtualenv poetry-and-colab in /content/gdrive/MyDrive/poetry-and-colab/.venv\n",
            "Installing dependencies from lock file\n",
            "\n",
            "Package operations: 6 installs, 1 update, 0 removals\n",
            "\n",
            "  • Updating setuptools (67.1.0 -> 67.0.0)\n",
            "  • Installing nvidia-cublas-cu11 (11.10.3.66)\n",
            "  • Installing nvidia-cuda-nvrtc-cu11 (11.7.99)\n",
            "  • Installing nvidia-cuda-runtime-cu11 (11.7.99)\n",
            "  • Installing nvidia-cudnn-cu11 (8.5.0.96)\n",
            "  • Installing typing-extensions (4.4.0)\n",
            "  • Installing torch (1.13.1)\n",
            "\n",
            "/content/gdrive/MyDrive/poetry-and-colab/poetry_and_colab does not contain any element\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Then to add a new package and run a project, it is the same steps as 1.3 and 1.4."
      ],
      "metadata": {
        "id": "9LAiFqSMNP9X"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2.3 Add a new package\n",
        "\n",
        "Go to [step 1.3](#add-new-package)"
      ],
      "metadata": {
        "id": "IFvmnj5Le6YI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2.4 Run project"
      ],
      "metadata": {
        "id": "qlTHziYKe-ah"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os, sys\n",
        "\n",
        "VENV_PATH = \"/content/gdrive/MyDrive/test-poetry/.venv/lib/python3.8/site-packages\"\n",
        "LOCAL_VENV_PATH = '/content/venv' # local notebook\n",
        "os.symlink(VENV_PATH, LOCAL_VENV_PATH) # connect to directory in drive\n",
        "sys.path.insert(0, LOCAL_VENV_PATH)"
      ],
      "metadata": {
        "id": "zlva3R9Pe_1T"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "print(torch.__version__)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xyz_bbzafp3D",
        "outputId": "d6b8b60d-c84d-420a-813c-13a51d2a365e"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1.13.1+cu116\n"
          ]
        }
      ]
    }
  ]
}