docs/tutorials/Getting_Started_2_Setting_Up_Agent.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "dBWyl9fx9C1c"
      },
      "source": [
        "<div align=\"center\">\n",
        "\n",
        "  <a href=\"https://clear.ml\" target=\"_blank\">\n",
        "    <img width=\"512\", src=\"https://github.com/allegroai/clearml/raw/master/docs/clearml-logo.svg\"></a>\n",
        "\n",
        "\n",
        "<br>\n",
        "\n",
        "<h1>Notebook 2: Remote Agent</h1>\n",
        "\n",
        "<br>\n",
        "\n",
        "Hi there! This is the second notebook in the ClearML getting started notebook series, meant to teach you the ropes. ClearML has a lot of modules that you can use, so in this notebook, we'll contiue the progress that we made in <a href=\"https://colab.research.google.com/github/allegroai/clearml/blob/master/docs/tutorials/Getting_Started_1_Experiment_Management.ipynb\" target=\"_blank\">our first notebook</a>: <a href=\"https://app.clear.ml/workers-and-queues/workers\" target=\"_blank\">Remote Agents.</a>\n",
        "\n",
        "You can find out more details about the other ClearML modules and the technical specifics of each in <a href=\"https://clear.ml/docs\" target=\"_blank\">our documentation.</a>\n",
        "\n",
        "<table>\n",
        "<tbody>\n",
        "  <tr>\n",
        "    <td>Step 1: Experiment Management</td>\n",
        "    <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/allegroai/clearml/blob/master/docs/tutorials/Getting_Started_1_Experiment_Management.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a></td>\n",
        "  </tr>\n",
        "  <tr>\n",
        "    <td><b>Step 2: Remote Agent</b></td>\n",
        "    <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/allegroai/clearml/blob/master/docs/tutorials/Getting_Started_2_Setting_Up_Agent.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a></td>\n",
        "  </tr>\n",
        "  <tr>\n",
        "    <td>Step 3: Remote Task Execution</td>\n",
        "    <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/allegroai/clearml/blob/master/docs/tutorials/Getting_Started_3_Remote_Execution.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a></td>\n",
        "  </tr>\n",
        "</tbody>\n",
        "</table>\n",
        "\n",
        "</div>"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "LjRqze6s96Ri"
      },
      "source": [
        "# 📦 Setup\n",
        "\n",
        "ClearML allows you to schedule and run your experiments on any number of clustered remote machines. In order to coordinate this, you will have to turn these remote machines into ClearML agents.\n",
        "\n",
        "A ClearML agent is any remote machine that is running the ClearML agent daemon. This agent can then listen to one or more queues and pull tasks to execute. Multiple agents can work together as a cluster to deal with large amounts of incoming tasks.\n",
        "\n",
        "To keep things simple, let's turn this very notebook into a ClearML agent! (I told you \"any\" remote machine would work, did I not 😉 That includes this colab machine)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MIf4PPrK_oHm"
      },
      "source": [
        "## 🔹 Install the ClearML agent package"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Zx1ZqnbRBRYp",
        "outputId": "cbc294b8-9140-4f0c-f348-12d7769c882b"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade clearml\n",
        "import clearml\n",
        "clearml.browser_login()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hfzUNGlB8-8L",
        "outputId": "fbab0523-a2b6-486d-aa82-49ce093c7118"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting clearml-agent\n",
            "  Downloading clearml_agent-1.5.1-py3-none-any.whl (397 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m397.7/397.7 KB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: furl<2.2.0,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from clearml-agent) (2.1.3)\n",
            "Requirement already satisfied: python-dateutil<2.9.0,>=2.4.2 in /usr/local/lib/python3.8/dist-packages (from clearml-agent) (2.8.2)\n",
            "Collecting virtualenv<21,>=16\n",
            "  Downloading virtualenv-20.19.0-py3-none-any.whl (8.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.7/8.7 MB\u001b[0m \u001b[31m23.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyparsing<3.1.0,>=2.0.3 in /usr/local/lib/python3.8/dist-packages (from clearml-agent) (3.0.9)\n",
            "Requirement already satisfied: pathlib2<2.4.0,>=2.3.0 in /usr/local/lib/python3.8/dist-packages (from clearml-agent) (2.3.7.post1)\n",
            "Requirement already satisfied: pyjwt<2.7.0,>=2.4.0 in /usr/local/lib/python3.8/dist-packages (from clearml-agent) (2.4.0)\n",
            "Requirement already satisfied: PyYAML<6.1,>=3.12 in /usr/local/lib/python3.8/dist-packages (from clearml-agent) (6.0)\n",
            "Requirement already satisfied: urllib3<1.27.0,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from clearml-agent) (1.24.3)\n",
            "Requirement already satisfied: jsonschema<5.0.0,>=2.6.0 in /usr/local/lib/python3.8/dist-packages (from clearml-agent) (4.3.3)\n",
            "Requirement already satisfied: attrs<23.0.0,>=18.0 in /usr/local/lib/python3.8/dist-packages (from clearml-agent) (22.2.0)\n",
            "Requirement already satisfied: six<1.17.0,>=1.13.0 in /usr/local/lib/python3.8/dist-packages (from clearml-agent) (1.15.0)\n",
            "Requirement already satisfied: requests<2.29.0,>=2.20.0 in /usr/local/lib/python3.8/dist-packages (from clearml-agent) (2.25.1)\n",
            "Requirement already satisfied: psutil<5.10.0,>=3.4.2 in /usr/local/lib/python3.8/dist-packages (from clearml-agent) (5.4.8)\n",
            "Requirement already satisfied: orderedmultidict>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from furl<2.2.0,>=2.0.0->clearml-agent) (1.0.1)\n",
            "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema<5.0.0,>=2.6.0->clearml-agent) (0.19.3)\n",
            "Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema<5.0.0,>=2.6.0->clearml-agent) (5.12.0)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests<2.29.0,>=2.20.0->clearml-agent) (4.0.0)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests<2.29.0,>=2.20.0->clearml-agent) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests<2.29.0,>=2.20.0->clearml-agent) (2022.12.7)\n",
            "Requirement already satisfied: platformdirs<4,>=2.4 in /usr/local/lib/python3.8/dist-packages (from virtualenv<21,>=16->clearml-agent) (3.0.0)\n",
            "Collecting distlib<1,>=0.3.6\n",
            "  Downloading distlib-0.3.6-py2.py3-none-any.whl (468 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m468.5/468.5 KB\u001b[0m \u001b[31m17.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: filelock<4,>=3.4.1 in /usr/local/lib/python3.8/dist-packages (from virtualenv<21,>=16->clearml-agent) (3.9.0)\n",
            "Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0->jsonschema<5.0.0,>=2.6.0->clearml-agent) (3.14.0)\n",
            "Installing collected packages: distlib, virtualenv, clearml-agent\n",
            "Successfully installed clearml-agent-1.5.1 distlib-0.3.6 virtualenv-20.19.0\n"
          ]
        }
      ],
      "source": [
        "%pip install --upgrade clearml-agent"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w4u0JDS-CIlh"
      },
      "source": [
        "# 🧠 Start the agent\n",
        "\n",
        "This section is actually pretty simple! To setup this notebook machine as a ClearML agent, run the following command.\n",
        "\n",
        "(We're adding `--foreground` so you can see your tasks being executed here later!)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Rf01oyleCL0a"
      },
      "outputs": [],
      "source": [
        "!clearml-agent daemon --queue \"default\" --foreground"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "rcN7wa3yDsNE"
      },
      "source": [
        "# 🥾 Next steps\n",
        "\n",
        "The agent will keep running unless manually stopped, but while it's running it is ready for tasks! So let's go to the [next notebook](https://colab.research.google.com/github/allegroai/clearml/blob/master/docs/tutorials/Getting_Started_3_Remote_Execution.ipynb) of this series and start enqueuing some tasks for this agent to work on!"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<table>\n",
        "<tbody>\n",
        "  <tr>\n",
        "    <td>Step 1: Experiment Management</td>\n",
        "    <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/allegroai/clearml/blob/master/docs/tutorials/Getting_Started_1_Experiment_Management.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a></td>\n",
        "  </tr>\n",
        "  <tr>\n",
        "    <td>Step 2: Remote Agent</td>\n",
        "    <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/allegroai/clearml/blob/master/docs/tutorials/Getting_Started_2_Setting_Up_Agent.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a></td>\n",
        "  </tr>\n",
        "  <tr>\n",
        "    <td><b>NEXT UP -> Step 3: Remote Task Execution</b></td>\n",
        "    <td><a target=\"_blank\" href=\"https://colab.research.google.com/github/allegroai/clearml/blob/master/docs/tutorials/Getting_Started_3_Remote_Execution.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a></td>\n",
        "  </tr>\n",
        "</tbody>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": ".venv",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0]"
    },
    "vscode": {
      "interpreter": {
        "hash": "8b483fbf9fa60c6c6195634afd5159f586a30c5c6a9d31fa17f93a17f02fdc40"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}