lightly_solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This jupyter notebook must run on a x86 CPU (Intel or AMD), which excludes Apple Mx CPUs. We recommend running it on a Linux machine. It works both with and without a Nvidia GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into './dataset_clothing_images'...\n",
      "Receiving objects: 100% (376/376), 9.55 MiB | 44.65 MiB/s, done.\n"
     ]
    }
   ],
   "source": [
    "# Set the path to the dataset.\n",
    "# Here we download and use a dataset containing 354 images showing clothes.\n",
    "# If you want to use your own dataset, just set the dataset_path to it.\n",
    "dataset_path = \"dataset_clothing_images\"\n",
    "!git clone https://github.com/lightly-ai/dataset_clothing_images.git {dataset_path}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "latest: Pulling from lightly/worker\n",
      "Status: Image is up to date for lightly/worker:latest\n",
      "[2024-03-26 12:27:06] Lightly Worker Solution v2.11.1\u001b[0m\n",
      "[2024-03-26 12:27:06] Congratulations! It looks like the Lightly container is running!\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Install the Lightly Python Client\n",
    "!pip3 install lightly\n",
    "\n",
    "# Install the Lightly Worker and do a quick sanity check\n",
    "# If these commands fail, follow our docker installation guide at https://docs.lightly.ai/docs/install-lightly#docker\n",
    "!docker pull lightly/worker:latest\n",
    "!docker run --shm-size=\"1024m\" --rm -it lightly/worker:latest sanity_check=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Schedule a run\n",
    "from lightly.api import ApiWorkflowClient\n",
    "from lightly.openapi_generated.swagger_client import DatasetType, DatasourcePurpose\n",
    "\n",
    "lightly_token = \"CHANGE_ME\" # Copy the API_TOKEN from https://app.lightly.ai/preferences\n",
    "\n",
    "# Create the Lightly client to connect to the API.\n",
    "client = ApiWorkflowClient(token=lightly_token)\n",
    "\n",
    "# Register the Lightly Worker.\n",
    "worker_id = client.register_compute_worker(name=\"clothing-worker\")\n",
    "\n",
    "# Create the dataset on the Lightly Platform.\n",
    "# If a dataset with this name already exists, you have to change the name.\n",
    "client.create_dataset(\n",
    "    dataset_name=\"clothing-small\",\n",
    "    dataset_type=DatasetType.IMAGES,\n",
    ")\n",
    "\n",
    "# Configure the datasource.\n",
    "client.set_local_config(\n",
    "    purpose=DatasourcePurpose.INPUT,\n",
    ")\n",
    "client.set_local_config(\n",
    "    purpose=DatasourcePurpose.LIGHTLY,\n",
    ")\n",
    "\n",
    "# Schedule a run on the dataset to select 50 samples.\n",
    "scheduled_run_id = client.schedule_compute_worker_run(\n",
    "    worker_config={\"shutdown_when_job_finished\": True},\n",
    "    selection_config={\n",
    "        \"n_samples\": 50,\n",
    "        \"strategies\": [\n",
    "            {\"input\": {\"type\": \"EMBEDDINGS\"}, \"strategy\": {\"type\": \"DIVERSITY\"}}\n",
    "        ],\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2024-03-26 12:27:54] Lightly Worker Solution v2.11.1\u001b[0m\n",
      "[2024-03-26 12:27:55] You are using docker build: Tue Mar 12 07:56:29 UTC 2024.\u001b[0m\n",
      "[2024-03-26 12:27:55] Starting worker with id '65806b455ca68c93b29ad6b3'...\u001b[0m\n",
      "\u001b[93m[2024-03-26 12:27:55] Worker 2.11.1 can only process jobs scheduled with Lightly Python client 1.5 or higher.\u001b[0m\n",
      "[2024-03-26 12:27:55] Worker with labels '[]' started. Waiting for jobs...\u001b[0m\n",
      "[2024-03-26 12:27:56] Found 1 open jobs.\u001b[0m\n",
      "[2024-03-26 12:27:57] Started job with job_id '6602bf3ef5e7d0833c9264e8'.\u001b[0m\n",
      "...\n",
      "...\n",
      "[2024-03-26 12:29:21] Done!\u001b[0m\n",
      "[2024-03-26 12:29:24] Finished compute worker run successfully.\u001b[0m\n",
      "[2024-03-26 12:29:24] Shutting down...\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Run the Lightly Worker to process the scheduled run.\n",
    "# If you have a Nvidia GPU, add the flag to use it: --gpus all\n",
    "\n",
    "from pathlib import Path\n",
    "absolute_dataset_path = Path(dataset_path).absolute()\n",
    "absolute_lightly_path = Path(\"lightly\").absolute()\n",
    "\n",
    "!docker run --shm-size=\"1024m\" --rm -it \\\n",
    "    -v {absolute_dataset_path}:/input_mount:ro \\\n",
    "    -v {absolute_lightly_path}:/lightly_mount \\\n",
    "    -e LIGHTLY_TOKEN={lightly_token} \\\n",
    "    -e LIGHTLY_WORKER_ID={worker_id} \\\n",
    "    lightly/worker:latest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! You succesfully ran the Lightly solution.\n",
    "Now you can view and explore the dataset interactively on the [Lightly Platform](https://app.lightly.ai).\n",
    "To not only see the metadata and distribution, but also the images itself, you need to serve them from your local disk to your local browser by using the `lightly-serve` CLI command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting server, listening at 'localhost:3456'\n",
      "Serving files in '/GitHub/lightly-solution-all-in-one-notebook/dataset_clothing_images' and '/GitHub/lightly-solution-all-in-one-notebook/lightly'\n"
     ]
    }
   ],
   "source": [
    "!lightly-serve input_mount={absolute_dataset_path} lightly_mount={absolute_lightly_path}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case your browser runs on a different machine than your notebook, you also need to forward a port, see our [docs](https://docs.lightly.ai/docs/local-storage#view-local-data-in-remote-machine-in-lightly-platform)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}