MatteAnything.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuromorph/Matte-Anything/blob/main/MatteAnything.ipynb)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "clXthc2IBs3z"
      },
      "source": [
        "**Colab to run [Matte Anything](https://github.com/hustvl/Matte-Anything) demo**\n",
        "\n",
        "This notebook goes through setting up Matte Anything demo along with its dependencies.\n",
        "In [this version](https://github.com/neuromorph/Matte-Anything), I have also added additional support for:\n",
        "\n",
        "*   Use text input for foreground objects (along with text settings), instead of selecting points in image\n",
        "*   Change transparency settings to better tune for images with transparent objects\n",
        "\n",
        "\n",
        "Follow the setup below in Colab after selecting GPU runtime and finally you will get a shareable link of Gradio app for the demo.\n",
        "In the app, follow steps below:\n",
        "\n",
        "1.   Click to upload an image for background removal/replacement\n",
        "2.   Click on 'foreground_point' radio button and then click on image to select some points as foreground object (one by one)\n",
        "3.   Do the same for background points by clicking on 'background_point' button\n",
        "4.   Instead of selecting points, you can also just provide text input describing foreground objects. Be concise and precise.\n",
        "5.   Use sliders for settings:\n",
        "     * Thresholds for boxes selected as per input text\n",
        "     * Trimap: morphology operations like erosion, dilation\n",
        "     * Transparency: input text and box thresholds\n",
        "5.   Click on Start button\n",
        "6.   Explore tabs below the source image once the output images are loaded\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "1f2bGe4sDX4F"
      },
      "outputs": [],
      "source": [
        "import os"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-yNiZOtQBrA8",
        "outputId": "dacc5a86-210d-45a4-fb36-32363c67fae8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "sample_data\n"
          ]
        }
      ],
      "source": [
        "!ls"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jS3YUJ9YCRHd"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/neuromorph/Matte-Anything.git"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "DgslgTZtDbIw"
      },
      "outputs": [],
      "source": [
        "os.chdir(\"./Matte-Anything\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TnRi4MOkCmSY",
        "outputId": "28f13f5e-81cb-4757-a418-8cc948b87210"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "configs  figs\t  main.py\t       matte_anything.py  README.md\n",
            "engine\t LICENSE  MatteAnything.ipynb  modeling\t\t  requirements.txt\n"
          ]
        }
      ],
      "source": [
        "!ls"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xx768PK2Hq2z",
        "outputId": "744b6591-bb32-4d86-8e51-429ea8ff168f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (58.2.0)\n"
          ]
        }
      ],
      "source": [
        "!pip install setuptools"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I12e-cs4JGKr"
      },
      "outputs": [],
      "source": [
        "!pip install git+https://github.com/facebookresearch/segment-anything.git"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "o8KL833gJSFL"
      },
      "outputs": [],
      "source": [
        "!python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'\n",
        "# !pip install -r requirements.txt\n",
        "# Error installing opencv so skipping requirements.txt, check next cell note."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ia5bjW3_LIT8",
        "outputId": "d893fefd-fc5a-4c7d-d211-21f3000fdbb3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: opencv-python in /usr/local/lib/python3.10/dist-packages (4.7.0.72)\n",
            "Requirement already satisfied: numpy>=1.21.2 in /usr/local/lib/python3.10/dist-packages (from opencv-python) (1.22.4)\n"
          ]
        }
      ],
      "source": [
        "# NOTE! The requirements.txt has a specific version of opencv-python which was consistently failing to install. However going with default version also works.\n",
        "!pip install opencv-python"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "S9weXSGtLogg"
      },
      "outputs": [],
      "source": [
        "# Installing packages from requirements.txt\n",
        "!pip install torch torchvision tensorboard timm==0.5.4 easydict wget scikit-image gradio fairscale"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ir5gC44iMGH3"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/IDEA-Research/GroundingDINO.git\n",
        "os.chdir(\"./GroundingDINO\")\n",
        "!pip install -e ."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "_ARSblSUMw_s"
      },
      "outputs": [],
      "source": [
        "# Make directory 'pretrained' inside 'Matte-Anything'\n",
        "os.chdir(\"../\")\n",
        "os.mkdir(\"./pretrained\")\n",
        "os.chdir(\"./pretrained\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TP1q9iN0N_z_"
      },
      "outputs": [],
      "source": [
        "# Download pretrained models for Segment Anything, VitMatte and GroundingDINO under 'pretrained' directory\n",
        "!wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth\n",
        "!wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth\n",
        "!gdown 1d97oKuITCeWgai2Tf3iNilt6rMSSYzkW"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GKslRRFjQEJ8"
      },
      "outputs": [],
      "source": [
        "# Check that you have 'GroundingDINO' and 'pretrained' directories inside Matte-Anything dir\n",
        "os.chdir(\"../\")\n",
        "!ls"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HNeHXSbKQHfk"
      },
      "outputs": [],
      "source": [
        "# Running this cell will start a Gradio webapp and the url for the same can be found at the end of the output\n",
        "!python matte_anything.py\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DWH4zjCAROTe"
      },
      "outputs": [],
      "source": [
        "# This step is only needed to pull updated code from git when some code changes are to be tested\n",
        "!git pull origin main"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}