AICU_SDLoRA2_Lora_Trainer_Hollowstrawberry.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/aicuai/Book-StartGuideSDXL/blob/main/AICU_SDLoRA2_Lora_Trainer_Hollowstrawberry.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 「画像生成AI Stable Diffusion スタートガイド」第3刷\n",
        "\n",
        "## 6.3 自分の画風のLoRAを作る　https://j.aicu.ai/SDLoRA2\n",
        "\n",
        "Ver.20241118\n",
        "\n",
        "初版の [Linaqruf版](https://j.aicu.ai/SDLoRA1x)アーカイブ化により、こちらのHollowstrawberry版への移行を進めています。\n",
        "\n",
        "こちらは解説つきSD1.5系スクリプトになります。\n",
        "SDXL系の場合は原作の[こちら](https://colab.research.google.com/github/hollowstrawberry/kohya-colab/blob/main/Lora_Trainer_XL.ipynb)をお使いください\n",
        "\n"
      ],
      "metadata": {
        "id": "FgM6ACtBdh_G"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rmCPmqFL6hCQ"
      },
      "source": [
        "# ⭐ Lora Trainer by Hollowstrawberry\n",
        "\n",
        "This is based on the work of [Kohya-ss](https://github.com/kohya-ss/sd-scripts) and [Linaqruf](https://colab.research.google.com/github/Linaqruf/kohya-trainer/blob/main/kohya-LoRA-dreambooth.ipynb). Thank you!\n",
        "\n",
        "訳注: 20241118版より原作の Kohya-ssではなく [uYouUs版](https://github.com/uYouUs/sd-scripts) を使っています。\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vJ8clWTZEu-g"
      },
      "source": [
        "### ⭕ Disclaimer 【免責事項】\n",
        "The purpose of this document is to research bleeding-edge technologies in the field of machine learning inference.  \n",
        "Please read and follow the [Google Colab guidelines](https://research.google.com/colaboratory/faq.html) and its [Terms of Service](https://research.google.com/colaboratory/tos_v3.html).\n",
        "\n",
        "このドキュメントの目的は、機械学習推論の分野における最先端の技術を調査することです。Google Colab のガイドラインと利用規約を読んで遵守してください。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dPQlB4djNm3C"
      },
      "source": [
        "| |GitHub|🇬🇧 English|🇪🇸 Spanish|\n",
        "|:--|:-:|:-:|:-:|\n",
        "| 🏠 **Homepage** | [![GitHub](https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/assets/github.svg)](https://github.com/hollowstrawberry/kohya-colab) | | |\n",
        "| 📊 **Dataset Maker** | [![GitHub](https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/assets/github.svg)](https://github.com/hollowstrawberry/kohya-colab/blob/main/Dataset_Maker.ipynb) | [![Open in Colab](https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/assets/colab-badge.svg)](https://colab.research.google.com/github/hollowstrawberry/kohya-colab/blob/main/Dataset_Maker.ipynb) | [![Abrir en Colab](https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/assets/colab-badge-spanish.svg)](https://colab.research.google.com/github/hollowstrawberry/kohya-colab/blob/main/Spanish_Dataset_Maker.ipynb) |\n",
        "| ⭐ **Lora Trainer** | [![GitHub](https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/assets/github.svg)](https://github.com/hollowstrawberry/kohya-colab/blob/main/Lora_Trainer.ipynb) | [![Open in Colab](https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/assets/colab-badge.svg)](https://colab.research.google.com/github/hollowstrawberry/kohya-colab/blob/main/Lora_Trainer.ipynb) | [![Abrir en Colab](https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/assets/colab-badge-spanish.svg)](https://colab.research.google.com/github/hollowstrawberry/kohya-colab/blob/main/Spanish_Lora_Trainer.ipynb) |\n",
        "| 🌟 **XL Lora Trainer** | [![GitHub](https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/assets/github.svg)](https://github.com/hollowstrawberry/kohya-colab/blob/main/Lora_Trainer_XL.ipynb) | [![Open in Colab](https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/assets/colab-badge.svg)](https://colab.research.google.com/github/hollowstrawberry/kohya-colab/blob/main/Lora_Trainer_XL.ipynb) |  |"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "![](https://huggingface.co/AICU/SDXL-LoRA/resolve/main/Chapter6.png)\n",
        "\n",
        "\n",
        "サンプルデータが必要な方は AICUの HuggingFace\n",
        "\n",
        "https://huggingface.co/AICU/SDXL-LoRA/tree/main\n",
        "\n",
        "ZIPファイルはこちらのURLになります\n",
        "\n",
        "https://huggingface.co/AICU/SDXL-LoRA/resolve/main/9shoku0219.zip\n",
        "\n",
        "サンプルデータはSDXLでの学習でも使えるように 1024px × 1024px にトリミングした画像 40枚 を用意してあります。\n",
        "このスクリプトでの学習はSD1.5系なので 512 x 512 で行うことをおすすめしますが、設定を変えて 1024 x 1024 で実施することも可能です（綺麗です）。その場合、時間はだいたい T4 GPUで 1エポックあたり15分以上かかります。\n"
      ],
      "metadata": {
        "id": "-dzKVQOaZSQM"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "WDjkp4scvPgE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "collapsed": true,
        "outputId": "4e10bacd-3ddb-49e8-ce2b-0d96cd9646c8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "📂 Google Drive に接続中...\n",
            "Mounted at /content/drive\n",
            "mkdir: cannot create directory ‘/content/drive/MyDrive/Loras’: File exists\n",
            "/content/drive/MyDrive/Loras\n",
            "--2024-11-18 07:04:00--  https://huggingface.co/AICU/SDXL-LoRA/resolve/main/9shoku0219.zip\n",
            "Resolving huggingface.co (huggingface.co)... 3.171.171.128, 3.171.171.6, 3.171.171.65, ...\n",
            "Connecting to huggingface.co (huggingface.co)|3.171.171.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://cdn-lfs-us-1.hf.co/repos/15/99/15991d153f50a7b2277059791040f32902fc56f04850c101d5fd6ddcf9c1df10/7a0fc3c64061ed228293a381017407bd260ed67742cc6c903c02b223a99ea221?response-content-disposition=inline%3B+filename*%3DUTF-8%27%279shoku0219.zip%3B+filename%3D%229shoku0219.zip%22%3B&response-content-type=application%2Fzip&Expires=1732172640&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTczMjE3MjY0MH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmhmLmNvL3JlcG9zLzE1Lzk5LzE1OTkxZDE1M2Y1MGE3YjIyNzcwNTk3OTEwNDBmMzI5MDJmYzU2ZjA0ODUwYzEwMWQ1ZmQ2ZGRjZjljMWRmMTAvN2EwZmMzYzY0MDYxZWQyMjgyOTNhMzgxMDE3NDA3YmQyNjBlZDY3NzQyY2M2YzkwM2MwMmIyMjNhOTllYTIyMT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSomcmVzcG9uc2UtY29udGVudC10eXBlPSoifV19&Signature=Xg%7EvAZWp8EIr4wZ44qUDVYIYZdkEFsaX5xisJMng8fmAiu1BBIRv1AfkSkYLwtgAWocL7xoZod3QNPiLroDYAHvFEYONH7hAqDKVny8T8DQWHa9LurQkibeIieIxIdIHsoX%7EKJALDb2D7hhLBjIj-NN79QMb8ubrcuhhOHkJu6iVjbwKEMybfPF8Sxzy5H5T1yNWJiE5MLdcjmvfK7bwdT5GxO-7FPQ4Zdadq9BeCHk38RyjLbBRDKIRIxc2H6UORaBCwcGeccMC3lj%7E5ZEFZnpS8p25ZFF7SAvpsbZp-JoSj98qGtUN%7E1qHml4-AKv3ji49ECiwZCELmFC1L1iVKA__&Key-Pair-Id=K24J24Z295AEI9 [following]\n",
            "--2024-11-18 07:04:01--  https://cdn-lfs-us-1.hf.co/repos/15/99/15991d153f50a7b2277059791040f32902fc56f04850c101d5fd6ddcf9c1df10/7a0fc3c64061ed228293a381017407bd260ed67742cc6c903c02b223a99ea221?response-content-disposition=inline%3B+filename*%3DUTF-8%27%279shoku0219.zip%3B+filename%3D%229shoku0219.zip%22%3B&response-content-type=application%2Fzip&Expires=1732172640&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTczMjE3MjY0MH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmhmLmNvL3JlcG9zLzE1Lzk5LzE1OTkxZDE1M2Y1MGE3YjIyNzcwNTk3OTEwNDBmMzI5MDJmYzU2ZjA0ODUwYzEwMWQ1ZmQ2ZGRjZjljMWRmMTAvN2EwZmMzYzY0MDYxZWQyMjgyOTNhMzgxMDE3NDA3YmQyNjBlZDY3NzQyY2M2YzkwM2MwMmIyMjNhOTllYTIyMT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSomcmVzcG9uc2UtY29udGVudC10eXBlPSoifV19&Signature=Xg%7EvAZWp8EIr4wZ44qUDVYIYZdkEFsaX5xisJMng8fmAiu1BBIRv1AfkSkYLwtgAWocL7xoZod3QNPiLroDYAHvFEYONH7hAqDKVny8T8DQWHa9LurQkibeIieIxIdIHsoX%7EKJALDb2D7hhLBjIj-NN79QMb8ubrcuhhOHkJu6iVjbwKEMybfPF8Sxzy5H5T1yNWJiE5MLdcjmvfK7bwdT5GxO-7FPQ4Zdadq9BeCHk38RyjLbBRDKIRIxc2H6UORaBCwcGeccMC3lj%7E5ZEFZnpS8p25ZFF7SAvpsbZp-JoSj98qGtUN%7E1qHml4-AKv3ji49ECiwZCELmFC1L1iVKA__&Key-Pair-Id=K24J24Z295AEI9\n",
            "Resolving cdn-lfs-us-1.hf.co (cdn-lfs-us-1.hf.co)... 18.164.78.17, 18.164.78.6, 18.164.78.101, ...\n",
            "Connecting to cdn-lfs-us-1.hf.co (cdn-lfs-us-1.hf.co)|18.164.78.17|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 25609274 (24M) [application/zip]\n",
            "Saving to: ‘/content/drive/MyDrive/Loras/9shoku0219.zip’\n",
            "\n",
            "/content/drive/MyDr 100%[===================>]  24.42M  38.0MB/s    in 0.6s    \n",
            "\n",
            "2024-11-18 07:04:02 (38.0 MB/s) - ‘/content/drive/MyDrive/Loras/9shoku0219.zip’ saved [25609274/25609274]\n",
            "\n",
            "Archive:  /content/drive/MyDrive/Loras/9shoku0219.zip\n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/1.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/10.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/10c.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/11.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/11c.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/12.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/12c.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/13.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/13c.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/14.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/14c.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/15.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/15c.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/1c.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/2.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/2c.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/3.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/3c.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/4.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/4c.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/5.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/5c.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/6.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/6c.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/7.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/7c.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/8.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/8c.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/9.png  \n",
            "  inflating: /content/drive/MyDrive/Loras/9shoku0219/dataset/9c.png  \n"
          ]
        }
      ],
      "source": [
        "#@markdown ### 📂 AICUデータセットの準備\n",
        "#@markdown あなたの Google Driveに接続し、マイドライブのトップ「Loras」フォルダに[サンプルデータセット](https://huggingface.co/AICU/SDXL-LoRA/tree/main)をダウンロードし、展開します。\n",
        "import os, zipfile\n",
        "\n",
        "download_from = \"https://huggingface.co/AICU/SDXL-LoRA/resolve/main/9shoku0219.zip\" #@param  {type:\"string\"}\n",
        "zip = \"/content/drive/MyDrive/Loras/9shoku0219.zip\" #@param {type:\"string\"}\n",
        "extract_to = \"/content/drive/MyDrive/Loras/9shoku0219/dataset\" #@param {type:\"string\"}\n",
        "\n",
        "if not os.path.exists('/content/drive'):\n",
        "  from google.colab import drive\n",
        "  print(\"📂 Google Drive に接続中...\")\n",
        "  drive.mount('/content/drive')\n",
        "\n",
        "!mkdir /content/drive/MyDrive/Loras\n",
        "!mkdir -p '{extract_to}'\n",
        "%cd /content/drive/MyDrive/Loras\n",
        "!wget '{download_from}' -O '{zip}'\n",
        "#指定ディレクトリにフラットに展開したい\n",
        "!unzip -j '{zip}' -d '{extract_to}'\n",
        "\n",
        "#os.makedirs(extract_to, exist_ok=True)\n",
        "#with zipfile.ZipFile(zip, 'r') as f:\n",
        "#  f.extractall(extract_to)\n",
        "#\n",
        "##print(\"✅ Done\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OglZzI_ujZq-",
        "cellView": "form",
        "collapsed": true,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a7a890b8-612e-4b65-e069-bf9f4804fbe2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "💿 Checking dataset...\n",
            "📁MyDrive/Loras/9shoku0219/dataset\n",
            "📈 Found 30 images with 10 repeats, equaling 300 steps.\n",
            "📉 Divide 300 steps by 2 batch size to get 150.0 steps per epoch.\n",
            "🔮 There will be 10 epochs, for around 1500 total training steps.\n",
            "\n",
            "🏭 Installing dependencies...\n",
            "\n",
            "Cloning into '/content/kohya-trainer'...\n",
            "remote: Enumerating objects: 6254, done.\u001b[K\n",
            "remote: Total 6254 (delta 0), reused 0 (delta 0), pack-reused 6254 (from 1)\u001b[K\n",
            "Receiving objects: 100% (6254/6254), 9.45 MiB | 17.34 MiB/s, done.\n",
            "Resolving deltas: 100% (4478/4478), done.\n",
            "56 packages can be upgraded. Run 'apt list --upgradable' to see them.\n",
            "\u001b[1;33mW: \u001b[0mSkipping acquire of configured file 'main/source/Sources' as repository 'https://r2u.stat.illinois.edu/ubuntu jammy InRelease' does not seem to provide it (sources.list entry misspelt?)\u001b[0m\n",
            "The following additional packages will be installed:\n",
            "  libaria2-0 libc-ares2\n",
            "The following NEW packages will be installed:\n",
            "  aria2 libaria2-0 libc-ares2\n",
            "0 upgraded, 3 newly installed, 0 to remove and 56 not upgraded.\n",
            "Need to get 1,513 kB of archives.\n",
            "After this operation, 5,441 kB of additional disk space will be used.\n",
            "Selecting previously unselected package libc-ares2:amd64.\n",
            "(Reading database ... 123629 files and directories currently installed.)\n",
            "Preparing to unpack .../libc-ares2_1.18.1-1ubuntu0.22.04.3_amd64.deb ...\n",
            "Unpacking libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.3) ...\n",
            "Selecting previously unselected package libaria2-0:amd64.\n",
            "Preparing to unpack .../libaria2-0_1.36.0-1_amd64.deb ...\n",
            "Unpacking libaria2-0:amd64 (1.36.0-1) ...\n",
            "Selecting previously unselected package aria2.\n",
            "Preparing to unpack .../aria2_1.36.0-1_amd64.deb ...\n",
            "Unpacking aria2 (1.36.0-1) ...\n",
            "Setting up libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.3) ...\n",
            "Setting up libaria2-0:amd64 (1.36.0-1) ...\n",
            "Setting up aria2 (1.36.0-1) ...\n",
            "Processing triggers for man-db (2.10.2-1) ...\n",
            "Processing triggers for libc-bin (2.35-0ubuntu3.4) ...\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtcm_debug.so.1 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtcm.so.1 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libur_adapter_opencl.so.0 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libumf.so.0 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libur_loader.so.0 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libhwloc.so.15 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libur_adapter_level_zero.so.0 is not a symbolic link\n",
            "\n",
            "Looking in indexes: https://download.pytorch.org/whl/cu121\n",
            "Collecting torch==2.4\n",
            "  Downloading https://download.pytorch.org/whl/cu121/torch-2.4.0%2Bcu121-cp310-cp310-linux_x86_64.whl (799.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m799.1/799.1 MB\u001b[0m \u001b[31m2.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting xformers\n",
            "  Downloading https://download.pytorch.org/whl/cu121/xformers-0.0.28.post3-cp310-cp310-manylinux_2_28_x86_64.whl (16.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m16.7/16.7 MB\u001b[0m \u001b[31m79.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting triton\n",
            "  Downloading https://download.pytorch.org/whl/triton-3.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (209.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m209.5/209.5 MB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting torchvision==0.19\n",
            "  Downloading https://download.pytorch.org/whl/cu121/torchvision-0.19.0%2Bcu121-cp310-cp310-linux_x86_64.whl (7.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.1/7.1 MB\u001b[0m \u001b[31m89.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch==2.4) (3.16.1)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch==2.4) (4.12.2)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch==2.4) (1.13.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch==2.4) (3.4.2)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch==2.4) (3.1.4)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch==2.4) (2024.10.0)\n",
            "Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch==2.4)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m23.7/23.7 MB\u001b[0m \u001b[31m51.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cuda-runtime-cu12==12.1.105 (from torch==2.4)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m823.6/823.6 kB\u001b[0m \u001b[31m51.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cuda-cupti-cu12==12.1.105 (from torch==2.4)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.1/14.1 MB\u001b[0m \u001b[31m64.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cudnn-cu12==9.1.0.70 (from torch==2.4)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl (664.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m664.8/664.8 MB\u001b[0m \u001b[31m2.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cublas-cu12==12.1.3.1 (from torch==2.4)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m410.6/410.6 MB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cufft-cu12==11.0.2.54 (from torch==2.4)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m121.6/121.6 MB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-curand-cu12==10.3.2.106 (from torch==2.4)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.5/56.5 MB\u001b[0m \u001b[31m13.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cusolver-cu12==11.4.5.107 (from torch==2.4)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m124.2/124.2 MB\u001b[0m \u001b[31m7.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cusparse-cu12==12.1.0.106 (from torch==2.4)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m196.0/196.0 MB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-nccl-cu12==2.20.5 (from torch==2.4)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl (176.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m176.2/176.2 MB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-nvtx-cu12==12.1.105 (from torch==2.4)\n",
            "  Downloading https://download.pytorch.org/whl/cu121/nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m99.1/99.1 kB\u001b[0m \u001b[31m8.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting triton\n",
            "  Downloading https://download.pytorch.org/whl/triton-3.0.0-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (209.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m209.4/209.4 MB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from torchvision==0.19) (1.26.4)\n",
            "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.10/dist-packages (from torchvision==0.19) (11.0.0)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch==2.4) (12.6.77)\n",
            "INFO: pip is looking at multiple versions of xformers to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting xformers\n",
            "  Downloading https://download.pytorch.org/whl/cu121/xformers-0.0.28.post2-cp310-cp310-manylinux_2_28_x86_64.whl (16.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m16.7/16.7 MB\u001b[0m \u001b[31m70.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading https://download.pytorch.org/whl/cu121/xformers-0.0.28.post1-cp310-cp310-manylinux_2_28_x86_64.whl (16.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m16.7/16.7 MB\u001b[0m \u001b[31m64.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading https://download.pytorch.org/whl/cu121/xformers-0.0.28-cp310-cp310-manylinux_2_28_x86_64.whl (16.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m16.7/16.7 MB\u001b[0m \u001b[31m86.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading https://download.pytorch.org/whl/cu121/xformers-0.0.27.post2-cp310-cp310-manylinux2014_x86_64.whl (20.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m20.8/20.8 MB\u001b[0m \u001b[31m84.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch==2.4) (3.0.2)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch==2.4) (1.3.0)\n",
            "Installing collected packages: triton, nvidia-nvtx-cu12, nvidia-nccl-cu12, nvidia-cusparse-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, nvidia-cusolver-cu12, nvidia-cudnn-cu12, torch, xformers, torchvision\n",
            "  Attempting uninstall: nvidia-nccl-cu12\n",
            "    Found existing installation: nvidia-nccl-cu12 2.23.4\n",
            "    Uninstalling nvidia-nccl-cu12-2.23.4:\n",
            "      Successfully uninstalled nvidia-nccl-cu12-2.23.4\n",
            "  Attempting uninstall: nvidia-cusparse-cu12\n",
            "    Found existing installation: nvidia-cusparse-cu12 12.5.4.2\n",
            "    Uninstalling nvidia-cusparse-cu12-12.5.4.2:\n",
            "      Successfully uninstalled nvidia-cusparse-cu12-12.5.4.2\n",
            "  Attempting uninstall: nvidia-curand-cu12\n",
            "    Found existing installation: nvidia-curand-cu12 10.3.7.77\n",
            "    Uninstalling nvidia-curand-cu12-10.3.7.77:\n",
            "      Successfully uninstalled nvidia-curand-cu12-10.3.7.77\n",
            "  Attempting uninstall: nvidia-cufft-cu12\n",
            "    Found existing installation: nvidia-cufft-cu12 11.3.0.4\n",
            "    Uninstalling nvidia-cufft-cu12-11.3.0.4:\n",
            "      Successfully uninstalled nvidia-cufft-cu12-11.3.0.4\n",
            "  Attempting uninstall: nvidia-cuda-runtime-cu12\n",
            "    Found existing installation: nvidia-cuda-runtime-cu12 12.6.77\n",
            "    Uninstalling nvidia-cuda-runtime-cu12-12.6.77:\n",
            "      Successfully uninstalled nvidia-cuda-runtime-cu12-12.6.77\n",
            "  Attempting uninstall: nvidia-cuda-cupti-cu12\n",
            "    Found existing installation: nvidia-cuda-cupti-cu12 12.6.80\n",
            "    Uninstalling nvidia-cuda-cupti-cu12-12.6.80:\n",
            "      Successfully uninstalled nvidia-cuda-cupti-cu12-12.6.80\n",
            "  Attempting uninstall: nvidia-cublas-cu12\n",
            "    Found existing installation: nvidia-cublas-cu12 12.6.3.3\n",
            "    Uninstalling nvidia-cublas-cu12-12.6.3.3:\n",
            "      Successfully uninstalled nvidia-cublas-cu12-12.6.3.3\n",
            "  Attempting uninstall: nvidia-cusolver-cu12\n",
            "    Found existing installation: nvidia-cusolver-cu12 11.7.1.2\n",
            "    Uninstalling nvidia-cusolver-cu12-11.7.1.2:\n",
            "      Successfully uninstalled nvidia-cusolver-cu12-11.7.1.2\n",
            "  Attempting uninstall: nvidia-cudnn-cu12\n",
            "    Found existing installation: nvidia-cudnn-cu12 9.5.1.17\n",
            "    Uninstalling nvidia-cudnn-cu12-9.5.1.17:\n",
            "      Successfully uninstalled nvidia-cudnn-cu12-9.5.1.17\n",
            "  Attempting uninstall: torch\n",
            "    Found existing installation: torch 2.5.1+cu121\n",
            "    Uninstalling torch-2.5.1+cu121:\n",
            "      Successfully uninstalled torch-2.5.1+cu121\n",
            "  Attempting uninstall: torchvision\n",
            "    Found existing installation: torchvision 0.20.1+cu121\n",
            "    Uninstalling torchvision-0.20.1+cu121:\n",
            "      Successfully uninstalled torchvision-0.20.1+cu121\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "torchaudio 2.5.1+cu121 requires torch==2.5.1, but you have torch 2.4.0+cu121 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-9.1.0.70 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.20.5 nvidia-nvtx-cu12-12.1.105 torch-2.4.0+cu121 torchvision-0.19.0+cu121 triton-3.0.0 xformers-0.0.27.post2\n",
            "Collecting accelerate==0.25.0\n",
            "  Downloading accelerate-0.25.0-py3-none-any.whl.metadata (18 kB)\n",
            "Collecting transformers==4.36.2\n",
            "  Downloading transformers-4.36.2-py3-none-any.whl.metadata (126 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m126.8/126.8 kB\u001b[0m \u001b[31m3.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting diffusers==0.25.0 (from diffusers[torch]==0.25.0)\n",
            "  Downloading diffusers-0.25.0-py3-none-any.whl.metadata (19 kB)\n",
            "Collecting ftfy==6.1.1\n",
            "  Downloading ftfy-6.1.1-py3-none-any.whl.metadata (6.1 kB)\n",
            "Collecting opencv-python==4.8.1.78\n",
            "  Downloading opencv_python-4.8.1.78-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB)\n",
            "Collecting einops==0.7.0\n",
            "  Downloading einops-0.7.0-py3-none-any.whl.metadata (13 kB)\n",
            "Collecting pytorch-lightning==1.9.0\n",
            "  Downloading pytorch_lightning-1.9.0-py3-none-any.whl.metadata (23 kB)\n",
            "Collecting bitsandbytes==0.43.0\n",
            "  Downloading bitsandbytes-0.43.0-py3-none-manylinux_2_24_x86_64.whl.metadata (1.8 kB)\n",
            "Collecting prodigyopt==1.0\n",
            "  Downloading prodigyopt-1.0-py3-none-any.whl.metadata (1.2 kB)\n",
            "Collecting lion-pytorch==0.0.6\n",
            "  Downloading lion_pytorch-0.0.6-py3-none-any.whl.metadata (620 bytes)\n",
            "Requirement already satisfied: tensorboard in /usr/local/lib/python3.10/dist-packages (2.17.1)\n",
            "Collecting safetensors==0.4.2\n",
            "  Downloading safetensors-0.4.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)\n",
            "Requirement already satisfied: altair==4.2.2 in /usr/local/lib/python3.10/dist-packages (4.2.2)\n",
            "Collecting easygui==0.98.3\n",
            "  Downloading easygui-0.98.3-py2.py3-none-any.whl.metadata (8.4 kB)\n",
            "Requirement already satisfied: toml==0.10.2 in /usr/local/lib/python3.10/dist-packages (0.10.2)\n",
            "Collecting voluptuous==0.13.1\n",
            "  Downloading voluptuous-0.13.1-py3-none-any.whl.metadata (20 kB)\n",
            "Collecting huggingface-hub==0.20.1\n",
            "  Downloading huggingface_hub-0.20.1-py3-none-any.whl.metadata (12 kB)\n",
            "Requirement already satisfied: imagesize==1.4.1 in /usr/local/lib/python3.10/dist-packages (1.4.1)\n",
            "Collecting rich==13.7.1\n",
            "  Downloading rich-13.7.1-py3-none-any.whl.metadata (18 kB)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.25.0) (1.26.4)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.25.0) (24.2)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate==0.25.0) (5.9.5)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate==0.25.0) (6.0.2)\n",
            "Requirement already satisfied: torch>=1.10.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.25.0) (2.4.0+cu121)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.36.2) (3.16.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.36.2) (2024.9.11)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.36.2) (2.32.3)\n",
            "Collecting tokenizers<0.19,>=0.14 (from transformers==4.36.2)\n",
            "  Downloading tokenizers-0.15.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.36.2) (4.66.6)\n",
            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.10/dist-packages (from diffusers==0.25.0->diffusers[torch]==0.25.0) (8.5.0)\n",
            "Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from diffusers==0.25.0->diffusers[torch]==0.25.0) (11.0.0)\n",
            "Requirement already satisfied: wcwidth>=0.2.5 in /usr/local/lib/python3.10/dist-packages (from ftfy==6.1.1) (0.2.13)\n",
            "Requirement already satisfied: fsspec>2021.06.0 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]>2021.06.0->pytorch-lightning==1.9.0) (2024.10.0)\n",
            "Collecting torchmetrics>=0.7.0 (from pytorch-lightning==1.9.0)\n",
            "  Downloading torchmetrics-1.6.0-py3-none-any.whl.metadata (20 kB)\n",
            "Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.9.0) (4.12.2)\n",
            "Collecting lightning-utilities>=0.4.2 (from pytorch-lightning==1.9.0)\n",
            "  Downloading lightning_utilities-0.11.8-py3-none-any.whl.metadata (5.2 kB)\n",
            "Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair==4.2.2) (0.4)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from altair==4.2.2) (3.1.4)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair==4.2.2) (4.23.0)\n",
            "Requirement already satisfied: pandas>=0.18 in /usr/local/lib/python3.10/dist-packages (from altair==4.2.2) (2.2.2)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair==4.2.2) (0.12.1)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich==13.7.1) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich==13.7.1) (2.18.0)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (1.4.0)\n",
            "Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (1.67.1)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (3.7)\n",
            "Requirement already satisfied: protobuf!=4.24.0,>=3.19.6 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (4.25.5)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (75.1.0)\n",
            "Requirement already satisfied: six>1.9 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (1.16.0)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (0.7.2)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard) (3.1.3)\n",
            "Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]>2021.06.0->pytorch-lightning==1.9.0) (3.10.10)\n",
            "Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair==4.2.2) (24.2.0)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair==4.2.2) (2024.10.1)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair==4.2.2) (0.35.1)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair==4.2.2) (0.21.0)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich==13.7.1) (0.1.2)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas>=0.18->altair==4.2.2) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas>=0.18->altair==4.2.2) (2024.2)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas>=0.18->altair==4.2.2) (2024.2)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.0) (1.13.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.0) (3.4.2)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.0) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.0) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.0) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.0) (9.1.0.70)\n",
            "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.0) (12.1.3.1)\n",
            "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.0) (11.0.2.54)\n",
            "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.0) (10.3.2.106)\n",
            "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.0) (11.4.5.107)\n",
            "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.0) (12.1.0.106)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.0) (2.20.5)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.0) (12.1.105)\n",
            "Requirement already satisfied: triton==3.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.25.0) (3.0.0)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.10.0->accelerate==0.25.0) (12.6.77)\n",
            "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard) (3.0.2)\n",
            "Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata->diffusers==0.25.0->diffusers[torch]==0.25.0) (3.21.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.36.2) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.36.2) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.36.2) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.36.2) (2024.8.30)\n",
            "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0) (2.4.3)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0) (1.3.1)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0) (1.5.0)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0) (6.1.0)\n",
            "Requirement already satisfied: yarl<2.0,>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0) (1.17.1)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0) (4.0.3)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.10.0->accelerate==0.25.0) (1.3.0)\n",
            "Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from yarl<2.0,>=1.12.0->aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]>2021.06.0->pytorch-lightning==1.9.0) (0.2.0)\n",
            "Downloading accelerate-0.25.0-py3-none-any.whl (265 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m265.7/265.7 kB\u001b[0m \u001b[31m13.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading transformers-4.36.2-py3-none-any.whl (8.2 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.2/8.2 MB\u001b[0m \u001b[31m70.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading diffusers-0.25.0-py3-none-any.whl (1.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m61.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading ftfy-6.1.1-py3-none-any.whl (53 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.1/53.1 kB\u001b[0m \u001b[31m5.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading opencv_python-4.8.1.78-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.7/61.7 MB\u001b[0m \u001b[31m13.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading einops-0.7.0-py3-none-any.whl (44 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m44.6/44.6 kB\u001b[0m \u001b[31m3.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pytorch_lightning-1.9.0-py3-none-any.whl (825 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m825.8/825.8 kB\u001b[0m \u001b[31m54.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading bitsandbytes-0.43.0-py3-none-manylinux_2_24_x86_64.whl (102.2 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m102.2/102.2 MB\u001b[0m \u001b[31m8.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading prodigyopt-1.0-py3-none-any.whl (5.5 kB)\n",
            "Downloading lion_pytorch-0.0.6-py3-none-any.whl (4.2 kB)\n",
            "Downloading safetensors-0.4.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m74.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading easygui-0.98.3-py2.py3-none-any.whl (92 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m92.7/92.7 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading voluptuous-0.13.1-py3-none-any.whl (29 kB)\n",
            "Downloading huggingface_hub-0.20.1-py3-none-any.whl (330 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m330.1/330.1 kB\u001b[0m \u001b[31m29.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading rich-13.7.1-py3-none-any.whl (240 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m240.7/240.7 kB\u001b[0m \u001b[31m22.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading lightning_utilities-0.11.8-py3-none-any.whl (26 kB)\n",
            "Downloading tokenizers-0.15.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.6/3.6 MB\u001b[0m \u001b[31m86.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading torchmetrics-1.6.0-py3-none-any.whl (926 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m926.4/926.4 kB\u001b[0m \u001b[31m62.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: voluptuous, easygui, safetensors, prodigyopt, opencv-python, lightning-utilities, ftfy, einops, rich, huggingface-hub, tokenizers, diffusers, transformers, torchmetrics, lion-pytorch, bitsandbytes, accelerate, pytorch-lightning\n",
            "  Attempting uninstall: safetensors\n",
            "    Found existing installation: safetensors 0.4.5\n",
            "    Uninstalling safetensors-0.4.5:\n",
            "      Successfully uninstalled safetensors-0.4.5\n",
            "  Attempting uninstall: opencv-python\n",
            "    Found existing installation: opencv-python 4.10.0.84\n",
            "    Uninstalling opencv-python-4.10.0.84:\n",
            "      Successfully uninstalled opencv-python-4.10.0.84\n",
            "  Attempting uninstall: einops\n",
            "    Found existing installation: einops 0.8.0\n",
            "    Uninstalling einops-0.8.0:\n",
            "      Successfully uninstalled einops-0.8.0\n",
            "  Attempting uninstall: rich\n",
            "    Found existing installation: rich 13.9.4\n",
            "    Uninstalling rich-13.9.4:\n",
            "      Successfully uninstalled rich-13.9.4\n",
            "  Attempting uninstall: huggingface-hub\n",
            "    Found existing installation: huggingface-hub 0.26.2\n",
            "    Uninstalling huggingface-hub-0.26.2:\n",
            "      Successfully uninstalled huggingface-hub-0.26.2\n",
            "  Attempting uninstall: tokenizers\n",
            "    Found existing installation: tokenizers 0.20.3\n",
            "    Uninstalling tokenizers-0.20.3:\n",
            "      Successfully uninstalled tokenizers-0.20.3\n",
            "  Attempting uninstall: diffusers\n",
            "    Found existing installation: diffusers 0.31.0\n",
            "    Uninstalling diffusers-0.31.0:\n",
            "      Successfully uninstalled diffusers-0.31.0\n",
            "  Attempting uninstall: transformers\n",
            "    Found existing installation: transformers 4.46.2\n",
            "    Uninstalling transformers-4.46.2:\n",
            "      Successfully uninstalled transformers-4.46.2\n",
            "  Attempting uninstall: accelerate\n",
            "    Found existing installation: accelerate 1.1.1\n",
            "    Uninstalling accelerate-1.1.1:\n",
            "      Successfully uninstalled accelerate-1.1.1\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "sentence-transformers 3.2.1 requires transformers<5.0.0,>=4.41.0, but you have transformers 4.36.2 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed accelerate-0.25.0 bitsandbytes-0.43.0 diffusers-0.25.0 easygui-0.98.3 einops-0.7.0 ftfy-6.1.1 huggingface-hub-0.20.1 lightning-utilities-0.11.8 lion-pytorch-0.0.6 opencv-python-4.8.1.78 prodigyopt-1.0 pytorch-lightning-1.9.0 rich-13.7.1 safetensors-0.4.2 tokenizers-0.15.2 torchmetrics-1.6.0 transformers-4.36.2 voluptuous-0.13.1\n",
            "Obtaining file:///content/kohya-trainer\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Installing collected packages: library\n",
            "  Running setup.py develop for library\n",
            "Successfully installed library-0.0.0\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.\n",
            "  _torch_pytree._register_pytree_node(\n",
            "/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.\n",
            "  _torch_pytree._register_pytree_node(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "✅ Installation finished in 251 seconds.\n",
            "\n",
            "🔄 Downloading model...\n",
            "\u001b[0m\n",
            "Download Results:\n",
            "gid   |stat|avg speed  |path/URI\n",
            "======+====+===========+=======================================================\n",
            "7197bb|\u001b[1;32mOK\u001b[0m  |   246MiB/s|//content/AnyLoRA_noVae_fp16-pruned.ckpt\n",
            "\n",
            "Status Legend:\n",
            "(OK):download completed.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-2-f57c8d58e051>:482: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
            "  test = load_ckpt(model_file)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "\n",
            "📄 Config saved to /content/drive/MyDrive/Loras/9shoku0219/training_config.toml\n",
            "📄 Dataset config saved to /content/drive/MyDrive/Loras/9shoku0219/dataset_config.toml\n",
            "\n",
            "⭐ Starting trainer...\n",
            "\n",
            "/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.\n",
            "  _torch_pytree._register_pytree_node(\n",
            "/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.\n",
            "  _torch_pytree._register_pytree_node(\n",
            "/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.\n",
            "  _torch_pytree._register_pytree_node(\n",
            "/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.\n",
            "  _torch_pytree._register_pytree_node(\n",
            "/usr/local/lib/python3.10/dist-packages/diffusers/utils/outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.\n",
            "  torch.utils._pytree._register_pytree_node(\n",
            "/usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.\n",
            "  @torch.library.impl_abstract(\"xformers_flash::flash_fwd\")\n",
            "/usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.\n",
            "  @torch.library.impl_abstract(\"xformers_flash::flash_bwd\")\n",
            "/usr/local/lib/python3.10/dist-packages/diffusers/utils/outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.\n",
            "  torch.utils._pytree._register_pytree_node(\n",
            "/usr/local/lib/python3.10/dist-packages/diffusers/utils/outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.\n",
            "  torch.utils._pytree._register_pytree_node(\n",
            "/usr/local/lib/python3.10/dist-packages/diffusers/utils/outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.\n",
            "  torch.utils._pytree._register_pytree_node(\n",
            "Loading settings from /content/drive/MyDrive/Loras/9shoku0219/training_config.toml...\n",
            "/content/drive/MyDrive/Loras/9shoku0219/training_config\n",
            "prepare tokenizer\n",
            "tokenizer_config.json: 100% 905/905 [00:00<00:00, 3.77MB/s]\n",
            "vocab.json: 100% 961k/961k [00:00<00:00, 13.0MB/s]\n",
            "merges.txt: 100% 525k/525k [00:00<00:00, 8.36MB/s]\n",
            "special_tokens_map.json: 100% 389/389 [00:00<00:00, 1.63MB/s]\n",
            "tokenizer.json: 100% 2.22M/2.22M [00:00<00:00, 52.0MB/s]\n",
            "update token length: 225\n",
            "Loading dataset config from /content/drive/MyDrive/Loras/9shoku0219/dataset_config.toml\n",
            "prepare images.\n",
            "found directory /content/drive/MyDrive/Loras/9shoku0219/dataset contains 30 image files\n",
            "No caption file found for 30 images. Training will continue without captions for these images. If class token exists, it will be used. / 30枚の画像にキャプションファイルが見つかりませんでした。これらの画像についてはキャプションなしで学習を続行します。class tokenが存在する場合はそれを使います。\n",
            "/content/drive/MyDrive/Loras/9shoku0219/dataset/1.png\n",
            "/content/drive/MyDrive/Loras/9shoku0219/dataset/10.png\n",
            "/content/drive/MyDrive/Loras/9shoku0219/dataset/10c.png\n",
            "/content/drive/MyDrive/Loras/9shoku0219/dataset/11.png\n",
            "/content/drive/MyDrive/Loras/9shoku0219/dataset/11c.png\n",
            "/content/drive/MyDrive/Loras/9shoku0219/dataset/12.png... and 25 more\n",
            "300 train images with repeating.\n",
            "0 reg images.\n",
            "no regularization images / 正則化画像が見つかりませんでした\n",
            "[Dataset 0]\n",
            "  batch_size: 2\n",
            "  resolution: (1024, 1024)\n",
            "  enable_bucket: True\n",
            "  network_multiplier: 1.0\n",
            "  min_bucket_reso: 320\n",
            "  max_bucket_reso: 1280\n",
            "  bucket_reso_steps: 64\n",
            "  bucket_no_upscale: False\n",
            "\n",
            "  [Subset 0 of Dataset 0]\n",
            "    image_dir: \"/content/drive/MyDrive/Loras/9shoku0219/dataset\"\n",
            "    image_count: 30\n",
            "    num_repeats: 10\n",
            "    shuffle_caption: True\n",
            "    keep_tokens: 1\n",
            "    keep_tokens_separator: \n",
            "    secondary_separator: None\n",
            "    enable_wildcard: False\n",
            "    caption_dropout_rate: 0.0\n",
            "    caption_dropout_every_n_epoches: 0\n",
            "    caption_tag_dropout_rate: 0.0\n",
            "    caption_prefix: None\n",
            "    caption_suffix: None\n",
            "    color_aug: False\n",
            "    flip_aug: False\n",
            "    face_crop_aug_range: None\n",
            "    random_crop: False\n",
            "    token_warmup_min: 1,\n",
            "    token_warmup_step: 0,\n",
            "    is_reg: False\n",
            "    class_tokens: 9shoku0219\n",
            "    caption_extension: \n",
            "\n",
            "\n",
            "[Dataset 0]\n",
            "loading image sizes.\n",
            "100% 30/30 [00:00<00:00, 123.36it/s]\n",
            "make buckets\n",
            "number of images (including repeats) / 各bucketの画像枚数（繰り返し回数を含む）\n",
            "bucket 0: resolution (1024, 1024), count: 300\n",
            "mean ar error (without repeats): 0.0\n",
            "preparing accelerator\n",
            "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py:439: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.\n",
            "  self.scaler = torch.cuda.amp.GradScaler(**kwargs)\n",
            "accelerator device: cuda\n",
            "loading model for process 0/1\n",
            "load StableDiffusion checkpoint: /content/AnyLoRA_noVae_fp16-pruned.ckpt\n",
            "/content/kohya-trainer/library/model_util.py:978: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
            "  checkpoint = torch.load(ckpt_path, map_location=device)\n",
            "UNet2DConditionModel: 64, 8, 768, False, False\n",
            "loading u-net: <All keys matched successfully>\n",
            "loading vae: <All keys matched successfully>\n",
            "loading text encoder: <All keys matched successfully>\n",
            "Enable xformers for U-Net\n",
            "import network module: networks.lora\n",
            "[Dataset 0]\n",
            "caching latents.\n",
            "checking cache validity...\n",
            "100% 30/30 [00:00<00:00, 358487.52it/s]\n",
            "caching latents...\n",
            "100% 30/30 [00:22<00:00,  1.32it/s]\n",
            "create LoRA network. base dim (rank): 16, alpha: 8\n",
            "neuron dropout: p=None, rank dropout: p=None, module dropout: p=None\n",
            "create LoRA for Text Encoder:\n",
            "create LoRA for Text Encoder: 72 modules.\n",
            "create LoRA for U-Net: 192 modules.\n",
            "enable LoRA for text encoder\n",
            "enable LoRA for U-Net\n",
            "prepare optimizer, data loader etc.\n",
            "use 8-bit AdamW optimizer | {}\n",
            "override steps. steps for 10 epochs is / 指定エポックまでのステップ数: 1500\n",
            "running training / 学習開始\n",
            "  num train images * repeats / 学習画像の数×繰り返し回数: 300\n",
            "  num reg images / 正則化画像の数: 0\n",
            "  num batches per epoch / 1epochのバッチ数: 150\n",
            "  num epochs / epoch数: 10\n",
            "  batch size per device / バッチサイズ: 2\n",
            "  gradient accumulation steps / 勾配を合計するステップ数 = 1\n",
            "  total optimization steps / 学習ステップ数: 1500\n",
            "steps:   0% 0/1500 [00:00<?, ?it/s]\n",
            "epoch 1/10\n",
            "steps:  10% 150/1500 [15:37<2:20:40,  6.25s/it, avr_loss=0.07]\n",
            "saving checkpoint: /content/drive/MyDrive/Loras/9shoku0219/output/9shoku0219-01.safetensors\n",
            "\n",
            "epoch 2/10\n",
            "steps:  20% 300/1500 [31:17<2:05:10,  6.26s/it, avr_loss=0.0689]\n",
            "saving checkpoint: /content/drive/MyDrive/Loras/9shoku0219/output/9shoku0219-02.safetensors\n",
            "\n",
            "epoch 3/10\n",
            "steps:  30% 450/1500 [46:57<1:49:34,  6.26s/it, avr_loss=0.0674]\n",
            "saving checkpoint: /content/drive/MyDrive/Loras/9shoku0219/output/9shoku0219-03.safetensors\n",
            "\n",
            "epoch 4/10\n",
            "steps:  40% 600/1500 [1:02:36<1:33:54,  6.26s/it, avr_loss=0.0616]\n",
            "saving checkpoint: /content/drive/MyDrive/Loras/9shoku0219/output/9shoku0219-04.safetensors\n",
            "\n",
            "epoch 5/10\n",
            "steps:  50% 750/1500 [1:18:15<1:18:15,  6.26s/it, avr_loss=0.0691]\n",
            "saving checkpoint: /content/drive/MyDrive/Loras/9shoku0219/output/9shoku0219-05.safetensors\n",
            "\n",
            "epoch 6/10\n",
            "steps:  60% 900/1500 [1:33:54<1:02:36,  6.26s/it, avr_loss=0.0617]\n",
            "saving checkpoint: /content/drive/MyDrive/Loras/9shoku0219/output/9shoku0219-06.safetensors\n",
            "\n",
            "epoch 7/10\n",
            "steps:  63% 945/1500 [1:38:36<57:54,  6.26s/it, avr_loss=0.0643]"
          ]
        }
      ],
      "source": [
        "import os\n",
        "import re\n",
        "import toml\n",
        "from time import time\n",
        "from IPython.display import Markdown, display\n",
        "\n",
        "# These carry information from past executions\n",
        "if \"model_url\" in globals():\n",
        "  old_model_url = model_url\n",
        "else:\n",
        "  old_model_url = None\n",
        "if \"dependencies_installed\" not in globals():\n",
        "  dependencies_installed = False\n",
        "if \"model_file\" not in globals():\n",
        "  model_file = None\n",
        "\n",
        "# These may be set by other cells, some are legacy\n",
        "if \"custom_dataset\" not in globals():\n",
        "  custom_dataset = None\n",
        "if \"override_dataset_config_file\" not in globals():\n",
        "  override_dataset_config_file = None\n",
        "if \"override_config_file\" not in globals():\n",
        "  override_config_file = None\n",
        "if \"optimizer\" not in globals():\n",
        "  optimizer = \"AdamW8bit\"\n",
        "if \"optimizer_args\" not in globals():\n",
        "  optimizer_args = None\n",
        "if \"continue_from_lora\" not in globals():\n",
        "  continue_from_lora = \"\"\n",
        "if \"weighted_captions\" not in globals():\n",
        "  weighted_captions = False\n",
        "if \"adjust_tags\" not in globals():\n",
        "  adjust_tags = False\n",
        "if \"keep_tokens_weight\" not in globals():\n",
        "  keep_tokens_weight = 1.0\n",
        "\n",
        "COLAB = True # low ram\n",
        "XFORMERS = True\n",
        "SOURCE = \"https://github.com/uYouUs/sd-scripts\"\n",
        "COMMIT = None\n",
        "BETTER_EPOCH_NAMES = True\n",
        "LOAD_TRUNCATED_IMAGES = True\n",
        "\n",
        "#@title ## 🚩 Start Here\n",
        "\n",
        "#@markdown ### ▶️ Setup\n",
        "#@markdown Your project name will be the same as the folder containing your images. Spaces aren't allowed.\n",
        "#@markdown プロジェクト名は、画像を含むフォルダーと同じになります。スペースは使用できません。\n",
        "project_name = \"9shoku0219\" #@param {type:\"string\"}\n",
        "#@markdown The folder structure doesn't matter and is purely for comfort. Make sure to always pick the same one. I like organizing by project.\n",
        "#@markdown フォルダー構造は重要ではなく、単に利便性のためです。常に同じものを選択するようにしてください。プロジェクト別に整理するのがオススメです。\n",
        "folder_structure = \"Organize by project (MyDrive/Loras/project_name/dataset)\" #@param [\"Organize by category (MyDrive/lora_training/datasets/project_name)\", \"Organize by project (MyDrive/Loras/project_name/dataset)\"]\n",
        "#@markdown Decide the model that will be downloaded and used for training. These options should produce clean and consistent results. You can also choose your own by pasting its download link.\n",
        "#@markdown ダウンロードしてトレーニングに使用するモデルを決定します。これらのオプションにより、クリーンで一貫性のある結果が得られます。ダウンロード リンクを貼り付けて独自のモデルを選択することもできます。\n",
        "training_model = \"AnyLora (AnyLoRA_noVae_fp16-pruned.ckpt)\" #@param [\"Anime (animefull-final-pruned-fp16.safetensors)\", \"AnyLora (AnyLoRA_noVae_fp16-pruned.ckpt)\", \"Stable Diffusion (sd-v1-5-pruned-noema-fp16.safetensors)\"]\n",
        "optional_custom_training_model_url = \"\" #@param {type:\"string\"}\n",
        "custom_model_is_based_on_sd2 = False #@param {type:\"boolean\"}\n",
        "\n",
        "if optional_custom_training_model_url:\n",
        "  model_url = optional_custom_training_model_url\n",
        "elif \"AnyLora\" in training_model:\n",
        "  model_url = \"https://huggingface.co/Lykon/AnyLoRA/resolve/main/AnyLoRA_noVae_fp16-pruned.ckpt\"\n",
        "elif \"Anime\" in training_model:\n",
        "  model_url = \"https://huggingface.co/hollowstrawberry/stable-diffusion-guide/resolve/main/models/animefull-final-pruned-fp16.safetensors\"\n",
        "else:\n",
        "  model_url = \"https://huggingface.co/hollowstrawberry/stable-diffusion-guide/resolve/main/models/sd-v1-5-pruned-noema-fp16.safetensors\"\n",
        "\n",
        "#@markdown ### ▶️ Processing\n",
        "#@markdown Resolution of 512 is standard for Stable Diffusion 1.5. Higher resolution training is much slower but can lead to better details. <p>\n",
        "#@markdown Images will be automatically scaled while training to produce the best results, so you don't need to crop or resize anything yourself.\n",
        "#@markdown (ここでは1024を選択可能ですが)解像度 512px は Stable Diffusion 1.5 モデルの標準です。高解像度のトレーニングははるかに時間がかかりますが、詳細度が向上します。<p>\n",
        "#@markdown トレーニング中に画像は自動的に拡大縮小され、最良の結果が得られます。そのため、自分で切り取ったりサイズを変更したりする必要はありません。\n",
        "resolution = 512 #@param {type:\"slider\", min:512, max:1024, step:128}\n",
        "#@markdown This option will train your images both normally and flipped, for no extra cost, to learn more from them. Turn it on specially if you have less than 20 images. <p>\n",
        "#@markdown **Turn it off if you care about asymmetrical elements in your Lora**.\n",
        "#@markdown <p>このオプションは、追加コストなしで、画像を通常と反転の両方でトレーニングし、画像からより多くのことを学習します。画像が 20 枚未満の場合は特にオンにしてください。Lora の非対称要素が気になる場合はオフにしてください。\n",
        "flip_aug = False #@param {type:\"boolean\"}\n",
        "#markdown Leave empty for no captions.\n",
        "caption_extension = \".txt\" #param {type:\"string\"}\n",
        "#@markdown Shuffling anime tags in place improves learning and prompting. An activation tag goes at the start of every text file and will not be shuffled.\n",
        "#@markdown animeタグを所定の位置にシャッフルすると、学習とプロンプトが向上します。activationタグは各テキストファイルの先頭に配置します(シャッフルされません)。\n",
        "shuffle_tags = True #@param {type:\"boolean\"}\n",
        "shuffle_caption = shuffle_tags\n",
        "activation_tags = \"1\" #@param [0,1,2,3]\n",
        "keep_tokens = int(activation_tags)\n",
        "\n",
        "#@markdown ### ▶️ Steps <p>\n",
        "#@markdown Your images will repeat this number of times during training. I recommend that your images multiplied by their repeats is between 200 and 400.\n",
        "num_repeats = 10 #@param {type:\"number\"}\n",
        "#@markdown Choose how long you want to train for. A good starting point is around 10 epochs or around 2000 steps.<p>\n",
        "#@markdown One epoch is a number of steps equal to: your number of images multiplied by their repeats, divided by batch size. <p>\n",
        "#@markdown トレーニング中に、画像はこの回数繰り返されます。画像の数と繰り返し回数を掛け合わせた回数は、200 ～ 400 にすることをお勧めします。\n",
        "num_repeats = 10 #@param {type:\"number\"}\n",
        "#@markdown トレーニングする時間を選択します。開始点としては、約 10 エポックまたは約 2000 ステップが適切です。<p>\n",
        "#@markdown 1 エポックは、画像の数と繰り返し回数を掛け合わせた回数をバッチ サイズで割った回数に等しいステップ数です。<p>\n",
        "preferred_unit = \"Epochs\" #@param [\"Epochs\", \"Steps\"]\n",
        "how_many = 10 #@param {type:\"number\"}\n",
        "max_train_epochs = how_many if preferred_unit == \"Epochs\" else None\n",
        "max_train_steps = how_many if preferred_unit == \"Steps\" else None\n",
        "#@markdown Saving more epochs will let you compare your Lora's progress better.\n",
        "#@markdown より多くのエポックを保存すると、Lora の進行状況をより適切に比較できるようになります。\n",
        "save_every_n_epochs = 1 #@param {type:\"number\"}\n",
        "keep_only_last_n_epochs = 10 #@param {type:\"number\"}\n",
        "if not save_every_n_epochs:\n",
        "  save_every_n_epochs = max_train_epochs\n",
        "if not keep_only_last_n_epochs:\n",
        "  keep_only_last_n_epochs = max_train_epochs\n",
        "#@markdown Increasing the batch size makes training faster, but may make learning worse. Recommended 2 or 3.\n",
        "#@markdown バッチ サイズを大きくするとトレーニングは速くなりますが、学習が悪くなる可能性があります。2 または 3 を推奨します。\n",
        "train_batch_size = 2 #@param {type:\"slider\", min:1, max:8, step:1}\n",
        "\n",
        "#@markdown ### ▶️ Learning\n",
        "#@markdown The learning rate is the most important for your results. If you want to train slower with lots of images, or if your dim and alpha are high, move the unet to 2e-4 or lower. <p>\n",
        "#@markdown The text encoder helps your Lora learn concepts slightly better. It is recommended to make it half or a fifth of the unet. If you're training a style you can even set it to 0.\n",
        "#@markdown 学習率(lr:learning rate)は最も重要です。多くの画像を使用してよりゆっくりとトレーニングしたい場合、または dim と alpha が高い場合は、unet を 2e-4 以下に動かします。<p>\n",
        "#@markdown テキスト エンコーダーにより、Lora は概念を少しだけよく学習できます。unet の半分または 5 分の 1 にすることをお勧めします。スタイルをトレーニングする場合は、0 に設定することもできます。\n",
        "unet_lr = 5e-4 #@param {type:\"number\"}\n",
        "text_encoder_lr = 1e-4 #@param {type:\"number\"}\n",
        "#@markdown The scheduler is the algorithm that guides the learning rate. If you're not sure, pick `constant` and ignore the number. I personally recommend `cosine_with_restarts` with 3 restarts.\n",
        "#@markdown スケジューラは学習率を導くアルゴリズムです。よくわからない場合は、`constant` を選択して数値を無視してください。個人的には、3 回の再起動を伴う `cosine_with_restarts` をお勧めします。\n",
        "lr_scheduler = \"cosine_with_restarts\" #@param [\"constant\", \"cosine\", \"cosine_with_restarts\", \"constant_with_warmup\", \"linear\", \"polynomial\"]\n",
        "lr_scheduler_number = 3 #@param {type:\"number\"}\n",
        "lr_scheduler_num_cycles = lr_scheduler_number if lr_scheduler == \"cosine_with_restarts\" else 0\n",
        "lr_scheduler_power = lr_scheduler_number if lr_scheduler == \"polynomial\" else 0\n",
        "#@markdown Steps spent \"warming up\" the learning rate during training for efficiency. I recommend leaving it at 5%.\n",
        "#@markdown 効率化のためにトレーニング中に学習率を「ウォーミングアップ」するために費やされるステップ。5% のままにしておくことをお勧めします。\n",
        "lr_warmup_ratio = 0.05 #@param {type:\"slider\", min:0.0, max:0.5, step:0.01}\n",
        "lr_warmup_steps = 0\n",
        "#@markdown New feature that adjusts loss over time, makes learning much more efficient, and training can be done with about half as many epochs. Uses a value of 5.0 as recommended by [the paper](https://arxiv.org/abs/2303.09556).\n",
        "#@markdown 時間の経過とともに損失を調整し、学習の効率を大幅に高め、トレーニングを約半分のエポックで実行できる新機能。[論文](https://arxiv.org/abs/2303.09556)で推奨されている値 5.0 を使用します。\n",
        "min_snr_gamma = True #@param {type:\"boolean\"}\n",
        "min_snr_gamma_value = 5.0 if min_snr_gamma else None\n",
        "\n",
        "#@markdown ### ▶️ Structure\n",
        "#@markdown LoRA is the classic type and good for a variety of purposes. LoCon is good with artstyles as it has more layers to learn more aspects of the dataset.\n",
        "#@markdown LoRA はクラシックなタイプで、さまざまな目的に適しています。LoCon は、データセットのより多くの側面を学習するためのレイヤーが多いため、アートスタイルに適しています。\n",
        "lora_type = \"LoRA\" #@param [\"LoRA\", \"LoCon\"]\n",
        "\n",
        "#@markdown Below are some recommended values for the following settings:\n",
        "#@markdown 以下の設定に推奨される値を以下に示します。\n",
        "\n",
        "#@markdown | type | network_dim | network_alpha | conv_dim | conv_alpha |\n",
        "#@markdown | :---: | :---: | :---: | :---: | :---: |\n",
        "#@markdown | LoRA | 16 | 8 |   |   |\n",
        "#@markdown | LoCon | 16 | 8 | 8 | 4 |\n",
        "\n",
        "#@markdown More dim means larger Lora, it can hold more information but more isn't always better. A dim between 8-32 is recommended, and alpha equal to half the dim.\n",
        "#@markdown dim が大きいほど Lora が大きくなり、より多くの情報を保持できますが、多ければ多いほど良いというわけではありません。dim は 8 ～ 32 が推奨され、alpha は dim の半分に等しくなります。\n",
        "network_dim = 16 #@param {type:\"slider\", min:1, max:128, step:1}\n",
        "network_alpha = 8 #@param {type:\"slider\", min:1, max:128, step:1}\n",
        "#@markdown The following two values only apply to the additional layers of LoCon.\n",
        "#@markdown 次の 2 つの値は、LoCon の追加レイヤーにのみ適用されます。\n",
        "conv_dim = 8 #@param {type:\"slider\", min:1, max:64, step:1}\n",
        "conv_alpha = 4 #@param {type:\"slider\", min:1, max:64, step:1}\n",
        "\n",
        "network_module = \"networks.lora\"\n",
        "network_args = None\n",
        "if lora_type.lower() == \"locon\":\n",
        "  network_args = [f\"conv_dim={conv_dim}\", f\"conv_alpha={conv_alpha}\"]\n",
        "\n",
        "#@markdown ### ▶️ Ready\n",
        "#@markdown You can now run this cell to cook your Lora. Good luck! <p>\n",
        "#@markdown さあ このセルを実行して Lora ができあがります。グッドラック! <p>\n",
        "\n",
        "# 👩‍💻 Cool code goes here\n",
        "\n",
        "if optimizer.lower() == \"prodigy\" or \"dadapt\" in optimizer.lower():\n",
        "  if override_values_for_dadapt_and_prodigy:\n",
        "    unet_lr = 0.5\n",
        "    text_encoder_lr = 0.5\n",
        "    lr_scheduler = \"constant_with_warmup\"\n",
        "    lr_warmup_ratio = 0.05\n",
        "    network_alpha = network_dim\n",
        "\n",
        "  if not optimizer_args:\n",
        "    optimizer_args = [\"decouple=True\",\"weight_decay=0.01\",\"betas=[0.9,0.999]\"]\n",
        "    if optimizer == \"Prodigy\":\n",
        "      optimizer_args.extend([\"d_coef=2\",\"use_bias_correction=True\"])\n",
        "      if lr_warmup_ratio > 0:\n",
        "        optimizer_args.append(\"safeguard_warmup=True\")\n",
        "      else:\n",
        "        optimizer_args.append(\"safeguard_warmup=False\")\n",
        "\n",
        "root_dir = \"/content\" if COLAB else \"~/Loras\"\n",
        "deps_dir = os.path.join(root_dir, \"deps\")\n",
        "repo_dir = os.path.join(root_dir, \"kohya-trainer\")\n",
        "\n",
        "if \"/Loras\" in folder_structure:\n",
        "  main_dir      = os.path.join(root_dir, \"drive/MyDrive/Loras\") if COLAB else root_dir\n",
        "  log_folder    = os.path.join(main_dir, \"_logs\")\n",
        "  config_folder = os.path.join(main_dir, project_name)\n",
        "  images_folder = os.path.join(main_dir, project_name, \"dataset\")\n",
        "  output_folder = os.path.join(main_dir, project_name, \"output\")\n",
        "else:\n",
        "  main_dir      = os.path.join(root_dir, \"drive/MyDrive/lora_training\") if COLAB else root_dir\n",
        "  images_folder = os.path.join(main_dir, \"datasets\", project_name)\n",
        "  output_folder = os.path.join(main_dir, \"output\", project_name)\n",
        "  config_folder = os.path.join(main_dir, \"config\", project_name)\n",
        "  log_folder    = os.path.join(main_dir, \"log\")\n",
        "\n",
        "config_file = os.path.join(config_folder, \"training_config.toml\")\n",
        "dataset_config_file = os.path.join(config_folder, \"dataset_config.toml\")\n",
        "accelerate_config_file = os.path.join(repo_dir, \"accelerate_config/config.yaml\")\n",
        "\n",
        "\n",
        "def install_dependencies():\n",
        "  os.chdir(root_dir)\n",
        "  !git clone {SOURCE} {repo_dir}\n",
        "  os.chdir(repo_dir)\n",
        "  if COMMIT:\n",
        "    !git reset --hard {COMMIT}\n",
        "  !wget https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/train_network_wrapper.py -q -O train_network_wrapper.py\n",
        "  !wget https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/dracula.py -q -O dracula.py\n",
        "\n",
        "  !apt -y update -qq\n",
        "  !apt -y install aria2 -qq\n",
        "  !pip install -U torch==2.4 xformers triton torchvision==0.19 --index-url https://download.pytorch.org/whl/cu121\n",
        "  !pip install accelerate==0.25.0 transformers==4.36.2 diffusers[torch]==0.25.0 ftfy==6.1.1 \\\n",
        "    opencv-python==4.8.1.78 einops==0.7.0 pytorch-lightning==1.9.0 bitsandbytes==0.43.0 \\\n",
        "    prodigyopt==1.0 lion-pytorch==0.0.6 tensorboard safetensors==0.4.2 altair==4.2.2 \\\n",
        "    easygui==0.98.3 toml==0.10.2 voluptuous==0.13.1 huggingface-hub==0.20.1 imagesize==1.4.1 rich==13.7.1\n",
        "  !pip install -e .\n",
        "\n",
        "  # patch kohya for minor stuff\n",
        "  if COLAB:\n",
        "    !sed -i \"s@cpu@cuda@\" library/model_util.py # low ram\n",
        "  if LOAD_TRUNCATED_IMAGES:\n",
        "    !sed -i 's/from PIL import Image/from PIL import Image, ImageFile\\nImageFile.LOAD_TRUNCATED_IMAGES=True/g' library/train_util.py # fix truncated jpegs error\n",
        "  if BETTER_EPOCH_NAMES:\n",
        "    !sed -i 's/{:06d}/{:02d}/g' library/train_util.py # make epoch names shorter\n",
        "    !sed -i 's/\".\" + args.save_model_as)/\"-{:02d}.\".format(num_train_epochs) + args.save_model_as)/g' train_network.py # name of the last epoch will match the rest\n",
        "\n",
        "  from accelerate.utils import write_basic_config\n",
        "  if not os.path.exists(accelerate_config_file):\n",
        "    write_basic_config(save_location=accelerate_config_file)\n",
        "\n",
        "  os.environ[\"TF_CPP_MIN_LOG_LEVEL\"] = \"3\"\n",
        "  os.environ[\"BITSANDBYTES_NOWELCOME\"] = \"1\"\n",
        "  os.environ[\"SAFETENSORS_FAST_GPU\"] = \"1\"\n",
        "\n",
        "def validate_dataset():\n",
        "  global lr_warmup_steps, lr_warmup_ratio, caption_extension, keep_tokens, keep_tokens_weight, weighted_captions, adjust_tags\n",
        "  supported_types = (\".png\", \".jpg\", \".jpeg\", \".webp\", \".bmp\")\n",
        "\n",
        "  print(\"\\n💿 Checking dataset...\")\n",
        "  if not project_name.strip() or any(c in project_name for c in \" .()\\\"'\\\\/\"):\n",
        "    print(\"💥 Error: Please choose a valid project name.\")\n",
        "    return\n",
        "\n",
        "  if custom_dataset:\n",
        "    try:\n",
        "      datconf = toml.loads(custom_dataset)\n",
        "      datasets = [d for d in datconf[\"datasets\"][0][\"subsets\"]]\n",
        "    except:\n",
        "      print(f\"💥 Error: Your custom dataset is invalid or contains an error! Please check the original template.\")\n",
        "      return\n",
        "    reg = [d.get(\"image_dir\") for d in datasets if d.get(\"is_reg\", False)]\n",
        "    datasets_dict = {d[\"image_dir\"]: d[\"num_repeats\"] for d in datasets}\n",
        "    folders = datasets_dict.keys()\n",
        "    files = [f for folder in folders for f in os.listdir(folder)]\n",
        "    images_repeats = {folder: (len([f for f in os.listdir(folder) if f.lower().endswith(supported_types)]), datasets_dict[folder]) for folder in folders}\n",
        "  else:\n",
        "    reg = []\n",
        "    folders = [images_folder]\n",
        "    files = os.listdir(images_folder)\n",
        "    images_repeats = {images_folder: (len([f for f in files if f.lower().endswith(supported_types)]), num_repeats)}\n",
        "\n",
        "  for folder in folders:\n",
        "    if not os.path.exists(folder):\n",
        "      print(f\"💥 Error: The folder {folder.replace('/content/drive/', '')} doesn't exist.\")\n",
        "      return\n",
        "  for folder, (img, rep) in images_repeats.items():\n",
        "    if not img:\n",
        "      print(f\"💥 Error: Your {folder.replace('/content/drive/', '')} folder is empty.\")\n",
        "      return\n",
        "  for f in files:\n",
        "    if not f.lower().endswith((\".txt\", \".npz\")) and not f.lower().endswith(supported_types):\n",
        "      print(f\"💥 Error: Invalid file in dataset: \\\"{f}\\\". Aborting.\")\n",
        "      return\n",
        "\n",
        "  if not [txt for txt in files if txt.lower().endswith(\".txt\")]:\n",
        "    caption_extension = \"\"\n",
        "  if continue_from_lora and not (continue_from_lora.endswith(\".safetensors\") and os.path.exists(continue_from_lora)):\n",
        "    print(f\"💥 Error: Invalid path to existing Lora. Example: /content/drive/MyDrive/Loras/example.safetensors\")\n",
        "    return\n",
        "\n",
        "  pre_steps_per_epoch = sum(img*rep for (img, rep) in images_repeats.values())\n",
        "  steps_per_epoch = pre_steps_per_epoch/train_batch_size\n",
        "  total_steps = max_train_steps or int(max_train_epochs*steps_per_epoch)\n",
        "  estimated_epochs = int(total_steps/steps_per_epoch)\n",
        "  lr_warmup_steps = int(total_steps*lr_warmup_ratio)\n",
        "\n",
        "  for folder, (img, rep) in images_repeats.items():\n",
        "    print(\"📁\"+folder.replace(\"/content/drive/\", \"\") + (\" (Regularization)\" if folder in reg else \"\"))\n",
        "    print(f\"📈 Found {img} images with {rep} repeats, equaling {img*rep} steps.\")\n",
        "  print(f\"📉 Divide {pre_steps_per_epoch} steps by {train_batch_size} batch size to get {steps_per_epoch} steps per epoch.\")\n",
        "  if max_train_epochs:\n",
        "    print(f\"🔮 There will be {max_train_epochs} epochs, for around {total_steps} total training steps.\")\n",
        "  else:\n",
        "    print(f\"🔮 There will be {total_steps} steps, divided into {estimated_epochs} epochs and then some.\")\n",
        "\n",
        "  if total_steps > 10000:\n",
        "    print(\"💥 Error: Your total steps are too high. You probably made a mistake. Aborting...\")\n",
        "    return\n",
        "\n",
        "  if adjust_tags:\n",
        "    print(f\"\\n📎 Weighted tags: {'ON' if weighted_captions else 'OFF'}\")\n",
        "    if weighted_captions:\n",
        "      print(f\"📎 Will use {keep_tokens_weight} weight on {keep_tokens} activation tag(s)\")\n",
        "    print(\"📎 Adjusting tags...\")\n",
        "    adjust_weighted_tags(folders, keep_tokens, keep_tokens_weight, weighted_captions)\n",
        "\n",
        "  return True\n",
        "\n",
        "def adjust_weighted_tags(folders, keep_tokens: int, keep_tokens_weight: float, weighted_captions: bool):\n",
        "  weighted_tag = re.compile(r\"\\((.+?):[.\\d]+\\)(,|$)\")\n",
        "  for folder in folders:\n",
        "    for txt in [f for f in os.listdir(folder) if f.lower().endswith(\".txt\")]:\n",
        "      with open(os.path.join(folder, txt), 'r') as f:\n",
        "        content = f.read()\n",
        "      # reset previous changes\n",
        "      content = content.replace('\\\\', '')\n",
        "      content = weighted_tag.sub(r'\\1\\2', content)\n",
        "      if weighted_captions:\n",
        "        # re-apply changes\n",
        "        content = content.replace(r'(', r'\\(').replace(r')', r'\\)').replace(r':', r'\\:')\n",
        "        if keep_tokens_weight > 1:\n",
        "          tags = [s.strip() for s in content.split(\",\")]\n",
        "          for i in range(min(keep_tokens, len(tags))):\n",
        "            tags[i] = f'({tags[i]}:{keep_tokens_weight})'\n",
        "          content = \", \".join(tags)\n",
        "      with open(os.path.join(folder, txt), 'w') as f:\n",
        "        f.write(content)\n",
        "\n",
        "def create_config():\n",
        "  global dataset_config_file, config_file, model_file\n",
        "\n",
        "  if override_config_file:\n",
        "    config_file = override_config_file\n",
        "    print(f\"\\n⭕ Using custom config file {config_file}\")\n",
        "  else:\n",
        "    config_dict = {\n",
        "      \"additional_network_arguments\": {\n",
        "        \"unet_lr\": unet_lr,\n",
        "        \"text_encoder_lr\": text_encoder_lr,\n",
        "        \"network_dim\": network_dim,\n",
        "        \"network_alpha\": network_alpha,\n",
        "        \"network_module\": network_module,\n",
        "        \"network_args\": network_args,\n",
        "        \"network_train_unet_only\": True if text_encoder_lr == 0 else None,\n",
        "        \"network_weights\": continue_from_lora if continue_from_lora else None\n",
        "      },\n",
        "      \"optimizer_arguments\": {\n",
        "        \"learning_rate\": unet_lr,\n",
        "        \"lr_scheduler\": lr_scheduler,\n",
        "        \"lr_scheduler_num_cycles\": lr_scheduler_num_cycles if lr_scheduler == \"cosine_with_restarts\" else None,\n",
        "        \"lr_scheduler_power\": lr_scheduler_power if lr_scheduler == \"polynomial\" else None,\n",
        "        \"lr_warmup_steps\": lr_warmup_steps if lr_scheduler != \"constant\" else None,\n",
        "        \"optimizer_type\": optimizer,\n",
        "        \"optimizer_args\": optimizer_args if optimizer_args else None,\n",
        "      },\n",
        "      \"training_arguments\": {\n",
        "        \"max_train_steps\": max_train_steps,\n",
        "        \"max_train_epochs\": max_train_epochs,\n",
        "        \"save_every_n_epochs\": save_every_n_epochs,\n",
        "        \"save_last_n_epochs\": keep_only_last_n_epochs,\n",
        "        \"train_batch_size\": train_batch_size,\n",
        "        \"noise_offset\": None,\n",
        "        \"clip_skip\": 2,\n",
        "        \"min_snr_gamma\": min_snr_gamma_value,\n",
        "        \"weighted_captions\": weighted_captions,\n",
        "        \"seed\": 42,\n",
        "        \"max_token_length\": 225,\n",
        "        \"xformers\": XFORMERS,\n",
        "        \"lowram\": COLAB,\n",
        "        \"max_data_loader_n_workers\": 8,\n",
        "        \"persistent_data_loader_workers\": True,\n",
        "        \"save_precision\": \"fp16\",\n",
        "        \"mixed_precision\": \"fp16\",\n",
        "        \"output_dir\": output_folder,\n",
        "        \"logging_dir\": log_folder,\n",
        "        \"output_name\": project_name,\n",
        "        \"log_prefix\": project_name,\n",
        "      },\n",
        "      \"model_arguments\": {\n",
        "        \"pretrained_model_name_or_path\": model_file,\n",
        "        \"v2\": custom_model_is_based_on_sd2,\n",
        "        \"v_parameterization\": True if custom_model_is_based_on_sd2 else None,\n",
        "      },\n",
        "      \"saving_arguments\": {\n",
        "        \"save_model_as\": \"safetensors\",\n",
        "      },\n",
        "      \"dreambooth_arguments\": {\n",
        "        \"prior_loss_weight\": 1.0,\n",
        "      },\n",
        "      \"dataset_arguments\": {\n",
        "        \"cache_latents\": True,\n",
        "      },\n",
        "    }\n",
        "\n",
        "    for key in config_dict:\n",
        "      if isinstance(config_dict[key], dict):\n",
        "        config_dict[key] = {k: v for k, v in config_dict[key].items() if v is not None}\n",
        "\n",
        "    with open(config_file, \"w\") as f:\n",
        "      f.write(toml.dumps(config_dict))\n",
        "    print(f\"\\n📄 Config saved to {config_file}\")\n",
        "\n",
        "  if override_dataset_config_file:\n",
        "    dataset_config_file = override_dataset_config_file\n",
        "    print(f\"⭕ Using custom dataset config file {dataset_config_file}\")\n",
        "  else:\n",
        "    dataset_config_dict = {\n",
        "      \"general\": {\n",
        "        \"resolution\": resolution,\n",
        "        \"shuffle_caption\": shuffle_caption,\n",
        "        \"keep_tokens\": keep_tokens,\n",
        "        \"flip_aug\": flip_aug,\n",
        "        \"caption_extension\": caption_extension,\n",
        "        \"enable_bucket\": True,\n",
        "        \"bucket_reso_steps\": 64,\n",
        "        \"bucket_no_upscale\": False,\n",
        "        \"min_bucket_reso\": 320 if resolution > 640 else 256,\n",
        "        \"max_bucket_reso\": 1280 if resolution > 640 else 1024,\n",
        "      },\n",
        "      \"datasets\": toml.loads(custom_dataset)[\"datasets\"] if custom_dataset else [\n",
        "        {\n",
        "          \"subsets\": [\n",
        "            {\n",
        "              \"num_repeats\": num_repeats,\n",
        "              \"image_dir\": images_folder,\n",
        "              \"class_tokens\": None if caption_extension else project_name\n",
        "            }\n",
        "          ]\n",
        "        }\n",
        "      ]\n",
        "    }\n",
        "\n",
        "    for key in dataset_config_dict:\n",
        "      if isinstance(dataset_config_dict[key], dict):\n",
        "        dataset_config_dict[key] = {k: v for k, v in dataset_config_dict[key].items() if v is not None}\n",
        "\n",
        "    with open(dataset_config_file, \"w\") as f:\n",
        "      f.write(toml.dumps(dataset_config_dict))\n",
        "    print(f\"📄 Dataset config saved to {dataset_config_file}\")\n",
        "\n",
        "def download_model():\n",
        "  global old_model_url, model_url, model_file\n",
        "  real_model_url = model_url.strip()\n",
        "\n",
        "  if real_model_url.lower().endswith((\".ckpt\", \".safetensors\")):\n",
        "    model_file = f\"/content{real_model_url[real_model_url.rfind('/'):]}\"\n",
        "  else:\n",
        "    model_file = \"/content/downloaded_model.safetensors\"\n",
        "    if os.path.exists(model_file):\n",
        "      !rm \"{model_file}\"\n",
        "\n",
        "  if m := re.search(r\"(?:https?://)?(?:www\\.)?huggingface\\.co/[^/]+/[^/]+/blob\", model_url):\n",
        "    real_model_url = real_model_url.replace(\"blob\", \"resolve\")\n",
        "  elif m := re.search(r\"(?:https?://)?(?:www\\\\.)?civitai\\.com/models/([0-9]+)(/[A-Za-z0-9-_]+)?\", model_url):\n",
        "    if m.group(2):\n",
        "      model_file = f\"/content{m.group(2)}.safetensors\"\n",
        "    if m := re.search(r\"modelVersionId=([0-9]+)\", model_url):\n",
        "      real_model_url = f\"https://civitai.com/api/download/models/{m.group(1)}\"\n",
        "    else:\n",
        "      raise ValueError(\"optional_custom_training_model_url contains a civitai link, but the link doesn't include a modelVersionId. You can also right click the download button to copy the direct download link.\")\n",
        "\n",
        "  !aria2c \"{real_model_url}\" --console-log-level=warn -c -s 16 -x 16 -k 10M -d / -o \"{model_file}\"\n",
        "\n",
        "  if model_file.lower().endswith(\".safetensors\"):\n",
        "    from safetensors.torch import load_file as load_safetensors\n",
        "    try:\n",
        "      test = load_safetensors(model_file)\n",
        "      del test\n",
        "    except:\n",
        "      #if \"HeaderTooLarge\" in str(e):\n",
        "      new_model_file = os.path.splitext(model_file)[0]+\".ckpt\"\n",
        "      !mv \"{model_file}\" \"{new_model_file}\"\n",
        "      model_file = new_model_file\n",
        "      print(f\"Renamed model to {os.path.splitext(model_file)[0]}.ckpt\")\n",
        "\n",
        "  if model_file.lower().endswith(\".ckpt\"):\n",
        "    from torch import load as load_ckpt\n",
        "    try:\n",
        "      test = load_ckpt(model_file)\n",
        "      del test\n",
        "    except:\n",
        "      return False\n",
        "\n",
        "  return True\n",
        "\n",
        "def main():\n",
        "  global dependencies_installed\n",
        "\n",
        "  if COLAB and not os.path.exists('/content/drive'):\n",
        "    from google.colab import drive\n",
        "    print(\"📂 Connecting to Google Drive...\")\n",
        "    drive.mount('/content/drive')\n",
        "\n",
        "  for dir in (main_dir, deps_dir, repo_dir, log_folder, images_folder, output_folder, config_folder):\n",
        "    os.makedirs(dir, exist_ok=True)\n",
        "\n",
        "  if not validate_dataset():\n",
        "    return\n",
        "\n",
        "  if not dependencies_installed:\n",
        "    print(\"\\n🏭 Installing dependencies...\\n\")\n",
        "    t0 = time()\n",
        "    install_dependencies()\n",
        "    t1 = time()\n",
        "    dependencies_installed = True\n",
        "    print(f\"\\n✅ Installation finished in {int(t1-t0)} seconds.\")\n",
        "  else:\n",
        "    print(\"\\n✅ Dependencies already installed.\")\n",
        "\n",
        "  if old_model_url != model_url or not model_file or not os.path.exists(model_file):\n",
        "    print(\"\\n🔄 Downloading model...\")\n",
        "    if not download_model():\n",
        "      print(\"\\n💥 Error: The model you selected is invalid or corrupted, or couldn't be downloaded. You can use a civitai or huggingface link, or any direct download link.\")\n",
        "      return\n",
        "    print()\n",
        "  else:\n",
        "    print(\"\\n🔄 Model already downloaded.\\n\")\n",
        "\n",
        "  create_config()\n",
        "\n",
        "  print(\"\\n⭐ Starting trainer...\\n\")\n",
        "  os.chdir(repo_dir)\n",
        "\n",
        "  !accelerate launch --config_file={accelerate_config_file} --num_cpu_threads_per_process=1 train_network_wrapper.py --dataset_config={dataset_config_file} --config_file={config_file}\n",
        "\n",
        "main()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mBMUJ7BuvNcn"
      },
      "source": [
        "## *️⃣ Extras\n",
        "\n",
        "You can run these before starting the training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "sy9jU2yrdYar"
      },
      "outputs": [],
      "source": [
        "#@markdown ### 🔮 Optimizer\n",
        "#@markdown If you run this cell you will change the optimizer used for training. The default will be `AdamW8bit` otherwise, which is recommended.<p>\n",
        "#@markdown * Dadapt and Prodigy manage learning rate automatically and are very good for small datasets. You can use it without changing anything else here.<p>\n",
        "#@markdown For Dadapt and Prodigy, the following values will be overriden if the box is checked:<p>\n",
        "#@markdown `learning_rate=0.5`, `network_alpha=network_dim`, `lr_scheduler=\"constant_with_warmup\"`, `lr_warmup_ratio=0.05`<p>\n",
        "#@markdown For Dadapt and Prodigy, if `optimizer_args` is left empty the default will be `decouple=True, weight_decay=0.01, betas=[0.9,0.999]`<p>\n",
        "#@markdown And additionally for Prodigy: `d_coef=2, use_bias_correction=True, safeguard_warmup=True`<p>\n",
        "#@markdown このセルを実行すると、トレーニングに使用されるオプティマイザー(最適化)が変更されます。それ以外の場合は、デフォルトは `AdamW8bit` になりますが、これが推奨されます。<p>\n",
        "#@markdown * Dadapt と Prodigy は学習率を自動的に管理し、小さなデータセットに非常に適しています。ここでは他に何も変更せずに使用できます。<p>\n",
        "#@markdown Dadapt および Prodigy の場合、チェックボックスをオンにすると、次の値がオーバーライドされます:<p>\n",
        "#@markdown `learning_rate=0.5`、`network_alpha=network_dim`、`lr_scheduler=\"constant_with_warmup\"`、`lr_warmup_ratio=0.05`<p>\n",
        "#@markdown Dadapt および Prodigy の場合、`optimizer_args` が空のままの場合、デフォルトは `decouple=True、weight_decay=0.01、betas=[0.9,0.999]` になります<p>\n",
        "#@markdown さらに Prodigy の場合: `d_coef=2、use_bias_correction=True、safeguard_warmup=True`<p>\n",
        "\n",
        "optimizer = \"Prodigy\" #@param [\"AdamW8bit\", \"Prodigy\", \"DAdaptation\", \"DadaptAdam\", \"DadaptLion\", \"AdamW\", \"Lion\", \"SGDNesterov\", \"SGDNesterov8bit\", \"AdaFactor\"]\n",
        "optimizer_args = \"\" #@param {type:\"string\"}\n",
        "splitter = \", \" if \", \" in optimizer_args else \",\"\n",
        "optimizer_args = [a.strip() for a in optimizer_args.split(splitter) if a]\n",
        "override_values_for_dadapt_and_prodigy = True #@param {type:\"boolean\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wd4916Eu1tb9"
      },
      "source": [
        "### 📚 Multiple folders in dataset データセット内の複数のフォルダー\n",
        "Below is a template allowing you to define multiple folders in your dataset. You must include the location of each folder and you can set different number of repeats for each one. To add more folders simply copy and paste the sections starting with `[[datasets.subsets]]`.\n",
        "\n",
        "When enabling this, the number of repeats set in the main cell will be ignored, and the main folder set by the project name will also be ignored.\n",
        "\n",
        "You can make one of them a regularization folder by adding `is_reg = true`  \n",
        "You can also set different `keep_tokens`, `flip_aug`, etc.\n",
        "\n",
        "以下は、データセット内に複数のフォルダーを定義できるテンプレートです。各フォルダーの場所を含める必要があり、フォルダーごとに異なる繰り返し回数を設定できます。フォルダーを追加するには、`[[datasets.subsets]]` で始まるセクションをコピーして貼り付けるだけです。\n",
        "\n",
        "これを有効にすると、メインセルで設定された繰り返し回数は無視され、プロジェクト名で設定されたメインフォルダーも無視されます。\n",
        "\n",
        "`is_reg = true` を追加することで、そのうちの 1 つを正規化フォルダーにすることができます\n",
        "\n",
        "異なる `keep_tokens`、`flip_aug` などを設定することもできます。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y037lagnJWmn",
        "cellView": "form"
      },
      "outputs": [],
      "source": [
        "# @title\n",
        "custom_dataset = \"\"\"\n",
        "[[datasets]]\n",
        "\n",
        "[[datasets.subsets]]\n",
        "image_dir = \"/content/drive/MyDrive/Loras/example/dataset/good_images\"\n",
        "num_repeats = 3\n",
        "\n",
        "[[datasets.subsets]]\n",
        "image_dir = \"/content/drive/MyDrive/Loras/example/dataset/normal_images\"\n",
        "num_repeats = 1\n",
        "\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "W84Jxf-U2TIU"
      },
      "outputs": [],
      "source": [
        "custom_dataset = None"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "-Yq5mNvcCy2l"
      },
      "outputs": [],
      "source": [
        "#@markdown ### 🤓 Other\n",
        "#@markdown These are kept here for the few people that use them.\n",
        "#@markdown ごく一部のユーザーのために残しておきます。\n",
        "\n",
        "#@markdown Weighted captions is a new feature that allows you to use (parentheses) to give more weight to certain tags in your dataset, same as in your webui prompts. <p>\n",
        "#@markdown Normal parentheses in your tags such as `(series names)` will need to be escaped like `\\(series names\\)`\n",
        "\n",
        "#@markdown 重み付けされたキャプションは、WebUI プロンプトと同様に、(括弧) を使用してデータセット内の特定のタグに重みを付けることができる新しい機能です。<p>\n",
        "#@markdown タグ内の通常の括弧 (`(シリーズ名)` など) は、`\\(シリーズ名\\)` のようにエスケープする必要があります。\n",
        "\n",
        "weighted_captions = False #@param {type:\"boolean\"}\n",
        "\n",
        "#markdown By enabling `adjust_tags`, you will let this colab modify your tags before running to automatically adjust to `weighted_captions` being on or off. <p>\n",
        "#markdown Then, you may increase `activation_tag_weight` to improve how effective your activation tag is.\n",
        "#markdown `adjust_tags` を有効にすると、この colab は実行前にタグを変更して、`weighted_captions` のオン/オフに自動的に調整します。<p>\n",
        "#markdown 次に、`activation_tag_weight` を増やして、アクティベーション タグの効果を高めることができます。\n",
        "adjust_tags = False #param {type:\"boolean\"}\n",
        "activation_tag_weight = \"1.0\" #param [\"1.0\",\"1.1\",\"1.2\"]\n",
        "keep_tokens_weight = float(activation_tag_weight)\n",
        "\n",
        "#@markdown Here you can write a path in your Google Drive to load an existing Lora file to continue training on.<p>\n",
        "#@markdown **Warning:** It's not the same as one long training session. The epochs start from scratch, and it may have worse results.\n",
        "\n",
        "#@markdown ここで、Google ドライブにパスを記述して、既存の Lora ファイルを読み込み、トレーニングを続行できます。<p>\n",
        "#@markdown **注意:** 1 回の長いトレーニング セッションと同等ではありません。エポックは最初から開始され、結果が悪くなる可能性もあります。\n",
        "continue_from_lora = \"\" #@param {type:\"string\"}\n",
        "if continue_from_lora and not continue_from_lora.startswith(\"/content/drive/MyDrive\"):\n",
        "  import os\n",
        "  continue_from_lora = os.path.join(\"/content/drive/MyDrive\", continue_from_lora)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "aKWlpsG0jrX3"
      },
      "outputs": [],
      "source": [
        "#@markdown ### 🔢 Count datasets\n",
        "#@markdown Google Drive makes it impossible to count the files in a folder, so this will show you the file counts in all folders and subfolders.\n",
        "\n",
        "#@markdown Google ドライブではフォルダ内のファイルを数えることができないため、これによりすべてのフォルダとサブフォルダ内のファイル数が表示されます。\n",
        "folder = \"/content/drive/MyDrive/Loras\" #@param {type:\"string\"}\n",
        "\n",
        "import os\n",
        "from google.colab import drive\n",
        "\n",
        "if not os.path.exists('/content/drive'):\n",
        "    print(\"📂 Connecting to Google Drive...\\n\")\n",
        "    drive.mount('/content/drive')\n",
        "\n",
        "tree = {}\n",
        "exclude = (\"_logs\", \"/output\")\n",
        "for i, (root, dirs, files) in enumerate(os.walk(folder, topdown=True)):\n",
        "  dirs[:] = [d for d in dirs if all(ex not in d for ex in exclude)]\n",
        "  images = len([f for f in files if f.lower().endswith((\".png\", \".jpg\", \".jpeg\"))])\n",
        "  captions = len([f for f in files if f.lower().endswith(\".txt\")])\n",
        "  others = len(files) - images - captions\n",
        "  path = root[folder.rfind(\"/\")+1:]\n",
        "  tree[path] = None if not images else f\"{images:>4} images | {captions:>4} captions |\"\n",
        "  if tree[path] and others:\n",
        "    tree[path] += f\" {others:>4} other files\"\n",
        "\n",
        "pad = max(len(k) for k in tree)\n",
        "print(\"\\n\".join(f\"📁{k.ljust(pad)} | {v}\" for k, v in tree.items() if v))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G7Us_RSCPLsF"
      },
      "source": [
        "# 📈 Plot training results トレーニング結果をプロットする\n",
        "You can do this after running the trainer. You don't need this unless you know what you're doing.  \n",
        "The first cell below may fail to load all your logs. Keep trying the second cell until all data has loaded.\n",
        "\n",
        "トレーナーを実行した後にこれを行うことができます。何をしているのかわかっていない限り、これは必要ありません。\n",
        "以下の最初のセルでは、すべてのログを読み込めない可能性があります。すべてのデータが読み込まれるまで、2 番目のセルを試し続けてください。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Zwiit6YzPLsG"
      },
      "outputs": [],
      "source": [
        "%load_ext tensorboard\n",
        "%tensorboard --logdir={log_folder}/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PU7FP9ooPLsG",
        "cellView": "form"
      },
      "outputs": [],
      "source": [
        "# @title\n",
        "from tensorboard import notebook\n",
        "notebook.display(port=6006, height=800)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}