OpenSource_RAG_LLM_JAP_at_4_Bits.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "<h1>Fully open-source RAG for Japanese in a low-resource environment</h1>\n",
        "\n",
        "<h1>低コスト環境でも処理可能な日本語対応完全オープンソースRAG</h1>\n",
        "\n",
        "---\n"
      ],
      "metadata": {
        "id": "zxtavHKf_6zN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "According to <b><i>Goldman Sachs Research</b></i> in April 2023, Generative AI could raise global GDP by `7%` (or almost `$7` trillion). On a total addressable market of `$685` billion for the global software industry, the generative AI software market is estimated to be `$150` billion. Large Language Models (LLMs) are eating software.\n",
        "\n",
        "<p align=\"justify\">Nevertheless, autoregressive LLMs alone have three main problems. LLMs are sometimes generating non-factual or nonsensical statements called <b><i>“stochastic parrots”</b></i>, often called by an anthropomorphism <b><i>“hallucination”</b></i>. LLMs are trained on 7 billion to 120 billion parameters, requiring a huge amount of computing power and energy. Finally, companies are reluctant to leverage LLM solutions due to the governance of data and privacy concerns.\n",
        "\n",
        "<p align=\"justify\">The current best Generative AI solutions are using LLMs helped by the information retrieval framework called <b><i>Retrieval-Augmented Generation (RAG)</b></i>. RAG is dynamically augmenting the generative capabilities of LLMs with external or/and proprietary knowledge. A real open-source RAG coupled with open-source LLM could minimize stochastic parrots while optimizing costs and maintaining data privacy for a real democratization of Generative AI.\n",
        "\n",
        "<p align=\"justify\">This notebook showcases a quick implementation for open-source RAG and LLM in a low-resource environment for Japanese.\n",
        "\n",
        "- The first element of this RAG is <b><i>Llama-Index</b></i> with vanilla <i>Hybrid search</i> (combining retrieval for text search and vector search)\n",
        "\n",
        "- The second element is a Japanese LLM “ELYZA-japanese-Llama-2-7b-instruct” created by Japanese startup, <b><i>Elyza Inc.</b></i>\n",
        "\n",
        "- The third element is an open-source database <b><i>PostgreSQL</b></i> transformed into a vector database by the library <b><i>PGVector</b></i>\n",
        "\n",
        "<p align=\"justify\">As a dataset, this notebook explores 5 documents from the public hearings “AI Strategy for 2023” created by the <b><i>Cabinet Office</b></i> in Japan available here (https://www8.cao.go.jp/cstp/ai/ai_senryaku/ai_senryaku.html)\n",
        "\n",
        "<p align=\"justify\">In the first part, I will explain the technical details of an open-source RAG. In the second part, I will try to explain the business implications of the “token economy”. In the last part, I will try to give a threefold perspective of business innovation and economic competitiveness between the USA, France, and Japan."
      ],
      "metadata": {
        "id": "lmS1ucVr_6vi"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dNqJOHs8dZtQ",
        "outputId": "4915452e-1e26-41e5-bedd-dcd33100dc53"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ],
      "source": [
        "!pip install git+https://github.com/huggingface/transformers --quiet\n",
        "!pip install \"llama-index==0.8.63.post2\" --quiet\n",
        "!pip install accelerate safetensors bitsandbytes --upgrade --quiet\n",
        "!pip install sentence_transformers cohere pdfminer.six --upgrade --quiet"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "I269d8GtOawX"
      },
      "outputs": [],
      "source": [
        "import nest_asyncio\n",
        "nest_asyncio.apply()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## - Llama Index | ラマ・インデックス\n",
        "\n",
        "<p align=\"justify\"><b><i>Llama-Index</b></i> is a key data framework for a structured retrieval-augmentation generative. Formerly known as GPT-Index, Llama-Index created by <b><i>Jerry Liu</b></i> has been quickly morphing into a model-agnostic framework.\n",
        "\n",
        "<p align=\"justify\">While <b><i>LangChain</b></i> might be easier for beginners, Llama-Index gives a great lower-level API for robust indices or metadata, advanced retrieval, or query interface. Informational retrieval is nothing new, the BM25 retriever is derived from TF-IDF, and Re-Ranker is derived from cross-encoder.\n",
        "\n",
        "<p align=\"justify\">To manage the tradeoff between accuracy and speed, Llama-Index enables us to optimize fundamental components often disregarded in my opinion, a hierarchical ingestion of data and metadata filters."
      ],
      "metadata": {
        "id": "4FVQ6qlyBJ4F"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "ZV4nMJFFdZx-"
      },
      "outputs": [],
      "source": [
        "import sys, logging, warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "\n",
        "from llama_index import ServiceContext, StorageContext\n",
        "from llama_index.indices.vector_store import VectorStoreIndex\n",
        "from llama_index.vector_stores import PGVectorStore\n",
        "import textwrap\n",
        "\n",
        "import tqdm, ipywidgets\n",
        "from IPython.display import Markdown, display"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## - Vector embeddings | 埋め込み\n",
        "\n",
        "<p align=\"justify\">Tokenization has evolved quickly on the last 10 years from character-based, to sub-word units in 2015, to SentencePiece by <b><i>Taku Kudo</b></i> in 2018, and SentenceTransformers by <b><i>Nils Reimers</b></i> in 2019. We now have not word-level or sentence-level but massive text embedding in more than 100 languages in 2023. Embeddings are a key building block of large language models. For RAG, text embeddings are important for semantic search to calculate similarity metrics such as <i>cosine distance, inner product, L2 (Euclidean distance),</i> or <i>L1 (taxi cab or Manhattan distance)</i> between various query embeddings and various corpus embeddings.\n",
        "\n",
        "<p align=\"justify\">According to the <b><i>MTEB Benchmark</b></i>, we have a mix of proprietary embeddings (<b><i>OpenAi, Cohere, VoyageAI</b></i>) and open-source embeddings (<b><i>BAAI, Infloat, Instructor, Jina</b></i>) sometimes of equal performance. Embedding models are the lock on the usage of LLMs and an important factor in the quality of inference of RAGs.\n",
        "\n",
        "<p align=\"justify\">With an open-source perspective in Japanese, we will use the great multilingual <b><i>“Multilingual-e5-large”</b></i> with more than 100 languages by <b><i>Dr. Furu Wei</b></i> and his team at <b><i>Microsoft Research Asia</b></i>.  Based on the paper, <i>“Text Embeddings by Weakly-Supervised Contrastive Pre-training”</i>, <b><i>multilingual-e5-Large</b></i> has 24 layers and an embedding size of 1024. Performances are great with 62.5 for Japanese and 70.5 on average mean reciprocal rank (Avg MRR). However, for a size of 2.24GB, the <b><i>multilingual-e5-Large</b></i> is consuming a lot of computing power.\n"
      ],
      "metadata": {
        "id": "_HLHc10HBYjA"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "bzz-tJzvE3Pu"
      },
      "outputs": [],
      "source": [
        "from llama_index.embeddings import HuggingFaceEmbedding\n",
        "embed_model = HuggingFaceEmbedding(model_name=\"intfloat/multilingual-e5-large\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## - Quantized LLM | 量子化の大規模言語モデル\n",
        "\n",
        "<p align=\"justify\">From a perspective of low-resource, I will load the model in a quantized version of 4-Bits by leveraging <b><i>Bitsandbytes</b></i> and <b><i>Accelerated</b></i> by <b><i>Hugging Face</b></i>. Developed by <b><i>Tim Dettmers</b></i> of the <i>University of Washington</i>, Bitsandbytes is optimizing 4-bit CUDA functions from <b><i>Nvidia</b></i> for the <b><i>PyTorch</b></i> framework.\n",
        "\n",
        "<p align=\"justify\">Tim Dettmers' research on low-bit quantization of large language models and hardware-optimized deep learning has been the real factor of democratization for LLM. Another actor of low-resource LLM, <b><i>Tom Jobbins (The Bloke)</b></i> has been releasing many quantized models financed by the venture capital firm, <b><i>Andreessen Horowitz</b></i>."
      ],
      "metadata": {
        "id": "W0PXYIflB1sb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "from transformers import BitsAndBytesConfig\n",
        "\n",
        "nf4_config = BitsAndBytesConfig(\n",
        "    load_in_4bit=True,\n",
        "    bnb_4bit_quant_type=\"nf4\",\n",
        "    #bnb_4bit_use_double_quant=True  #Nested Quantize is possible to save even more RAM\n",
        ")"
      ],
      "metadata": {
        "id": "_3bUZPMY-4Hs"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## - LLM from Elyza | 大規模言語モデル、イライザ\n",
        "\n",
        "<p align=\"justify\">According to the <b><i>Rakuda Ranking</b></i>, I chose the <b><i>“ELYZA-japanese-Llama-2-7b-fast-instruct”</b></i> from Tokyo-based startup <b><i>Elyza, Inc</b></i>. Elyza is based on the English model, <b><i>LlaMA-2</b></i> developed by <b><i>Facebook/Meta AI</b></i> with a community license. The AI team of Elyza has expanded the vocabulary size in Japanese for better inference time. This open-source model at only 7B could be compared to GPT-3.5 in Japanese."
      ],
      "metadata": {
        "id": "LkKpUUUoB_rU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index.llms import HuggingFaceLLM\n",
        "\n",
        "llm = HuggingFaceLLM(\n",
        "    model_name=\"elyza/ELYZA-japanese-Llama-2-7b-instruct\",\n",
        "    tokenizer_name=\"elyza/ELYZA-japanese-Llama-2-7b-instruct\",\n",
        "    context_window=3900,\n",
        "    max_new_tokens=1024,\n",
        "    model_kwargs={\"quantization_config\": nf4_config},\n",
        "    generate_kwargs={\"temperature\": 0.1,\"do_sample\": True},\n",
        "    device_map=\"auto\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "07d518b4cd084374b707a611132375d4",
            "ba7e62a913e240d393f468e2cc0a5c1a",
            "177b57b682f14d8288c5e1ed51d66d6b",
            "ba1f11e4330a4f64a40cdaa16a733ce8",
            "c2f7bc3c64d64d34b5f751989ab7f956",
            "052bbd1effda46fbbebb1570e03af885",
            "3ef76f4d4dd0486b9a6c8d9799661898",
            "8d18f57d5d134024a7703ab7e791d6a5",
            "8ef82c9e11364e58a09bd6ee1a13bcbd",
            "7c2e9000f6894d71b09bb70f344256d1",
            "5e01480fbded4e13b895c360719e2528"
          ]
        },
        "id": "uOvUDX1u-4yu",
        "outputId": "41fa0502-0645-41ca-ce23-ab668db5f9a7"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "07d518b4cd084374b707a611132375d4"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "roRyq7Zveh0h"
      },
      "outputs": [],
      "source": [
        "from llama_index.text_splitter import SentenceSplitter\n",
        "from llama_index.node_parser import SimpleNodeParser\n",
        "\n",
        "text_splitter = SentenceSplitter(\n",
        "    chunk_size=512,\n",
        "    chunk_overlap=20,\n",
        "    paragraph_separator=\"\\n\\n\\n\",\n",
        ")\n",
        "node_parser = SimpleNodeParser.from_defaults(text_splitter=text_splitter)\n",
        "\n",
        "service_context = ServiceContext.from_defaults(llm=llm,embed_model=embed_model,node_parser=node_parser)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## - CJKPDFReader | CJKPDFリーダー\n",
        "\n",
        "<p align=\"justify\">For the specificity of Japanese a particular PDF loader is required for shift-jis encoded documents. The Llama-Hub offers <b><i>CJKPDFReader</b></i> a document for Japanese, Chinese, and Korean based on pdfminer.six.  \n",
        "\n",
        "<p align=\"justify\">Unfortunately, we encounter multiple problems due to the formatting of the administrative documents that will create problems with embedding, indexing, and final results.\n",
        "\n",
        "- Titles are neither in Japanese, nor in English. <b><i>“Giji”</b></i> is a Hepburn reading of <b>“議事”</b> proceedings (e.g. government proceedings). The second title is slightly better with <b><i>“gijiyoushi2kai”</b></i> but still wrong as Hepburn of <b>“議事要旨第2回</b> (proceedings second meeting)\n",
        "\n",
        "- The administrative document is dated with the Japanese era calendar scheme as <b>“令和 5 年</b> (era number 5 of Reiwa)” as 2023\n",
        "\n",
        "- The Japanese language doesn’t have white space. However, the formatting of the Word file and the transformation into PDF with Acrobat Maker 23 (library 23.1.206) is creating space between words\n",
        "\n",
        "- In the very important first-page introduction participants such as academic researchers, company leaders, and government officials are presented in a justify and minimalistic way (creating space)\n",
        "\n",
        "<p align=\"justify\">As the financial sector is heavily regulated, even transcripts of earning calls are carefully structured, perfect for metadata such as title, date, company, board of directors, and core chapters. In the USA, the <b><i>Securities and Exchange Commission (SEC)</b></i> required a clear and strict format for 10-Q, 10-K, and 8-K. In France, the reporting format is getting more regulated with a universal registration document (URD) by the <b><i>European Securities Markets Authority (ESMA)</b></i>, while in Japan the <b><i>Financial Services Agency (FSA)</b></i> is producing guidelines and starting in 2019 self-regulatory activities with the <b><i>“Open Policy Lab”</b></i> initiative."
      ],
      "metadata": {
        "id": "Xp32sslpCHQQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from pathlib import Path\n",
        "from llama_index import download_loader\n",
        "\n",
        "CJKPDFReader = download_loader(\"CJKPDFReader\")\n",
        "loader = CJKPDFReader()\n",
        "\n",
        "giji_01 = loader.load_data(file=Path('/content/giji.pdf'))\n",
        "giji_02 = loader.load_data(file=Path('/content/gijiyoushi2kai.pdf'))"
      ],
      "metadata": {
        "id": "3shVuZPdbDTY"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "2Pmb6g8eWlFR"
      },
      "outputs": [],
      "source": [
        "!pip install psycopg2-binary pgvector asyncpg \"sqlalchemy[asyncio]\" greenlet --upgrade --quiet"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## - Vector database | ベクトルデータベース (PostgreSQL + PGVector)\n",
        "\n",
        "<p align=\"justify\">The boom in Generative AI created a spark in the vector database space with popular startups <b><i>Chroma DB, Qdrant, Zilliz, Pinecone,</b></i> etc. but also major players such as <b><i>Elasticsearch, Redis, MongoDB,</b></i> etc.\n",
        "\n",
        "<p align=\"justify\">Powered by the open-source community, <b><i>PostgreSQL</b></i> enabled vector similarity search library <b><i>PGVector</b></i> developed by <b><i>Andrew Kane</b></i>. The combination of Postgres and PGVector offers a self-hosted and cloud, enabled hybrid search, store documents, and work in async. The installation is very easy with a Docker image.\n",
        "\n",
        "## - Hierarchical Navigable Small World | 階層的なスモールワールド\n",
        "\n",
        "<p align=\"justify\">PGVector is compatible with the <b><i>IVF Flat (Inverted File with Flat Compression)</b></i> format and the <b><i>HNSW (Hierarchical Navigable Small World)</b></i> format released in August 2023 (version 0.50). HNSW is an approximate K-nearest neighbor search based on navigable small world graphs among top-performing indexes for vector similarity search developed by <b><i>Dmitry Yashunin</b></i> and <b><i>Yury Malkov</b></i>. HNSW is slower to build but reduces memory requirements.\n",
        "\n",
        "<p align=\"justify\">Postgres is fully open-source, ACID compliant, implemented in Japanese via <b><i>MeCab</b></i>, and cost-efficient. In more than 20 vector databases available with LLama-Index only <b><i>Elasticsearch, Pinecone, Weaviate, Postgres, Azure Cognitive Search, TencentVector DB, MyScale,</b></i> and <b><i>Lantern</b></i> can use hybrid search.\n",
        "<p align=\"justify\">Among them, only <b><i>Postgres + PGVector</b></i> is truly open-source and can leverage the cutting-edge graph-based algorithm of <b\n",
        "><i>HNSW</b></i>."
      ],
      "metadata": {
        "id": "igWbQYpGCuNd"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "8OvMUmuwKSBk"
      },
      "outputs": [],
      "source": [
        "import psycopg2\n",
        "\n",
        "connection_string = \"postgresql://postgres:test@0.tcp.eu.ngrok.io:19642\" #Ngrok TCP tuneling\n",
        "db_name = \"vector_db\"\n",
        "conn = psycopg2.connect(connection_string)\n",
        "conn.autocommit = True\n",
        "\n",
        "with conn.cursor() as c:\n",
        "    c.execute(f\"DROP DATABASE IF EXISTS {db_name}\")\n",
        "    c.execute(f\"CREATE DATABASE {db_name}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## - Ngrok | エングロック\n",
        "\n",
        "<p align=\"justify\">I am using the well-secured ingress platform, <b><i>Ngrok</b></i> for tunneling to connect the local Postgres built via <b><i>Docker</b></i> to my Google Colab Pro."
      ],
      "metadata": {
        "id": "X86BS59-C_lr"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "s424yv5-10A5"
      },
      "outputs": [],
      "source": [
        "from sqlalchemy import make_url\n",
        "from llama_index.vector_stores import PGVectorStore\n",
        "\n",
        "url = make_url(connection_string)\n",
        "hybrid_vector_store = PGVectorStore.from_params(\n",
        "    database=db_name,\n",
        "    host=url.host,\n",
        "    password=url.password,\n",
        "    port=url.port,\n",
        "    user=url.username,\n",
        "    table_name=\"ai_law_japan\",\n",
        "    embed_dim=1024,\n",
        "    hybrid_search=True,\n",
        "    #text_search_config=\"japanese\", #Thx to MeCab integration into Postgres\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## - Hybrid approach of RAG | RAGシステムのハイブリッド検索\n",
        "- Text plus vector search | テキスト検索とベクトル検索\n",
        "\n",
        "<p align=\"justify\">I believe that a <b><i>hybrid search</b></i> approach should be mandatory for proper RAG. The hybrid option must be enabled at the construction of the <b><i>PGVector</b></i> store and stated in the query engine to combine retrieval from both text search and vector search. The implementation of a hybrid search is straightforward. It can be improved in many aspects.\n"
      ],
      "metadata": {
        "id": "kHe1sfdDDMdL"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "E3OoelXwAL6j"
      },
      "outputs": [],
      "source": [
        "storage_context = StorageContext.from_defaults(vector_store=hybrid_vector_store)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "idU8rWOFbOxj"
      },
      "outputs": [],
      "source": [
        "from llama_index.tools import QueryEngineTool, ToolMetadata\n",
        "from llama_index.schema import MetadataMode\n",
        "\n",
        "giji_01_hybrid_index = VectorStoreIndex.from_documents(giji_01,storage_context=storage_context, service_context=service_context, metadata=ToolMetadata(name = \"giji_01\",description = \"第1回AI戦略会議 内閣府 2023年5月11日 AIを巡る主な論点について\"))\n",
        "\n",
        "giji_02_hybrid_index = VectorStoreIndex.from_documents(giji_02, storage_context=storage_context, service_context=service_context, metadata=ToolMetadata(name = \"giji_02\",description = \"第2回AI戦略会議 内閣府 2023年5月26日 AIに関する暫定的な論点整理について\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## - Prompt engineering | プロンプトエンジニアリング\n",
        "\n",
        "<p align=\"justify\">Llama-Index provides great prompt templates, unfortunately only in English. For Japanese, we can override the template with a Q&A template for example. I used many different templates, it was rather difficult to have a clear understanding. <b><i>Prompt engineering feels more like trial and error than programming</b></i>. I used the prompt guidelines of Llama 2 mixed with the default prompt system by Elyza for Japanese. Asking for responses in bullet points (マークダウン形式) gave me better results."
      ],
      "metadata": {
        "id": "FEkkTlLWDSsO"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "UYItlpbb4JPp"
      },
      "outputs": [],
      "source": [
        "from llama_index.prompts.prompts import QuestionAnswerPrompt\n",
        "\n",
        "qa_template = QuestionAnswerPrompt(\"\"\"\n",
        "<s>[INST] <<SYS>>\n",
        "あなたは誠実で優秀な日本人のアシスタントです。\n",
        "マークダウン形式で以下のコンテキスト情報を元に質問に回答してください。\n",
        "<</SYS>>\n",
        "\n",
        "{context_str}\n",
        "\n",
        "{query_str}\n",
        "[/INST]\"\"\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "3WwU8aoKdaCW"
      },
      "outputs": [],
      "source": [
        "giji_01_hyb_engine = giji_01_hybrid_index.as_query_engine(vector_store_query_mode=\"hybrid\",\n",
        "                                                      sparse_top_k=2,\n",
        "                                                      text_qa_template=qa_template,\n",
        "                                                      )\n",
        "\n",
        "giji_02_hyb_engine = giji_02_hybrid_index.as_query_engine(vector_store_query_mode=\"hybrid\",\n",
        "                                                      sparse_top_k=2,\n",
        "                                                      text_qa_template=qa_template,\n",
        "                                                       )"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## - Q&A over documents | データを回答させてみる"
      ],
      "metadata": {
        "id": "kP6rGz_HEoV-"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "SC25f9HFfBVs",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "405e69cd-f691-4883-91c9-b05d4c3428ab"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " 構成員は以下の通りです。\n",
            "\n",
            "- 座長: 松尾豊 東京大学大学院工学系研究科教授\n",
            "- 構成員: 江間有沙 東京大学未来ビジョン研究センター准教授、岡田淳 東京大学大学院工学系研究科教授、北野宏明 東京大学大学院工学系研究科教授、佐渡島庸平 株式会社コルク代表取締役社長、田中邦裕 さくらインターネット株式会社代表取締役社長\n",
            "- 政府側参加者: 岸田文雄 内閣総理大臣、高市早苗 科学技術政策担当大臣、尾崎正直 国務大臣（サイバーセキュリティ戦略本部副本部長）、松本剛明 総務大臣、簗和生 文部科学副大臣、西村康稔 経済産業大臣、中谷真一 経済産業副大臣、村井英樹 内閣総理大臣補佐官\n",
            "\n",
            "なお、議事録は作成されておりません。\n"
          ]
        }
      ],
      "source": [
        "response1 = giji_02_hyb_engine.query(\"人工知能関連の政策を議論する内閣府の「第2回AI戦略会議」の構成員は誰ですか？\")\n",
        "print(response1)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<b><i>Question 1:</b></i> Who are the members of the second AI strategy meeting?\n",
        "\n",
        "<b><i>Answer 1:</b></i>Correct! (正解!)\n",
        "\n",
        "<p align=\"justify\">The first page doesn’t give a relation between the full name and the company. The impressive RAG understood the linear relation between researchers and CEOs to universities or companies. The RAG gave a full list with chairman, the members, and the government officials."
      ],
      "metadata": {
        "id": "zMdZmuACEvlA"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "imP2b-dFdaN-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "32772fa6-b751-4fd1-8a6d-de068fafb295"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " 第1回AI戦略会議の座長は、松尾豊氏です。\n"
          ]
        }
      ],
      "source": [
        "response2 = giji_01_hyb_engine.query(\"第1回AI戦略会議の座長は誰でしょうか？\")\n",
        "print(response2)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<b><i>Question 2:</b></i> Who was the chairman of the first AI strategy meeting?\n",
        "\n",
        "<b><i>Answer 2:</b></i> Correct! (正解!)\n",
        "\n",
        "<p align=\"justify\">Even with the double space created by the formatting, the chairman is 松尾豊 (Matsuo Yutaka) is correctly found by the LLM. The model deleted the white space between the family name “松尾(Matsu)” and the personal name “豊(Yutaka)”. It even introduced the honorific particle “氏” to be polite in the answer."
      ],
      "metadata": {
        "id": "Xf05Pb8KFGpH"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "kkd6ZJ_xday9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d3842e2e-aeff-48d2-cfb3-b831ef7478b3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " 質問者さんにお返事いたします。\n",
            "\n",
            "株式会社ソニーリサーチの代表取締役は、北野宏明さんです。\n"
          ]
        }
      ],
      "source": [
        "response3 = giji_02_hyb_engine.query(\"株式会社ソニーリサーチの代表取締役は誰ですか？\")\n",
        "print(response3)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<b><i>Question 3:</b></i> Who is the representative director of Sony Research?\n",
        "\n",
        "<b><i>Answer 3:</b></i> Correct! (正解!)\n",
        "\n",
        "The model is finding correctly the CEO of Sony Research, Mr. Kitano Hiroaki (北野宏明) regardless of the space."
      ],
      "metadata": {
        "id": "Mx27lO5-FUF4"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "5wLMSb7Tojge",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e761546f-8f61-4ca6-80b9-92903352cfd3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " 広島AIプロセスの目的について、G7サミットで議論するために作成した暫定的な論点整理案に基づいて説明し.ます。\n",
            "\n",
            "1. 我が国が世界に誇るべきAIの力を活用し、課題を解決し、社会の課題を解決し、国民の安心・安全を担保し、経済の発展による豊かな社会を実現する。\n",
            "2. 国民のAIの理解を深め、公正な社会の実現に向けた議論を深める。\n",
            "3. 国際社会でのAIの活用における課題を共有し、国際的なルールの枠組み作りに貢献する。\n",
            "4. 我が国のAI戦略の基本方針を踏まえ、G7広島AIプロセスで議論を深め、我が国がAI戦略会議の主導的な役割を担い、G7の議論を主導し、G7広島AIプロセスの成果をG7サミットで報告し、G7サミットの成果を我が国のAI戦略の基本方針に反映させる。\n"
          ]
        }
      ],
      "source": [
        "response4 = giji_02_hyb_engine.query(\"G7サミットで議論する広島AIプロセスの目的を説明してください。 \")\n",
        "print(response4)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<b><i>Question 4:</b></i> Can you explain the Hiroshima A.I Process from G7 summit?\n",
        "\n",
        "<b><i>Answer 4:</b></i> Correct! (正解!)\n",
        "\n",
        "<p align=\"justify\">The answer is correct. The model does explain the <b><i>“Hiroshima AI Process”</b></i> is a comprehensive policy framework. The reply is pretty great but I would love to know \"Hiroshima AI Process\" is a in partnership with the <b><i>Global Partnership for Artificial Intelligence (GPAI)</b></i> and the <b><i>Organisation for Economic Co-operation and Development (OECD)</b></i> that will take place in Hiroshima in September 2023."
      ],
      "metadata": {
        "id": "p3k0bONjFe_i"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "wrFvioipwSZE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "212ffce1-f574-4a4d-ef7b-8e80470813d1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " 質問に回答いたします。\n",
            "\n",
            "ITサービスの貿易赤字は、2030年には10兆円近く生じると予測されています。\n"
          ]
        }
      ],
      "source": [
        "response5 = giji_02_hyb_engine.query(\"ITサービスの貿易赤字はいくらですか？\")\n",
        "print(response5)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<b><i>Question 5:</b></i> Can you tell me the trade deficit for the IT service?\n",
        "\n",
        "<b><i>Answer 5:</b></i> Incorrect! (不正解!!!)\n",
        "\n",
        "<p align=\"justify\">The model is giving back the 10 trilion of yen. Correct answer is that the Japanese trade deficit of IT (3 trilion of yen) and Governement cloud (8 trilion yen). The Minister of Internal Affairs and Communications, <b><i> Mr. Matsumoto</b></i>  complained about government spending for ChatGpt of OpenAI, when a similar system could be created by local companies."
      ],
      "metadata": {
        "id": "kUo5AIDCFsln"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "response6 = giji_02_hyb_engine.query(\"里見経済産業大臣政務官の結論を要約してください。\")\n",
        "print(response6)"
      ],
      "metadata": {
        "id": "dy1WRo8pVqfE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0b4d6326-0f63-4487-d071-c6eed1a5b387"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " 与えられたコンテキスト情報を元に質問に回答します。\n",
            "\n",
            "里見経済産業大臣政務官は、AIの利活用による産業競争力の強化を重要視しています。具体的には、安全性、信頼性に十分に留意しながらAIの開発にタイムリーに関与し、最先端の技術情報にアクセスする機会を得ることが重要だと述べています。\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<b><i>Question 6:</b></i> Can you summarize the conclusion of Mr. Satomi of the Parliamentary Vice-Minister for Economy, Trade and Industry?\n",
        "\n",
        "<b><i>Answer 6:</b></i> 正解 | Correct!\n",
        "\n",
        "<p align=\"justify\">The answer is correct despite that I didn't put in place a \"Document Summary Index\" dedicated to summarization in Llama-Index.\n",
        "\n",
        "---\n"
      ],
      "metadata": {
        "id": "9iFLrW5DFypJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## - Technical observations | 技術ノート\n",
        "\n",
        "<p align=\"justify\">5 good answers to 6 questions with a fully open-source RAG project (7B LLM quantized at 4-Bits) and low resource setup (T4 Nvidia GPU) is quite amazing. By fine-tuning embedding, optimizing retrieval, metadata filters, or organizing document hierarchies, I believe it would achieve a perfect score.\n",
        "\n",
        "<p align=\"justify\"> 8-Bits is still fitting on the T4 Nvidia GPU and give better responses. By instantiating at higher floating-point precision such as 16 bits (lower precision) with a V100 Nvidia and 32 bits (normal precision) with a more powerful A100 GPU, it will use more memory, and cost more but enable quicker and better responses.\n",
        "\n",
        "## - Business implications | ビジネスインパクト分析\n",
        "\n",
        "<p align=\"justify\">As of November 2023, we are seeing a pricing war among proprietary systems to reduce barriers to entry and engage the adoption of Generative AI solutions. Newcomer <b><i>Voyage AI</b></i> co-created by an associate professor at <b><i>Stanford</b></i>, <b><i>Tengyu Ma</b></i> is cost-cutting with `$0.1` (per million tokens) on a maximum context length of 4096 (first 50 million tokens are free as part of a trial period). As a comparison, <b><i>GPT-4 of OpenAI</b></i> `$0.03` for inputs and `$0.06` for outputs (per 1K tokens) as of November 2023.\n",
        "\n",
        "<p align=\"justify\">Using an open-source embedding model breaks the “token-based business model” giving total freedom to experiment and scale. Alternatively, open-source embedding models created by the university could finance PhD students or professors working at AI labs.\n",
        "\n",
        "<p align=\"justify\">According to the price of Replicate, the Nvidia T4 GPU is at `$0.000225`/sec (`$0.81`/hour) for less than `$600` per month, or `$7,200` per year to run on 4bits or 8bits quantized LLM. PostgresSQL is probably the most stable and available open-source database in the world. A fully open-source RAG can empower software, machine learning, or NLP engineers supported by the open-source community."
      ],
      "metadata": {
        "id": "t3fcCmrQGNL4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## - Generative A.I strategy for U.S.A, France, and Japan | アメリカ、フランス、日本の生成ＡＩ戦略\n",
        "\n",
        "<p align=\"justify\">In the USA, since 2021, the success of GPT-3 by OpenAI reoriented the research towards autoregressive architecture and opened a wave of huge investments in the private sector. At <b><i>Y Combinator</b></i>, more than 59 generative AI-focused startups (22%) in the 272 companies in the Winter 2023 cohort.\n",
        "\n",
        "<p align=\"justify\">In France, the most successful AI startup Hugging Face (built by 3 French entrepreneurs) was financed by American venture capital firms in 2019. In the latest Series D funding, in August 2023, Hugging Face raised `$235` million led by tech giants such as <b><i>Google, Amazon, Nvidia, etc</b></i> for a `$4.5` billion valuation. The <b><i>Confiance.AI</b></i> strategy of the French government focuses on the excellence of higher education and partnerships with century-old groups such as <b><i>Airbus, Thales, Renault, Air Liquide, etc.</b></i>\n",
        "\n",
        "<p align=\"justify\">The powerful ecosystem of Silicon Valley, cool management style, and high salaries are attracting top talents from around the world in California. France is more or less a laggard in AI-related investments and is the first provider of engineers for leading AI companies.\n",
        "\n",
        "<p align=\"justify\">Due to the language barrier, Japan has a hybrid ecosystem composed of academic labs of prestigious universities, upcoming startups such as <b><i>Elyza, Stockmark, Rinna,</b></i> and tech giants such as <b><i>NTT Data A.I labs, Line A.I (Naver), Megagon Labs (Recruit), CyberAgent (Abema), etc.</b></i>\n",
        "\n",
        "<p align=\"justify\">Back in 2021, the popular Bert models of <i>Tohoku University</i> were created with Cloud TPUs provided by the <i>TensorFlow Research Cloud program</i> of <b><i>Google Inc</b></i>. The most prolific Japanese researcher is <b><i>Tatsunori Hashimoto</b></i> at <i>Stanford University</i>, while the most influential Japanese researcher is <b><i>Taku Kudo</b></i> (creator of <b><i>MeCab, CaboCha, and CRF++</b></i>) at <b><i>Google Inc</b></i>.\n",
        "\n",
        "<p align=\"justify\">In 2023, the <b><i>city of Yokosuka</b></i> (Kanagawa Prefecture) in Japan was the first city to use ChatGPT for administrative tasks. The creator of popular anime One Piece, <b><i>Eiichiro Oda</b></i> has used ChatGPT 4.0 to create a new manga chapter. This extremely fast adoption of autoregressive models sparked a copyright debate in Japan. According to <b><i>Professor Tatsuhiro Ueno</b></i> of <i>Waseda University</i>, the Japanese Copyright Act has an explicit provision on copyright exception for text and data mining (Art.47septies 著作権法 - 美術の著作物等の展示に伴う複製等 - 第四十七条）under which it shall be allowed to copy any copyright-protected work to promote machine learning without authorization of copyright holders.\n",
        "\n",
        "<p align=\"justify\">The merger <b><i>LY Corporation</b></i> (LINEヤフー株式会社) between <b><i>LINE</b></i> and <b><i>Yahoo Japan</b></i> could be a game changer in the creation of corpus in Japanese. Since embedding models of <i>Kindai University</i> and <i>Nagoya University</i> for Japanese are still based on SentenceTransformers architecture, a bilingual embedding model in Japanese and English could be a great open-source project to eliminate the first bottleneck.\n",
        "\n",
        "<p align=\"justify\">Consisting of over 500 participants including researchers in natural language processing and computer systems from universities and corporations, <b><i>LLM Study Group</b></i> has been an extraordinary project for Japan that unveiled a great model called LLM-JP 13B. The LLM Study Group can be comparable to the <b><i>BigScience</b></i> project led by <b><i>Hugging Face</b></i> with the Bloom models.\n",
        "\n",
        "<p align=\"justify\">By reading the guidelines of the French and Japanese governments, the words <b><i>“open-source”</b></i> were never used as a viable solution for promoting innovation and economic development that could be implemented at a large scale. Open source ensures that AI progress benefits all, in the government for administrative tasks, in small and medium companies for business applications, and for research in graduate schools or public hospitals.\n",
        "\n",
        "<p align=\"justify\">Japanese language specificities and academic excellence could give an advantage to local startups but the clock is ticking."
      ],
      "metadata": {
        "id": "BnjsdrUXGjik"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Thank you for reading!\n",
        "\n",
        "Please feel free to contact me if you have any questions.<br>\n",
        "\n",
        "<b><i>Akim Mousterou</b></i>\n",
        "<br><br><br>\n",
        "\n",
        "---\n",
        "\n",
        "<p align=\"justify\">\n",
        "<b><i>Disclaimer:</b></i> <i>None of the content published on this notebook constitutes a recommendation that any particular security, portfolio of securities, transaction, or investment strategy is suitable for any specific person. None of the information providers or their affiliates will advise you personally concerning the nature, potential, value, or suitability of any particular security, portfolio of securities, transaction, investment strategy, or other matter.</i>"
      ],
      "metadata": {
        "id": "LM_RbasDHPdi"
      }
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "machine_shape": "hm",
      "gpuType": "T4"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "07d518b4cd084374b707a611132375d4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_ba7e62a913e240d393f468e2cc0a5c1a",
              "IPY_MODEL_177b57b682f14d8288c5e1ed51d66d6b",
              "IPY_MODEL_ba1f11e4330a4f64a40cdaa16a733ce8"
            ],
            "layout": "IPY_MODEL_c2f7bc3c64d64d34b5f751989ab7f956"
          }
        },
        "ba7e62a913e240d393f468e2cc0a5c1a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_052bbd1effda46fbbebb1570e03af885",
            "placeholder": "​",
            "style": "IPY_MODEL_3ef76f4d4dd0486b9a6c8d9799661898",
            "value": "Loading checkpoint shards: 100%"
          }
        },
        "177b57b682f14d8288c5e1ed51d66d6b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8d18f57d5d134024a7703ab7e791d6a5",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8ef82c9e11364e58a09bd6ee1a13bcbd",
            "value": 2
          }
        },
        "ba1f11e4330a4f64a40cdaa16a733ce8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7c2e9000f6894d71b09bb70f344256d1",
            "placeholder": "​",
            "style": "IPY_MODEL_5e01480fbded4e13b895c360719e2528",
            "value": " 2/2 [00:11&lt;00:00,  5.28s/it]"
          }
        },
        "c2f7bc3c64d64d34b5f751989ab7f956": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "052bbd1effda46fbbebb1570e03af885": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3ef76f4d4dd0486b9a6c8d9799661898": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8d18f57d5d134024a7703ab7e791d6a5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8ef82c9e11364e58a09bd6ee1a13bcbd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "7c2e9000f6894d71b09bb70f344256d1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5e01480fbded4e13b895c360719e2528": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}