examples/python/transformers/onnx/HuggingFace_ONNX_in_Spark_NLP_T5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![JohnSnowLabs](https://sparknlp.org/assets/images/logo.png)\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp/blob/master/examples/python/transformers/onnx/HuggingFace_ONNX_in_Spark_NLP_T5.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import ONNX T5 models from HuggingFace 🤗 into Spark NLP 🚀\n",
    "\n",
    "Let's keep in mind a few things before we start 😊\n",
    "\n",
    "- ONNX support was introduced in  `Spark NLP 5.0.0`, enabling high performance inference for models.\n",
    "- ONNX support for the `T5Transformer` is only available since in `Spark NLP 5.2.0` and after. So please make sure you have upgraded to the latest Spark NLP release\n",
    "- You can import T5 models via `T5Model`. These models are usually under `Text2Text Generation` category and have `T5` in their labels\n",
    "- This is a very computationally expensive module especially on larger sequence. The use of an accelerator such as GPU is recommended.\n",
    "- Reference: [T5Model](https://huggingface.co/docs/transformers/model_doc/t5#transformers.T5Model)\n",
    "- Some [example models](https://huggingface.co/models?other=T5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export and Save HuggingFace model\n",
    "\n",
    "- Let's install `transformers` package with the `onnx` extension and it's dependencies. You don't need `onnx` to be installed for Spark NLP, however, we need it to load and save models from HuggingFace.\n",
    "- We lock `transformers` on version `4.35.2`. This doesn't mean it won't work with the future releases\n",
    "- We will also need `sentencepiece` for tokenization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m400.9/400.9 kB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m10.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m84.5/84.5 kB\u001b[0m \u001b[31m8.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m454.7/454.7 kB\u001b[0m \u001b[31m12.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.4/6.4 MB\u001b[0m \u001b[31m21.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m212.7/212.7 kB\u001b[0m \u001b[31m15.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m3.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m521.2/521.2 kB\u001b[0m \u001b[31m24.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m15.7/15.7 MB\u001b[0m \u001b[31m27.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m55.5/55.5 kB\u001b[0m \u001b[31m3.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.8/86.8 kB\u001b[0m \u001b[31m7.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m115.3/115.3 kB\u001b[0m \u001b[31m5.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m52.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "tensorflow 2.14.0 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3, but you have protobuf 3.20.2 which is incompatible.\n",
      "tensorflow-metadata 1.14.0 requires protobuf<4.21,>=3.20.3, but you have protobuf 3.20.2 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip install -q --upgrade transformers[onnx]==4.35.2 optimum sentencepiece onnx==1.14.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- HuggingFace has an extension called Optimum which offers specialized model inference, including ONNX. We can use this to import and export ONNX models with `from_pretrained` and `save_pretrained`.\n",
    "- We'll use [google/flan-t5-base](https://huggingface.co/google/flan-t5-base) model from HuggingFace as an example\n",
    "- In addition to `T5Model` we also need to save the tokenizer. This is the same for every model, these are assets needed for tokenization inside Spark NLP.\n",
    "- If we want to optimize the model, a GPU will be needed. Make sure to select the correct runtime.\n",
    "0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import transformers\n",
    "# Model name, either HF (e.g. \"google/flan-t5-base\") or a local path\n",
    "MODEL_NAME = \"google/flan-t5-base\"\n",
    "\n",
    "\n",
    "# Path to store the exported models\n",
    "EXPORT_PATH = f\"onnx_models/{MODEL_NAME}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-12-09 15:50:28.712604: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2023-12-09 15:50:28.712694: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2023-12-09 15:50:28.712744: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2023-12-09 15:50:31.175343: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
      "Framework not specified. Using pt to export to ONNX.\n",
      "config.json: 100% 1.40k/1.40k [00:00<00:00, 5.73MB/s]\n",
      "model.safetensors: 100% 990M/990M [00:12<00:00, 82.4MB/s]\n",
      "generation_config.json: 100% 147/147 [00:00<00:00, 555kB/s]\n",
      "tokenizer_config.json: 100% 2.54k/2.54k [00:00<00:00, 8.77MB/s]\n",
      "spiece.model: 100% 792k/792k [00:00<00:00, 138MB/s]\n",
      "tokenizer.json: 100% 2.42M/2.42M [00:00<00:00, 15.2MB/s]\n",
      "special_tokens_map.json: 100% 2.20k/2.20k [00:00<00:00, 9.60MB/s]\n",
      "Using the export variant default. Available variants are:\n",
      "    - default: The default ONNX variant.\n",
      "Using framework PyTorch: 2.1.0+cu118\n",
      "Overriding 1 configuration item(s)\n",
      "\t- use_cache -> False\n",
      "Using framework PyTorch: 2.1.0+cu118\n",
      "Overriding 1 configuration item(s)\n",
      "\t- use_cache -> True\n",
      "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py:873: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  if causal_mask.shape[1] < attention_mask.shape[1]:\n",
      "Using framework PyTorch: 2.1.0+cu118\n",
      "Overriding 1 configuration item(s)\n",
      "\t- use_cache -> True\n",
      "/usr/local/lib/python3.10/dist-packages/transformers/models/t5/modeling_t5.py:508: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  elif past_key_value.shape[2] != key_value_states.shape[1]:\n",
      "In-place op on output of tensor.shape. See https://pytorch.org/docs/master/onnx.html#avoid-inplace-operations-when-using-tensor-shape-in-tracing-mode\n",
      "In-place op on output of tensor.shape. See https://pytorch.org/docs/master/onnx.html#avoid-inplace-operations-when-using-tensor-shape-in-tracing-mode\n",
      "Post-processing the exported models...\n",
      "Weight deduplication check in the ONNX export requires accelerate. Please install accelerate to run it.\n",
      "The two models proto have different outputs (49 and 25 outputs). Constant outputs will be added to unify the two models outputs.\n",
      "Adding a constant output for present.0.encoder.key of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.0.encoder.value of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.1.encoder.key of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.1.encoder.value of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.2.encoder.key of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.2.encoder.value of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.3.encoder.key of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.3.encoder.value of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.4.encoder.key of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.4.encoder.value of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.5.encoder.key of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.5.encoder.value of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.6.encoder.key of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.6.encoder.value of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.7.encoder.key of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.7.encoder.value of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.8.encoder.key of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.8.encoder.value of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.9.encoder.key of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.9.encoder.value of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.10.encoder.key of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.10.encoder.value of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.11.encoder.key of shape [0, 12, 1, 64] in model2.\n",
      "Adding a constant output for present.11.encoder.value of shape [0, 12, 1, 64] in model2.\n",
      "Validating ONNX model onnx_models/google/flan-t5-base/encoder_model.onnx...\n",
      "\t-[✓] ONNX model output names match reference model (last_hidden_state)\n",
      "\t- Validating ONNX Model output \"last_hidden_state\":\n",
      "\t\t-[✓] (2, 16, 768) matches (2, 16, 768)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "Validating ONNX model onnx_models/google/flan-t5-base/decoder_model_merged.onnx...\n",
      "2023-12-09 15:51:48.208017351 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.3/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208083363 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.1/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208097809 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.9/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208108009 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.5/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208123335 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.11/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208132392 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.7/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208142039 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.4/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208152218 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.8/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208160006 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.1/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208180803 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.3/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208190583 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.2/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208198618 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.4/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208206693 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.0/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208214643 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.4/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208238111 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.5/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208248040 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.0/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208267097 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.2/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208274621 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.6/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208280709 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.3/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208287100 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.2/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208295365 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.1/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208305471 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.10/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208317459 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.11/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208324941 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.0/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208336444 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.11/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208347444 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.6/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208357823 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.9/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208366278 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.10/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208374195 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.5/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208399398 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.6/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208409515 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.7/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208421604 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.7/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208440000 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.8/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208450547 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.8/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208457592 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.9/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208477670 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.10/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.208494899 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/final_layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209574821 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.11/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209602649 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.11/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209614041 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.11/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209628454 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.3/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209654487 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.9/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209673964 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.0/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209681506 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.10/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209690866 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.7/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209699687 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.4/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209715509 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.8/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209722772 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.1/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209736676 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.3/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209751723 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.2/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209759121 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.4/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209769884 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.4/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209783392 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.1/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209811926 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.0/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209822486 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.2/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209829954 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.2/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209845010 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.10/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209857264 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.1/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209863876 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.0/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209871035 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.6/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209886358 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.5/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209895598 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.9/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209904553 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.10/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209912621 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.5/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209922981 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.5/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209939977 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.3/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209947278 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.6/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209966706 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.7/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209977370 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.7/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.209986458 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.8/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.210008378 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.8/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.210016191 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.9/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.210024666 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.6/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.210033910 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/final_layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.296212541 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Unsqueeze_370'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.296270003 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Unsqueeze_368'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.296279095 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/Constant_8_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.296296726 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/Shape_3_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.297386573 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Unsqueeze_314'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.297433644 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/Constant_7_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.297470013 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Unsqueeze_316'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.297530880 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/Shape_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.338965280 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Unsqueeze_343'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.339025410 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.340208075 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Unsqueeze_295'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.340237966 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.427374890 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/ConstantOfShape_1_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:48.428639761 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/ConstantOfShape_1_output_0'. It is not used by any node and should be removed from the model.\n",
      "\t-[✓] ONNX model output names match reference model (present.2.encoder.key, present.0.decoder.key, present.8.decoder.value, present.11.encoder.key, logits, present.7.encoder.value, present.11.decoder.key, present.9.decoder.value, present.1.decoder.value, present.4.encoder.value, present.4.decoder.value, present.6.encoder.value, present.2.encoder.value, present.6.encoder.key, present.10.encoder.key, present.11.encoder.value, present.9.decoder.key, present.7.decoder.value, present.4.encoder.key, present.9.encoder.key, present.10.decoder.key, present.3.decoder.key, present.8.encoder.value, present.1.encoder.key, present.2.decoder.key, present.1.encoder.value, present.6.decoder.key, present.11.decoder.value, present.5.decoder.key, present.7.decoder.key, present.7.encoder.key, present.8.decoder.key, present.0.encoder.key, present.10.decoder.value, present.5.decoder.value, present.5.encoder.key, present.1.decoder.key, present.3.encoder.value, present.0.decoder.value, present.2.decoder.value, present.3.encoder.key, present.9.encoder.value, present.10.encoder.value, present.0.encoder.value, present.4.decoder.key, present.8.encoder.key, present.5.encoder.value, present.6.decoder.value, present.3.decoder.value)\n",
      "\t- Validating ONNX Model output \"logits\":\n",
      "\t\t-[✓] (2, 16, 32128) matches (2, 16, 32128)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.0.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.0.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.0.encoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.0.encoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.1.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.1.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.1.encoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.1.encoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.2.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.2.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.2.encoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.2.encoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.3.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.3.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.3.encoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.3.encoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.4.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.4.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[x] values not close enough, max diff: 1.9073486328125e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.4.encoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.4.encoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.5.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.5.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.5.encoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.5.encoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.6.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.6.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[x] values not close enough, max diff: 2.3603439331054688e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.6.encoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.6.encoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.7.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.7.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[x] values not close enough, max diff: 1.811981201171875e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.7.encoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.7.encoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.8.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.8.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[x] values not close enough, max diff: 2.193450927734375e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.8.encoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.8.encoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.9.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.9.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[x] values not close enough, max diff: 2.9087066650390625e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.9.encoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.9.encoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.10.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[x] values not close enough, max diff: 2.574920654296875e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.10.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[x] values not close enough, max diff: 5.14984130859375e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.10.encoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.10.encoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[x] values not close enough, max diff: 2.09808349609375e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.11.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[x] values not close enough, max diff: 2.384185791015625e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.11.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[x] values not close enough, max diff: 4.1484832763671875e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.11.encoder.key\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.11.encoder.value\":\n",
      "\t\t-[✓] (2, 12, 16, 64) matches (2, 12, 16, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "Validating ONNX model onnx_models/google/flan-t5-base/decoder_model_merged.onnx...\n",
      "2023-12-09 15:51:51.066662621 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.3/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.066740990 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.1/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.066755838 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.9/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.066766224 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.5/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.066783678 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.11/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.066792789 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.7/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.066807718 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.4/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.066828873 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.8/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.066844908 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.1/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.066873884 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.3/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.066893121 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.2/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.066902640 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.4/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.066915783 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.0/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.066928136 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.4/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.066970164 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.5/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.066983878 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.0/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067010352 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.2/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067021723 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.6/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067031786 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.3/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067041416 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.2/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067053275 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.1/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067067646 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.10/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067084830 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.11/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067096963 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.0/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067114183 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.11/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067130196 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.6/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067146084 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.9/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067158086 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.10/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067170672 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.5/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067205436 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.6/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067221892 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.7/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067238753 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.7/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067264981 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.8/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067279333 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.8/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067290760 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.9/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067318324 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.10/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.067342512 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/final_layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068437655 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.11/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068472192 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.11/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068485686 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.11/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068513648 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.3/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068553098 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.9/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068581661 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.0/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068594421 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.10/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068608852 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.7/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068622518 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.4/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068646811 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.8/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068659238 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.1/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068676697 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.3/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068696910 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.2/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068709118 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.4/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068724187 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.4/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068744855 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.1/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068769920 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.0/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068783680 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.2/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068795337 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.2/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068814117 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.10/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068838786 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.1/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068849310 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.0/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068860510 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.6/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068882381 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.5/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068896055 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.9/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068910675 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.10/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068923158 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.5/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068946958 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.5/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068958596 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.3/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068968973 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.6/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.068995012 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.7/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.069013807 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.7/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.069027040 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.8/layer.1/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.069057772 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.8/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.069069730 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.9/layer.0/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.069082472 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/block.6/layer.2/layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.069096216 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/final_layer_norm/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.161041863 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Unsqueeze_370'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.161099643 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Unsqueeze_368'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.161109168 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/Constant_8_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.161131091 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/Shape_3_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.162257670 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Unsqueeze_314'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.162316903 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/Constant_7_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.162365825 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Unsqueeze_316'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.162445897 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/Shape_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.203729989 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Unsqueeze_343'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.203793895 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.205036049 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer 'onnx::Unsqueeze_295'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.205074263 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/Constant_2_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.288856449 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/ConstantOfShape_1_output_0'. It is not used by any node and should be removed from the model.\n",
      "2023-12-09 15:51:51.290187950 [W:onnxruntime:, graph.cc:3553 CleanUnusedInitializersAndNodeArgs] Removing initializer '/decoder/ConstantOfShape_1_output_0'. It is not used by any node and should be removed from the model.\n",
      "\t-[✓] ONNX model output names match reference model (present.0.decoder.key, present.8.decoder.value, logits, present.11.decoder.key, present.9.decoder.value, present.1.decoder.value, present.4.decoder.value, present.9.decoder.key, present.7.decoder.value, present.10.decoder.key, present.3.decoder.key, present.2.decoder.key, present.6.decoder.key, present.11.decoder.value, present.5.decoder.key, present.7.decoder.key, present.8.decoder.key, present.10.decoder.value, present.5.decoder.value, present.1.decoder.key, present.0.decoder.value, present.2.decoder.value, present.4.decoder.key, present.6.decoder.value, present.3.decoder.value)\n",
      "\t- Validating ONNX Model output \"logits\":\n",
      "\t\t-[✓] (2, 1, 32128) matches (2, 1, 32128)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.0.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.0.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.1.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.1.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.2.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.2.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.3.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.3.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.4.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.4.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[x] values not close enough, max diff: 1.6510486602783203e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.5.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.5.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[x] values not close enough, max diff: 2.0802021026611328e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.6.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.6.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[x] values not close enough, max diff: 1.9609928131103516e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.7.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.7.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[x] values not close enough, max diff: 2.6941299438476562e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.8.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[✓] all values close (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.8.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[x] values not close enough, max diff: 3.266334533691406e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.9.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[x] values not close enough, max diff: 1.811981201171875e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.9.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[x] values not close enough, max diff: 5.53131103515625e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.10.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[x] values not close enough, max diff: 2.086162567138672e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.10.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[x] values not close enough, max diff: 6.723403930664062e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.11.decoder.key\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[x] values not close enough, max diff: 2.8252601623535156e-05 (atol: 1e-05)\n",
      "\t- Validating ONNX Model output \"present.11.decoder.value\":\n",
      "\t\t-[✓] (2, 12, 17, 64) matches (2, 12, 17, 64)\n",
      "\t\t-[x] values not close enough, max diff: 6.0439109802246094e-05 (atol: 1e-05)\n",
      "Validation for the model onnx_models/google/flan-t5-base/decoder_model_merged.onnx raised: The maximum absolute difference between the output of the reference model and the ONNX exported model is not within the set tolerance 1e-05:\n",
      "- present.4.decoder.value: max diff = 1.9073486328125e-05\n",
      "- present.6.decoder.value: max diff = 2.3603439331054688e-05\n",
      "- present.7.decoder.value: max diff = 1.811981201171875e-05\n",
      "- present.8.decoder.value: max diff = 2.193450927734375e-05\n",
      "- present.9.decoder.value: max diff = 2.9087066650390625e-05\n",
      "- present.10.decoder.key: max diff = 2.574920654296875e-05\n",
      "- present.10.decoder.value: max diff = 5.14984130859375e-05\n",
      "- present.10.encoder.value: max diff = 2.09808349609375e-05\n",
      "- present.11.decoder.key: max diff = 2.384185791015625e-05\n",
      "- present.11.decoder.value: max diff = 4.1484832763671875e-05\n",
      "The ONNX export succeeded with the warning: The maximum absolute difference between the output of the reference model and the ONNX exported model is not within the set tolerance 1e-05:\n",
      "- present.4.decoder.value: max diff = 1.6510486602783203e-05\n",
      "- present.5.decoder.value: max diff = 2.0802021026611328e-05\n",
      "- present.6.decoder.value: max diff = 1.9609928131103516e-05\n",
      "- present.7.decoder.value: max diff = 2.6941299438476562e-05\n",
      "- present.8.decoder.value: max diff = 3.266334533691406e-05\n",
      "- present.9.decoder.key: max diff = 1.811981201171875e-05\n",
      "- present.9.decoder.value: max diff = 5.53131103515625e-05\n",
      "- present.10.decoder.key: max diff = 2.086162567138672e-05\n",
      "- present.10.decoder.value: max diff = 6.723403930664062e-05\n",
      "- present.11.decoder.key: max diff = 2.8252601623535156e-05\n",
      "- present.11.decoder.value: max diff = 6.0439109802246094e-05.\n",
      " The exported model was saved at: onnx_models/google/flan-t5-base\n"
     ]
    }
   ],
   "source": [
    "# Export the model to ONNX using optimum\n",
    "\n",
    "# Export with optimizations (uncomment next line)\n",
    "# !optimum-cli export onnx --task text2text-generation-with-past --model {MODEL_NAME} --optimize O2 {EXPORT_PATH}\n",
    "# IMPORTANT - there is a bug in onnxruntime which crashes it when trying to optimize a T5 small model (or any derivative of it)\n",
    "# There are two ways to addess the problem:\n",
    "# 1. Go to onnx_model_bert.py in the onnxruntime module (the full path depends on the module version),\n",
    "#    find the BertOnnxModel class and comment the following line in the constructor:\n",
    "#    assert (num_heads == 0 and hidden_size == 0) or (num_heads > 0 and hidden_size % num_heads == 0)\n",
    "# 2. Disable optimization by removing '--optimize O2' (use line below).\n",
    "\n",
    "# Export without optimizations\n",
    "!optimum-cli export onnx --task text2text-generation-with-past --model {MODEL_NAME} {EXPORT_PATH}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look inside these two directories and see what we are dealing with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 2283400\n",
      "-rw-r--r-- 1 root root      1529 Dec  9 15:50 config.json\n",
      "-rw-r--r-- 1 root root 651182887 Dec  9 15:51 decoder_model_merged.onnx\n",
      "-rw-r--r-- 1 root root 650848962 Dec  9 15:51 decoder_model.onnx\n",
      "-rw-r--r-- 1 root root 594197310 Dec  9 15:51 decoder_with_past_model.onnx\n",
      "-rw-r--r-- 1 root root 438697389 Dec  9 15:50 encoder_model.onnx\n",
      "-rw-r--r-- 1 root root       142 Dec  9 15:50 generation_config.json\n",
      "-rw-r--r-- 1 root root      2201 Dec  9 15:50 special_tokens_map.json\n",
      "-rw-r--r-- 1 root root    791656 Dec  9 15:50 spiece.model\n",
      "-rw-r--r-- 1 root root     20771 Dec  9 15:50 tokenizer_config.json\n",
      "-rw-r--r-- 1 root root   2422256 Dec  9 15:50 tokenizer.json\n"
     ]
    }
   ],
   "source": [
    "!ls -l {EXPORT_PATH}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- As you can see, we need to move the sentence piece models `spiece.model` from the tokenizer to assets folder which Spark NLP will look for"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! mkdir -p {EXPORT_PATH}/assets\n",
    "! mv -t {EXPORT_PATH}/assets {EXPORT_PATH}/spiece.model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 776\n",
      "-rw-r--r-- 1 root root 791656 Dec  9 15:50 spiece.model\n"
     ]
    }
   ],
   "source": [
    "!ls -l {EXPORT_PATH}/assets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import and Save T5 in Spark NLP\n",
    "\n",
    "- Let's install and setup Spark NLP in Google Colab\n",
    "- This part is pretty easy via our simple script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing PySpark 3.2.3 and Spark NLP 5.2.0\n",
      "setup Colab for PySpark 3.2.3 and Spark NLP 5.2.0\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m281.5/281.5 MB\u001b[0m \u001b[31m4.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m548.5/548.5 kB\u001b[0m \u001b[31m41.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m199.7/199.7 kB\u001b[0m \u001b[31m20.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
     ]
    }
   ],
   "source": [
    "! wget -q http://setup.johnsnowlabs.com/colab.sh -O - | bash"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start Spark with Spark NLP included via our simple `start()` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sparknlp\n",
    "\n",
    "# let's start Spark with Spark NLP\n",
    "spark = sparknlp.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let's use `loadSavedModel` functon in `T5Transformer` which allows us to load the ONNX model\n",
    "- Most params will be set automatically. They can also be set later after loading the model in `T5Transformer` during runtime, so don't worry about setting them now\n",
    "- `loadSavedModel` accepts two params, first is the path to the exported model. The second is the SparkSession that is `spark` variable we previously started via `sparknlp.start()`\n",
    "- NOTE: `loadSavedModel` accepts local paths in addition to distributed file systems such as `HDFS`, `S3`, `DBFS`, etc. This feature was introduced in Spark NLP 4.2.2 release. Keep in mind the best and recommended way to move/share/reuse Spark NLP models is to use `write.save` so you can use `.load()` from any file systems natively.st and recommended way to move/share/reuse Spark NLP models is to use `write.save` so you can use `.load()` from any file systems natively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sparknlp.annotator import *\n",
    "\n",
    "T5 = T5Transformer.loadSavedModel(EXPORT_PATH, spark)\\\n",
    "  .setUseCache(True) \\\n",
    "  .setTask(\"summarize:\") \\\n",
    "  .setMaxOutputLength(200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's save it on disk so it is easier to be moved around and also be used later via `.load` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "T5.write().overwrite().save(f\"{MODEL_NAME}_spark_nlp\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's clean up stuff we don't need anymore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf {EXPORT_PATH}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Awesome  😎 !\n",
    "\n",
    "This is your ONNX T5 model from HuggingFace 🤗  loaded and saved by Spark NLP 🚀"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 1065292\n",
      "-rw-r--r-- 1 root root 651282390 Dec  9 16:07 decoder.onxx\n",
      "-rw-r--r-- 1 root root 438764467 Dec  9 16:07 encoder.onxx\n",
      "drwxr-xr-x 2 root root      4096 Dec  9 16:07 metadata\n",
      "-rw-r--r-- 1 root root    791656 Dec  9 16:07 t5_spp\n"
     ]
    }
   ],
   "source": [
    "! ls -l {MODEL_NAME}_spark_nlp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see how we can use it on other machines, clusters, or any place you wish to use your new and shiny T5 model 😊"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------------------------------------------------------------------------------------------------------+\n",
      "|result                                                                                                     |\n",
      "+-----------------------------------------------------------------------------------------------------------+\n",
      "|[We introduce a unified framework that converts text-to-text language problems into a text-to-text format.]|\n",
      "+-----------------------------------------------------------------------------------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import sparknlp\n",
    "from sparknlp.base import *\n",
    "from sparknlp.annotator import *\n",
    "from pyspark.ml import Pipeline\n",
    "\n",
    "test_data = spark.createDataFrame([\n",
    "    [\"Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a \" +\n",
    "       \"downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness\" +\n",
    "       \" of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this \" +\n",
    "       \"paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework \" +\n",
    "       \"that converts all text-based language problems into a text-to-text format. Our systematic study compares \" +\n",
    "       \"pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens \" +\n",
    "       \"of language understanding tasks. By combining the insights from our exploration with scale and our new \" +\n",
    "       \"Colossal Clean Crawled Corpus, we achieve state-of-the-art results on many benchmarks covering \" +\n",
    "       \"summarization, question answering, text classification, and more. To facilitate future work on transfer \" +\n",
    "       \"learning for NLP, we release our data set, pre-trained models, and code.\"]\n",
    "]).toDF(\"text\")\n",
    "\n",
    "\n",
    "document_assembler = DocumentAssembler() \\\n",
    "    .setInputCol(\"text\")\\\n",
    "    .setOutputCol(\"document\")\n",
    "\n",
    "T5 = T5Transformer.load(f\"{MODEL_NAME}_spark_nlp\") \\\n",
    "  .setInputCols([\"document\"]) \\\n",
    "  .setOutputCol(\"summary\")\n",
    "\n",
    "pipeline = Pipeline().setStages([document_assembler, T5])\n",
    "\n",
    "result = pipeline.fit(test_data).transform(test_data)\n",
    "result.select(\"summary.result\").show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it! You can now go wild and use hundreds of T5 models from HuggingFace 🤗 in Spark NLP 🚀\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}