OpenELM.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "machine_shape": "hm",
      "gpuType": "L4",
      "authorship_tag": "ABX9TyMnRaZlcgcYfgYHhF23twN9",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/frank-morales2020/MLxDL/blob/main/OpenELM.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "https://huggingface.co/apple/OpenELM\n",
        "\n",
        "https://huggingface.co/apple/OpenELM-3B"
      ],
      "metadata": {
        "id": "VILThFV3mBh2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install colab-env --quiet\n",
        "!pip install tiktoken -q\n",
        "!pip install accelerate -q"
      ],
      "metadata": {
        "id": "sJac91eSyUoR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import colab_env\n",
        "import os\n",
        "\n",
        "access_token_write = os.getenv(\"HUGGINGFACE_ACCESS_TOKEN_WRITE\")\n",
        "\n",
        "from huggingface_hub import login\n",
        "\n",
        "login(\n",
        "  token=access_token_write,\n",
        "  add_to_git_credential=True\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-_3n0OBdycr7",
        "outputId": "ccbbcb66-e36d-4da1-85a9-40d5dc1a965e"
      },
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/gdrive; to attempt to forcibly remount, call drive.mount(\"/content/gdrive\", force_remount=True).\n",
            "Token is valid (permission: write).\n",
            "Your token has been saved in your configured git credential helpers (store).\n",
            "Your token has been saved to /root/.cache/huggingface/token\n",
            "Login successful\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%cd /content/\n",
        "!git lfs install\n",
        "!git clone https://huggingface.co/apple/OpenELM-3B"
      ],
      "metadata": {
        "id": "wUsWhL0iwbgj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!echo $HUGGINGFACE_ACCESS_TOKEN_WRITE"
      ],
      "metadata": {
        "id": "4w-vDIW6_zUT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!python /content/OpenELM-3B/generate_openelm.py --model apple/OpenELM-3B --hf_access_token $HUGGINGFACE_ACCESS_TOKEN_WRITE --prompt \"Okay, how about a more detailed explanation to a high schooler?\" --generate_kwargs repetition_penalty=1.2"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_HQCHNSLxe4u",
        "outputId": "ab628106-cff7-44bc-d03d-2ea2a821a703"
      },
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "WARNING:root:inference device is not set, using cuda:0, NVIDIA L4\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "Loading checkpoint shards: 100% 3/3 [00:02<00:00,  1.48it/s]\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "2024-05-12 18:10:44.338179: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
            "2024-05-12 18:10:44.338235: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
            "2024-05-12 18:10:44.340046: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
            "2024-05-12 18:10:45.461394: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "\n",
            "\n",
            "\u001b[1m Prompt + Generated Output\u001b[0m\n",
            "\n",
            "Okay, how about a more detailed explanation to a high schooler?\n",
            "\n",
            "I'm not sure if you're aware of this, but the US is a federal republic.  The states are the \"states\" and they have their own governments.  The federal government has its own government as well.  The federal government is made up of three branches: the executive, the legislative, and the judicial.  The executive branch is made up of the president and the vice president.  The legislative branch is made up of the House of Representatives and the Senate.  The judicial branch is made up of the Supreme Court and the lower courts.\n",
            "\n",
            "The executive branch is responsible for enforcing the laws passed by Congress.  The legislative branch is responsible for passing new laws.  The judicial branch is responsible for interpreting the laws passed by Congress and the laws passed by the states.\n",
            "\n",
            "The executive branch is made up of the president and the vice president.  The president is the head of the executive branch.  The vice president is the second in command.  The president and the vice president are elected by the people.  The president and the vice president are responsible for appointing cabinet members and other officials.\n",
            "\n",
            "\n",
            "Generation took\u001b[1m\u001b[92m 14.75 \u001b[0mseconds.\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#harness_repo=\"public-lm-eval-harness\"\n",
        "!git clone https://github.com/EleutherAI/lm-evaluation-harness/\n",
        "%cd /content/lm-evaluation-harness\n",
        "# use main branch on 03-15-2024, SHA is dc90fec\n",
        "#!git checkout dc90fec\n",
        "!pip install -e .\n",
        "#!cd .."
      ],
      "metadata": {
        "id": "DKG-jllArzaL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install datasets@git+https://github.com/huggingface/datasets.git@66d6242 -q\n",
        "!pip install tokenizers>=0.15.2 transformers>=4.38.2 sentencepiece>=0.2.0 -q"
      ],
      "metadata": {
        "id": "S1fCVQRXuN1B"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%mkdir /content/lm_eval_output"
      ],
      "metadata": {
        "id": "DwcneIsgvBYk"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%cd /content/lm_eval_output"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VBjRpw4jvJe0",
        "outputId": "13dea679-6329-4bed-ddb1-eb5b91677bdf"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/lm_eval_output\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "os.environ['hf_model'] = 'apple/OpenELM-3B'\n",
        "os.environ['task']='mmlu,winogrande'\n",
        "os.environ['shot']='5'\n",
        "\n",
        "os.environ['tokenizer']='meta-llama/Llama-2-7b-hf'\n",
        "os.environ['add_bos_token']='True'\n",
        "os.environ['batch_size']='1'"
      ],
      "metadata": {
        "id": "VHXBLEmqHbrB"
      },
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!echo $hf_model\n",
        "!echo $task\n",
        "!echo $shot\n",
        "\n",
        "!lm_eval --model hf --verbosity DEBUG --model_args pretrained=${hf_model},trust_remote_code=True,add_bos_token=${add_bos_token},tokenizer=${tokenizer} --tasks ${task} --device cuda:0 --num_fewshot ${shot} --output_path /content/lm_eval_output/${hf_model//\\//_}_${task//,/_}-${shot}shot --batch_size ${batch_size} 2>&1 | tee /content/lm_eval_output/eval-${hf_model//\\//_}_${task//,/_}-${shot}shot.log\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Wpbs38z1vRK8",
        "outputId": "c4cc44e7-7251-4428-92eb-9b3bfecd26a1"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "apple/OpenELM-3B\n",
            "mmlu,winogrande\n",
            "5\n",
            "2024-05-12 14:09:25.948337: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
            "2024-05-12 14:09:25.948379: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
            "2024-05-12 14:09:25.950387: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
            "2024-05-12 14:09:27.165210: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "2024-05-12:14:09:30,774 INFO     [__main__.py:254] Verbosity set to DEBUG\n",
            "2024-05-12:14:09:37,154 INFO     [__main__.py:341] Selected Tasks: ['mmlu', 'winogrande']\n",
            "2024-05-12:14:09:37,160 INFO     [evaluator.py:141] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234\n",
            "2024-05-12:14:09:37,160 INFO     [evaluator.py:178] Initializing hf model, with arguments: {'pretrained': 'apple/OpenELM-3B', 'trust_remote_code': True, 'add_bos_token': True, 'tokenizer': 'meta-llama/Llama-2-7b-hf'}\n",
            "2024-05-12:14:09:37,232 INFO     [huggingface.py:165] Using device 'cuda:0'\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
            "  warnings.warn(\n",
            "Loading checkpoint shards: 100%|██████████| 3/3 [00:04<00:00,  1.63s/it]\n",
            "/usr/local/lib/python3.10/dist-packages/datasets/load.py:1483: FutureWarning: The repository for hails/mmlu_no_train contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/hails/mmlu_no_train\n",
            "You can avoid this message in future by passing the argument `trust_remote_code=True`.\n",
            "Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.\n",
            "  warnings.warn(\n",
            "Downloading builder script: 100%|██████████| 5.86k/5.86k [00:00<00:00, 25.8MB/s]\n",
            "Downloading readme: 100%|██████████| 1.11k/1.11k [00:00<00:00, 7.03MB/s]\n",
            "Downloading data: 100%|██████████| 166M/166M [00:01<00:00, 96.2MB/s]\n",
            "Generating test split: 378 examples [00:00, 3052.62 examples/s]\n",
            "Generating validation split: 41 examples [00:00, 6210.64 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 50.72 examples/s]\n",
            "Generating test split: 152 examples [00:00, 1738.32 examples/s]\n",
            "Generating validation split: 16 examples [00:00, 5101.40 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 57.32 examples/s]\n",
            "Generating test split: 100 examples [00:00, 1116.79 examples/s]\n",
            "Generating validation split: 11 examples [00:00, 2351.67 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.65 examples/s]\n",
            "Generating test split: 151 examples [00:00, 1633.35 examples/s]\n",
            "Generating validation split: 17 examples [00:00, 3821.79 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 57.31 examples/s]\n",
            "Generating test split: 144 examples [00:00, 1679.15 examples/s]\n",
            "Generating validation split: 16 examples [00:00, 3000.08 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 60.23 examples/s]\n",
            "Generating test split: 310 examples [00:00, 3183.29 examples/s]\n",
            "Generating validation split: 32 examples [00:00, 7037.05 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 57.91 examples/s]\n",
            "Generating test split: 270 examples [00:00, 2852.93 examples/s]\n",
            "Generating validation split: 29 examples [00:00, 6830.73 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.53 examples/s]\n",
            "Generating test split: 100 examples [00:00, 1138.46 examples/s]\n",
            "Generating validation split: 9 examples [00:00, 3066.01 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 56.95 examples/s]\n",
            "Generating test split: 102 examples [00:00, 1149.06 examples/s]\n",
            "Generating validation split: 11 examples [00:00, 3347.41 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 59.35 examples/s]\n",
            "Generating test split: 100 examples [00:00, 1175.94 examples/s]\n",
            "Generating validation split: 8 examples [00:00, 1653.01 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.10 examples/s]\n",
            "Generating test split: 235 examples [00:00, 2455.03 examples/s]\n",
            "Generating validation split: 26 examples [00:00, 7125.25 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 55.85 examples/s]\n",
            "Generating test split: 203 examples [00:00, 2229.34 examples/s]\n",
            "Generating validation split: 22 examples [00:00, 4200.60 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 60.73 examples/s]\n",
            "Generating test split: 100 examples [00:00, 1105.87 examples/s]\n",
            "Generating validation split: 11 examples [00:00, 2947.13 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.95 examples/s]\n",
            "Generating test split: 100 examples [00:00, 1154.91 examples/s]\n",
            "Generating validation split: 11 examples [00:00, 2723.41 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 57.49 examples/s]\n",
            "Generating test split: 112 examples [00:00, 1286.47 examples/s]\n",
            "Generating validation split: 11 examples [00:00, 3355.69 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.54 examples/s]\n",
            "Generating test split: 100 examples [00:00, 1159.20 examples/s]\n",
            "Generating validation split: 11 examples [00:00, 2984.50 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 57.97 examples/s]\n",
            "Generating test split: 135 examples [00:00, 1547.05 examples/s]\n",
            "Generating validation split: 14 examples [00:00, 4456.95 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 59.56 examples/s]\n",
            "Generating test split: 145 examples [00:00, 1685.22 examples/s]\n",
            "Generating validation split: 16 examples [00:00, 4121.91 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 59.31 examples/s]\n",
            "Generating test split: 216 examples [00:00, 2226.64 examples/s]\n",
            "Generating validation split: 23 examples [00:00, 4949.67 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.69 examples/s]\n",
            "Generating test split: 306 examples [00:00, 2825.02 examples/s]\n",
            "Generating validation split: 33 examples [00:00, 5982.02 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 49.53 examples/s]\n",
            "Generating test split: 265 examples [00:00, 2676.98 examples/s]\n",
            "Generating validation split: 29 examples [00:00, 6320.00 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.91 examples/s]\n",
            "Generating test split: 100 examples [00:00, 1186.92 examples/s]\n",
            "Generating validation split: 10 examples [00:00, 3253.92 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 56.31 examples/s]\n",
            "Generating test split: 100 examples [00:00, 1103.44 examples/s]\n",
            "Generating validation split: 11 examples [00:00, 3672.77 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.58 examples/s]\n",
            "Generating test split: 282 examples [00:00, 2818.24 examples/s]\n",
            "Generating validation split: 31 examples [00:00, 3999.49 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 59.07 examples/s]\n",
            "Generating test split: 173 examples [00:00, 1901.21 examples/s]\n",
            "Generating validation split: 22 examples [00:00, 5357.33 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 57.46 examples/s]\n",
            "Generating test split: 223 examples [00:00, 2376.88 examples/s]\n",
            "Generating validation split: 23 examples [00:00, 4698.24 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 59.42 examples/s]\n",
            "Generating test split: 234 examples [00:00, 2469.74 examples/s]\n",
            "Generating validation split: 25 examples [00:00, 4267.36 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 59.48 examples/s]\n",
            "Generating test split: 166 examples [00:00, 1828.41 examples/s]\n",
            "Generating validation split: 18 examples [00:00, 3792.89 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 59.27 examples/s]\n",
            "Generating test split: 272 examples [00:00, 2704.34 examples/s]\n",
            "Generating validation split: 31 examples [00:00, 5393.60 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.86 examples/s]\n",
            "Generating test split: 100 examples [00:00, 1151.10 examples/s]\n",
            "Generating validation split: 11 examples [00:00, 3753.75 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.53 examples/s]\n",
            "Generating test split: 103 examples [00:00, 1176.53 examples/s]\n",
            "Generating validation split: 11 examples [00:00, 2612.24 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 59.78 examples/s]\n",
            "Generating test split: 783 examples [00:00, 6037.29 examples/s]\n",
            "Generating validation split: 86 examples [00:00, 11581.64 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.47 examples/s]\n",
            "Generating test split: 193 examples [00:00, 2138.00 examples/s]\n",
            "Generating validation split: 21 examples [00:00, 5636.06 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 59.29 examples/s]\n",
            "Generating test split: 390 examples [00:00, 3877.97 examples/s]\n",
            "Generating validation split: 43 examples [00:00, 7650.92 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 59.77 examples/s]\n",
            "Generating test split: 100 examples [00:00, 1087.62 examples/s]\n",
            "Generating validation split: 11 examples [00:00, 2559.91 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 57.72 examples/s]\n",
            "Generating test split: 545 examples [00:00, 4978.69 examples/s]\n",
            "Generating validation split: 60 examples [00:00, 7674.38 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.95 examples/s]\n",
            "Generating test split: 245 examples [00:00, 2655.10 examples/s]\n",
            "Generating validation split: 27 examples [00:00, 3974.95 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.69 examples/s]\n",
            "Generating test split: 612 examples [00:00, 5215.48 examples/s]\n",
            "Generating validation split: 69 examples [00:00, 8230.91 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 57.61 examples/s]\n",
            "Generating test split: 131 examples [00:00, 1171.77 examples/s]\n",
            "Generating validation split: 12 examples [00:00, 2288.01 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 45.29 examples/s]\n",
            "Generating test split: 198 examples [00:00, 2101.74 examples/s]\n",
            "Generating validation split: 22 examples [00:00, 6577.89 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 59.28 examples/s]\n",
            "Generating test split: 114 examples [00:00, 1314.89 examples/s]\n",
            "Generating validation split: 12 examples [00:00, 3223.49 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.82 examples/s]\n",
            "Generating test split: 201 examples [00:00, 2226.17 examples/s]\n",
            "Generating validation split: 22 examples [00:00, 3691.14 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.27 examples/s]\n",
            "Generating test split: 110 examples [00:00, 1156.77 examples/s]\n",
            "Generating validation split: 12 examples [00:00, 2576.22 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 50.46 examples/s]\n",
            "Generating test split: 238 examples [00:00, 2412.96 examples/s]\n",
            "Generating validation split: 26 examples [00:00, 4394.60 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.47 examples/s]\n",
            "Generating test split: 163 examples [00:00, 1829.53 examples/s]\n",
            "Generating validation split: 18 examples [00:00, 3698.14 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 59.16 examples/s]\n",
            "Generating test split: 311 examples [00:00, 3177.98 examples/s]\n",
            "Generating validation split: 34 examples [00:00, 7703.04 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 57.84 examples/s]\n",
            "Generating test split: 171 examples [00:00, 1875.01 examples/s]\n",
            "Generating validation split: 19 examples [00:00, 4548.36 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 56.46 examples/s]\n",
            "Generating test split: 237 examples [00:00, 2350.46 examples/s]\n",
            "Generating validation split: 26 examples [00:00, 3828.80 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.01 examples/s]\n",
            "Generating test split: 126 examples [00:00, 1369.90 examples/s]\n",
            "Generating validation split: 14 examples [00:00, 3417.35 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.77 examples/s]\n",
            "Generating test split: 165 examples [00:00, 1774.62 examples/s]\n",
            "Generating validation split: 18 examples [00:00, 2908.45 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.96 examples/s]\n",
            "Generating test split: 324 examples [00:00, 3165.84 examples/s]\n",
            "Generating validation split: 35 examples [00:00, 6157.49 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.82 examples/s]\n",
            "Generating test split: 121 examples [00:00, 1375.63 examples/s]\n",
            "Generating validation split: 13 examples [00:00, 3072.75 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 56.51 examples/s]\n",
            "Generating test split: 1534 examples [00:00, 8223.32 examples/s]\n",
            "Generating validation split: 170 examples [00:00, 10890.47 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 57.25 examples/s]\n",
            "Generating test split: 895 examples [00:00, 6555.58 examples/s]\n",
            "Generating validation split: 100 examples [00:00, 10805.05 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.24 examples/s]\n",
            "Generating test split: 108 examples [00:00, 1240.28 examples/s]\n",
            "Generating validation split: 11 examples [00:00, 2133.52 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.43 examples/s]\n",
            "Generating test split: 204 examples [00:00, 2118.47 examples/s]\n",
            "Generating validation split: 22 examples [00:00, 3434.88 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 58.46 examples/s]\n",
            "Generating test split: 346 examples [00:00, 3419.14 examples/s]\n",
            "Generating validation split: 38 examples [00:00, 5361.94 examples/s]\n",
            "Generating dev split: 5 examples [00:00, 56.81 examples/s]\n",
            "Downloading data: 100%|██████████| 2.06M/2.06M [00:00<00:00, 5.93MB/s]\n",
            "Downloading data: 100%|██████████| 118k/118k [00:00<00:00, 464kB/s]\n",
            "Downloading data: 100%|██████████| 85.9k/85.9k [00:00<00:00, 316kB/s]\n",
            "Generating train split: 100%|██████████| 40398/40398 [00:00<00:00, 994701.86 examples/s]\n",
            "Generating test split: 100%|██████████| 1767/1767 [00:00<00:00, 644183.85 examples/s]\n",
            "Generating validation split: 100%|██████████| 1267/1267 [00:00<00:00, 529301.11 examples/s]\n",
            "2024-05-12:14:13:05,569 WARNING  [evaluator.py:240] Overwriting default num_fewshot of winogrande from None to 5\n",
            "2024-05-12:14:13:05,570 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,570 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_moral_disputes from None to 5\n",
            "2024-05-12:14:13:05,570 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,570 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_high_school_us_history from None to 5\n",
            "2024-05-12:14:13:05,570 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,570 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_jurisprudence from None to 5\n",
            "2024-05-12:14:13:05,570 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,570 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_moral_scenarios from None to 5\n",
            "2024-05-12:14:13:05,570 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,570 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_professional_law from None to 5\n",
            "2024-05-12:14:13:05,570 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,570 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_international_law from None to 5\n",
            "2024-05-12:14:13:05,570 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,570 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_prehistory from None to 5\n",
            "2024-05-12:14:13:05,570 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,570 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_high_school_european_history from None to 5\n",
            "2024-05-12:14:13:05,571 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,571 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_formal_logic from None to 5\n",
            "2024-05-12:14:13:05,571 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,571 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_high_school_world_history from None to 5\n",
            "2024-05-12:14:13:05,571 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,571 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_world_religions from None to 5\n",
            "2024-05-12:14:13:05,571 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,571 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_philosophy from None to 5\n",
            "2024-05-12:14:13:05,571 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,571 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_logical_fallacies from None to 5\n",
            "2024-05-12:14:13:05,571 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,571 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_high_school_microeconomics from None to 5\n",
            "2024-05-12:14:13:05,571 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,571 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_public_relations from None to 5\n",
            "2024-05-12:14:13:05,571 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,571 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_sociology from None to 5\n",
            "2024-05-12:14:13:05,571 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,571 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_econometrics from None to 5\n",
            "2024-05-12:14:13:05,571 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,572 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_high_school_geography from None to 5\n",
            "2024-05-12:14:13:05,572 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,572 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_human_sexuality from None to 5\n",
            "2024-05-12:14:13:05,572 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,572 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_professional_psychology from None to 5\n",
            "2024-05-12:14:13:05,572 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,572 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_security_studies from None to 5\n",
            "2024-05-12:14:13:05,572 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,572 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_high_school_psychology from None to 5\n",
            "2024-05-12:14:13:05,572 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,572 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_us_foreign_policy from None to 5\n",
            "2024-05-12:14:13:05,572 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,572 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_high_school_macroeconomics from None to 5\n",
            "2024-05-12:14:13:05,572 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,572 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_high_school_government_and_politics from None to 5\n",
            "2024-05-12:14:13:05,572 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,572 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_miscellaneous from None to 5\n",
            "2024-05-12:14:13:05,572 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,572 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_management from None to 5\n",
            "2024-05-12:14:13:05,572 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,572 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_medical_genetics from None to 5\n",
            "2024-05-12:14:13:05,572 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,572 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_professional_medicine from None to 5\n",
            "2024-05-12:14:13:05,572 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,572 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_virology from None to 5\n",
            "2024-05-12:14:13:05,572 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,572 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_marketing from None to 5\n",
            "2024-05-12:14:13:05,572 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,573 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_human_aging from None to 5\n",
            "2024-05-12:14:13:05,573 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,573 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_college_medicine from None to 5\n",
            "2024-05-12:14:13:05,573 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,573 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_professional_accounting from None to 5\n",
            "2024-05-12:14:13:05,573 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,573 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_business_ethics from None to 5\n",
            "2024-05-12:14:13:05,573 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,573 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_global_facts from None to 5\n",
            "2024-05-12:14:13:05,573 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,573 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_clinical_knowledge from None to 5\n",
            "2024-05-12:14:13:05,573 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,573 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_nutrition from None to 5\n",
            "2024-05-12:14:13:05,573 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,573 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_high_school_statistics from None to 5\n",
            "2024-05-12:14:13:05,573 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,573 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_electrical_engineering from None to 5\n",
            "2024-05-12:14:13:05,573 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,573 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_anatomy from None to 5\n",
            "2024-05-12:14:13:05,573 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,573 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_abstract_algebra from None to 5\n",
            "2024-05-12:14:13:05,573 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,573 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_machine_learning from None to 5\n",
            "2024-05-12:14:13:05,573 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,573 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_computer_security from None to 5\n",
            "2024-05-12:14:13:05,573 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,573 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_college_mathematics from None to 5\n",
            "2024-05-12:14:13:05,573 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,573 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_high_school_chemistry from None to 5\n",
            "2024-05-12:14:13:05,573 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,573 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_conceptual_physics from None to 5\n",
            "2024-05-12:14:13:05,574 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,574 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_college_chemistry from None to 5\n",
            "2024-05-12:14:13:05,574 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,574 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_college_physics from None to 5\n",
            "2024-05-12:14:13:05,574 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,574 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_high_school_computer_science from None to 5\n",
            "2024-05-12:14:13:05,574 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,574 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_high_school_mathematics from None to 5\n",
            "2024-05-12:14:13:05,574 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,574 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_high_school_biology from None to 5\n",
            "2024-05-12:14:13:05,574 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,574 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_college_biology from None to 5\n",
            "2024-05-12:14:13:05,574 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,574 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_high_school_physics from None to 5\n",
            "2024-05-12:14:13:05,574 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,574 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_college_computer_science from None to 5\n",
            "2024-05-12:14:13:05,574 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,574 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_astronomy from None to 5\n",
            "2024-05-12:14:13:05,574 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,574 WARNING  [evaluator.py:240] Overwriting default num_fewshot of mmlu_elementary_mathematics from None to 5\n",
            "2024-05-12:14:13:05,574 INFO     [evaluator.py:245] Setting fewshot random generator seed to 1234\n",
            "2024-05-12:14:13:05,580 DEBUG    [cache.py:33] requests-winogrande-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:13:05,581 INFO     [task.py:398] Building contexts for winogrande on rank 0...\n",
            "100%|██████████| 1267/1267 [00:00<00:00, 15311.25it/s]\n",
            "2024-05-12:14:13:05,729 DEBUG    [evaluator.py:366] Task: winogrande; number of requests on this rank: 2534\n",
            "2024-05-12:14:13:05,730 DEBUG    [cache.py:33] requests-mmlu_moral_disputes-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:13:05,730 INFO     [task.py:398] Building contexts for mmlu_moral_disputes on rank 0...\n",
            "100%|██████████| 346/346 [00:05<00:00, 66.14it/s]\n",
            "2024-05-12:14:13:10,977 DEBUG    [evaluator.py:366] Task: mmlu_moral_disputes; number of requests on this rank: 1384\n",
            "2024-05-12:14:13:10,978 DEBUG    [cache.py:33] requests-mmlu_high_school_us_history-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:13:10,978 INFO     [task.py:398] Building contexts for mmlu_high_school_us_history on rank 0...\n",
            "100%|██████████| 204/204 [00:03<00:00, 65.53it/s]\n",
            "2024-05-12:14:13:14,101 DEBUG    [evaluator.py:366] Task: mmlu_high_school_us_history; number of requests on this rank: 816\n",
            "2024-05-12:14:13:14,101 DEBUG    [cache.py:33] requests-mmlu_jurisprudence-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:13:14,101 INFO     [task.py:398] Building contexts for mmlu_jurisprudence on rank 0...\n",
            "100%|██████████| 108/108 [00:01<00:00, 66.27it/s]\n",
            "2024-05-12:14:13:15,736 DEBUG    [evaluator.py:366] Task: mmlu_jurisprudence; number of requests on this rank: 432\n",
            "2024-05-12:14:13:15,736 DEBUG    [cache.py:33] requests-mmlu_moral_scenarios-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:13:15,737 INFO     [task.py:398] Building contexts for mmlu_moral_scenarios on rank 0...\n",
            "100%|██████████| 895/895 [00:13<00:00, 66.50it/s]\n",
            "2024-05-12:14:13:29,236 DEBUG    [evaluator.py:366] Task: mmlu_moral_scenarios; number of requests on this rank: 3580\n",
            "2024-05-12:14:13:29,237 DEBUG    [cache.py:33] requests-mmlu_professional_law-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:13:29,237 INFO     [task.py:398] Building contexts for mmlu_professional_law on rank 0...\n",
            "100%|██████████| 1534/1534 [00:22<00:00, 66.80it/s]\n",
            "2024-05-12:14:13:52,270 DEBUG    [evaluator.py:366] Task: mmlu_professional_law; number of requests on this rank: 6136\n",
            "2024-05-12:14:13:52,272 DEBUG    [cache.py:33] requests-mmlu_international_law-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:13:52,272 INFO     [task.py:398] Building contexts for mmlu_international_law on rank 0...\n",
            "100%|██████████| 121/121 [00:01<00:00, 66.67it/s]\n",
            "2024-05-12:14:13:54,093 DEBUG    [evaluator.py:366] Task: mmlu_international_law; number of requests on this rank: 484\n",
            "2024-05-12:14:13:54,093 DEBUG    [cache.py:33] requests-mmlu_prehistory-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:13:54,093 INFO     [task.py:398] Building contexts for mmlu_prehistory on rank 0...\n",
            "100%|██████████| 324/324 [00:04<00:00, 67.00it/s]\n",
            "2024-05-12:14:13:58,944 DEBUG    [evaluator.py:366] Task: mmlu_prehistory; number of requests on this rank: 1296\n",
            "2024-05-12:14:13:58,944 DEBUG    [cache.py:33] requests-mmlu_high_school_european_history-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:13:58,944 INFO     [task.py:398] Building contexts for mmlu_high_school_european_history on rank 0...\n",
            "100%|██████████| 165/165 [00:02<00:00, 66.87it/s]\n",
            "2024-05-12:14:14:01,420 DEBUG    [evaluator.py:366] Task: mmlu_high_school_european_history; number of requests on this rank: 660\n",
            "2024-05-12:14:14:01,420 DEBUG    [cache.py:33] requests-mmlu_formal_logic-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:14:01,421 INFO     [task.py:398] Building contexts for mmlu_formal_logic on rank 0...\n",
            "100%|██████████| 126/126 [00:01<00:00, 66.56it/s]\n",
            "2024-05-12:14:14:03,320 DEBUG    [evaluator.py:366] Task: mmlu_formal_logic; number of requests on this rank: 504\n",
            "2024-05-12:14:14:03,320 DEBUG    [cache.py:33] requests-mmlu_high_school_world_history-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:14:03,320 INFO     [task.py:398] Building contexts for mmlu_high_school_world_history on rank 0...\n",
            "100%|██████████| 237/237 [00:03<00:00, 66.55it/s]\n",
            "2024-05-12:14:14:06,893 DEBUG    [evaluator.py:366] Task: mmlu_high_school_world_history; number of requests on this rank: 948\n",
            "2024-05-12:14:14:06,893 DEBUG    [cache.py:33] requests-mmlu_world_religions-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:14:06,893 INFO     [task.py:398] Building contexts for mmlu_world_religions on rank 0...\n",
            "100%|██████████| 171/171 [00:02<00:00, 66.26it/s]\n",
            "2024-05-12:14:14:09,482 DEBUG    [evaluator.py:366] Task: mmlu_world_religions; number of requests on this rank: 684\n",
            "2024-05-12:14:14:09,483 DEBUG    [cache.py:33] requests-mmlu_philosophy-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:14:09,483 INFO     [task.py:398] Building contexts for mmlu_philosophy on rank 0...\n",
            "100%|██████████| 311/311 [00:04<00:00, 65.88it/s]\n",
            "2024-05-12:14:14:14,218 DEBUG    [evaluator.py:366] Task: mmlu_philosophy; number of requests on this rank: 1244\n",
            "2024-05-12:14:14:14,218 DEBUG    [cache.py:33] requests-mmlu_logical_fallacies-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:14:14,218 INFO     [task.py:398] Building contexts for mmlu_logical_fallacies on rank 0...\n",
            "100%|██████████| 163/163 [00:02<00:00, 66.53it/s]\n",
            "2024-05-12:14:14:16,676 DEBUG    [evaluator.py:366] Task: mmlu_logical_fallacies; number of requests on this rank: 652\n",
            "2024-05-12:14:14:16,676 DEBUG    [cache.py:33] requests-mmlu_high_school_microeconomics-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:14:16,677 INFO     [task.py:398] Building contexts for mmlu_high_school_microeconomics on rank 0...\n",
            "100%|██████████| 238/238 [00:03<00:00, 66.36it/s]\n",
            "2024-05-12:14:14:20,274 DEBUG    [evaluator.py:366] Task: mmlu_high_school_microeconomics; number of requests on this rank: 952\n",
            "2024-05-12:14:14:20,275 DEBUG    [cache.py:33] requests-mmlu_public_relations-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:14:20,275 INFO     [task.py:398] Building contexts for mmlu_public_relations on rank 0...\n",
            "100%|██████████| 110/110 [00:01<00:00, 66.67it/s]\n",
            "2024-05-12:14:14:21,931 DEBUG    [evaluator.py:366] Task: mmlu_public_relations; number of requests on this rank: 440\n",
            "2024-05-12:14:14:21,931 DEBUG    [cache.py:33] requests-mmlu_sociology-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:14:21,931 INFO     [task.py:398] Building contexts for mmlu_sociology on rank 0...\n",
            "100%|██████████| 201/201 [00:03<00:00, 66.39it/s]\n",
            "2024-05-12:14:14:24,968 DEBUG    [evaluator.py:366] Task: mmlu_sociology; number of requests on this rank: 804\n",
            "2024-05-12:14:14:24,968 DEBUG    [cache.py:33] requests-mmlu_econometrics-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:14:24,968 INFO     [task.py:398] Building contexts for mmlu_econometrics on rank 0...\n",
            "100%|██████████| 114/114 [00:01<00:00, 65.78it/s]\n",
            "2024-05-12:14:14:26,707 DEBUG    [evaluator.py:366] Task: mmlu_econometrics; number of requests on this rank: 456\n",
            "2024-05-12:14:14:26,707 DEBUG    [cache.py:33] requests-mmlu_high_school_geography-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:14:26,707 INFO     [task.py:398] Building contexts for mmlu_high_school_geography on rank 0...\n",
            "100%|██████████| 198/198 [00:02<00:00, 66.34it/s]\n",
            "2024-05-12:14:14:29,701 DEBUG    [evaluator.py:366] Task: mmlu_high_school_geography; number of requests on this rank: 792\n",
            "2024-05-12:14:14:29,702 DEBUG    [cache.py:33] requests-mmlu_human_sexuality-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:14:29,702 INFO     [task.py:398] Building contexts for mmlu_human_sexuality on rank 0...\n",
            "100%|██████████| 131/131 [00:01<00:00, 65.87it/s]\n",
            "2024-05-12:14:14:31,697 DEBUG    [evaluator.py:366] Task: mmlu_human_sexuality; number of requests on this rank: 524\n",
            "2024-05-12:14:14:31,697 DEBUG    [cache.py:33] requests-mmlu_professional_psychology-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:14:31,697 INFO     [task.py:398] Building contexts for mmlu_professional_psychology on rank 0...\n",
            "100%|██████████| 612/612 [00:09<00:00, 66.15it/s]\n",
            "2024-05-12:14:14:40,976 DEBUG    [evaluator.py:366] Task: mmlu_professional_psychology; number of requests on this rank: 2448\n",
            "2024-05-12:14:14:40,977 DEBUG    [cache.py:33] requests-mmlu_security_studies-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:14:40,977 INFO     [task.py:398] Building contexts for mmlu_security_studies on rank 0...\n",
            "100%|██████████| 245/245 [00:03<00:00, 66.23it/s]\n",
            "2024-05-12:14:14:44,688 DEBUG    [evaluator.py:366] Task: mmlu_security_studies; number of requests on this rank: 980\n",
            "2024-05-12:14:14:44,688 DEBUG    [cache.py:33] requests-mmlu_high_school_psychology-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:14:44,688 INFO     [task.py:398] Building contexts for mmlu_high_school_psychology on rank 0...\n",
            "100%|██████████| 545/545 [00:08<00:00, 66.27it/s]\n",
            "2024-05-12:14:14:52,937 DEBUG    [evaluator.py:366] Task: mmlu_high_school_psychology; number of requests on this rank: 2180\n",
            "2024-05-12:14:14:52,938 DEBUG    [cache.py:33] requests-mmlu_us_foreign_policy-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:14:52,938 INFO     [task.py:398] Building contexts for mmlu_us_foreign_policy on rank 0...\n",
            "100%|██████████| 100/100 [00:01<00:00, 66.20it/s]\n",
            "2024-05-12:14:14:54,454 DEBUG    [evaluator.py:366] Task: mmlu_us_foreign_policy; number of requests on this rank: 400\n",
            "2024-05-12:14:14:54,454 DEBUG    [cache.py:33] requests-mmlu_high_school_macroeconomics-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:14:54,454 INFO     [task.py:398] Building contexts for mmlu_high_school_macroeconomics on rank 0...\n",
            "100%|██████████| 390/390 [00:06<00:00, 63.51it/s]\n",
            "2024-05-12:14:15:00,613 DEBUG    [evaluator.py:366] Task: mmlu_high_school_macroeconomics; number of requests on this rank: 1560\n",
            "2024-05-12:14:15:00,614 DEBUG    [cache.py:33] requests-mmlu_high_school_government_and_politics-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:00,614 INFO     [task.py:398] Building contexts for mmlu_high_school_government_and_politics on rank 0...\n",
            "100%|██████████| 193/193 [00:02<00:00, 64.96it/s]\n",
            "2024-05-12:14:15:03,595 DEBUG    [evaluator.py:366] Task: mmlu_high_school_government_and_politics; number of requests on this rank: 772\n",
            "2024-05-12:14:15:03,595 DEBUG    [cache.py:33] requests-mmlu_miscellaneous-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:03,595 INFO     [task.py:398] Building contexts for mmlu_miscellaneous on rank 0...\n",
            "100%|██████████| 783/783 [00:11<00:00, 66.10it/s]\n",
            "2024-05-12:14:15:15,477 DEBUG    [evaluator.py:366] Task: mmlu_miscellaneous; number of requests on this rank: 3132\n",
            "2024-05-12:14:15:15,478 DEBUG    [cache.py:33] requests-mmlu_management-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:15,478 INFO     [task.py:398] Building contexts for mmlu_management on rank 0...\n",
            "100%|██████████| 103/103 [00:01<00:00, 66.16it/s]\n",
            "2024-05-12:14:15:17,040 DEBUG    [evaluator.py:366] Task: mmlu_management; number of requests on this rank: 412\n",
            "2024-05-12:14:15:17,040 DEBUG    [cache.py:33] requests-mmlu_medical_genetics-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:17,040 INFO     [task.py:398] Building contexts for mmlu_medical_genetics on rank 0...\n",
            "100%|██████████| 100/100 [00:01<00:00, 66.66it/s]\n",
            "2024-05-12:14:15:18,546 DEBUG    [evaluator.py:366] Task: mmlu_medical_genetics; number of requests on this rank: 400\n",
            "2024-05-12:14:15:18,546 DEBUG    [cache.py:33] requests-mmlu_professional_medicine-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:18,546 INFO     [task.py:398] Building contexts for mmlu_professional_medicine on rank 0...\n",
            "100%|██████████| 272/272 [00:04<00:00, 66.62it/s]\n",
            "2024-05-12:14:15:22,642 DEBUG    [evaluator.py:366] Task: mmlu_professional_medicine; number of requests on this rank: 1088\n",
            "2024-05-12:14:15:22,642 DEBUG    [cache.py:33] requests-mmlu_virology-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:22,642 INFO     [task.py:398] Building contexts for mmlu_virology on rank 0...\n",
            "100%|██████████| 166/166 [00:02<00:00, 66.55it/s]\n",
            "2024-05-12:14:15:25,144 DEBUG    [evaluator.py:366] Task: mmlu_virology; number of requests on this rank: 664\n",
            "2024-05-12:14:15:25,145 DEBUG    [cache.py:33] requests-mmlu_marketing-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:25,145 INFO     [task.py:398] Building contexts for mmlu_marketing on rank 0...\n",
            "100%|██████████| 234/234 [00:03<00:00, 66.07it/s]\n",
            "2024-05-12:14:15:28,697 DEBUG    [evaluator.py:366] Task: mmlu_marketing; number of requests on this rank: 936\n",
            "2024-05-12:14:15:28,698 DEBUG    [cache.py:33] requests-mmlu_human_aging-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:28,698 INFO     [task.py:398] Building contexts for mmlu_human_aging on rank 0...\n",
            "100%|██████████| 223/223 [00:03<00:00, 66.90it/s]\n",
            "2024-05-12:14:15:32,042 DEBUG    [evaluator.py:366] Task: mmlu_human_aging; number of requests on this rank: 892\n",
            "2024-05-12:14:15:32,042 DEBUG    [cache.py:33] requests-mmlu_college_medicine-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:32,042 INFO     [task.py:398] Building contexts for mmlu_college_medicine on rank 0...\n",
            "100%|██████████| 173/173 [00:02<00:00, 66.92it/s]\n",
            "2024-05-12:14:15:34,635 DEBUG    [evaluator.py:366] Task: mmlu_college_medicine; number of requests on this rank: 692\n",
            "2024-05-12:14:15:34,636 DEBUG    [cache.py:33] requests-mmlu_professional_accounting-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:34,636 INFO     [task.py:398] Building contexts for mmlu_professional_accounting on rank 0...\n",
            "100%|██████████| 282/282 [00:04<00:00, 66.47it/s]\n",
            "2024-05-12:14:15:38,891 DEBUG    [evaluator.py:366] Task: mmlu_professional_accounting; number of requests on this rank: 1128\n",
            "2024-05-12:14:15:38,892 DEBUG    [cache.py:33] requests-mmlu_business_ethics-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:38,892 INFO     [task.py:398] Building contexts for mmlu_business_ethics on rank 0...\n",
            "100%|██████████| 100/100 [00:01<00:00, 64.86it/s]\n",
            "2024-05-12:14:15:40,439 DEBUG    [evaluator.py:366] Task: mmlu_business_ethics; number of requests on this rank: 400\n",
            "2024-05-12:14:15:40,439 DEBUG    [cache.py:33] requests-mmlu_global_facts-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:40,439 INFO     [task.py:398] Building contexts for mmlu_global_facts on rank 0...\n",
            "100%|██████████| 100/100 [00:01<00:00, 66.32it/s]\n",
            "2024-05-12:14:15:41,952 DEBUG    [evaluator.py:366] Task: mmlu_global_facts; number of requests on this rank: 400\n",
            "2024-05-12:14:15:41,952 DEBUG    [cache.py:33] requests-mmlu_clinical_knowledge-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:41,952 INFO     [task.py:398] Building contexts for mmlu_clinical_knowledge on rank 0...\n",
            "100%|██████████| 265/265 [00:03<00:00, 66.82it/s]\n",
            "2024-05-12:14:15:45,931 DEBUG    [evaluator.py:366] Task: mmlu_clinical_knowledge; number of requests on this rank: 1060\n",
            "2024-05-12:14:15:45,931 DEBUG    [cache.py:33] requests-mmlu_nutrition-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:45,931 INFO     [task.py:398] Building contexts for mmlu_nutrition on rank 0...\n",
            "100%|██████████| 306/306 [00:04<00:00, 66.57it/s]\n",
            "2024-05-12:14:15:50,543 DEBUG    [evaluator.py:366] Task: mmlu_nutrition; number of requests on this rank: 1224\n",
            "2024-05-12:14:15:50,543 DEBUG    [cache.py:33] requests-mmlu_high_school_statistics-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:50,543 INFO     [task.py:398] Building contexts for mmlu_high_school_statistics on rank 0...\n",
            "100%|██████████| 216/216 [00:03<00:00, 65.79it/s]\n",
            "2024-05-12:14:15:53,837 DEBUG    [evaluator.py:366] Task: mmlu_high_school_statistics; number of requests on this rank: 864\n",
            "2024-05-12:14:15:53,838 DEBUG    [cache.py:33] requests-mmlu_electrical_engineering-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:53,838 INFO     [task.py:398] Building contexts for mmlu_electrical_engineering on rank 0...\n",
            "100%|██████████| 145/145 [00:02<00:00, 66.48it/s]\n",
            "2024-05-12:14:15:56,026 DEBUG    [evaluator.py:366] Task: mmlu_electrical_engineering; number of requests on this rank: 580\n",
            "2024-05-12:14:15:56,026 DEBUG    [cache.py:33] requests-mmlu_anatomy-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:56,026 INFO     [task.py:398] Building contexts for mmlu_anatomy on rank 0...\n",
            "100%|██████████| 135/135 [00:02<00:00, 66.64it/s]\n",
            "2024-05-12:14:15:58,059 DEBUG    [evaluator.py:366] Task: mmlu_anatomy; number of requests on this rank: 540\n",
            "2024-05-12:14:15:58,059 DEBUG    [cache.py:33] requests-mmlu_abstract_algebra-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:58,059 INFO     [task.py:398] Building contexts for mmlu_abstract_algebra on rank 0...\n",
            "100%|██████████| 100/100 [00:01<00:00, 66.88it/s]\n",
            "2024-05-12:14:15:59,559 DEBUG    [evaluator.py:366] Task: mmlu_abstract_algebra; number of requests on this rank: 400\n",
            "2024-05-12:14:15:59,559 DEBUG    [cache.py:33] requests-mmlu_machine_learning-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:15:59,559 INFO     [task.py:398] Building contexts for mmlu_machine_learning on rank 0...\n",
            "100%|██████████| 112/112 [00:01<00:00, 66.73it/s]\n",
            "2024-05-12:14:16:01,243 DEBUG    [evaluator.py:366] Task: mmlu_machine_learning; number of requests on this rank: 448\n",
            "2024-05-12:14:16:01,244 DEBUG    [cache.py:33] requests-mmlu_computer_security-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:16:01,244 INFO     [task.py:398] Building contexts for mmlu_computer_security on rank 0...\n",
            "100%|██████████| 100/100 [00:01<00:00, 66.83it/s]\n",
            "2024-05-12:14:16:02,745 DEBUG    [evaluator.py:366] Task: mmlu_computer_security; number of requests on this rank: 400\n",
            "2024-05-12:14:16:02,745 DEBUG    [cache.py:33] requests-mmlu_college_mathematics-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:16:02,745 INFO     [task.py:398] Building contexts for mmlu_college_mathematics on rank 0...\n",
            "100%|██████████| 100/100 [00:01<00:00, 65.68it/s]\n",
            "2024-05-12:14:16:04,273 DEBUG    [evaluator.py:366] Task: mmlu_college_mathematics; number of requests on this rank: 400\n",
            "2024-05-12:14:16:04,273 DEBUG    [cache.py:33] requests-mmlu_high_school_chemistry-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:16:04,273 INFO     [task.py:398] Building contexts for mmlu_high_school_chemistry on rank 0...\n",
            "100%|██████████| 203/203 [00:03<00:00, 66.17it/s]\n",
            "2024-05-12:14:16:07,351 DEBUG    [evaluator.py:366] Task: mmlu_high_school_chemistry; number of requests on this rank: 812\n",
            "2024-05-12:14:16:07,351 DEBUG    [cache.py:33] requests-mmlu_conceptual_physics-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:16:07,351 INFO     [task.py:398] Building contexts for mmlu_conceptual_physics on rank 0...\n",
            "100%|██████████| 235/235 [00:03<00:00, 66.47it/s]\n",
            "2024-05-12:14:16:10,897 DEBUG    [evaluator.py:366] Task: mmlu_conceptual_physics; number of requests on this rank: 940\n",
            "2024-05-12:14:16:10,897 DEBUG    [cache.py:33] requests-mmlu_college_chemistry-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:16:10,897 INFO     [task.py:398] Building contexts for mmlu_college_chemistry on rank 0...\n",
            "100%|██████████| 100/100 [00:01<00:00, 66.31it/s]\n",
            "2024-05-12:14:16:12,411 DEBUG    [evaluator.py:366] Task: mmlu_college_chemistry; number of requests on this rank: 400\n",
            "2024-05-12:14:16:12,411 DEBUG    [cache.py:33] requests-mmlu_college_physics-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:16:12,411 INFO     [task.py:398] Building contexts for mmlu_college_physics on rank 0...\n",
            "100%|██████████| 102/102 [00:01<00:00, 66.51it/s]\n",
            "2024-05-12:14:16:13,950 DEBUG    [evaluator.py:366] Task: mmlu_college_physics; number of requests on this rank: 408\n",
            "2024-05-12:14:16:13,950 DEBUG    [cache.py:33] requests-mmlu_high_school_computer_science-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:16:13,950 INFO     [task.py:398] Building contexts for mmlu_high_school_computer_science on rank 0...\n",
            "100%|██████████| 100/100 [00:01<00:00, 66.52it/s]\n",
            "2024-05-12:14:16:15,458 DEBUG    [evaluator.py:366] Task: mmlu_high_school_computer_science; number of requests on this rank: 400\n",
            "2024-05-12:14:16:15,458 DEBUG    [cache.py:33] requests-mmlu_high_school_mathematics-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:16:15,459 INFO     [task.py:398] Building contexts for mmlu_high_school_mathematics on rank 0...\n",
            "100%|██████████| 270/270 [00:04<00:00, 65.92it/s]\n",
            "2024-05-12:14:16:19,567 DEBUG    [evaluator.py:366] Task: mmlu_high_school_mathematics; number of requests on this rank: 1080\n",
            "2024-05-12:14:16:19,568 DEBUG    [cache.py:33] requests-mmlu_high_school_biology-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:16:19,568 INFO     [task.py:398] Building contexts for mmlu_high_school_biology on rank 0...\n",
            "100%|██████████| 310/310 [00:04<00:00, 66.68it/s]\n",
            "2024-05-12:14:16:24,231 DEBUG    [evaluator.py:366] Task: mmlu_high_school_biology; number of requests on this rank: 1240\n",
            "2024-05-12:14:16:24,232 DEBUG    [cache.py:33] requests-mmlu_college_biology-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:16:24,232 INFO     [task.py:398] Building contexts for mmlu_college_biology on rank 0...\n",
            "100%|██████████| 144/144 [00:02<00:00, 66.28it/s]\n",
            "2024-05-12:14:16:26,412 DEBUG    [evaluator.py:366] Task: mmlu_college_biology; number of requests on this rank: 576\n",
            "2024-05-12:14:16:26,412 DEBUG    [cache.py:33] requests-mmlu_high_school_physics-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:16:26,412 INFO     [task.py:398] Building contexts for mmlu_high_school_physics on rank 0...\n",
            "100%|██████████| 151/151 [00:02<00:00, 66.24it/s]\n",
            "2024-05-12:14:16:28,699 DEBUG    [evaluator.py:366] Task: mmlu_high_school_physics; number of requests on this rank: 604\n",
            "2024-05-12:14:16:28,699 DEBUG    [cache.py:33] requests-mmlu_college_computer_science-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:16:28,699 INFO     [task.py:398] Building contexts for mmlu_college_computer_science on rank 0...\n",
            "100%|██████████| 100/100 [00:01<00:00, 64.83it/s]\n",
            "2024-05-12:14:16:30,247 DEBUG    [evaluator.py:366] Task: mmlu_college_computer_science; number of requests on this rank: 400\n",
            "2024-05-12:14:16:30,247 DEBUG    [cache.py:33] requests-mmlu_astronomy-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:16:30,248 INFO     [task.py:398] Building contexts for mmlu_astronomy on rank 0...\n",
            "100%|██████████| 152/152 [00:02<00:00, 66.50it/s]\n",
            "2024-05-12:14:16:32,541 DEBUG    [evaluator.py:366] Task: mmlu_astronomy; number of requests on this rank: 608\n",
            "2024-05-12:14:16:32,541 DEBUG    [cache.py:33] requests-mmlu_elementary_mathematics-5shot-rank0-world_size1 is not cached, generating...\n",
            "2024-05-12:14:16:32,541 INFO     [task.py:398] Building contexts for mmlu_elementary_mathematics on rank 0...\n",
            "100%|██████████| 378/378 [00:05<00:00, 66.70it/s]\n",
            "2024-05-12:14:16:38,226 DEBUG    [evaluator.py:366] Task: mmlu_elementary_mathematics; number of requests on this rank: 1512\n",
            "2024-05-12:14:16:38,226 INFO     [evaluator.py:395] Running loglikelihood requests\n",
            "Running loglikelihood requests: 100%|██████████| 58702/58702 [2:56:45<00:00,  5.53it/s]\n",
            "fatal: not a git repository (or any of the parent directories): .git\n",
            "2024-05-12:17:16:54,624 INFO     [evaluation_tracker.py:132] Saving results aggregated\n",
            "hf (pretrained=apple/OpenELM-3B,trust_remote_code=True,add_bos_token=True,tokenizer=meta-llama/Llama-2-7b-hf), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: 1\n",
            "|                 Tasks                 |Version|Filter|n-shot|Metric|Value |   |Stderr|\n",
            "|---------------------------------------|-------|------|-----:|------|-----:|---|-----:|\n",
            "|mmlu                                   |N/A    |none  |     0|acc   |0.2676|±  |0.0037|\n",
            "|  - abstract_algebra                   |      0|none  |     5|acc   |0.2600|±  |0.0441|\n",
            "|  - anatomy                            |      0|none  |     5|acc   |0.2444|±  |0.0371|\n",
            "|  - astronomy                          |      0|none  |     5|acc   |0.2303|±  |0.0343|\n",
            "|  - business_ethics                    |      0|none  |     5|acc   |0.2900|±  |0.0456|\n",
            "|  - clinical_knowledge                 |      0|none  |     5|acc   |0.2566|±  |0.0269|\n",
            "|  - college_biology                    |      0|none  |     5|acc   |0.3125|±  |0.0388|\n",
            "|  - college_chemistry                  |      0|none  |     5|acc   |0.2300|±  |0.0423|\n",
            "|  - college_computer_science           |      0|none  |     5|acc   |0.3500|±  |0.0479|\n",
            "|  - college_mathematics                |      0|none  |     5|acc   |0.2800|±  |0.0451|\n",
            "|  - college_medicine                   |      0|none  |     5|acc   |0.2717|±  |0.0339|\n",
            "|  - college_physics                    |      0|none  |     5|acc   |0.2059|±  |0.0402|\n",
            "|  - computer_security                  |      0|none  |     5|acc   |0.2600|±  |0.0441|\n",
            "|  - conceptual_physics                 |      0|none  |     5|acc   |0.3447|±  |0.0311|\n",
            "|  - econometrics                       |      0|none  |     5|acc   |0.2632|±  |0.0414|\n",
            "|  - electrical_engineering             |      0|none  |     5|acc   |0.2828|±  |0.0375|\n",
            "|  - elementary_mathematics             |      0|none  |     5|acc   |0.2063|±  |0.0208|\n",
            "|  - formal_logic                       |      0|none  |     5|acc   |0.2302|±  |0.0376|\n",
            "|  - global_facts                       |      0|none  |     5|acc   |0.3300|±  |0.0473|\n",
            "|  - high_school_biology                |      0|none  |     5|acc   |0.2935|±  |0.0259|\n",
            "|  - high_school_chemistry              |      0|none  |     5|acc   |0.2365|±  |0.0299|\n",
            "|  - high_school_computer_science       |      0|none  |     5|acc   |0.3000|±  |0.0461|\n",
            "|  - high_school_european_history       |      0|none  |     5|acc   |0.2121|±  |0.0319|\n",
            "|  - high_school_geography              |      0|none  |     5|acc   |0.2475|±  |0.0307|\n",
            "|  - high_school_government_and_politics|      0|none  |     5|acc   |0.2694|±  |0.0320|\n",
            "|  - high_school_macroeconomics         |      0|none  |     5|acc   |0.2923|±  |0.0231|\n",
            "|  - high_school_mathematics            |      0|none  |     5|acc   |0.2630|±  |0.0268|\n",
            "|  - high_school_microeconomics         |      0|none  |     5|acc   |0.2815|±  |0.0292|\n",
            "|  - high_school_physics                |      0|none  |     5|acc   |0.3113|±  |0.0378|\n",
            "|  - high_school_psychology             |      0|none  |     5|acc   |0.2422|±  |0.0184|\n",
            "|  - high_school_statistics             |      0|none  |     5|acc   |0.2824|±  |0.0307|\n",
            "|  - high_school_us_history             |      0|none  |     5|acc   |0.2647|±  |0.0310|\n",
            "|  - high_school_world_history          |      0|none  |     5|acc   |0.2869|±  |0.0294|\n",
            "|  - human_aging                        |      0|none  |     5|acc   |0.3229|±  |0.0314|\n",
            "|  - human_sexuality                    |      0|none  |     5|acc   |0.2519|±  |0.0381|\n",
            "| - humanities                          |N/A    |none  |     5|acc   |0.2491|±  |0.0063|\n",
            "|  - international_law                  |      0|none  |     5|acc   |0.2314|±  |0.0385|\n",
            "|  - jurisprudence                      |      0|none  |     5|acc   |0.2315|±  |0.0408|\n",
            "|  - logical_fallacies                  |      0|none  |     5|acc   |0.2331|±  |0.0332|\n",
            "|  - machine_learning                   |      0|none  |     5|acc   |0.3125|±  |0.0440|\n",
            "|  - management                         |      0|none  |     5|acc   |0.2039|±  |0.0399|\n",
            "|  - marketing                          |      0|none  |     5|acc   |0.2991|±  |0.0300|\n",
            "|  - medical_genetics                   |      0|none  |     5|acc   |0.2800|±  |0.0451|\n",
            "|  - miscellaneous                      |      0|none  |     5|acc   |0.3001|±  |0.0164|\n",
            "|  - moral_disputes                     |      0|none  |     5|acc   |0.2486|±  |0.0233|\n",
            "|  - moral_scenarios                    |      0|none  |     5|acc   |0.2425|±  |0.0143|\n",
            "|  - nutrition                          |      0|none  |     5|acc   |0.2647|±  |0.0253|\n",
            "| - other                               |N/A    |none  |     5|acc   |0.2984|±  |0.0082|\n",
            "|  - philosophy                         |      0|none  |     5|acc   |0.2412|±  |0.0243|\n",
            "|  - prehistory                         |      0|none  |     5|acc   |0.2778|±  |0.0249|\n",
            "|  - professional_accounting            |      0|none  |     5|acc   |0.2376|±  |0.0254|\n",
            "|  - professional_law                   |      0|none  |     5|acc   |0.2419|±  |0.0109|\n",
            "|  - professional_medicine              |      0|none  |     5|acc   |0.4559|±  |0.0303|\n",
            "|  - professional_psychology            |      0|none  |     5|acc   |0.2402|±  |0.0173|\n",
            "|  - public_relations                   |      0|none  |     5|acc   |0.3091|±  |0.0443|\n",
            "|  - security_studies                   |      0|none  |     5|acc   |0.2653|±  |0.0283|\n",
            "| - social_sciences                     |N/A    |none  |     5|acc   |0.2610|±  |0.0079|\n",
            "|  - sociology                          |      0|none  |     5|acc   |0.2438|±  |0.0304|\n",
            "| - stem                                |N/A    |none  |     5|acc   |0.2712|±  |0.0079|\n",
            "|  - us_foreign_policy                  |      0|none  |     5|acc   |0.3100|±  |0.0465|\n",
            "|  - virology                           |      0|none  |     5|acc   |0.3133|±  |0.0361|\n",
            "|  - world_religions                    |      0|none  |     5|acc   |0.3275|±  |0.0360|\n",
            "|winogrande                             |      1|none  |     5|acc   |0.6725|±  |0.0132|\n",
            "\n",
            "|      Groups      |Version|Filter|n-shot|Metric|Value |   |Stderr|\n",
            "|------------------|-------|------|-----:|------|-----:|---|-----:|\n",
            "|mmlu              |N/A    |none  |     0|acc   |0.2676|±  |0.0037|\n",
            "| - humanities     |N/A    |none  |     5|acc   |0.2491|±  |0.0063|\n",
            "| - other          |N/A    |none  |     5|acc   |0.2984|±  |0.0082|\n",
            "| - social_sciences|N/A    |none  |     5|acc   |0.2610|±  |0.0079|\n",
            "| - stem           |N/A    |none  |     5|acc   |0.2712|±  |0.0079|\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install accelerate -q"
      ],
      "metadata": {
        "id": "zzSyfa8ApOfe"
      },
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "https://huggingface.co/docs/transformers/generation_strategies#default-text-generation-configuration"
      ],
      "metadata": {
        "id": "ecUMFrn0PJ9y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
        "\n",
        "#prompt = \"I look forward to\"\n",
        "prompt = \"What was the first album Beyoncé released as a solo artist?\"\n",
        "\n",
        "model_id = \"apple/OpenELM-3B\"\n",
        "model_id_Tokenizer = 'meta-llama/Llama-2-7b-hf'\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_id_Tokenizer)\n",
        "inputs = tokenizer(prompt, return_tensors=\"pt\")\n",
        "\n",
        "model = AutoModelForCausalLM.from_pretrained(model_id,trust_remote_code=True)\n",
        "outputs = model.generate(**inputs, max_new_tokens=30,  temperature=0.8, do_sample=True,  eos_token_id=pipe.tokenizer.eos_token_id, pad_token_id=pipe.tokenizer.eos_token_id)"
      ],
      "metadata": {
        "id": "cym1f00XNNZ6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.batch_decode(outputs, skip_special_tokens=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sSx36pHJN4T0",
        "outputId": "747a5f03-a9ac-45a7-d161-985a39951f1d"
      },
      "execution_count": 179,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['What was the first album Beyoncé released as a solo artist?\\n\\nI just found out she released that her first solo album \"Dangerously in Love\" in 2003. \\n\\n']"
            ]
          },
          "metadata": {},
          "execution_count": 179
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "prompt = \"What is the capital of CANADA?\"\n",
        "inputs = tokenizer(prompt, return_tensors=\"pt\")\n",
        "outputs = model.generate(**inputs, max_new_tokens=30, temperature=0.8, do_sample=True,  eos_token_id=pipe.tokenizer.eos_token_id, pad_token_id=pipe.tokenizer.eos_token_id)"
      ],
      "metadata": {
        "id": "jyLMYpgXTOrY"
      },
      "execution_count": 224,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.batch_decode(outputs, skip_special_tokens=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "s4VXCNxsTnD2",
        "outputId": "2af6ee60-b18e-42f0-923c-37769be4d886"
      },
      "execution_count": 225,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['What is the capital of CANADA?\\nWhat is the capital of CANADA?\\nOttawa, the capital of Canada, was founded in 1826.']"
            ]
          },
          "metadata": {},
          "execution_count": 225
        }
      ]
    }
  ]
}