PyTorch/llama2_fine_tuning_inference/llama2_fine_tuning_inference.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9599e2f3-6b9c-4578-9501-7d5c65df408a",
   "metadata": {},
   "source": [
    "Copyright (c) 2023 Habana Labs, Ltd. an Intel Company.\n",
    "\n",
    "#### Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a8d33aa-8f5b-4de3-bdaa-b2bfac88e1e8",
   "metadata": {},
   "source": [
    "# Using Paramater Efficient Fine Tuning on Llama 2 with 70B Parameters\n",
    "This example will Fine Tune the Llama2-70B model using Parameter Efficient Fine Tuining (PEFT) and then run inference on a text prompt.  This will be using the Llama2 model with two task examples from the Optimum Habana library on the Hugging Face model repository.   The Optimum Habana library is optimized for Deep Learning training and inference on First-gen Gaudi and Gaudi2 and offers tasks such as text generation, language modeling, question answering and more. For all the examples and models, please refer to the [Optimum Habana GitHub](https://github.com/huggingface/optimum-habana#validated-models).\n",
    "\n",
    "This example will Fine Tune the Llama2-70B model using Parameter Efficient Fine Tuining (PEFT) on the timdettmers/openassistant-guanaco dataset using the Language-Modeling Task in Optimum Habana."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f337ed16-dcfa-424f-a46d-933880d12d04",
   "metadata": {},
   "source": [
    "### Parameter Efficient Fine Tuning with Low Rank Adaptation\n",
    "Parameter Efficient Fine Tuning is a strategy for adapting large pre-trained language models to specific tasks while minimizing computational and memory demands.   It aims to reduce the computational cost and memory requirements associated with fine-tuning large models while maintaining or even improving their performance.  It does so by adding a smaller task-specific layer, leveraging knowledge distillation, and often relying on few-shot learning, resulting in efficient yet effective models for various natural language understanding tasks.   PEFT starts with a pre-trained language model that has already learned a wide range of language understanding tasks from a large corpus of text data. These models are usually large and computationally expensive.   Instead of fine-tuning the entire pre-trained model, PEFT adds a task-specific layer or a few task-specific layers on top of the pre-trained model. These additional layers are relatively smaller and have fewer parameters compared to the base model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "404692f3-1266-4dcb-8885-ec6016aa5bef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/root/Gaudi-tutorials/PyTorch/llama2_fine_tuning_inference\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/IPython/core/magics/osm.py:417: UserWarning: using dhist requires you to install the `pickleshare` library.\n",
      "  self.shell.db['dhist'] = compress_dhist(dhist)[-100:]\n"
     ]
    }
   ],
   "source": [
    "%cd ~/Gaudi-tutorials/PyTorch/llama2_fine_tuning_inference"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29ad8bfa-90ae-4e32-963f-821b92ddab0e",
   "metadata": {},
   "source": [
    "### Model Setup: "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4db7e60-ff31-4a76-9dd5-cb4eb76412b9",
   "metadata": {},
   "source": [
    "##### Install the latest version of the Habana Deepspeed Library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "aaab9f1b-f081-4156-b86f-b1e9b23f9a4f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip install -q git+https://github.com/HabanaAI/DeepSpeed.git@1.14.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9240274a-dd9a-4ca9-a73d-b4ff956c343d",
   "metadata": {},
   "source": [
    "##### Install the Parameter Efficient Fine Tuning Library methods\n",
    "This is taking the PEFT method from the Hugging Face repository and will be used to help create the PEFT Fine Tuning with the Llama2 model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7d21efb3-978e-4585-915a-4c8a9ba9b064",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'peft'...\n",
      "remote: Enumerating objects: 5894, done.\u001b[K\n",
      "remote: Counting objects: 100% (1638/1638), done.\u001b[K\n",
      "remote: Compressing objects: 100% (395/395), done.\u001b[K\n",
      "remote: Total 5894 (delta 1411), reused 1310 (delta 1225), pack-reused 4256\u001b[K\n",
      "Receiving objects: 100% (5894/5894), 8.39 MiB | 14.96 MiB/s, done.\n",
      "Resolving deltas: 100% (3833/3833), done.\n",
      "/root/Gaudi-tutorials/PyTorch/llama2_fine_tuning_inference/peft\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m/root/Gaudi-tutorials/PyTorch/llama2_fine_tuning_inference\n"
     ]
    }
   ],
   "source": [
    "!pip install peft==0.8.2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d91f0f3e-8bf8-4c1c-adbb-3926b292e5ed",
   "metadata": {},
   "source": [
    "##### Install the Optimum-Habana Library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c8408759-937d-472a-bd00-e67142a90fbb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip install -q optimum-habana==1.10.4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89b06c57-0d8c-4d40-85ae-1eea945a3ef9",
   "metadata": {},
   "source": [
    "##### Pull the Hugging Face Examples from GitHub\n",
    "These contain the working Hugging Face Task Examples that have been optimized for Gaudi.  For Fine Tuning, we'll use the language-modeling task. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "50af3c79-6641-47d7-a440-09eba2bd5765",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'optimum-habana'...\n",
      "remote: Enumerating objects: 8311, done.\u001b[K\n",
      "remote: Counting objects: 100% (3188/3188), done.\u001b[K\n",
      "remote: Compressing objects: 100% (812/812), done.\u001b[K\n",
      "remote: Total 8311 (delta 2597), reused 2673 (delta 2308), pack-reused 5123\u001b[K\n",
      "Receiving objects: 100% (8311/8311), 3.83 MiB | 9.24 MiB/s, done.\n",
      "Resolving deltas: 100% (5521/5521), done.\n"
     ]
    }
   ],
   "source": [
    "%cd ~/Gaudi-tutorials/PyTorch/llama2_fine_tuning_inference\n",
    "!git clone https://github.com/huggingface/optimum-habana.git\n",
    "%cd optimum-habana\n",
    "!git checkout v1.10.4 \n",
    "%cd .."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11e369a6-ce17-40a2-8a52-d735b7140e09",
   "metadata": {},
   "source": [
    "##### Go to the Language Modeling Task and install the model specific requirements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "441de6d9-2b6f-4cf2-8bfd-664cec1c4998",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/root/Gaudi-tutorials/PyTorch/llama2_fine_tuning_inference/optimum-habana/examples/language-modeling\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "%cd optimum-habana/examples/language-modeling/\n",
    "!pip install -q -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88ed7d45",
   "metadata": {},
   "source": [
    "##### How to access and Use the Llama 2 model\n",
    "\n",
    "Use of the pretrained model is subject to compliance with third party licenses, including the “Llama 2 Community License Agreement” (LLAMAV2). For guidance on the intended use of the LLAMA2 model, what will be considered misuse and out-of-scope uses, who are the intended users and additional terms please review and read the instructions in this link https://ai.meta.com/llama/license/.\n",
    "Users bear sole liability and responsibility to follow and comply with any third party licenses, and Habana Labs disclaims and will bear no liability with respect to users’ use or compliance with third party licenses.\n",
    "\n",
    "To be able to run gated models like this Llama-2-70b-hf, you need the following: \n",
    "- Have a HuggingFace account\n",
    "- Agree to the terms of use of the model in its model card on the HF Hub\n",
    "- set a read token\n",
    "- Login to your account using the HF CLI: run huggingface-cli login before launching your script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2aa53882-c834-4ff3-8fc4-742579ee8cea",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token will not been saved to git credential helper. Pass `add_to_git_credential=True` if you want to set the git credential as well.\n",
      "Token is valid (permission: read).\n",
      "Your token has been saved to /root/.cache/huggingface/token\n",
      "Login successful\n"
     ]
    }
   ],
   "source": [
    "!huggingface-cli login --token <your_token_here>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2d4c81e-c48d-45e0-93de-579e53995602",
   "metadata": {},
   "source": [
    "## Fine Tuning the model with PEFT and LoRA\n",
    "\n",
    "We'll now run the fine tuning with the PEFT method. Remember that the PEFT methods only fine-tune a small number of extra model parameters, thereby greatly decreasing the computational and storage costs. Recent State-of-the-Art PEFT techniques achieve performance comparable to that of full fine-tuning.\n",
    "\n",
    "##### Here's a summary of the command required to run the Fine Tuning, you'll run this in the next cell below. \n",
    "Note in this case the following: \n",
    "1. Using the language modeling with LoRA; `run_lora_clm.py`\n",
    "2. It's very efficient: only 0.02% of the total paramters are being fine tuned of the total 70B parameters.\n",
    "3. Using DeepSpeed has reduced the max amount of memory to ~73.9 GB out of a total memory available 94.6 GB\n",
    "4. Only 2 epochs are needed for fine tuning, it takes less than 38 minutes to run with the openassisant-guanaco dataset.  (Running the Llama 2 7B or 13B models will be faster, see the addendum below)\n",
    "\n",
    "NOTE: In some cases running the full Llama2 70B Fine Tuning in the Juypter Notebook may cause errors.   It's safer to run this in a standard terminal window.   Simply copy the command below and run this in a Terminal window in the same folder location:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ed5d6093-978c-4c4f-b2ab-c3f571e013d8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[WARNING|utils.py:198] 2023-12-21 01:48:07,136 >> optimum-habana v1.9.0 has been validated for SynapseAI v1.13.0 but the driver version is v1.12.0, this could lead to undefined behavior!\n",
      "DistributedRunner run(): command = deepspeed --num_nodes 1 --num_gpus 8 --no_local_rank run_lora_clm.py --model_name_or_path meta-llama/Llama-2-70b-hf --deepspeed llama2_ds_zero3_config.json --dataset_name timdettmers/openassistant-guanaco --bf16 True --output_dir ./model_lora_llama --num_train_epochs 2 --max_seq_len 2048 --per_device_train_batch_size 10 --per_device_eval_batch_size 10 --gradient_checkpointing --evaluation_strategy epoch --eval_delay 2 --save_strategy no --learning_rate 0.0018 --warmup_ratio 0.03 --lr_scheduler_type cosine --logging_steps 1 --dataset_concatenation --attn_softmax_bf16 True --do_train --do_eval --use_habana --use_lazy_mode --report_to none --pipelining_fwd_bwd --throughput_warmup_steps 3 --lora_rank 4 --lora_target_modules q_proj v_proj k_proj o_proj --validation_split_percentage 4\n",
      "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:252: UserWarning: Device capability of hccl unspecified, assuming `cpu` and `cuda`. Please specify it via the `devices` argument of `register_backend`.\n",
      "  warnings.warn(\n",
      "[2023-12-21 01:48:09,916] [INFO] [real_accelerator.py:175:get_accelerator] Setting ds_accelerator to hpu (auto detect)\n",
      "[2023-12-21 01:48:11,291] [WARNING] [runner.py:206:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.\n",
      "[2023-12-21 01:48:11,357] [INFO] [runner.py:585:main] cmd = /usr/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgMywgNCwgNSwgNiwgN119 --master_addr=127.0.0.1 --master_port=29500 --no_local_rank --enable_each_rank_log=None run_lora_clm.py --model_name_or_path meta-llama/Llama-2-70b-hf --deepspeed llama2_ds_zero3_config.json --dataset_name timdettmers/openassistant-guanaco --bf16 True --output_dir ./model_lora_llama --num_train_epochs 2 --max_seq_len 2048 --per_device_train_batch_size 10 --per_device_eval_batch_size 10 --gradient_checkpointing --evaluation_strategy epoch --eval_delay 2 --save_strategy no --learning_rate 0.0018 --warmup_ratio 0.03 --lr_scheduler_type cosine --logging_steps 1 --dataset_concatenation --attn_softmax_bf16 True --do_train --do_eval --use_habana --use_lazy_mode --report_to none --pipelining_fwd_bwd --throughput_warmup_steps 3 --lora_rank 4 --lora_target_modules q_proj v_proj k_proj o_proj --validation_split_percentage 4\n",
      "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:252: UserWarning: Device capability of hccl unspecified, assuming `cpu` and `cuda`. Please specify it via the `devices` argument of `register_backend`.\n",
      "  warnings.warn(\n",
      "[2023-12-21 01:48:14,189] [INFO] [real_accelerator.py:175:get_accelerator] Setting ds_accelerator to hpu (auto detect)\n",
      "[2023-12-21 01:48:15,548] [INFO] [launch.py:146:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3, 4, 5, 6, 7]}\n",
      "[2023-12-21 01:48:15,549] [INFO] [launch.py:152:main] nnodes=1, num_local_procs=8, node_rank=0\n",
      "[2023-12-21 01:48:15,549] [INFO] [launch.py:163:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3, 4, 5, 6, 7]})\n",
      "[2023-12-21 01:48:15,549] [INFO] [launch.py:164:main] dist_world_size=8\n",
      "[2023-12-21 01:48:15,549] [INFO] [launch.py:166:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7\n",
      "[WARNING|utils.py:198] 2023-12-21 01:48:20,790 >> optimum-habana v1.9.0 has been validated for SynapseAI v1.13.0 but the driver version is v1.12.0, this could lead to undefined behavior!\n",
      "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:252: UserWarning: Device capability of hccl unspecified, assuming `cpu` and `cuda`. Please specify it via the `devices` argument of `register_backend`.\n",
      "  warnings.warn(\n",
      "[WARNING|utils.py:198] 2023-12-21 01:48:21,587 >> optimum-habana v1.9.0 has been validated for SynapseAI v1.13.0 but the driver version is v1.12.0, this could lead to undefined behavior!\n",
      "[WARNING|utils.py:198] 2023-12-21 01:48:21,668 >> optimum-habana v1.9.0 has been validated for SynapseAI v1.13.0 but the driver version is v1.12.0, this could lead to undefined behavior!\n",
      "[WARNING|utils.py:198] 2023-12-21 01:48:21,697 >> optimum-habana v1.9.0 has been validated for SynapseAI v1.13.0 but the driver version is v1.12.0, this could lead to undefined behavior!\n",
      "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:252: UserWarning: Device capability of hccl unspecified, assuming `cpu` and `cuda`. Please specify it via the `devices` argument of `register_backend`.\n",
      "  warnings.warn(\n",
      "[WARNING|utils.py:198] 2023-12-21 01:48:21,865 >> optimum-habana v1.9.0 has been validated for SynapseAI v1.13.0 but the driver version is v1.12.0, this could lead to undefined behavior!\n",
      "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:252: UserWarning: Device capability of hccl unspecified, assuming `cpu` and `cuda`. Please specify it via the `devices` argument of `register_backend`.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:252: UserWarning: Device capability of hccl unspecified, assuming `cpu` and `cuda`. Please specify it via the `devices` argument of `register_backend`.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:252: UserWarning: Device capability of hccl unspecified, assuming `cpu` and `cuda`. Please specify it via the `devices` argument of `register_backend`.\n",
      "  warnings.warn(\n",
      "[WARNING|utils.py:198] 2023-12-21 01:48:22,530 >> optimum-habana v1.9.0 has been validated for SynapseAI v1.13.0 but the driver version is v1.12.0, this could lead to undefined behavior!\n",
      "[2023-12-21 01:48:22,574] [INFO] [real_accelerator.py:175:get_accelerator] Setting ds_accelerator to hpu (auto detect)\n",
      "[WARNING|utils.py:198] 2023-12-21 01:48:22,607 >> optimum-habana v1.9.0 has been validated for SynapseAI v1.13.0 but the driver version is v1.12.0, this could lead to undefined behavior!\n",
      "[WARNING|utils.py:198] 2023-12-21 01:48:22,619 >> optimum-habana v1.9.0 has been validated for SynapseAI v1.13.0 but the driver version is v1.12.0, this could lead to undefined behavior!\n",
      "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:252: UserWarning: Device capability of hccl unspecified, assuming `cpu` and `cuda`. Please specify it via the `devices` argument of `register_backend`.\n",
      "  warnings.warn(\n",
      "[2023-12-21 01:48:22,771] [INFO] [real_accelerator.py:175:get_accelerator] Setting ds_accelerator to hpu (auto detect)\n",
      "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:252: UserWarning: Device capability of hccl unspecified, assuming `cpu` and `cuda`. Please specify it via the `devices` argument of `register_backend`.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:252: UserWarning: Device capability of hccl unspecified, assuming `cpu` and `cuda`. Please specify it via the `devices` argument of `register_backend`.\n",
      "  warnings.warn(\n",
      "[2023-12-21 01:48:23,269] [INFO] [real_accelerator.py:175:get_accelerator] Setting ds_accelerator to hpu (auto detect)\n",
      "[2023-12-21 01:48:23,317] [INFO] [real_accelerator.py:175:get_accelerator] Setting ds_accelerator to hpu (auto detect)\n",
      "[2023-12-21 01:48:23,487] [INFO] [real_accelerator.py:175:get_accelerator] Setting ds_accelerator to hpu (auto detect)\n",
      "[2023-12-21 01:48:23,914] [WARNING] [comm.py:163:init_deepspeed_backend] HCCL backend in DeepSpeed not yet implemented\n",
      "[2023-12-21 01:48:23,914] [INFO] [comm.py:637:init_distributed] cdb=None\n",
      "[2023-12-21 01:48:23,914] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend hccl\n",
      "[2023-12-21 01:48:24,182] [WARNING] [comm.py:163:init_deepspeed_backend] HCCL backend in DeepSpeed not yet implemented\n",
      "[2023-12-21 01:48:24,182] [INFO] [comm.py:637:init_distributed] cdb=None\n",
      "[2023-12-21 01:48:24,239] [INFO] [real_accelerator.py:175:get_accelerator] Setting ds_accelerator to hpu (auto detect)\n",
      "12/21/2023 01:48:24 - WARNING - __main__ -   Process rank: 3, device: hpu:3, distributed training: True, 16-bits training: True\n",
      "[2023-12-21 01:48:24,315] [INFO] [real_accelerator.py:175:get_accelerator] Setting ds_accelerator to hpu (auto detect)\n",
      "[2023-12-21 01:48:24,333] [INFO] [real_accelerator.py:175:get_accelerator] Setting ds_accelerator to hpu (auto detect)\n",
      "Downloading config.json: 100%|█████████████████| 609/609 [00:00<00:00, 1.48MB/s]\n",
      "Downloading tokenizer_config.json: 100%|███████| 776/776 [00:00<00:00, 2.24MB/s]\n",
      "[2023-12-21 01:48:25,012] [WARNING] [comm.py:163:init_deepspeed_backend] HCCL backend in DeepSpeed not yet implemented\n",
      "[2023-12-21 01:48:25,012] [INFO] [comm.py:637:init_distributed] cdb=None\n",
      "[2023-12-21 01:48:25,041] [WARNING] [comm.py:163:init_deepspeed_backend] HCCL backend in DeepSpeed not yet implemented\n",
      "[2023-12-21 01:48:25,041] [INFO] [comm.py:637:init_distributed] cdb=None\n",
      "12/21/2023 01:48:25 - WARNING - __main__ -   Process rank: 2, device: hpu:2, distributed training: True, 16-bits training: True\n",
      "Downloading tokenizer.model:   0%|                   | 0.00/500k [00:00<?, ?B/s]12/21/2023 01:48:25 - WARNING - __main__ -   Process rank: 7, device: hpu:7, distributed training: True, 16-bits training: True\n",
      "Downloading tokenizer.model: 100%|███████████| 500k/500k [00:00<00:00, 23.7MB/s]\n",
      "[2023-12-21 01:48:25,228] [WARNING] [comm.py:163:init_deepspeed_backend] HCCL backend in DeepSpeed not yet implemented\n",
      "[2023-12-21 01:48:25,228] [INFO] [comm.py:637:init_distributed] cdb=None\n",
      "12/21/2023 01:48:25 - WARNING - __main__ -   Process rank: 6, device: hpu:6, distributed training: True, 16-bits training: True\n",
      "Downloading tokenizer.json: 100%|██████████| 1.84M/1.84M [00:00<00:00, 7.04MB/s]\n",
      "Downloading (…)cial_tokens_map.json: 100%|██████| 414/414 [00:00<00:00, 930kB/s]\n",
      "/usr/local/lib/python3.10/dist-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0.\n",
      "You can remove this warning by passing 'token=None' instead.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.10/dist-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0.\n",
      "You can remove this warning by passing 'token=None' instead.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.10/dist-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0.\n",
      "You can remove this warning by passing 'token=None' instead.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.10/dist-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0.\n",
      "You can remove this warning by passing 'token=None' instead.\n",
      "  warnings.warn(\n",
      "[2023-12-21 01:48:26,031] [WARNING] [comm.py:163:init_deepspeed_backend] HCCL backend in DeepSpeed not yet implemented\n",
      "[2023-12-21 01:48:26,031] [INFO] [comm.py:637:init_distributed] cdb=None\n",
      "12/21/2023 01:48:26 - WARNING - __main__ -   Process rank: 1, device: hpu:1, distributed training: True, 16-bits training: True\n",
      "[2023-12-21 01:48:26,402] [WARNING] [comm.py:163:init_deepspeed_backend] HCCL backend in DeepSpeed not yet implemented\n",
      "[2023-12-21 01:48:26,402] [INFO] [comm.py:637:init_distributed] cdb=None\n",
      "[2023-12-21 01:48:26,412] [WARNING] [comm.py:163:init_deepspeed_backend] HCCL backend in DeepSpeed not yet implemented\n",
      "[2023-12-21 01:48:26,412] [INFO] [comm.py:637:init_distributed] cdb=None\n",
      "12/21/2023 01:48:26 - WARNING - __main__ -   Process rank: 4, device: hpu:4, distributed training: True, 16-bits training: True\n",
      "/usr/local/lib/python3.10/dist-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0.\n",
      "You can remove this warning by passing 'token=None' instead.\n",
      "  warnings.warn(\n",
      "12/21/2023 01:48:26 - WARNING - __main__ -   Process rank: 0, device: hpu:0, distributed training: True, 16-bits training: True\n",
      "12/21/2023 01:48:26 - INFO - __main__ -   Training/evaluation parameters GaudiTrainingArguments(\n",
      "_n_gpu=1,\n",
      "adafactor=False,\n",
      "adam_beta1=0.9,\n",
      "adam_beta2=0.999,\n",
      "adam_epsilon=1e-06,\n",
      "adjust_throughput=False,\n",
      "auto_find_batch_size=False,\n",
      "bf16=True,\n",
      "data_seed=None,\n",
      "dataloader_drop_last=False,\n",
      "dataloader_num_workers=0,\n",
      "dataloader_pin_memory=True,\n",
      "ddp_backend=hccl,\n",
      "ddp_broadcast_buffers=None,\n",
      "ddp_bucket_cap_mb=230,\n",
      "ddp_find_unused_parameters=False,\n",
      "ddp_timeout=1800,\n",
      "debug=[],\n",
      "deepspeed=llama2_ds_zero3_config.json,\n",
      "disable_tensor_cache_hpu_graphs=False,\n",
      "disable_tqdm=False,\n",
      "dispatch_batches=None,\n",
      "distribution_strategy=ddp,\n",
      "do_eval=True,\n",
      "do_predict=False,\n",
      "do_train=True,\n",
      "eval_accumulation_steps=None,\n",
      "eval_delay=2.0,\n",
      "eval_steps=None,\n",
      "evaluation_strategy=epoch,\n",
      "fsdp_config=None,\n",
      "fsdp_min_num_params=0,\n",
      "fsdp_transformer_layer_cls_to_wrap=None,\n",
      "full_determinism=False,\n",
      "gaudi_config_name=None,\n",
      "gradient_accumulation_steps=1,\n",
      "gradient_checkpointing=True,\n",
      "greater_is_better=None,\n",
      "group_by_length=False,\n",
      "half_precision_backend=hpu_amp,\n",
      "hub_always_push=False,\n",
      "hub_model_id=None,\n",
      "hub_private_repo=False,\n",
      "hub_strategy=every_save,\n",
      "hub_token=<HUB_TOKEN>,\n",
      "ignore_data_skip=False,\n",
      "ignore_eos=True,\n",
      "include_inputs_for_metrics=False,\n",
      "include_tokens_per_second=False,\n",
      "jit_mode_eval=False,\n",
      "label_names=None,\n",
      "label_smoothing_factor=0.0,\n",
      "learning_rate=0.0018,\n",
      "length_column_name=length,\n",
      "load_best_model_at_end=False,\n",
      "local_rank=0,\n",
      "log_level=passive,\n",
      "log_level_replica=warning,\n",
      "log_on_each_node=True,\n",
      "logging_dir=./model_lora_llama/runs/Dec21_01-48-21_hls2-srv01-demolab,\n",
      "logging_first_step=False,\n",
      "logging_nan_inf_filter=False,\n",
      "logging_steps=1.0,\n",
      "logging_strategy=steps,\n",
      "lr_scheduler_type=cosine,\n",
      "max_grad_norm=1.0,\n",
      "max_hpu_graphs=None,\n",
      "max_steps=-1,\n",
      "metric_for_best_model=None,\n",
      "no_cuda=False,\n",
      "non_blocking_data_copy=False,\n",
      "num_train_epochs=2.0,\n",
      "optim=adamw_torch,\n",
      "optim_args=None,\n",
      "output_dir=./model_lora_llama,\n",
      "overwrite_output_dir=False,\n",
      "past_index=-1,\n",
      "per_device_eval_batch_size=10,\n",
      "per_device_train_batch_size=10,\n",
      "pipelining_fwd_bwd=True,\n",
      "prediction_loss_only=False,\n",
      "profiling_record_shapes=True,\n",
      "profiling_steps=0,\n",
      "profiling_warmup_steps=0,\n",
      "push_to_hub=False,\n",
      "push_to_hub_model_id=None,\n",
      "push_to_hub_organization=None,\n",
      "push_to_hub_token=<PUSH_TO_HUB_TOKEN>,\n",
      "ray_scope=last,\n",
      "remove_unused_columns=True,\n",
      "report_to=[],\n",
      "resume_from_checkpoint=None,\n",
      "run_name=./model_lora_llama,\n",
      "save_on_each_node=False,\n",
      "save_safetensors=False,\n",
      "save_steps=500,\n",
      "save_strategy=no,\n",
      "save_total_limit=None,\n",
      "seed=42,\n",
      "sharded_ddp=,\n",
      "skip_memory_metrics=True,\n",
      "throughput_warmup_steps=3,\n",
      "torch_compile=False,\n",
      "torch_compile_backend=None,\n",
      "torch_compile_mode=None,\n",
      "torchdynamo=None,\n",
      "use_cpu=False,\n",
      "use_habana=True,\n",
      "use_hpu_graphs=False,\n",
      "use_hpu_graphs_for_inference=False,\n",
      "use_hpu_graphs_for_training=False,\n",
      "use_ipex=False,\n",
      "use_lazy_mode=True,\n",
      "use_legacy_prediction_loop=False,\n",
      "use_mps_device=False,\n",
      "warmup_ratio=0.03,\n",
      "warmup_steps=0,\n",
      "weight_decay=0.0,\n",
      ")\n",
      "12/21/2023 01:48:26 - WARNING - __main__ -   Process rank: 5, device: hpu:5, distributed training: True, 16-bits training: True\n",
      "[INFO|configuration_utils.py:715] 2023-12-21 01:48:26,692 >> loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--meta-llama--Llama-2-70b-hf/snapshots/90052941a64de02075ca800b09fcea1bdaacb939/config.json\n",
      "[INFO|configuration_utils.py:775] 2023-12-21 01:48:26,696 >> Model config LlamaConfig {\n",
      "  \"_name_or_path\": \"meta-llama/Llama-2-70b-hf\",\n",
      "  \"architectures\": [\n",
      "    \"LlamaForCausalLM\"\n",
      "  ],\n",
      "  \"attention_bias\": false,\n",
      "  \"bos_token_id\": 1,\n",
      "  \"eos_token_id\": 2,\n",
      "  \"hidden_act\": \"silu\",\n",
      "  \"hidden_size\": 8192,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 28672,\n",
      "  \"max_position_embeddings\": 4096,\n",
      "  \"model_type\": \"llama\",\n",
      "  \"num_attention_heads\": 64,\n",
      "  \"num_hidden_layers\": 80,\n",
      "  \"num_key_value_heads\": 8,\n",
      "  \"pretraining_tp\": 1,\n",
      "  \"rms_norm_eps\": 1e-05,\n",
      "  \"rope_scaling\": null,\n",
      "  \"rope_theta\": 10000.0,\n",
      "  \"tie_word_embeddings\": false,\n",
      "  \"torch_dtype\": \"float16\",\n",
      "  \"transformers_version\": \"4.34.1\",\n",
      "  \"use_cache\": false,\n",
      "  \"vocab_size\": 32000\n",
      "}\n",
      "\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:26 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:26 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "[INFO|tokenization_utils_base.py:2015] 2023-12-21 01:48:26,799 >> loading file tokenizer.model from cache at /root/.cache/huggingface/hub/models--meta-llama--Llama-2-70b-hf/snapshots/90052941a64de02075ca800b09fcea1bdaacb939/tokenizer.model\n",
      "[INFO|tokenization_utils_base.py:2015] 2023-12-21 01:48:26,799 >> loading file tokenizer.json from cache at /root/.cache/huggingface/hub/models--meta-llama--Llama-2-70b-hf/snapshots/90052941a64de02075ca800b09fcea1bdaacb939/tokenizer.json\n",
      "[INFO|tokenization_utils_base.py:2015] 2023-12-21 01:48:26,799 >> loading file added_tokens.json from cache at None\n",
      "[INFO|tokenization_utils_base.py:2015] 2023-12-21 01:48:26,799 >> loading file special_tokens_map.json from cache at /root/.cache/huggingface/hub/models--meta-llama--Llama-2-70b-hf/snapshots/90052941a64de02075ca800b09fcea1bdaacb939/special_tokens_map.json\n",
      "[INFO|tokenization_utils_base.py:2015] 2023-12-21 01:48:26,799 >> loading file tokenizer_config.json from cache at /root/.cache/huggingface/hub/models--meta-llama--Llama-2-70b-hf/snapshots/90052941a64de02075ca800b09fcea1bdaacb939/tokenizer_config.json\n",
      "/usr/local/lib/python3.10/dist-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0.\n",
      "You can remove this warning by passing 'token=None' instead.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.10/dist-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0.\n",
      "You can remove this warning by passing 'token=None' instead.\n",
      "  warnings.warn(\n",
      "/usr/local/lib/python3.10/dist-packages/datasets/load.py:2089: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0.\n",
      "You can remove this warning by passing 'token=None' instead.\n",
      "  warnings.warn(\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:26 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:26 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:27 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:27 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:27 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:27 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:28 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:29 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:29 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:29 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:29 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:29 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:29 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:30 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:30 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:30 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:30 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:30 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:31 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:31 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:31 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Downloading (…)fetensors.index.json: 100%|█| 66.7k/66.7k [00:00<00:00, 37.9MB/s]\n",
      "Downloading shards:   0%|                                | 0/15 [00:00<?, ?it/s]Repo card metadata block was not found. Setting CardData to empty.\n",
      "12/21/2023 01:48:31 - WARNING - huggingface_hub.repocard -   Repo card metadata block was not found. Setting CardData to empty.\n",
      "Downloading shards:   0%|                                | 0/15 [00:00<?, ?it/s]\n",
      "Downloading (…)of-00015.safetensors:   0%|          | 0.00/9.85G [00:00<?, ?B/s]\u001b[A\n",
      "Downloading shards:   0%|                                | 0/15 [00:00<?, ?it/s]\u001b[A\n",
      "Downloading shards:   0%|                                | 0/15 [00:00<?, ?it/s]\u001b[A\n",
      "Downloading shards:   0%|                                | 0/15 [00:00<?, ?it/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   1%|  | 62.9M/9.85G [00:00<01:27, 112MB/s]\u001b[A[INFO|modeling_utils.py:2993] 2023-12-21 01:48:32,690 >> loading weights file model.safetensors from cache at /root/.cache/huggingface/hub/models--meta-llama--Llama-2-70b-hf/snapshots/90052941a64de02075ca800b09fcea1bdaacb939/model.safetensors.index.json\n",
      "Downloading shards:   0%|                                | 0/15 [00:00<?, ?it/s]\n",
      "Downloading (…)of-00015.safetensors:   1%|  | 83.9M/9.85G [00:00<01:26, 113MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   1%|   | 105M/9.85G [00:00<01:26, 113MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   1%|   | 126M/9.85G [00:01<01:26, 113MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   1%|   | 147M/9.85G [00:01<01:25, 113MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   2%|   | 168M/9.85G [00:01<01:25, 113MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   2%|   | 189M/9.85G [00:01<01:25, 113MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   2%|   | 210M/9.85G [00:01<01:25, 113MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   2%|   | 231M/9.85G [00:02<01:24, 114MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   3%|   | 252M/9.85G [00:02<01:23, 115MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   3%|   | 273M/9.85G [00:02<01:22, 116MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   3%|   | 294M/9.85G [00:02<01:22, 116MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   3%|   | 315M/9.85G [00:02<01:21, 116MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   3%|   | 336M/9.85G [00:02<01:21, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   4%|   | 357M/9.85G [00:03<01:21, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   4%|   | 377M/9.85G [00:03<01:21, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   4%|   | 398M/9.85G [00:03<01:20, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   4%|▏  | 419M/9.85G [00:03<01:20, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   4%|▏  | 440M/9.85G [00:03<01:20, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   5%|▏  | 461M/9.85G [00:04<01:20, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   5%|▏  | 482M/9.85G [00:04<01:19, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   5%|▏  | 503M/9.85G [00:04<01:19, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:   8%|▏  | 41.9M/524M [00:00<00:04, 102MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  12%|▎  | 62.9M/524M [00:00<00:04, 109MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  16%|▍  | 83.9M/524M [00:00<00:03, 112MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  20%|▊   | 105M/524M [00:00<00:03, 114MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  24%|▉   | 126M/524M [00:01<00:03, 114MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  28%|█   | 147M/524M [00:01<00:03, 115MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  32%|█▎  | 168M/524M [00:01<00:03, 116MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  36%|█▍  | 189M/524M [00:01<00:02, 116MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  40%|█▌  | 210M/524M [00:01<00:02, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  44%|█▊  | 231M/524M [00:02<00:02, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  48%|█▉  | 252M/524M [00:02<00:02, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  52%|██  | 273M/524M [00:02<00:02, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  56%|██▏ | 294M/524M [00:02<00:01, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  60%|██▍ | 315M/524M [00:02<00:01, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  64%|██▌ | 336M/524M [00:02<00:01, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  68%|██▋ | 357M/524M [00:03<00:01, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  72%|██▉ | 377M/524M [00:03<00:01, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  76%|███ | 398M/524M [00:03<00:01, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  80%|███▏| 419M/524M [00:03<00:00, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  84%|███▎| 440M/524M [00:03<00:00, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  88%|███▌| 461M/524M [00:04<00:00, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  92%|███▋| 482M/524M [00:04<00:00, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors:  96%|███▊| 503M/524M [00:04<00:00, 117MB/s]\u001b[A\n",
      "Downloading (…)of-00015.safetensors: 100%|████| 524M/524M [00:04<00:00, 115MB/s]\u001b[A\n",
      "Downloading shards: 100%|███████████████████████| 15/15 [20:37<00:00, 82.53s/it]\n",
      "Downloading shards: 100%|███████████████████████| 15/15 [20:37<00:00, 82.52s/it]\n",
      "Downloading shards: 100%|███████████████████████| 15/15 [20:38<00:00, 82.54s/it]\n",
      "Downloading shards: 100%|███████████████████████| 15/15 [20:38<00:00, 82.54s/it]\n",
      "Downloading shards: 100%|███████████████████████| 15/15 [20:38<00:00, 82.54s/it]\n",
      "Downloading shards: 100%|███████████████████████| 15/15 [20:37<00:00, 82.48s/it]\n",
      "Downloading shards: 100%|███████████████████████| 15/15 [20:37<00:00, 82.51s/it]\n",
      "[INFO|modeling_utils.py:1220] 2023-12-21 02:09:09,932 >> Instantiating GaudiLlamaForCausalLM model under default dtype torch.bfloat16.\n",
      "[INFO|modeling_utils.py:3076] 2023-12-21 02:09:09,933 >> Detected DeepSpeed ZeRO-3: activating zero.init() for this model\n",
      "Downloading shards: 100%|███████████████████████| 15/15 [20:37<00:00, 82.50s/it]\n",
      "[INFO|configuration_utils.py:770] 2023-12-21 02:09:09,961 >> Generate config GaudiGenerationConfig {\n",
      "  \"attn_softmax_bf16\": null,\n",
      "  \"bos_token_id\": 1,\n",
      "  \"bucket_size\": -1,\n",
      "  \"eos_token_id\": 2,\n",
      "  \"ignore_eos\": null,\n",
      "  \"kv_cache_fp8\": null,\n",
      "  \"limit_hpu_graphs\": null,\n",
      "  \"reuse_cache\": null,\n",
      "  \"static_shapes\": null,\n",
      "  \"trim_logits\": null,\n",
      "  \"use_cache\": false\n",
      "}\n",
      "\n",
      "============================= HABANA PT BRIDGE CONFIGURATION =========================== \n",
      " PT_HPU_LAZY_MODE = 1\n",
      " PT_RECIPE_CACHE_PATH = \n",
      " PT_CACHE_FOLDER_DELETE = 0\n",
      " PT_HPU_RECIPE_CACHE_CONFIG = \n",
      " PT_HPU_MAX_COMPOUND_OP_SIZE = 10\n",
      " PT_HPU_LAZY_ACC_PAR_MODE = 0\n",
      " PT_HPU_ENABLE_REFINE_DYNAMIC_SHAPES = 0\n",
      "---------------------------: System Configuration :---------------------------\n",
      "Num CPU Cores : 160\n",
      "CPU RAM       : 1056399508 KB\n",
      "------------------------------------------------------------------------------\n",
      "[2023-12-21 02:09:20,271] [INFO] [partition_parameters.py:350:__exit__] finished initializing model - num_params = 723, num_elems = 68.98B\n",
      "Loading checkpoint shards: 100%|████████████████| 15/15 [00:35<00:00,  2.36s/it]\n",
      "[INFO|modeling_utils.py:3775] 2023-12-21 02:09:55,995 >> All model checkpoint weights were used when initializing GaudiLlamaForCausalLM.\n",
      "\n",
      "[INFO|modeling_utils.py:3783] 2023-12-21 02:09:55,995 >> All the weights of GaudiLlamaForCausalLM were initialized from the model checkpoint at meta-llama/Llama-2-70b-hf.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use GaudiLlamaForCausalLM for predictions without further training.\n",
      "Loading checkpoint shards: 100%|████████████████| 15/15 [00:35<00:00,  2.36s/it]\n",
      "Loading checkpoint shards: 100%|████████████████| 15/15 [00:35<00:00,  2.36s/it]\n",
      "Loading checkpoint shards: 100%|████████████████| 15/15 [00:35<00:00,  2.36s/it]\n",
      "Loading checkpoint shards: 100%|████████████████| 15/15 [00:35<00:00,  2.36s/it]\n",
      "Loading checkpoint shards: 100%|████████████████| 15/15 [00:35<00:00,  2.36s/it]\n",
      "Loading checkpoint shards: 100%|████████████████| 15/15 [00:35<00:00,  2.36s/it]\n",
      "Loading checkpoint shards: 100%|████████████████| 15/15 [00:35<00:00,  2.36s/it]\n",
      "Downloading generation_config.json: 100%|███████| 188/188 [00:00<00:00, 580kB/s]\n",
      "[INFO|configuration_utils.py:730] 2023-12-21 02:09:56,220 >> loading configuration file generation_config.json from cache at /root/.cache/huggingface/hub/models--meta-llama--Llama-2-70b-hf/snapshots/90052941a64de02075ca800b09fcea1bdaacb939/generation_config.json\n",
      "[INFO|configuration_utils.py:770] 2023-12-21 02:09:56,221 >> Generate config GaudiGenerationConfig {\n",
      "  \"attn_softmax_bf16\": null,\n",
      "  \"bos_token_id\": 1,\n",
      "  \"bucket_size\": -1,\n",
      "  \"do_sample\": true,\n",
      "  \"eos_token_id\": 2,\n",
      "  \"ignore_eos\": null,\n",
      "  \"kv_cache_fp8\": null,\n",
      "  \"limit_hpu_graphs\": null,\n",
      "  \"max_length\": 4096,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"reuse_cache\": null,\n",
      "  \"static_shapes\": null,\n",
      "  \"temperature\": 0.6,\n",
      "  \"top_p\": 0.9,\n",
      "  \"trim_logits\": null\n",
      "}\n",
      "\n",
      "Map: 100%|█████████████████████████| 9452/9452 [00:05<00:00, 1752.81 examples/s]\n",
      "Map: 100%|███████████████████████████| 518/518 [00:00<00:00, 1681.47 examples/s]\n",
      "Map: 100%|███████████████████████████| 394/394 [00:00<00:00, 1819.03 examples/s]\n",
      "Map: 100%|███████████████████████████| 518/518 [00:00<00:00, 1512.40 examples/s]\n",
      "Map: 100%|███████████████████████████| 518/518 [00:00<00:00, 1502.67 examples/s]\n",
      "Map: 100%|███████████████████████████| 518/518 [00:00<00:00, 1379.03 examples/s]\n",
      "Map: 100%|███████████████████████████| 518/518 [00:00<00:00, 1557.67 examples/s]\n",
      "Map: 100%|███████████████████████████| 518/518 [00:00<00:00, 1545.58 examples/s]\n",
      "Map: 100%|███████████████████████████| 518/518 [00:00<00:00, 1528.32 examples/s]\n",
      "Map: 100%|███████████████████████████| 518/518 [00:00<00:00, 1535.17 examples/s]\n",
      "12/21/2023 02:10:11 - INFO - __main__ -   Using data collator of type DataCollatorForLanguageModeling\n",
      "trainable params: 16,384,000 || all params: 68,993,032,192 || trainable%: 0.023747325605874423\n",
      "trainable params: 16,384,000 || all params: 68,993,032,192 || trainable%: 0.023747325605874423\n",
      "trainable params: 16,384,000 || all params: 68,993,032,192 || trainable%: 0.023747325605874423\n",
      "trainable params: 16,384,000 || all params: 68,993,032,192 || trainable%: 0.023747325605874423\n",
      "trainable params: 16,384,000 || all params: 68,993,032,192 || trainable%: 0.023747325605874423\n",
      "[2023-12-21 02:10:14,208] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.10.3+hpu.synapse.v1.13.0, git-hash=6522014, git-branch=1.13.0\n",
      "trainable params: 16,384,000 || all params: 68,993,032,192 || trainable%: 0.023747325605874423\n",
      "trainable params: 16,384,000 || all params: 68,993,032,192 || trainable%: 0.023747325605874423\n",
      "[2023-12-21 02:10:14,318] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False\n",
      "trainable params: 16,384,000 || all params: 68,993,032,192 || trainable%: 0.023747325605874423\n",
      "[2023-12-21 02:10:14,326] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the client Optimizer\n",
      "[2023-12-21 02:10:14,326] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer\n",
      "[2023-12-21 02:10:14,437] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdamW\n",
      "[2023-12-21 02:10:14,437] [INFO] [utils.py:61:is_zero_supported_optimizer] Checking ZeRO support for optimizer=FusedAdamW type=<class 'habana_frameworks.torch.hpex.optimizers.FusedAdamW.FusedAdamW'>\n",
      "[2023-12-21 02:10:14,437] [INFO] [logging.py:96:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False\n",
      "[2023-12-21 02:10:14,437] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer\n",
      "[2023-12-21 02:10:14,532] [INFO] [utils.py:866:see_memory_usage] Stage 3 initialize beginning\n",
      "[2023-12-21 02:10:14,536] [INFO] [utils.py:867:see_memory_usage] MA 16.25 GB         Max_MA 17.68 GB         CA 0.0 GB         Max_CA 0 GB \n",
      "[2023-12-21 02:10:14,536] [INFO] [utils.py:874:see_memory_usage] CPU Virtual Memory:  used = 63.05 GB, percent = 6.3%\n",
      "[2023-12-21 02:10:14,550] [INFO] [stage3.py:124:__init__] Reduce bucket size 500,000,000\n",
      "[2023-12-21 02:10:14,550] [INFO] [stage3.py:125:__init__] Prefetch bucket size 50,000,000\n",
      "[2023-12-21 02:10:14,660] [INFO] [utils.py:866:see_memory_usage] DeepSpeedZeRoOffload initialize [begin]\n",
      "[2023-12-21 02:10:14,665] [INFO] [utils.py:867:see_memory_usage] MA 16.25 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB \n",
      "[2023-12-21 02:10:14,665] [INFO] [utils.py:874:see_memory_usage] CPU Virtual Memory:  used = 63.05 GB, percent = 6.3%\n",
      "Parameter Offload: Total persistent parameters: 17702912 in 801 params\n",
      "[2023-12-21 02:10:15,918] [INFO] [utils.py:866:see_memory_usage] DeepSpeedZeRoOffload initialize [end]\n",
      "[2023-12-21 02:10:15,922] [INFO] [utils.py:867:see_memory_usage] MA 16.22 GB         Max_MA 16.25 GB         CA 0.0 GB         Max_CA 0 GB \n",
      "[2023-12-21 02:10:15,923] [INFO] [utils.py:874:see_memory_usage] CPU Virtual Memory:  used = 63.13 GB, percent = 6.3%\n",
      "[2023-12-21 02:10:16,041] [INFO] [utils.py:866:see_memory_usage] Before creating fp16 partitions\n",
      "[2023-12-21 02:10:16,046] [INFO] [utils.py:867:see_memory_usage] MA 16.22 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB \n",
      "[2023-12-21 02:10:16,046] [INFO] [utils.py:874:see_memory_usage] CPU Virtual Memory:  used = 63.13 GB, percent = 6.3%\n",
      "[2023-12-21 02:10:16,389] [INFO] [utils.py:866:see_memory_usage] After creating fp16 partitions: 1\n",
      "[2023-12-21 02:10:16,394] [INFO] [utils.py:867:see_memory_usage] MA 16.22 GB         Max_MA 16.22 GB         CA 0.0 GB         Max_CA 0 GB \n",
      "[2023-12-21 02:10:16,394] [INFO] [utils.py:874:see_memory_usage] CPU Virtual Memory:  used = 63.14 GB, percent = 6.3%\n",
      "[2023-12-21 02:10:16,512] [INFO] [utils.py:866:see_memory_usage] Before creating fp32 partitions\n",
      "[2023-12-21 02:10:16,516] [INFO] [utils.py:867:see_memory_usage] MA 16.22 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB \n",
      "[2023-12-21 02:10:16,517] [INFO] [utils.py:874:see_memory_usage] CPU Virtual Memory:  used = 63.14 GB, percent = 6.3%\n",
      "[2023-12-21 02:10:16,645] [INFO] [utils.py:866:see_memory_usage] After creating fp32 partitions\n",
      "[2023-12-21 02:10:16,649] [INFO] [utils.py:867:see_memory_usage] MA 16.22 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB \n",
      "[2023-12-21 02:10:16,649] [INFO] [utils.py:874:see_memory_usage] CPU Virtual Memory:  used = 63.14 GB, percent = 6.3%\n",
      "[2023-12-21 02:10:16,789] [INFO] [utils.py:866:see_memory_usage] Before initializing optimizer states\n",
      "[2023-12-21 02:10:16,795] [INFO] [utils.py:867:see_memory_usage] MA 16.22 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB \n",
      "[2023-12-21 02:10:16,795] [INFO] [utils.py:874:see_memory_usage] CPU Virtual Memory:  used = 63.16 GB, percent = 6.3%\n",
      "[2023-12-21 02:10:16,966] [INFO] [utils.py:866:see_memory_usage] After initializing optimizer states\n",
      "[2023-12-21 02:10:16,969] [INFO] [utils.py:867:see_memory_usage] MA 16.25 GB         Max_MA 16.25 GB         CA 0.0 GB         Max_CA 0 GB \n",
      "[2023-12-21 02:10:16,969] [INFO] [utils.py:874:see_memory_usage] CPU Virtual Memory:  used = 63.2 GB, percent = 6.3%\n",
      "[2023-12-21 02:10:16,971] [INFO] [stage3.py:446:_setup_for_real_optimizer] optimizer state initialized\n",
      "[2023-12-21 02:10:22,332] [INFO] [utils.py:866:see_memory_usage] After initializing ZeRO optimizer\n",
      "[2023-12-21 02:10:22,335] [INFO] [utils.py:867:see_memory_usage] MA 16.25 GB         Max_MA 16.25 GB         CA 0.0 GB         Max_CA 0 GB \n",
      "[2023-12-21 02:10:22,336] [INFO] [utils.py:874:see_memory_usage] CPU Virtual Memory:  used = 63.36 GB, percent = 6.3%\n",
      "[2023-12-21 02:10:22,337] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedZeroOptimizer_Stage3\n",
      "[2023-12-21 02:10:22,337] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using client LR scheduler\n",
      "[2023-12-21 02:10:22,337] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = None\n",
      "[2023-12-21 02:10:22,337] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0], mom=[(0.9, 0.999)]\n",
      "[2023-12-21 02:10:22,353] [INFO] [config.py:982:print] DeepSpeedEngine configuration:\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   activation_checkpointing_config  {\n",
      "    \"partition_activations\": false, \n",
      "    \"contiguous_memory_optimization\": false, \n",
      "    \"cpu_checkpointing\": false, \n",
      "    \"number_checkpoints\": null, \n",
      "    \"synchronize_checkpoint_boundary\": false, \n",
      "    \"profile\": false\n",
      "}\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   amp_enabled .................. False\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   amp_params ................... False\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   autotuning_config ............ {\n",
      "    \"enabled\": false, \n",
      "    \"start_step\": null, \n",
      "    \"end_step\": null, \n",
      "    \"metric_path\": null, \n",
      "    \"arg_mappings\": null, \n",
      "    \"metric\": \"throughput\", \n",
      "    \"model_info\": null, \n",
      "    \"results_dir\": \"autotuning_results\", \n",
      "    \"exps_dir\": \"autotuning_exps\", \n",
      "    \"overwrite\": true, \n",
      "    \"fast\": true, \n",
      "    \"start_profile_step\": 3, \n",
      "    \"end_profile_step\": 5, \n",
      "    \"tuner_type\": \"gridsearch\", \n",
      "    \"tuner_early_stopping\": 5, \n",
      "    \"tuner_num_trials\": 50, \n",
      "    \"model_info_path\": null, \n",
      "    \"mp_size\": 1, \n",
      "    \"max_train_batch_size\": null, \n",
      "    \"min_train_batch_size\": 1, \n",
      "    \"max_train_micro_batch_size_per_gpu\": 1.024000e+03, \n",
      "    \"min_train_micro_batch_size_per_gpu\": 1, \n",
      "    \"num_tuning_micro_batch_sizes\": 3\n",
      "}\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   bfloat16_accumulate_grads_via_hooks  False\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   bfloat16_enabled ............. True\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   checkpoint_parallel_write_pipeline  False\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   checkpoint_tag_validation_enabled  True\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   checkpoint_tag_validation_fail  False\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7fb6cb347280>\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   communication_data_type ...... None\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   curriculum_enabled_legacy .... False\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   curriculum_params_legacy ..... False\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   data_efficiency_enabled ...... False\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   dataloader_drop_last ......... False\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   disable_allgather ............ False\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   dump_state ................... False\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   dynamic_loss_scale_args ...... None\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   eigenvalue_enabled ........... False\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   eigenvalue_gas_boundary_resolution  1\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   eigenvalue_layer_name ........ bert.encoder.layer\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   eigenvalue_layer_num ......... 0\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   eigenvalue_max_iter .......... 100\n",
      "[2023-12-21 02:10:22,354] [INFO] [config.py:986:print]   eigenvalue_stability ......... 1e-06\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   eigenvalue_tol ............... 0.01\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   eigenvalue_verbose ........... False\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   elasticity_enabled ........... False\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   flops_profiler_config ........ {\n",
      "    \"enabled\": false, \n",
      "    \"recompute_fwd_factor\": 0.0, \n",
      "    \"profile_step\": 1, \n",
      "    \"module_depth\": -1, \n",
      "    \"top_modules\": 1, \n",
      "    \"detailed\": true, \n",
      "    \"output_file\": null\n",
      "}\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   fp16_auto_cast ............... None\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   fp16_enabled ................. False\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   fp16_master_weights_and_gradients  False\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   global_rank .................. 0\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   grad_accum_dtype ............. None\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   gradient_accumulation_steps .. 1\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   gradient_clipping ............ 1.0\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   gradient_predivide_factor .... 1.0\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   initial_dynamic_scale ........ 1\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   load_universal_checkpoint .... False\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   loss_scale ................... 1.0\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   memory_breakdown ............. False\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   mics_hierarchial_params_gather  False\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   mics_shard_size .............. -1\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   nebula_config ................ {\n",
      "    \"enabled\": false, \n",
      "    \"persistent_storage_path\": null, \n",
      "    \"persistent_time_interval\": 100, \n",
      "    \"num_of_version_in_retention\": 2, \n",
      "    \"enable_nebula_load\": true, \n",
      "    \"load_path\": null\n",
      "}\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   optimizer_legacy_fusion ...... False\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   optimizer_name ............... None\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   optimizer_params ............. None\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': 'auto', 'grad_partitioned': 'auto'}\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   pld_enabled .................. False\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   pld_params ................... False\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   prescale_gradients ........... False\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   scheduler_name ............... None\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   scheduler_params ............. None\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   sparse_attention ............. None\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   sparse_gradients_enabled ..... False\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   steps_per_print .............. inf\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   train_batch_size ............. 80\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   train_micro_batch_size_per_gpu  10\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   use_node_local_storage ....... False\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   wall_clock_breakdown ......... False\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   weight_quantization_config ... None\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   world_size ................... 8\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   zero_allow_comm_data_type_fp32  False\n",
      "[2023-12-21 02:10:22,355] [INFO] [config.py:986:print]   zero_allow_untested_optimizer  True\n",
      "[2023-12-21 02:10:22,356] [INFO] [config.py:986:print]   zero_config .................. stage=3 contiguous_gradients=False reduce_scatter=False reduce_bucket_size=500,000,000 allgather_partitions=True allgather_bucket_size=500,000,000 overlap_comm=False max_group_size=4000000000 load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True\n",
      "[2023-12-21 02:10:22,356] [INFO] [config.py:986:print]   zero_enabled ................. True\n",
      "[2023-12-21 02:10:22,356] [INFO] [config.py:986:print]   zero_force_ds_cpu_optimizer .. True\n",
      "[2023-12-21 02:10:22,356] [INFO] [config.py:986:print]   zero_optimization_stage ...... 3\n",
      "[2023-12-21 02:10:22,356] [INFO] [config.py:972:print_user_config]   json = {\n",
      "    \"steps_per_print\": inf, \n",
      "    \"train_batch_size\": 80, \n",
      "    \"train_micro_batch_size_per_gpu\": 10, \n",
      "    \"gradient_accumulation_steps\": 1, \n",
      "    \"bf16\": {\n",
      "        \"enabled\": true\n",
      "    }, \n",
      "    \"gradient_clipping\": 1.0, \n",
      "    \"zero_optimization\": {\n",
      "        \"stage\": 3, \n",
      "        \"overlap_comm\": false, \n",
      "        \"contiguous_gradients\": false\n",
      "    }, \n",
      "    \"fp16\": {\n",
      "        \"enabled\": false\n",
      "    }, \n",
      "    \"zero_allow_untested_optimizer\": true\n",
      "}\n",
      "[INFO|trainer.py:672] 2023-12-21 02:10:22,356 >> ***** Running training *****\n",
      "[INFO|trainer.py:673] 2023-12-21 02:10:22,356 >>   Num examples = 2,030\n",
      "[INFO|trainer.py:674] 2023-12-21 02:10:22,356 >>   Num Epochs = 2\n",
      "[INFO|trainer.py:675] 2023-12-21 02:10:22,356 >>   Instantaneous batch size per device = 10\n",
      "[INFO|trainer.py:678] 2023-12-21 02:10:22,356 >>   Total train batch size (w. parallel, distributed & accumulation) = 80\n",
      "[INFO|trainer.py:679] 2023-12-21 02:10:22,356 >>   Gradient Accumulation steps = 1\n",
      "[INFO|trainer.py:680] 2023-12-21 02:10:22,356 >>   Total optimization steps = 52\n",
      "[INFO|trainer.py:681] 2023-12-21 02:10:22,371 >>   Number of trainable parameters = 16,384,000\n",
      "  0%|                                                    | 0/52 [00:00<?, ?it/s][WARNING|logging.py:290] 2023-12-21 02:10:22,414 >> You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n",
      "You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n",
      "You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n",
      "You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n",
      "You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n",
      "You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n",
      "You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n",
      "You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n",
      "[2023-12-21 02:10:22,529] [INFO] [checkpointing.py:535:forward] Activation Checkpointing Information\n",
      "[2023-12-21 02:10:22,529] [INFO] [checkpointing.py:536:forward] ----Partition Activations False, CPU CHECKPOINTING False\n",
      "[2023-12-21 02:10:22,529] [INFO] [checkpointing.py:537:forward] ----contiguous Memory Checkpointing False with None total layers\n",
      "[2023-12-21 02:10:22,529] [INFO] [checkpointing.py:539:forward] ----Synchronization False\n",
      "[2023-12-21 02:10:22,530] [INFO] [checkpointing.py:540:forward] ----Profiling time in checkpointing False\n",
      "{'loss': 1.193, 'learning_rate': 0.0009, 'epoch': 0.04, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 69.27, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.1528, 'learning_rate': 0.0018, 'epoch': 0.08, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 72.4, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.2109, 'learning_rate': 0.0017982240555854444, 'epoch': 0.12, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 72.71, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0779, 'learning_rate': 0.0017929032311830302, 'epoch': 0.15, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.87, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0766, 'learning_rate': 0.00178405852565582, 'epoch': 0.19, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.87, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0486, 'learning_rate': 0.001771724845015768, 'epoch': 0.23, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0581, 'learning_rate': 0.0017559508646656382, 'epoch': 0.27, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0211, 'learning_rate': 0.001736798837299426, 'epoch': 0.31, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0375, 'learning_rate': 0.0017143443472194175, 'epoch': 0.35, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9831, 'learning_rate': 0.0016886760120394771, 'epoch': 0.38, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0186, 'learning_rate': 0.0016598951329518136, 'epoch': 0.42, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0058, 'learning_rate': 0.0016281152949374526, 'epoch': 0.46, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0109, 'learning_rate': 0.0015934619184982102, 'epoch': 0.5, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9605, 'learning_rate': 0.0015560717646792704, 'epoch': 0.54, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9743, 'learning_rate': 0.0015160923953358198, 'epoch': 0.58, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0403, 'learning_rate': 0.0014736815907738208, 'epoch': 0.62, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0054, 'learning_rate': 0.0014290067270632258, 'epoch': 0.65, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0218, 'learning_rate': 0.0013822441154810968, 'epoch': 0.69, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9739, 'learning_rate': 0.0013335783066915437, 'epoch': 0.73, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9541, 'learning_rate': 0.0012832013624085653, 'epoch': 0.77, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9803, 'learning_rate': 0.0012313120974162103, 'epoch': 0.81, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9911, 'learning_rate': 0.0011781152949374527, 'epoch': 0.85, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0279, 'learning_rate': 0.0011238208984483695, 'epoch': 0.88, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9762, 'learning_rate': 0.0010686431831271523, 'epoch': 0.92, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0026, 'learning_rate': 0.001012799910207874, 'epoch': 0.96, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0094, 'learning_rate': 0.0009565114675763821, 'epoch': 1.0, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.948, 'learning_rate': 0.0009, 'epoch': 1.04, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9711, 'learning_rate': 0.000843488532423618, 'epoch': 1.08, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9764, 'learning_rate': 0.000787200089792126, 'epoch': 1.12, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9921, 'learning_rate': 0.0007313568168728476, 'epoch': 1.15, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9792, 'learning_rate': 0.0006761791015516308, 'epoch': 1.19, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.954, 'learning_rate': 0.0006218847050625474, 'epoch': 1.23, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9957, 'learning_rate': 0.0005686879025837898, 'epoch': 1.27, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9643, 'learning_rate': 0.0005167986375914346, 'epoch': 1.31, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0023, 'learning_rate': 0.0004664216933084561, 'epoch': 1.35, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9594, 'learning_rate': 0.0004177558845189028, 'epoch': 1.38, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9749, 'learning_rate': 0.0003709932729367743, 'epoch': 1.42, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9789, 'learning_rate': 0.00032631840922617923, 'epoch': 1.46, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9384, 'learning_rate': 0.0002839076046641801, 'epoch': 1.5, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9708, 'learning_rate': 0.0002439282353207298, 'epoch': 1.54, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9732, 'learning_rate': 0.00020653808150178975, 'epoch': 1.58, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9761, 'learning_rate': 0.00017188470506254738, 'epoch': 1.62, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0026, 'learning_rate': 0.00014010486704818664, 'epoch': 1.65, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9906, 'learning_rate': 0.00011132398796052297, 'epoch': 1.69, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0107, 'learning_rate': 8.565565278058258e-05, 'epoch': 1.73, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0045, 'learning_rate': 6.320116270057379e-05, 'epoch': 1.77, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.972, 'learning_rate': 4.404913533436182e-05, 'epoch': 1.81, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9886, 'learning_rate': 2.827515498423203e-05, 'epoch': 1.85, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9593, 'learning_rate': 1.594147434418025e-05, 'epoch': 1.88, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0161, 'learning_rate': 7.096768816970011e-06, 'epoch': 1.92, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 1.0146, 'learning_rate': 1.7759444145555968e-06, 'epoch': 1.96, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "{'loss': 0.9315, 'learning_rate': 0.0, 'epoch': 2.0, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "100%|███████████████████████████████████████████| 52/52 [33:37<00:00, 36.20s/it][INFO|trainer.py:1494] 2023-12-21 02:44:00,139 >> ***** Running Evaluation *****\n",
      "[INFO|trainer.py:1496] 2023-12-21 02:44:00,139 >>   Num examples = 112\n",
      "[INFO|trainer.py:1499] 2023-12-21 02:44:00,139 >>   Batch size = 10\n",
      "\n",
      "  0%|                                                     | 0/2 [00:00<?, ?it/s]\u001b[A\n",
      "100%|█████████████████████████████████████████████| 2/2 [00:23<00:00, 11.91s/it]\u001b[A\n",
      "{'eval_loss': 1.0001739263534546, 'eval_accuracy': 0.7373508269942075, 'eval_runtime': 40.9166, 'eval_samples_per_second': 2.737, 'eval_steps_per_second': 0.049, 'epoch': 2.0, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "\n",
      "100%|███████████████████████████████████████████| 52/52 [34:18<00:00, 36.20s/it]\u001b[A\n",
      "                                                                                \u001b[A[INFO|trainer.py:945] 2023-12-21 02:44:41,036 >> \n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n",
      "{'train_runtime': 2058.6656, 'train_samples_per_second': 2.107, 'train_steps_per_second': 0.027, 'train_loss': 1.004965705367235, 'epoch': 2.0, 'memory_allocated (GB)': 21.6, 'max_memory_allocated (GB)': 73.9, 'total_memory_available (GB)': 94.62}\n",
      "100%|███████████████████████████████████████████| 52/52 [34:18<00:00, 39.59s/it]\n",
      "/usr/local/lib/python3.10/dist-packages/safetensors/torch.py:17: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()\n",
      "  return tensor.storage().data_ptr()\n"
     ]
    }
   ],
   "source": [
    "!PT_HPU_MAX_COMPOUND_OP_SIZE=10  DEEPSPEED_HPU_ZERO3_SYNC_MARK_STEP_REQUIRED=1 \\\n",
    "python3 ../gaudi_spawn.py --use_deepspeed  --world_size 8 run_lora_clm.py \\\n",
    "  --model_name_or_path meta-llama/Llama-2-70b-hf \\\n",
    "  --deepspeed llama2_ds_zero3_config.json \\\n",
    "  --dataset_name timdettmers/openassistant-guanaco \\\n",
    "  --bf16 True\\\n",
    "  --output_dir ./model_lora_llama \\\n",
    "  --num_train_epochs 2 \\\n",
    "  --max_seq_len 2048 \\\n",
    "  --per_device_train_batch_size 10 \\\n",
    "  --per_device_eval_batch_size 10 \\\n",
    "  --gradient_checkpointing \\\n",
    "  --evaluation_strategy epoch \\\n",
    "  --eval_delay 2 \\\n",
    "  --save_strategy no \\\n",
    "  --learning_rate 0.0018 \\\n",
    "  --warmup_ratio 0.03 \\\n",
    "  --lr_scheduler_type \"cosine\" \\\n",
    "  --logging_steps 1 \\\n",
    "  --dataset_concatenation \\\n",
    "  --attn_softmax_bf16 True \\\n",
    "  --do_train \\\n",
    "  --do_eval \\\n",
    "  --use_habana \\\n",
    "  --use_lazy_mode \\\n",
    "  --report_to none \\\n",
    "  --pipelining_fwd_bwd \\\n",
    "  --throughput_warmup_steps 3 \\\n",
    "  --lora_rank 4\\\n",
    "  --lora_target_modules \"q_proj\" \"v_proj\" \"k_proj\" \"o_proj\" \\\n",
    "  --validation_split_percentage 4 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "340f4aa5-5dc0-4662-bbca-a7a156be37f2",
   "metadata": {},
   "source": [
    "#### LoRA Fine Tuning Completed\n",
    "You will now see a \"model_lora_llama\" folder created which contains the PEFT model `adapter_model.bin` which will be used in the inference example below. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2a5b5cf-e4a4-4d67-888b-e3ddcdff1a5d",
   "metadata": {},
   "source": [
    "## Inference with Llama 2\n",
    "\n",
    "We'll now use the Hugging Face `text-generation` task to run inference on the Llama2-70b model; we'll generate text based on an included prompt.  Notice that we've included a path to the PEFT model that we just created.\n",
    "\n",
    "First, we'll move to the text-generation examples folder and install the requirements. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14841b58-2697-459d-ace5-763d721468f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/Gaudi-tutorials/PyTorch/llama2_fine_tuning_inference/optimum-habana/examples/text-generation\n",
    "!pip install -q -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03479984-e872-48df-b106-cb9b1dd445d3",
   "metadata": {},
   "source": [
    "You will see that we are now running inference with the `run_generation.py` task and we are including the PEFT model that we Fine Tuned in the steps above. \n",
    "\n",
    "\n",
    "\n",
    "```\n",
    "python3 ../gaudi_spawn.py --use_deepspeed  --world_size 8  run_generation.py \\\n",
    "   --model_name_or_path meta-llama/Llama-2-70b-hf \\\n",
    "   --batch_size 1 \\\n",
    "   --do_sample \\\n",
    "   --max_new_tokens 250 \\\n",
    "   --n_iterations 4 \\\n",
    "   --use_kv_cache \\\n",
    "   --use_hpu_graphs_for_inference \\\n",
    "   --bf16 \\\n",
    "   --prompt \"I am a dog. Please help me plan a surprise birthday party for my human, including fun activities, games and decorations. And don't forget to order a big bone-shaped cake for me to share with my fur friends!\" \\\n",
    "   --peft_model /root/Gaudi-tutorials/PyTorch/llama2_fine_tuning_inference/optimum-habana/examples/language-modeling/model_lora_llama \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e32f89a0-4bdc-4ca4-9855-88060b7f140e",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = input(\"Enter a prompt for text generation: \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d466f4a2-3607-4e8f-8b2d-1aefcc7d81aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "cmd = f'python3 ../gaudi_spawn.py --use_deepspeed  --world_size 8  run_generation.py \\\n",
    "   --model_name_or_path meta-llama/Llama-2-70b-hf \\\n",
    "   --batch_size 1 --do_sample --max_new_tokens 250 --n_iterations 4 --use_kv_cache --use_hpu_graphs_for_inference --bf16 --prompt \"{prompt}\" \\\n",
    "   --peft_model /root/Gaudi-tutorials/PyTorch/llama2_fine_tuning_inference/optimum-habana/examples/language-modeling/model_lora_llama/'\n",
    "print(cmd)\n",
    "import os\n",
    "os.system(cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40985bfa",
   "metadata": {},
   "source": [
    "###### Inference Output with PEFT\n",
    "\n",
    "```\n",
    "input 1: (\"I am a dog. Please help me plan a surprise birthday party for my human, including fun activities, games and decorations. And don't forget to order a big bone-shaped cake for me to share with my fur friends!\",)\n",
    "output 1: ('I am a dog. Please help me plan a surprise birthday party for my human, including fun activities, games and decorations. And don\\'t forget to order a big bone-shaped cake for me to share with my fur friends!\n",
    "\n",
    "Assistant: Hey there pup! I can help you plan your human\\'s birthday party. Here are some ideas for fun activities and games you can play together:\\n\\n\n",
    "1. A \"Find the Treat\" scavenger hunt: Hide treats around your home or yard for your human to find. Provide clues and hints along the way.\\n\n",
    "2. \"Tug-of-War\": Play a game of tug-of-war with a rope tied to a tree stump or post.\\n\n",
    "3. \"Frisbee Fun\": Invite your human to a game of fetch with a Frisbee in the park or backyard.\\n\\n\n",
    "Decorations can include: Dog-shaped balloons, paw print streamers, and a banner saying \"Happy Birthday\" with your human\\'s name.\\n\\n\n",
    "And don\\'t forget to order a cake in the shape of a big bone for you and your fur friends to share!\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d269a38c-eeab-49d8-b12a-73599636e445",
   "metadata": {},
   "source": [
    "##### Comparison without PEFT and LoRA\n",
    "In this example, we're simply running the Llama2 7B model **without** including the PEFT fine tuned model, so the you are losing the additional detail that is brought to the model, and the results have signficantly less information and fidelity compared to the last model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a7a9804-cf3e-4bd3-adba-b8d39a9d55a9",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cmd = f'python3 ../gaudi_spawn.py --use_deepspeed  --world_size 8  run_generation.py \\\n",
    "   --model_name_or_path /root/data/llama_inference/Llama-2-70b-hf \\\n",
    "   --batch_size 1 \\\n",
    "   --do_sample \\\n",
    "   --max_new_tokens 250 \\\n",
    "   --n_iterations 4 \\\n",
    "   --use_kv_cache \\\n",
    "   --use_hpu_graphs_for_infernence \\\n",
    "   --bf16 \\\n",
    "   --prompt \"{prompt}\"'\n",
    "print(cmd)\n",
    "import os\n",
    "os.system(cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "560ba49f",
   "metadata": {},
   "source": [
    "###### Inference Output without PEFT (using just standard Llama 2 model)\n",
    "\n",
    "```\n",
    "input 1: (\"I am a dog. Please help me plan a surprise birthday party for my human, including fun activities, games and decorations. And don't forget to order a big bone-shaped cake for me to share with my fur friends!\",)\n",
    "output 1: (\"I am a dog. Please help me plan a surprise birthday party for my human, including fun activities, games and decorations. And don't forget to order a big bone-shaped cake for me to share with my fur friends!\\n\n",
    "\n",
    "Make sure that you do not make a big noise because my human doesn’t know that we are planning a birthday party. Thanks to your help now I am sure there are no more things to worry about.\\n\n",
    "The dog does not have to worry that the human will find out about the party. She should not worry about the noise while planning the party. There will be big bone-shaped cake for the guest of honor to share with his fur friends. There will be fun activities, games and decorations. The following items are tagged newsletter marketing:\\n\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a34e1747-57ea-44aa-a564-4af8d303d72a",
   "metadata": {},
   "outputs": [],
   "source": [
    "exit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d29233ce-11b6-4fb0-8750-ab534e79533a",
   "metadata": {},
   "source": [
    "#### ADDENDUM: Running with Llama 2 7B or 13B Parameters\n",
    "For a faster runtime you can use the smaller Llama 2 7B or 13B model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7d6b699-b2b0-4b46-b508-7b1e1d6a753f",
   "metadata": {},
   "outputs": [],
   "source": [
    "!python ../gaudi_spawn.py --use_deepspeed \\\n",
    "       --world_size 8 run_lora_clm.py \\\n",
    "       --model_name_or_path meta-llama/Llama-2-7b-hf  \\\n",
    "       --dataset_name timdettmers/openassistant-guanaco \\\n",
    "       --bf16 True \\\n",
    "       --output_dir ./model_lora_llama_7b \\\n",
    "       --num_train_epochs 2 \\\n",
    "       --per_device_train_batch_size 2 \\\n",
    "       --per_device_eval_batch_size 2 \\\n",
    "       --gradient_accumulation_steps 4 \\\n",
    "       --evaluation_strategy \"no\"\\\n",
    "       --save_strategy \"steps\"\\\n",
    "       --save_steps 2000 \\\n",
    "       --save_total_limit 1 \\\n",
    "       --learning_rate 1e-4 \\\n",
    "       --logging_steps 1 \\\n",
    "       --dataset_concatenation \\\n",
    "       --do_train \\\n",
    "       --report_to none \\\n",
    "       --use_habana \\\n",
    "       --use_lazy_mode \\\n",
    "       --throughput_warmup_steps 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca86c3a2-882a-42a6-aaba-4eb2ed968d42",
   "metadata": {},
   "source": [
    "Once the Fine Tuning is completed, you can move to the text-generation task to run inference and generate text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1304e527-436b-4547-be8d-4a0c3b26c28d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ~/Gaudi-tutorials/PyTorch/llama2_fine_tuning_inference/optimum-habana/examples/text-generation\n",
    "!pip install -q -r requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9bc04b3-be73-4b38-9f5a-6dba31a8efa8",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt7b = input(\"Enter a prompt for text generation: \")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ccd9612-ee3c-4642-a6f7-4f0a30013656",
   "metadata": {},
   "source": [
    "This is the output with the PEFT fine tuning added"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7115d1ea-8bab-4e29-8cef-f194b9c0c3c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "cmd = f'python run_generation.py --model_name_or_path meta-llama/Llama-2-7b-hf --batch_size 1 --do_sample --max_new_tokens 250 --n_iterations 4 --use_kv_cache --use_hpu_graphs \\\n",
    "--bf16 --prompt \"{prompt7b}\" --peft_model /root/Gaudi-tutorials/PyTorch/llama2_fine_tuning_inference/optimum-habana/examples/language-modeling/model_lora_llama_7b/'\n",
    "print(cmd)\n",
    "import os\n",
    "os.system(cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02fc3067-72fa-43c6-aa49-a092a2eb8049",
   "metadata": {},
   "source": [
    "This is the output without the PEFT fine tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64f603ec-952b-42a3-a4ab-e4edbe5ba1ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "cmd = f'python run_generation.py --model_name_or_path meta-llama/Llama-2-7b-hf --batch_size 1 --do_sample --max_new_tokens 250 --n_iterations 4 --use_kv_cache --use_hpu_graphs \\\n",
    "--bf16 --prompt \"{prompt7b}\" '\n",
    "print(cmd)\n",
    "import os\n",
    "os.system(cmd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef0c837d-cc23-4404-9069-b1fa66a341eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "exit()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}