This is the repo for reproducing Stanford Alpaca : An Instruction-following LLaMA Model. We finetune some of LlaMa2-based large language model using medical QA dataset. The repo contains:
- The 5K data conversations between patients and physicians used for fine-tuning the model.
- The code for Preparation data.
- The code for Fine Tuning the Model.
- The link for Testing the Model.
We using the 5k generated dataset by Chat Doctor. The dataset is a generated conversations between patients and physicians from ChatGPT GenMedGPT-5k and disease database. Dataset also currated and modified to Indonesian Language Based.
GenMedGPT-5k-id.json
contains 5K instruction-following data we used for fine-tuning the LlaMa model. This JSON file is a list of dictionaries, each dictionary contains the following fields:
instruction
:str
, describes the task the model should perform. Each of the 52K instructions is unique.input
:str
, optional context or input for the task. For example, when the instruction is "Summarize the following article", the input is the article. Around 40% of the examples have an input.output
:str
, the answer to the instruction as generated bytext-davinci-003
.
If you're interested in fine-tuning with your own data, it's essential to adhere to the default prompt format that the model used during its pre-training phase. The prompt for LlaMa 2 is structured similarly to this:
<s>[INST] <<SYS>>
{{ instruction }}
<</SYS>>
{{ input }} [/INST] {{ output }} </s>
Meanwhile, the prompt for PolyLM and InternLM (adapted to Indonesian) is structured similarly to this:
Di bawah ini adalah instruksi yang menjelaskan tugas, dipasangkan dengan masukan yang memberikan konteks lebih lanjut. Tulis tanggapan yang melengkapi permintaan dengan tepat.
Instruksi:
{instruction}
Masukan:
{input}
Tanggapan:
{output}
We fine-tune our models based on the step from Stanford Alpaca. We choose to train some LLama-based model. The model that we finetune are PolyLM-1.7B, LlaMa-2-7B, InternLM-7B with the following hyperparameters:
Hyperparameter | PolyLM-1.7B | LLaMA-7B | InternLM-7B |
---|---|---|---|
Batch size | 128 | 128 | 128 |
Learning rate | 3e-4 | 3e-4 | 3e-4 |
Epochs | 3 | 3 | 3 |
Max length | 256 | 256 | 256 |
Weight decay | 0 | 0 | 0 |
To reproduce our fine-tuning runs for LLaMA, first install the requirements
pip install -r requirements.txt
The code for finetuning is available at fine-tuning.ipynb
with four sections of pre-preocessing data, fine-tuning with LlaMa 2, fine-tuning with PolyLM, and fine-tuning with InternLM.
These are link for test the fine-tuned model :
All interns below contributed equally and the order is determined by random draw.
All advised by Firqa Aqilla Noor Arasyi