README_EN.md

DevOps-Model

🤗 Hugging Face • 🤖 ModelScope

中文 | English

DevOps-Model is a series of industrial-fist Chinese DevOps large language models, mainly dedicated to exerting practical value in the field of DevOps. Currently, DevOps-Model can help engineers answer questions encountered in the all DevOps life cycle.

Based on the Qwen series of models, we output the Base model after additional training with high-quality Chinese DevOps corpus, and then output the Chat model after alignment with DevOps QA data. Our Base model and Chat model can achieve the best results among models of the same scale based on evaluation data related to the DevOps fields.

At the same time, we are also building an evaluation benchmark DevOpsEval exclusive to the DevOps field to better evaluate the effect of the DevOps field model.

Update

• [2023.10.31] Open source DevOps-Model-14B Base and Chat models.
• [2023.10.30] Open source DevOps-Model-7B Base and Chat models.

Download

Open source models and download links are shown in the table below: 🤗 Huggingface

	Base Model	Chat Model	Chat Model(Int4)
7B	Coming Soon	Coming Soon	Coming Soon
14B	Coming Soon	Coming Soon	Coming Soon

🤖 ModelScope

	Base Model	Chat Model	Chat Model(Int4)
7B	DevOps-Model-7B-Base	DevOps-Model-7B-Chat	Coming Soon
14B	DevOps-Model-14B-Base	DevOps-Model-14B-Chat	Coming Soon

Evaluation

We first selected a total of six exams related to DevOps in the two evaluation data sets of CMMLU and CEval. There are a total of 574 multiple-choice questions. The specific information is as follows:

Evaluation dataset	Exam subjects	Number of questions
CMMLU	Computer science	204
CMMLU	Computer security	171
CMMLU	Machine learning	122
CEval	College programming	37
CEval	Computer architecture	21
CEval	Computernetwork	19

We tested the results of Zero-shot and Five-shot respectively. Our 7B and 14B series models can achieve the best results among the tested models. More tests will be released later.

Base Model	Zero-shot Score	Five-shot Score
DevOps-Model-14B-Base	70.73	73.00
Qwen-14B-Base	69.16	71.25
Baichuan2-13B-Base	55.75	61.15
DevOps-Model-7B-Base	62.72	62.02
Qwen-7B-Base	55.75	56.00
Baichuan2-7B-Base	49.30	55.4
Internlm-7B-Base	47.56	52.6

Chat Model	Zero-shot Score	Five-shot Score
DevOps-Model-14B-Chat	74.04	75.96
Qwen-14B-Chat	69.16	70.03
Baichuan2-13B-Chat	52.79	55.23
DevOps-Model-7B-Chat	62.20	64.11
Qwen-7B-Chat	46.00	52.44
Baichuan2-7B-Chat	52.56	55.75
Internlm-7B-Chat	52.61	55.75

Quickstart

We provide simple examples to illustrate how to quickly use Devops-Model-Chat models with 🤗 Transformers.

Requirement

pip install -r requirements.txt

Chat Model Example

from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import GenerationConfig

tokenizer = AutoTokenizer.from_pretrained("path_to_DevOps-Model-Chat", trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained("path_to_DevOps-Model-Chat", device_map="auto", trust_remote_code=True, bf16=True).eval()

model.generation_config = GenerationConfig.from_pretrained("path_to_DevOps-Model-Chat", trust_remote_code=True)

resp2, hist2 = model.chat(query='What is the difference between HashMap and Hashtable in Java', tokenizer=tokenizer, history=hist)

Model Finetune

Data

The code internally reads data by calling datasets.load_dataset, and supports the data reading methods supported by load_dataset, such as json, csv, custom reading scripts, etc. (but it is recommended that the data be prepared in jsonl format files). Then you also need to update the data/dataset_info.json file. For details, please refer to data/README.md.

Pretrain

If you have collected a batch of documents and other corpus (such as company internal product documents) and want to train based on our model, you can execute scripts/devops-model-pt.sh to initiate an additional training to let the model learn The specific codes for the knowledge of this batch of documents are as follows:

set -v 

torchrun --nproc_per_node=8 --nnodes=$WORLD_SIZE --master_port=$MASTER_PORT --master_addr=$MASTER_ADDR --node_rank=$RANK src/train_bash.py \
    --deepspeed conf/deepspeed_config.json 
	--stage pt \
    --model_name_or_path path_to_model \
    --do_train \
    --report_to 'tensorboard' \
    --dataset your_corpus \
    --template default \
    --finetuning_type full \
    --output_dir path_to_output_checkpoint_path \
    --overwrite_cache \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --gradient_accumulation_steps 1 \
    --lr_scheduler_type cosine \
    --warmup_ratio 0.05 \
    --evaluation_strategy steps \
    --logging_steps 10 \
    --max_steps 1000 \
    --save_steps 1000 \
    --eval_steps 1000 \
    --learning_rate 5e-6 \
    --plot_loss \
    --max_source_length 2048 \
    --dataloader_num_workers 8 \
    --val_size 0.01 \
    --bf16 \
    --overwrite_output_dir

Users can adjust on this basis to initiate their own training. For more detailed configurations, it is recommended to obtain the complete parameter list through python src/train_bash.py -h.

Supervised Fine-Tuning

If you collect a batch of QA data and want to align it for devopspal, you can execute scripts/devops-model-sft.sh to initiate an additional training to align the model on the collected model. The specific code is as follows:

set -v 

torchrun --nproc_per_node=8 --nnodes=$WORLD_SIZE --master_port=$MASTER_PORT --master_addr=$MASTER_ADDR --node_rank=$RANK src/train_bash.py \
    --deepspeed conf/deepspeed_config.json \
    --stage sft \
    --model_name_or_path path_to_model \
    --do_train \
    --report_to 'tensorboard' \
    --dataset your_corpus \
    --template chatml \
    --finetuning_type full \
    --output_dir /mnt/llm/devopspal/model/trained \
    --overwrite_cache \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --gradient_accumulation_steps 1 \
    --lr_scheduler_type cosine \
    --warmup_ratio 0.05 \
    --evaluation_strategy steps \
    --logging_steps 10 \
    --max_steps 1000 \
    --save_steps 100 \
    --eval_steps 100 \
    --learning_rate 5e-5 \
    --plot_loss \
    --max_source_length 2048 \
    --dataloader_num_workers 8 \
    --val_size 0.01 \
    --bf16 \
    --overwrite_output_dir

Users can adjust on this basis to initiate their own SFT. For more detailed configurations, it is recommended to obtain the complete parameter list through python src/train_bash.py -h.

Quantilization

We will provide quantitative models of the DevOps-Model-Chat series. Of course, you can also quantify your own trained models through the following code

from transformers import AutoModelForCausalLM, AutoTokenizer
from optimum.gptq import GPTQQuantizer, load_quantized_model
import torch

model_name = "path_of_your_model"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16)

quantizer = GPTQQuantizer(bits=4, dataset="c4", block_name_to_quantize = "model.decoder.layers", model_seqlen = 2048)
quantized_model = quantizer.quantize_model(model, tokenizer)

out_dir = 'save_path_of_your_quantized_model'
quantized_model.save_quantized(out_dir)

Contact Us

Disclaimer

Due to the characteristics of language models, the content generated by the model may contain hallucinations or discriminatory remarks. Please use the content generated by the DevOps-Model family of models with caution. If you want to use this model service publicly or commercially, please note that the service provider needs to bear the responsibility for the adverse effects or harmful remarks caused by it. The developer of this project does not assume any responsibility for any consequences caused by the use of this project (including but not limited to data, models, codes, etc.) ) resulting in harm or loss.

Acknowledgments

This project refers to the following open source projects, and I would like to express my gratitude to the relevant projects and research and development personnel.

• LLaMA-Efficient-Tuning
• QwenLM