How can I configure the parameters of llama

[xiangxinhello]

How can I configure the parameters to make llama-cpp-python faster? I have done some tests and found that the inference speed is about the same as the original model from Hugging Face.
How to configure llama parameters to maximize inference speed. Thanks
qwen1.5-7b model
I use a message
llm = Llama(
model_path="/llama.cpp/ggml-qwen-chat-f16.gguf",
n_gpu_layers=-1,
chat_format="qwen",
logits_all=False
verbose=False
)

start_time = time.time()
x = llm.create_chat_completion(
messages=messages,
top_p=1.0,
top_k=50,
temperature=1.0,
max_tokens=512
)
end_time = time.time()
elapsed_time = end_time - start_time
print(f"generate time: {elapsed_time:.6f} s")
llama-cpp-python need 5.1s

#huggingface qwen1.5-7b model
I use a same message
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
start_time = time.time()
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(device)
generated_ids = model.generate(
**model_inputs,
max_new_tokens=512
)
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
end_time = time.time()
elapsed_time = end_time - start_time
print(f"generate time: {elapsed_time:.6f} s")
huggingface qwen1.5-7b model need 6.5s

[abetlen]

@xiangxinhello the main things you can do here are using a quantized version of the model (f16 inference is expected to be the same as pytorch as both are memory bound) I would recommend starting with Q8 as this has roughly the same accuracy as f16 with a decent speedup / memory footprint reduction.

Additionally you should enable flash attention with flash_attn=True if it's supported by the model / your machine.

[xiangxinhello]

@abetlen I have a question:
I've noticed that the GPU utilization is very low during model inference, with a maximum of only 80%, but I want to increase the GPU utilization to 99%. How can I adjust the parameters?
GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A100-PCI... Off | 00000000:8A:00.0 Off | 0 |
| N/A 66C P0 205W / 250W | 14807MiB / 40960MiB | 78% Default

    N_THREADS = multiprocessing.cpu_count()
    self.runner = Llama(
        model_path=self.model_name,
        n_gpu_layers=-1,
        chat_format=self.generating_args["chat_format"],
        tokenizer=self.llama_tokenizer,
        flash_attn=True,
        verbose=False,
        n_ctx=1024,
        n_threads=N_THREADS // 2,
        n_threads_batch=N_THREADS
    )
    x = runner.create_chat_completion(
        messages=messages,
        top_p=0.0,
        top_k=1,
        temperature=1,
        max_tokens=512,
        seed=1337
    )

[yhyu13]

Here it is FYI https://llama-cpp-python.readthedocs.io/en/latest/api-reference/