sarkar/cache_optimization

[ssarkar2]

What does this PR do?

PR-942 brought upto speed to main, with some fixes

test using:

python run_generation.py --model_name_or_path /mnt/weka/data/llama_inference/Llama-2-7b-hf --use_hpu_graphs --use_kv_cache --max_input_tokens 1024 --max_new_tokens 3072 --batch_size 39 --attn_softmax_bf16 --trim_logits --bf16 --warmup 2 --n_iterations 2 --limit_hpu_graphs --bucket_internal --bucket_size 128

note no --reuse_cache in above cmd

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[ssarkar2]

Needs a README change for llama, to mention --reuse_cache isnt needed

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.