Added AWQ option to llm-chatbot notebook

[nikita-savelyevv]

Add an option to run AWQ algorithm during INT4 model compression in llm-chatbot and llm-rag-langchain notebooks. Applying AWQ slightly improves model generation quality, but requires significant amount of additional memory and time so it is disabled by default.

Some evaluation results are below. The wikitext task is considered as a more accurate one. If not stated explicitly, AWQ was calibrated on wikitext2 dataset.

Model	Compression	PPL on lambada-openai	PPL on wikitext	Compression Time
gemma-2b-it	FP16	8.23
gemma-2b-it	INT8_asym	8.36
gemma-2b-it	INT4_sym group size 64 ratio 60%	8.9		59 sec.
gemma-2b-it	INT4_sym group size 64 ratio 60% + AWQ	8.62		202 sec.
llama-2-chat-7b	FP16	3.26	11.6
llama-2-chat-7b	INT8_asym	3.27	11.6
llama-2-chat-7b	INT4_sym group size 128 ratio 80%	3.38	11.95	215 sec.
llama-2-chat-7b	INT4_sym group size 128 ratio 80% + AWQ (wikitext2)	3.44	11.88	768 sec.
llama-2-chat-7b	INT4_sym group size 128 ratio 80% + AWQ (ptb)	3.42	11.87
llama-3-8b-instruct	FP16	3.1
llama-3-8b-instruct	INT8_asym	3.08
llama-3-8b-instruct	INT4_sym group size 128 ratio 80%	3.38		242 sec.
llama-3-8b-instruct	INT4_sym group size 128 ratio 80% + AWQ	3.26		956 sec.

Ticket
141233

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[andreyanufr]

Maybe it’s worth writing on the web page exactly what precision of the model is used for chat.

[eaidova]

Please high-light note about unapplicability AWQ and could you please provide details what do you mean under skip (there will be some warning message, there is explicit configuration that skip it or something else?).

Also please provide details about used dataset

Reply via ReviewNB

[nikita-savelyevv]

Highlighted the note about skipping of the algorithm and added information on which dataset is used for calibration.

When AWQ is skipped, there will be an NNCF INFO level log message: "No matching patterns were found for applying AWQ algorithm, it will be skipped."