feat: Openvino runtime for transformer backend and streaming support for Openvino and CUDA

[fakezeta]

Description

This PR add OpenVINO Runtime through Optimum Intel with acceleration for CPU (Intel and ARM)/iGPU/dGPU/NPU
Detailed list of supported Hardware from OpenVINO site.

Dynamic quantization and automatic conversion of Transformers models is not implemented because is slow so the model shall be converted and quantized first: instruction in Optimum Documentation.

I leave here two model for testing, one with 4bit quantization and another with 8bit.

• https://huggingface.co/fakezeta/openchat-3.5-0106-openvino-int8
• https://huggingface.co/fakezeta/openchat-3.5-0106-openvino-int4

Performance on iGPU UHD 770 (32 EU) with int8 quantized model above is >10tk/s.

Notes for Reviewers
Regression tested properly for CUDA, could not perform extensive testing on XPU due to missing hardware.

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai canceled.

Name	Link
🔨 Latest commit	afa088d
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/660344c9440e8a0008c70d54

[fakezeta]

Example of model definition for Openchat
OpenVINO Runtime is selected with type: OVModelForCausalLM

backend: transformers
parameters:
  model: fakezeta/openchat-3.5-0106-openvino-int8
context_size: 8192
threads: 6
f16: true
type: OVModelForCausalLM
stopwords:
- <|end_of_turn|>
- <|endoftext|>
prompt_cache_path: "cache"
prompt_cache_all: true
template:
  chat_message: |
    {{if eq .RoleName "system"}}{{.Content}}<|end_of_turn|>{{end}}{{if eq .RoleName "assistant"}}<|end_of_turn|>GPT4 Correct Assistant: {{.Content}}<|end_of_turn|>{{end}}{{if eq .RoleName "user"}}GPT4 Correct User: {{.Content}}{{end}}

  chat: |
    {{.Input}}<|end_of_turn|>GPT4 Correct Assistant:

  completion: |
    {{.Input}}

[mudler]

Great add, thanks @fakezeta !

[M0Rf30]

whisper.cpp also support -DWHISPER_OPENVINO=on