Quantized LLMs on Hailo-8L

[bkawakami]

Hello Hailo Team,

I recently acquired the Raspberry Pi AI Kit, which includes the Hailo-8L AI Accelerator. I am exploring the potential of running quantized large language models (LLMs) on this setup. Could you provide any guidance or examples of successfully implementing quantized LLMs with the Hailo-8L on a Raspberry Pi? Your assistance and any documentation on this would be greatly appreciated.

Thank you!