groq.mp4
This demonstrates how to use the Groq Python API Library to access the Groq LPU™ Inference Engine, which provides ultra-low latency inference for state-of-the-art large language models (LLMs) such as Llama-2 and Mixtral. The Groq LPU is a single-core unit based on the Tensor-Streaming Processor (TSP) architecture, which achieves 750 TOPS at INT8 and 188 TeraFLOPS at FP16.
The Groq Python API Library supports all request parameters and response fields, and offers both synchronous and asynchronous clients. It also provides a convenient wrapper for the Llama-2 model, which allows you to generate natural language text from a given prompt.
To install the Groq Python API Library, run the following command:
pip install groq