Nicol is an open-source web service, developed using the Kotlin programming language,
that enables streaming Server Stream Events and supports CPU inference for the port of llama2.c model.
- 🤖 Local Model: A Kotlin port of llama2.c by @karpathy .
- 🎢 Server Stream Event: Implemented a streaming inference interface similar to OpenAI's style .
- 🔒 Authorization: Provides basic interface token validation functionality based on the Spring Framework.
| Showcasing in Web Demo | Server Stream Event Restful Api |
|---|---|
 |
 |
Note
中国大陆 Ip 请保证网络能够正常访问 Github && Hugging face.
China Ip should make sure your network can access Github && Hugging face properly.
git clone https://github.com/WackyGem/Nicol.git
cd Nicol
docker-compose up// web demo
http://127.0.0.1:8005/
// openapi
http://127.0.0.1:9005/swagger-ui/index.html
- 🤖 llama2.c : Inference a baby Llama 2 model in pure C .
- 💡 jimmer : A revolutionary ORM framework for both java and kotlin, and a complete integrated solution.