🚀 Octet.Chat

This is a LLMs project implemented in Java. You can use it to deploy your own private services, supports the Llama3 and GPT models and other open-source models.

Provides

• llama-java-core Simple Java library
• octet-chat-app Octet desktop
• octet-chat-app API servers

Features

• 🦙 Built on llama.cpp
• 😊 Support AI Agent and implements Function calling based on Qwen-chat
• 🤖 Supports parallel inference, continuous conversation and text generation
• 📦 Supports the Llama3 and GPT models, such as Baichuan 7B,Qwen 7B

---

Last updated

...

• 🚀 Supported dynamic temperature sampling.
• 🚀 Added WebUI to octet-chat-app.
• 🚀 Updated API parameters.
• 🚀 Optimized chat formatter and Windows Cli.
• 🚀 Refactored function calls, Optimized chat formatter and APIs.

Quick start

Note

You can quantify the original model yourself or search for huggingface to obtain open-source models.

🖥 Octet desktop

💡 API servers

How to use

Set a custom AI character and Launch the app, open browser enjoy it now http://YOUR_IP_ADDR:8152/

# Default URL: http://YOUR_IP_ADDR:8152/

cd <YOUR_PATH>/octet-chat-app
bash app_server.sh start YOUR_CHARACTER

Tip

It can be integrated into your services, such as VsCode, App, Wechat, etc.

How to call API

Api docs: http://127.0.0.1:8152/swagger-ui.html

curl --location 'http://127.0.0.1:8152/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data '{
    "messages": [
        {
            "role": "USER",
            "content": "Who are you?"
        }
    ],
    "user": "User",
    "stream": true
}'

The API will return data in a stream format:

{
    "id": "octetchat-98fhd2dvj7",
    "model": "Llama2-chat",
    "created": 1695614393810,
    "choices": [
        {
            "index": 0,
            "delta": {
                "content": "Hi"
            },
            "finish_reason": "NONE"
        }
    ]
}

Documentation

Development

• 开发手册
• Development manual

Characters config

• Llama Java Parameter
• characters.template.json

Disclaimer

Important

• This project does not provide any models. Please obtain the model files yourself and comply with relevant agreements.
• Please do not use this project for illegal purposes, including but not limited to commercial use, profit-making use, or use that violates laws and regulations.
• Any legal liability arising from the use of this project shall be borne by the user, and this project shall not bear any legal liability.

Feedback

• If you have any questions, please submit them in GitHub Issue.