Local AI to use with for Synology AI and Synology Chat
-Future Goal is to make this a PC app or a docker app, but Help is needed before that will happen
Only tested on Windows 11 with python 3.11, builds on llama-ccp-python
Install
-
install visual studio community 2022 (I checked python development and C++ developement)
-
clone repository
-
create virtual envirement in folder
python -m venv venv
- activate virual envirement
venv/Scripts/activate
- install the requirements
pip install -r requirements.txt
-added cuda install directly into requirements, change to your prefered backend as needed
Setup
-
place your LLM in the model folder and copy that file name to the settings file (MODEL_FILENAME="model name.gguf")
-
setup a new bot in your synology chat app
-
copy the Token and the incoming URL to the settings file (SYNOCHAT_TOKEN="Token" and SYNOCHAT_WEBHOOK_URL="incoming url")
-Now is a good time to change the settings file defaults from what I used for testing on my lowend laptop
-
the outgoing URL in synology integration will be http://HOST_IP:HOST_PORT/SynologyLLM change HOST_IP and HOST_PORT to what it is on your local PC your running the model on
-
Use either synologyLLM.bat file or command
python synologyLLM.py
Features
-
Loads any llama.cpp model that is supported
-
To see list of commands
/commands
- To Reset your conversation and stored setting
/reset
- To change system prompt
/system {new system prompt message}
- to change how many chat turns is stored in system prompt
/chat_turns {number of turns}
- To toggle thinking if using a thinking model
/think {true|false}
-
Uses a Chat message queue system
-
Added tool usage and some sample tools in the tools.py file
-tools included are wiki_tool, ddg_tool, weather_tool, time_tool, news_tool
- Added multiple user capability, it keeps track of all the indivual users settings persistantly
To use with synology ai console
-
Select OpenAI as the provider
-
put in the API key used in settings HOST_API_SECRET
-default is sk-xxx
- Under advanced settins use this address
- http://Host IP address : host IP port
- change the values to what you are using