Skip to content
/ NPU-Chat Public

Web chat front end for rk3588_npu_llm_server / RK3588 LLM chat interface

License

Notifications You must be signed in to change notification settings

av1d/NPU-Chat

Repository files navigation

NPU Chat

a Web UI for rk3588_npu_llm_server

Chat interface for LLM running on RK3588 NPU. Responsive design for desktop & mobile.

Recent updates:

2024-05-17

  • Added automated installer script to make installation much easier.

2024-05-16

  • New version: 0.27
  • New alternate UI theme availalble. Select it in settings.ini.
  • New visual UI enhancements (touched up bloated rounded styles), better user-input text formatting.
  • Front-end error handling in JS now alerts if any servers are offline.
  • New context command prints current context.
2024-05-16
  • New version: 0.26
  • Experimental: You can now (optionally) use chat contexts. Use the commands clear, off, on to manipulate the state.
  • Improved error handling for non-2xx HTTP status codes in case of error or server offline.
  • Added new parameters to settings.ini pertaining to contexts and error handling.
  • Code cleanup: Added docstrings and improved commenting for better code readability.

Screenshot 01

install:
You must first be running https://github.com/av1d/rk3588_npu_llm_server
then:
python3 -m pip install -r requirements.txt

configure:
Edit settings.ini with your information.

run:
python3 npuchat.py

then navigate to the IP/port in your browser.

Works in any browser besides Firefox desktop or mobile (but maybe nightly, with the correct settings).

Known issues:
Some models don't output valid markdown for everything. For example, if you ask Qwen show me a list of emoji, it will not be valid markdown.
The problem is it uses newline (\n) after each item in the emoji list, but we can't convert it to <br> without potentially destroying all code that passes through. It would be a bad idea. The problem is further complicated by markdown validators do not consider this emoji list to be invalid markdown, so we cannot apply conditional rules. If you have a solution I would love to hear it.

About

Web chat front end for rk3588_npu_llm_server / RK3588 LLM chat interface

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Sponsor this project

Packages

No packages published