diff --git a/.gitignore b/.gitignore index ab28c2e03..4f9359c78 100644 --- a/.gitignore +++ b/.gitignore @@ -292,4 +292,5 @@ XMLs logs wwwroot appsettings.Production.json -*.csproj.user \ No newline at end of file +*.csproj.user +env/ \ No newline at end of file diff --git a/docs/index.rst b/docs/index.rst index cc504b66b..b7c8b78ef 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -96,6 +96,15 @@ The main documentation for the site is organized into the following sections: llm/few-shot-learning llm/provider +.. _llamasharp: + +.. toctree:: + :maxdepth: 2 + :caption: Use Local LLM Models + + llama-sharp/config-llamasharp + llama-sharp/use-llamasharp-in-ui + .. _architecture-docs: .. toctree:: diff --git a/docs/llama-sharp/assets/check-llamasharp-version.png b/docs/llama-sharp/assets/check-llamasharp-version.png new file mode 100644 index 000000000..b44498ea2 Binary files /dev/null and b/docs/llama-sharp/assets/check-llamasharp-version.png differ diff --git a/docs/llama-sharp/assets/choose-llamasharp-as-provider.png b/docs/llama-sharp/assets/choose-llamasharp-as-provider.png new file mode 100644 index 000000000..e4177ba93 Binary files /dev/null and b/docs/llama-sharp/assets/choose-llamasharp-as-provider.png differ diff --git a/docs/llama-sharp/assets/click-test-button.png b/docs/llama-sharp/assets/click-test-button.png new file mode 100644 index 000000000..b5bb123d2 Binary files /dev/null and b/docs/llama-sharp/assets/click-test-button.png differ diff --git a/docs/llama-sharp/assets/console-output-in-botsharp.png b/docs/llama-sharp/assets/console-output-in-botsharp.png new file mode 100644 index 000000000..6860144dc Binary files /dev/null and b/docs/llama-sharp/assets/console-output-in-botsharp.png differ diff --git a/docs/llama-sharp/assets/converstaion-examples.png b/docs/llama-sharp/assets/converstaion-examples.png new file mode 100644 index 000000000..b119c61bb Binary files /dev/null and b/docs/llama-sharp/assets/converstaion-examples.png differ diff --git a/docs/llama-sharp/assets/edit-agent.png b/docs/llama-sharp/assets/edit-agent.png new file mode 100644 index 000000000..5ae0d986b Binary files /dev/null and b/docs/llama-sharp/assets/edit-agent.png differ diff --git a/docs/llama-sharp/assets/install-llamasharp-plugin.png b/docs/llama-sharp/assets/install-llamasharp-plugin.png new file mode 100644 index 000000000..eb7f208c9 Binary files /dev/null and b/docs/llama-sharp/assets/install-llamasharp-plugin.png differ diff --git a/docs/llama-sharp/config-llamasharp.md b/docs/llama-sharp/config-llamasharp.md new file mode 100644 index 000000000..a6c661837 --- /dev/null +++ b/docs/llama-sharp/config-llamasharp.md @@ -0,0 +1,60 @@ +# Config LLamaSharp + +BotSharp contains LLamaSharp plugin that allows you to run local llm models. To use the LLamaSharp, you need to config the BotSharp project with few steps. + +## Install LLamaSharp Backend + +Before use LLamaSharp plugin, you need to install one of the LLamaSharp backend services that suits your environment. + +- [`LLamaSharp.Backend.Cpu`](https://www.nuget.org/packages/LLamaSharp.Backend.Cpu): Pure CPU for Windows & Linux. Metal for Mac. +- [`LLamaSharp.Backend.Cuda11`](https://www.nuget.org/packages/LLamaSharp.Backend.Cuda11): CUDA 11 for Windows and Linux +- [`LLamaSharp.Backend.Cuda12`](https://www.nuget.org/packages/LLamaSharp.Backend.Cuda12): CUDA 12 for Windows and Linux + +**Please install the same version of LLamaSharp Backend with the LLamaSharp in BotSharp.Plugin.LLamaSharp.csproj.** + +![Check LLamaSharp Version](assets/check-llamasharp-version.png) + +```shell +# move to the LLamaSharp Plugin Project +$ cd src/Plugins/BotSharp.Plugin.LLamaSharp +# Install the LLamaSharp Backend +$ dotnet add package LLamaSharp.Backend.Cpu --version 0.9.1 +``` + +## Download and Config Local LLM Models + +LLamaSharp supports many LLM Models like LLaMA and Alpaca. Download the `gguf` format models and save them in your machine. + +We will use a [Llama 2](https://huggingface.co/TheBloke/llama-2-7B-Guanaco-QLoRA-GGUF) model in this tutorial. + +After downloading the model, open the `src/WebStarter/appsettings.json` file to config the LLamaSharp models. Set the `LlmProviders` and `LlamaSharp` fields to correct settings as your computer. For example: + +```json +{ + ..., + "LlmProviders": [ + ..., + { + "Provider": "llama-sharp", + "Models": [ + { + "Name": "llama-2-7b.Q2_K.gguf", + "Type": "chat" + } + ] + }, + ... + ], + ..., + "LlamaSharp": { + "Interactive": true, + "ModelDir": "/Users/wenwei/Desktop/LLM", + "DefaultModel": "llama-2-7b.Q2_K.gguf", + "MaxContextLength": 1024, + "NumberOfGpuLayer": 20 + }, + ... +} +``` + +For more details about LLamaSharp, visit [LLamaSharp - GitHub](https://github.com/SciSharp/LLamaSharp). diff --git a/docs/llama-sharp/use-llamasharp-in-ui.md b/docs/llama-sharp/use-llamasharp-in-ui.md new file mode 100644 index 000000000..9dd98e07c --- /dev/null +++ b/docs/llama-sharp/use-llamasharp-in-ui.md @@ -0,0 +1,29 @@ +# Use LLamaSharp in BotSharp + +Start the BotSharp backend and frontend services, and follow this tutorial. + +## Install LLamaSharp Plugin in UI. + +Go to the Plugin page and install LLamaSharp Plugin. + +![Install LlamaSharp Plugin](assets/install-llamasharp-plugin.png) + +## Config LLamaSharp as LLM Providers for Agents + +Edit or create an agent in Agents page, and config the agent. + +![Edit Agent](assets/edit-agent.png) + +In the edit page, config the provider as llama-sharp. + +![Choose LLamaSharp as Provider](assets/choose-llamasharp-as-provider.png) + +Then test the agent. + +![Click Test Agent Button](assets/click-test-button.png) + +![Test Agent Example](assets/converstaion-examples.png) + +If run successfully, you will see log like this in BotSharp service's console. + +![Console Output](assets/console-output-in-botsharp.png)