Skip to content
This repository was archived by the owner on Jul 4, 2025. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions docs/docs/demos/chatbox-vid.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
title: Run local chatbox under 1 minute on MacOS with Nitro
---

<iframe width="675" height="380" src="https://www.youtube.com/embed/apfI13ijvjc" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

## Links

- [Download Nitro](https://github.com/janhq/nitro/releases)
- [Download Chatbox](https://github.com/Bin-Huang/chatbox)

## Commands

```bash title="Load model"
curl http://localhost:3928/inferences/llamacpp/loadmodel \
-H 'Content-Type: application/json' \
-d '{
"llama_model_path": "model/llama-2-7b-chat.Q5_K_M.gguf",
"ctx_len": 512,
"ngl": 100,
}'
```

For more information, please refer to the [Nitro with Chatbox](examples/chatbox.md) documentation.
24 changes: 18 additions & 6 deletions docs/docs/examples/chatbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,25 @@ To download and install Chatbox, follow the instructions available at this [link

## Using Nitro as a Backend

1. Start Nitro server
**1. Start Nitro server**

Open your command line tool and enter:
```
nitro
```

> Ensure you are using the latest version of [Nitro](new/install.md)
**2. Download Model**

2. Run the Model
Use these commands to download and save the [Llama2 7B chat model](https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/tree/main):

```bash
mkdir model && cd model
wget -O llama-2-7b-model.gguf https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q5_K_M.gguf?download=true
```

> For more GGUF model, please look at [The Bloke](https://huggingface.co/TheBloke).

**3. Run the Model**

To load the model, use the following command:

Expand All @@ -39,13 +48,16 @@ curl http://localhost:3928/inferences/llamacpp/loadmodel \
}'
```

3. Config chatbox
**4. Config chatbox**

Adjust the `settings` in Chatbox to connect with Nitro. Change your settings to match the configuration shown in the image below:

![Settings](img/chatbox.PNG)

4. Chat with the Model
**5. Chat with the Model**

Once the setup is complete, you can start chatting with the model using Chatbox. All functions of Chatbox are now enabled with Nitro as the backend.

## Video demo
## Futher Usage

For convenient usage, you can utilize [Jan](https://jan.ai/), as it is integrated with Nitro.
18 changes: 18 additions & 0 deletions docs/docs/examples/jan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
title: Nitro with Jan
---

You can effortlessly utilize Nitro through [Jan](https://jan.ai/), as it is fully integrated with all its functions. With Jan, using Nitro becomes straightforward without the need for any coding.


<iframe width="675" height="380" src="https://www.youtube.com/embed/hw9w8_54OIc" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

## What is Jan?

Jan is a ChatGPT-alternative that runs on your own computer, with a local API server.

Jan uses open-source AI models, stores data in open file formats, is highly customizable via extensions.

For additional details, please consult the [Jan Documenation](https://jan.ai/docs).

> [Download Jan](https://jan.ai/)
54 changes: 0 additions & 54 deletions docs/docs/examples/llm.md

This file was deleted.

2 changes: 1 addition & 1 deletion docs/docs/new/about.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: About Nitro
slug: /about
slug: /docs
---

Nitro is a high-efficiency C++ inference engine for edge computing, powering [Jan](https://jan.ai/). It is lightweight and embeddable, ideal for product integration.
Expand Down
4 changes: 2 additions & 2 deletions docs/docs/new/build-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,13 +72,13 @@ Time to build Nitro!
- **On Linux:**

```bash
make -j $(%NUMBER_OF_PROCESSORS%)
make -j $(nproc)
```

- **On Windows:**

```bash
cmake --build . --config Release
make -j $(%NUMBER_OF_PROCESSORS%)
```

## Start process
Expand Down
23 changes: 21 additions & 2 deletions docs/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,19 @@ const sidebars = {
{
type: "category",
label: "Features",
collapsible: false,
collapsed: false,
link: { type: "doc", id: "features/feat" },
items: [
"features/chat",
"features/embed",
],
},
{
type: "category",
label: "Advanced Features",
link: { type: "doc", id: "features/feat" },
items: [
"features/multi-thread",
"features/cont-batch",
"features/load-unload",
Expand All @@ -46,10 +55,11 @@ const sidebars = {
},
{
type: "category",
label: "Guides",
label: "Integrations",
collapsible: false,
collapsed: false,
items: [
"examples/jan",
"examples/chatbox",
"examples/openai-node",
"examples/openai-python",
Expand All @@ -62,7 +72,16 @@ const sidebars = {
// collapsed: false,
// items: [{ type: "doc", id: "new/architecture", label: "Architecture" }],
// },
"new/faq"
{
type: "category",
label: "Demos",
collapsible: true,
collapsed: true,
items: [
"demos/chatbox-vid",
],
},
"new/faq",
],

apiSidebar: [
Expand Down