Skip to content
This repository was archived by the owner on Jul 4, 2025. It is now read-only.

sync dev to main 1.0.3-rc1 #1704

Merged
merged 4 commits into from
Nov 19, 2024
Merged

sync dev to main 1.0.3-rc1 #1704

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
640d66a
docs: add proxy docs
namchuai Nov 18, 2024
53cfc42
Merge pull request #1700 from janhq/j/proxy-docs
namchuai Nov 19, 2024
9ff93c5
feat: install script for linux (#1677)
hiento09 Nov 19, 2024
37fecbd
feat: API to configure huggingface_token (#1699)
vansangpfiev Nov 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 31 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,14 +56,18 @@ Cortex also has a [Network Installer](#network-installer) which downloads the ne

<h4>
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:15px; width: 15px" />
Linux:
<a href='https://app.cortexcpp.com/download/latest/linux-amd64-local'><b>cortex.deb</b></a> (Coming soon: Linux installation script)
Linux debian based distros:
<a href='https://app.cortexcpp.com/download/latest/linux-amd64-local'><b>cortex-linux-local-installer.deb</b></a>
</h4>

- For Linux: Download the installer and run the following command in terminal:

```bash
sudo apt install ./cortex-local-installer.deb
# Linux debian based distros
curl -s https://raw.githubusercontent.com/janhq/cortex/main/engine/templates/linux/install.sh | sudo bash -s -- --deb_local

# Other Linux distros
curl -s https://raw.githubusercontent.com/janhq/cortex/main/engine/templates/linux/install.sh | sudo bash -s
```

- The binary will be installed in the `/usr/bin/` directory.
Expand Down Expand Up @@ -149,6 +153,28 @@ Select a model (1-9):
```

## Advanced Installation

### Network Installer (Stable)

Cortex.cpp is available with a Network Installer, which is a smaller installer but requires internet connection during installation to download the necessary dependencies.

<h4>
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
Windows:
<a href='https://app.cortexcpp.com/download/latest/windows-amd64-local'><b>cortex-windows-network-installer.exe</b></a>
</h4>

<h4>
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
MacOS (Universal):
<a href='https://app.cortexcpp.com/download/latest/mac-universal-network'><b>cortex-mac-network-installer.pkg</b></a>
</h4>

<h4>
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 15px" />
Linux debian based distros:
<a href='https://app.cortexcpp.com/download/latest/linux-amd64-network'><b>cortex-linux-network-installer.deb</b></a>
</h4>

### Beta & Nightly Versions (Local Installer)

Expand All @@ -163,7 +189,7 @@ Cortex releases Beta and Nightly versions for advanced users to try new features
<td style="text-align:center"><b>Version</b></td>
<td style="text-align:center"><b>Windows</b></td>
<td style="text-align:center"><b>MacOS</b></td>
<td style="text-align:center"><b>Linux</b></td>
<td style="text-align:center"><b>Linux debian based distros</b></td>
</tr>
<tr style="text-align:center">
<td style="text-align:center"><b>Beta (Preview)</b></td>
Expand Down Expand Up @@ -218,7 +244,7 @@ Cortex.cpp is available with a Network Installer, which is a smaller installer b
<td style="text-align:center"><b>Version Type</b></td>
<td style="text-align:center"><b>Windows</b></td>
<td style="text-align:center"><b>MacOS</b></td>
<td style="text-align:center"><b>Linux</b></td>
<td style="text-align:center"><b>Linux debian based distros</b></td>
</tr>
<tr style="text-align:center">
<td style="text-align:center"><b>Stable (Recommended)</b></td>
Expand Down
1 change: 1 addition & 0 deletions docs/docs/configurations/index.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,6 @@ title: Cortex configurations
Welcome to the Cortex configurations documentation. Here you will find detailed guides and references for configuring various aspects of Cortex, including:

- **CORS**: Learn how to set up Cross-Origin Resource Sharing.
- **Proxy**: Configure the proxy for Cortex.

Use the sidebar to navigate through the different configuration topics.
207 changes: 207 additions & 0 deletions docs/docs/configurations/proxy.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
---
title: Proxy
description: Setting up Proxy
slug: "proxy"
---

import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem";

:::warning
🚧 Cortex.cpp is currently under development. Our documentation outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase.
:::

# Proxy Configuration Guide

This document describes how to configure proxy settings for Cortex to be able to work behind your proxy.

## Command Line Interface (CLI)

### Basic Usage

```bash
cortex config [OPTIONS] [COMMAND]
```

### Commands

- `status`: Display all current configurations

```bash
cortex config status
```

Example Output:

```bash
+-----------------------+------------------------+
| Config name | Value |
+-----------------------+------------------------+
| no_proxy | localhost,127.0.0.1 |
+-----------------------+------------------------+
| proxy_password | |
+-----------------------+------------------------+
| proxy_url | http://localhost:8080 |
+-----------------------+------------------------+
| proxy_username | |
+-----------------------+------------------------+
| verify_host_ssl | true |
+-----------------------+------------------------+
| verify_peer_ssl | false |
+-----------------------+------------------------+
| verify_proxy_host_ssl | true |
+-----------------------+------------------------+
| verify_proxy_ssl | true |
+-----------------------+------------------------+
```

### Options

| Option | Description | Example |
| ----------------------------------- | --------------------------- | ------------------------------------------------- |
| `-h, --help` | Print help message and exit |
| `--proxy_url <proxy_url>` | Set the proxy URL | `cortex config --proxy_url http://localhost:8080` |
| `--proxy_username <proxy_username>` | Set the username for proxy | `cortex config --proxy_username my_username` |
| `--proxy_password <proxy_password>` | Set the password for proxy | `cortex config --proxy_password my_password` |
| `--no_proxy <no_proxy>` | Set the no_proxy list | `cortex config --no_proxy localhost,127.0.0.1` |
| `--verify_proxy_ssl [on/off]` | Verify proxy SSL | `cortex config --verify_proxy_ssl on` |
| `--verify_proxy_host_ssl [on/off]` | Verify proxy host SSL | `cortex config --verify_proxy_host_ssl on` |
| `--verify_peer_ssl [on/off]` | Verify peer SSL | `cortex config --verify_peer_ssl off` |
| `--verify_host_ssl [on/off]` | Verify host SSL | `cortex config --verify_host_ssl on` |

## Proxy API Configuration

### Endpoints

#### Get Current Configuration

```http
GET /v1/configs
```

Retrieves the current configuration settings.

##### Response

```json
{
"allowed_origins": [
"http://localhost:39281",
"http://127.0.0.1:39281",
"http://0.0.0.0:39281"
],
"cors": true,
"no_proxy": "localhost,127.0.0.1",
"proxy_password": "",
"proxy_url": "http://localhost:8080",
"proxy_username": "",
"verify_host_ssl": true,
"verify_peer_ssl": false,
"verify_proxy_host_ssl": true,
"verify_proxy_ssl": true
}
```

#### Update Configuration

```http
PATCH /v1/configs
```

Updates proxy configuration settings.

##### Request Headers

```
Content-Type: application/json
```

##### Request Body

```json
{
"no_proxy": "localhost",
"proxy_url": "http://localhost:8080",
"proxy_username": "my_username",
"proxy_password": "my_password",
"verify_host_ssl": false,
"verify_peer_ssl": false,
"verify_proxy_host_ssl": false,
"verify_proxy_ssl": false
}
```

##### Parameters

| Field | Type | Description |
| ----------------------- | ------- | -------------------------------------------------------------------------------------- |
| `no_proxy` | string | List of origins which request do not need to go through a proxy. Comma separated value |
| `proxy_url` | string | Proxy URL |
| `proxy_username` | string | Username for proxy authentication |
| `proxy_password` | string | Password for proxy authentication |
| `verify_host_ssl` | boolean | Verify host SSL |
| `verify_peer_ssl` | boolean | Verify peer SSL |
| `verify_proxy_host_ssl` | boolean | Verify proxy host SSL |
| `verify_proxy_ssl` | boolean | Verify proxy SSL |

##### Response

```json
{
"config": {
"allowed_origins": [
"http://localhost:39281",
"http://127.0.0.1:39281",
"http://0.0.0.0:39281"
],
"cors": true,
"no_proxy": "localhost",
"proxy_password": "my_password",
"proxy_url": "http://localhost:8080",
"proxy_username": "my_username",
"verify_host_ssl": false,
"verify_peer_ssl": false,
"verify_proxy_host_ssl": false,
"verify_proxy_ssl": false
},
"message": "Configuration updated successfully"
}
```

## Testing proxy configuration

You can test your proxy configuration using [mitmproxy](https://docs.mitmproxy.org/stable). This guide is written on macOS, but you can use it on any other platform.

### Install mitmproxy

```bash
brew install mitmproxy
```

### Start mitmproxy

```bash
mitmproxy --set stream_large_bodies=1m
```

mitmproxy will start on port `8080`. After mitmproxy started, you can adding options by pressing `O`. mitmproxy will display an option screen. You can check their document to learn more about mitmproxy. But let's take a simple option for now by setting the `proxyauth` for our local proxy. Inside the option screen, search for `proxyauth` and hit enter. Then, type `username:password` and hit enter again. You will see your newly added option is red-colored.

### Configuring Cortex to use that proxy

Let's using CLI to configure Cortex to use that proxy.

```bash
cortex config --proxy_url http://localhost:8080 --proxy_username username --proxy_password password
```

### Testing the proxy

Now, let's test the proxy. If you are setting the username and password correctly (same with `proxyauth` in mitmproxy), you will see the request in mitmproxy. For example, command `cortex pull tinyllama` should be successfully and returns a list of selectable models. Also, you will see your request in mitmproxy CLI screen.

Let's try to use a wrong authentication for your proxy.

```bash
cortex config --proxy_password wrong_pw
```

Now, let's test the proxy again. You will see the request is failed and returns an error.
Loading
Loading