You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Jul 4, 2025. It is now read-only.
> ⚠️ **Cortex is currently in Development**: Expect breaking changes and bugs!
12
12
13
13
## About
14
-
Cortex is an openAI-compatible local AI server that developers can use to build LLM apps. It is packaged with a Docker-inspired command-line interface and a Typescript client library. It can be used as a standalone server, or imported as a library.
14
+
Cortex is an OpenAI-compatible AI engine that developers can use to build LLM apps. It is packaged with a Docker-inspired command-line interface and client libraries. It can be used as a standalone server or imported as a library.
15
15
16
-
Cortex currently supports two inference engines:
16
+
Cortex currently supports 3 inference engines:
17
17
18
18
- Llama.cpp
19
+
- ONNX Runtime
19
20
- TensorRT-LLM
20
21
21
-
> Read more about Cortex at https://jan.ai/cortex
22
-
23
22
## Quicklinks
24
-
Cortex
25
-
-[Website](https://jan.ai/)
26
-
-[GitHub](https://github.com/janhq/cortex)
27
-
-[User Guides](https://jan.ai/cortex)
28
-
-[API reference](https://jan.ai/api-reference)
29
-
30
-
## Prerequisites
31
-
32
-
### **Dependencies**
33
-
34
-
Before installation, ensure that you have installed the following:
35
-
-**Node.js**: version 18 and above is required to run the installation.
36
-
-**NPM**: Needed to manage packages.
37
-
-**CPU Instruction Sets**: Available for download from the [Cortex GitHub Releases](https://github.com/janhq/cortex/releases) page.
38
-
39
-
40
-
>💡 The **CPU instruction sets** are not required for the initial installation of Cortex. This dependency will be automatically installed during the Cortex initialization if they are not already on your system.
41
-
42
23
43
-
### **Hardware**
44
-
45
-
Ensure that your system meets the following requirements to run Cortex:
46
-
47
-
-**OS**:
48
-
- MacOSX 13.6 or higher.
49
-
- Windows 10 or higher.
50
-
- Ubuntu 22.04 and later.
51
-
-**RAM (CPU Mode):**
52
-
- 8GB for running up to 3B models.
53
-
- 16GB for running up to 7B models.
54
-
- 32GB for running up to 13B models.
55
-
-**VRAM (GPU Mode):**
56
-
57
-
- 6GB can load the 3B model (int4) with `ngl` at 120 ~ full speed on CPU/ GPU.
58
-
- 8GB can load the 7B model (int4) with `ngl` at 120 ~ full speed on CPU/ GPU.
59
-
- 12GB can load the 13B model (int4) with `ngl` at 120 ~ full speed on CPU/ GPU.
60
-
61
-
-**Disk**: At least 10GB for app and model download.
24
+
-[Homepage](https://cortex.jan.ai/)
25
+
-[Docs](https://cortex.jan.ai/docs/)
62
26
63
27
## Quickstart
64
-
To install Cortex CLI, follow the steps below:
65
-
1. Install the Cortex NPM package globally:
66
-
```bash
67
-
npm i -g cortexso
68
-
```
69
-
> Cortex automatically detects your CPU and GPU, downloading the appropriate CPU instruction sets and required dependencies to optimize GPU performance.
70
28
71
-
2. Download a GGUF model from Hugging Face:
72
-
```bash
73
-
# Pull a model most compatible with your hardware
74
-
cortex pull llama3
75
-
76
-
# Pull a specific variant with `repo_name:branch`
77
-
cortex pull llama3:7b
29
+
Visit [Quickstart](https://cortex.jan.ai/docs/quickstart) to get started.
78
30
79
-
# Pull a model with the HuggingFace `model_id`
80
-
cortex pull microsoft/Phi-3-mini-4k-instruct-gguf
81
-
```
82
-
3. Load the model:
83
31
```bash
84
-
cortex models start llama3:7b
32
+
npm i -g @janhq/cortex
33
+
cortex run llama3
85
34
```
86
-
87
-
4. Start chatting with the model:
88
-
```bash
89
-
cortex chat tell me a joke
90
-
```
91
-
92
-
93
-
## Run as an API server
94
35
To run Cortex as an API server:
95
36
```bash
96
37
cortex serve
38
+
39
+
# Output
40
+
# Started server at http://localhost:1337
41
+
# Swagger UI available at http://localhost:1337/api
97
42
```
98
43
44
+
You can now access the Cortex API server at `http://localhost:1337`,
45
+
and the Swagger UI at `http://localhost:1337/api`.
46
+
99
47
## Build from Source
100
48
101
49
To install Cortex from the source, follow the steps below:
0 commit comments