You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Jul 4, 2025. It is now read-only.
> ⚠️ **Cortex is currently in Development**: Expect breaking changes and bugs!
12
12
13
13
## About
14
-
Cortex is an openAI-compatible local AI server that developers can use to build LLM apps. It is packaged with a Docker-inspired command-line interface and a Typescript client library. It can be used as a standalone server, or imported as a library.
14
+
Cortex is an OpenAI-compatible AI engine that developers can use to build LLM apps. It is packaged with a Docker-inspired command-line interface and client libraries. It can be used as a standalone server or imported as a library.
15
15
16
-
Cortex currently supports two inference engines:
16
+
Cortex currently supports 3 inference engines:
17
17
18
18
- Llama.cpp
19
+
- ONNX Runtime
19
20
- TensorRT-LLM
20
21
21
-
> Read more about Cortex at https://jan.ai/cortex
22
-
23
22
## Quicklinks
24
-
**Cortex**:
25
-
-[Website](https://jan.ai/)
26
-
-[GitHub](https://github.com/janhq/cortex)
27
-
-[User Guides](https://jan.ai/cortex)
28
-
-[API reference](https://jan.ai/api-reference)
29
-
30
-
## Prerequisites
31
-
32
-
### **Dependencies**
33
-
34
-
Before installation, ensure that you have installed the following:
35
-
36
-
-**Node.js**: version 18 and above is required to run the installation.
37
-
-**NPM**: Needed to manage packages.
38
-
-**CPU Instruction Sets**: Available for download from the [Cortex GitHub Releases](https://github.com/janhq/cortex/releases) page.
39
-
40
-
41
-
>💡 The **CPU instruction sets** are not required for the initial installation of Cortex. This dependency will be automatically installed during the Cortex initialization if they are not already on your system.
42
23
43
-
44
-
### **Hardware**
45
-
46
-
Ensure that your system meets the following requirements to run Cortex:
47
-
48
-
-**OS**:
49
-
- MacOSX 13.6 or higher.
50
-
- Windows 10 or higher.
51
-
- Ubuntu 12.04 and later.
52
-
-**RAM (CPU Mode):**
53
-
- 8GB for running up to 3B models.
54
-
- 16GB for running up to 7B models.
55
-
- 32GB for running up to 13B models.
56
-
-**VRAM (GPU Mode):**
57
-
58
-
- 6GB can load the 3B model (int4) with `ngl` at 120 ~ full speed on CPU/ GPU.
59
-
- 8GB can load the 7B model (int4) with `ngl` at 120 ~ full speed on CPU/ GPU.
60
-
- 12GB can load the 13B model (int4) with `ngl` at 120 ~ full speed on CPU/ GPU.
61
-
62
-
-**Disk**: At least 10GB for app and model download.
24
+
-[Homepage](https://cortex.jan.ai/)
25
+
-[Docs](https://cortex.jan.ai/docs/)
63
26
64
27
## Quickstart
65
-
To install Cortex CLI, follow the steps below:
66
-
1. Install the Cortex NPM package globally:
67
-
```bash
68
-
npm i -g @janhq/cortex
69
-
```
70
-
> Cortex automatically detects your CPU and GPU, downloading the appropriate CPU instruction sets and required dependencies to optimize GPU performance.
71
-
2. Download a GGUF model from Hugging Face:
72
-
```bash
73
-
# Pull a model most compatible with your hardware
74
-
cortex pull llama3
75
-
76
-
# Pull a specific variant with `repo_name:branch`
77
-
cortex pull llama3:7b
78
28
79
-
# Pull a model with the HuggingFace `model_id`
80
-
cortex pull microsoft/Phi-3-mini-4k-instruct-gguf
81
-
```
82
-
3. Load the model:
83
-
```bash
84
-
cortex models start llama3:7b
85
-
```
29
+
Visit [Quickstart](https://cortex.jan.ai/docs/quickstart) to get started.
0 commit comments