From 14dddee7b3a5743f11e33efb4d35ff27a77977fd Mon Sep 17 00:00:00 2001 From: irfanpena Date: Thu, 30 May 2024 11:20:59 +0700 Subject: [PATCH 01/11] docs: update the README.md --- README.md | 145 +++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 126 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index e6b453e98..ab27b6c19 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,69 @@ -# Cortex Monorepo +# Cortex - CLI -# Installation +

+ Documentation - API Reference + - Changelog - Bug reports - Discord +

-## Prerequisites +> ⚠️ **Cortex is currently in Development**: Expect breaking changes and bugs! -### **Dependencies** +## About +Cortex is an openAI-compatible local AI server that developers can use to build LLM apps. It is packaged with a Docker-inspired command-line interface and a Typescript client library. It can be used as a standalone server, or imported as a library. + +Cortex currently supports two inference engines: + +- Llama.cpp +- TensorRT-LLM + +> Read more about Cortex at https://jan.ai/cortex + +### Repo Structure +``` +# Entity Definitions +domain/ # This is the core directory where the domains are defined. + abstracts/ # Abstract base classes for common attributes and methods. + models/ # Domain interface definitions, e.g. model, assistant. + repositories/ # Extensions abstract and interface + +# Business Rules +usecases/ # Application logic + assistants/ # CRUD logic (invokes dtos, entities). + chat/ # Logic for chat functionalities. + models/ # Logic for model operations. + +# Adapters & Implementations +infrastructure/ # Implementations for Cortex interactions + commanders/ # CLI handlers + models/ + questions/ # CLI installation UX + shortcuts/ # CLI chained syntax + types/ + usecases/ # Invokes UseCases + + controllers/ # Nest controllers and HTTP routes + assistants/ # Invokes UseCases + chat/ # Invokes UseCases + models/ # Invokes UseCases + + database/ # Database providers (mysql, sqlite) + + # Framework specific object definitions + dtos/ # DTO definitions (data transfer & validation) + entities/ # TypeORM entity definitions (db schema) + + # Providers + providers/cortex # Cortex [server] provider (a core extension) + repositories/extensions # Extension provider (core & external extensions) + +extensions/ # External extensions +command.module.ts # CLI Commands List +main.ts # Entrypoint +``` +## Installation + +### Prerequisites + +#### **Dependencies** Before installation, ensure that you have installed the following: @@ -17,7 +76,7 @@ Before installation, ensure that you have installed the following: -### **Hardware** +#### **Hardware** Ensure that your system meets the following requirements to run Cortex: @@ -37,11 +96,11 @@ Ensure that your system meets the following requirements to run Cortex: - **Disk**: At least 10GB for app and model download. -## Cortex Installation +### Cortex Installation To install Cortex, follow the steps below: -### Step 1: Install Cortex +#### Step 1: Install Cortex Run the following command to install Cortex globally on your machine: @@ -50,7 +109,7 @@ Run the following command to install Cortex globally on your machine: npm i -g @janhq/cortex ``` -### Step 2: Verify the Installation +#### Step 2: Verify the Installation After installation, you can verify that Cortex is installed correctly by getting help information. @@ -59,7 +118,7 @@ After installation, you can verify that Cortex is installed correctly by getting cortex -h ``` -### Step 3: Initialize Cortex +#### Step 3: Initialize Cortex Once verified, you need to initialize the Cortex engine. @@ -69,7 +128,7 @@ Once verified, you need to initialize the Cortex engine. cortex init ``` -1. Select between `CPU` and `GPU` modes. +2. Select between `CPU` and `GPU` modes. ```bash ? Select run mode (Use arrow keys) @@ -77,7 +136,7 @@ cortex init GPU ``` -2. Select between GPU types. +3. Select between GPU types. ```bash ? Select GPU types (Use arrow keys) @@ -85,7 +144,7 @@ cortex init Others (Vulkan) ``` -3. Select CPU instructions (will be deprecated soon). +4. Select CPU instructions (will be deprecated soon). ```bash ? Select CPU instructions (Use arrow keys) @@ -94,10 +153,10 @@ cortex init AVX-512 ``` -1. Cortex will download the required CPU instruction sets if you choose `CPU` mode. If you choose `GPU` mode, Cortex will download the necessary dependencies to use your GPU. -2. Once downloaded, Cortex is ready to use! +5. Cortex will download the required CPU instruction sets if you choose `CPU` mode. If you choose `GPU` mode, Cortex will download the necessary dependencies to use your GPU. +6. Once downloaded, Cortex is ready to use! -### Step 4: Pull a model +#### Step 4: Pull a model From HuggingFace @@ -111,14 +170,14 @@ From Jan Hub (TBD) cortex pull llama3 ``` -### Step 5: Chat +#### Step 5: Chat ```bash cortex run janhq/phi-3-medium-128k-instruct-GGUF ``` ## Run as an API server - +To run Cortex as an API server: ```bash cortex serve ``` @@ -135,18 +194,62 @@ To install Cortex from the source, follow the steps below: npx nest build ``` -1. Make the `command.js` executable: +4. Make the `command.js` executable: ```bash chmod +x '[path-to]/cortex/cortex-js/dist/src/command.js' ``` -1. Link the package globally: +5. Link the package globally: ```bash npm link ``` +## Cortex CLI Command +The following CLI commands are currently available: +> ⚠️ **Cortex is currently in Development**: More commands will be added soon! + +```bash +cortex --help +Usage: cortex + +Commands: + chat Send a query to the chat service. + Example: cortex chat "tell me a joke" --stream + + models list List all available models. + Example: cortex models list + + models pull Download a specified model. + Example: cortex models pull llama3:8b + + models remove Delete a specified model. + Example: cortex models remove llama3:8b + + models get Retrieve the configuration of a specified model. + Example: cortex models config llama3:8b + + models start Start a specified model. + Example: cortex models start llama3:8b + + models stop Stop a specified model. + Example: cortex models stop llama3:8b + + models update Update the configuration of a specified model. + Example: cortex models update llama3:8b --ngl 32 + + engines Execute a specified command related to engines. + Example: cortex engines llamacpp + + engines list List all available engines. + Example: cortex engines list + +Options: + -h, --help Show this help message and exit. + + +``` ## Uninstall Cortex Run the following command to uninstall Cortex globally on your machine: @@ -155,3 +258,7 @@ Run the following command to uninstall Cortex globally on your machine: # Uninstall globally using NPM npm uninstall -g @janhq/cortex ``` +## Contact Support +- For support, please file a GitHub ticket. +- For questions, join our Discord [here](https://discord.gg/FTk2MvZwJH). +- For long-form inquiries, please email [hello@jan.ai](mailto:hello@jan.ai). \ No newline at end of file From cda53d755a5384250ba932e7df1260c259fb82a9 Mon Sep 17 00:00:00 2001 From: irfanpena Date: Thu, 30 May 2024 13:36:41 +0700 Subject: [PATCH 02/11] docs: Update the Cortex and Cortex-cpp readme files --- README.md | 136 +++++++++++-------------------------------- cortex-cpp/README.md | 71 +++++++++++----------- 2 files changed, 70 insertions(+), 137 deletions(-) diff --git a/README.md b/README.md index ab27b6c19..707599773 100644 --- a/README.md +++ b/README.md @@ -59,11 +59,16 @@ extensions/ # External extensions command.module.ts # CLI Commands List main.ts # Entrypoint ``` -## Installation +## Quicklinks +Cortex +- [Website](https://jan.ai/) +- [GitHub](https://github.com/janhq/cortex) +- [User Guides](https://jan.ai/cortex) +- [API reference](https://jan.ai/api-reference) -### Prerequisites +## Prerequisites -#### **Dependencies** +### **Dependencies** Before installation, ensure that you have installed the following: @@ -71,12 +76,11 @@ Before installation, ensure that you have installed the following: - **NPM**: Needed to manage packages. - **CPU Instruction Sets**: Available for download from the [Cortex GitHub Releases](https://github.com/janhq/cortex/releases) page. - +>💡 The **CPU instruction sets** are not required for the initial installation of Cortex. This dependency will be automatically installed during the Cortex initialization if they are not already on your system. -#### **Hardware** + +### **Hardware** Ensure that your system meets the following requirements to run Cortex: @@ -96,85 +100,31 @@ Ensure that your system meets the following requirements to run Cortex: - **Disk**: At least 10GB for app and model download. -### Cortex Installation - -To install Cortex, follow the steps below: - -#### Step 1: Install Cortex - -Run the following command to install Cortex globally on your machine: - -```bash -# Install using NPM globally +## Quickstart +1. Install the NPM package: +``` bash npm i -g @janhq/cortex ``` -#### Step 2: Verify the Installation - -After installation, you can verify that Cortex is installed correctly by getting help information. - -```bash -# Get the help information -cortex -h -``` - -#### Step 3: Initialize Cortex - -Once verified, you need to initialize the Cortex engine. - -1. Initialize the Cortex engine: - -``` +2. Initialize a compatible engine: +``` bash cortex init ``` -2. Select between `CPU` and `GPU` modes. - -```bash -? Select run mode (Use arrow keys) -> CPU - GPU +3. Download a GGUF model from Hugging Face +``` bash +cortex models pull janhq/TinyLlama-1.1B-Chat-v1.0-GGUF ``` - -3. Select between GPU types. - -```bash -? Select GPU types (Use arrow keys) -> Nvidia - Others (Vulkan) +4. Load the model +``` bash +cortex models start janhq/TinyLlama-1.1B-Chat-v1.0-GGUF ``` -4. Select CPU instructions (will be deprecated soon). - -```bash -? Select CPU instructions (Use arrow keys) -> AVX2 - AVX - AVX-512 +5. Start chatting with the model +``` bash +cortex chat tell me a joke ``` -5. Cortex will download the required CPU instruction sets if you choose `CPU` mode. If you choose `GPU` mode, Cortex will download the necessary dependencies to use your GPU. -6. Once downloaded, Cortex is ready to use! - -#### Step 4: Pull a model - -From HuggingFace - -```bash -cortex pull janhq/phi-3-medium-128k-instruct-GGUF -``` - -From Jan Hub (TBD) - -```bash -cortex pull llama3 -``` - -#### Step 5: Chat - -```bash -cortex run janhq/phi-3-medium-128k-instruct-GGUF -``` ## Run as an API server To run Cortex as an API server: @@ -211,44 +161,24 @@ The following CLI commands are currently available: > ⚠️ **Cortex is currently in Development**: More commands will be added soon! ```bash -cortex --help -Usage: cortex - -Commands: - chat Send a query to the chat service. - Example: cortex chat "tell me a joke" --stream + serve Providing API endpoint for Cortex backend + chat Send a chat request to a model + init|setup Init settings and download cortex's dependencies + ps Show running models and their status + kill Kill running cortex processes + pull|download Download a model. Working with HuggingFace model id. + run [options] EXPERIMENTAL: Shortcut to start a model and chat + models Subcommands for managing models models list List all available models. - Example: cortex models list - models pull Download a specified model. - Example: cortex models pull llama3:8b - models remove Delete a specified model. - Example: cortex models remove llama3:8b - models get Retrieve the configuration of a specified model. - Example: cortex models config llama3:8b - models start Start a specified model. - Example: cortex models start llama3:8b - models stop Stop a specified model. - Example: cortex models stop llama3:8b - models update Update the configuration of a specified model. - Example: cortex models update llama3:8b --ngl 32 - engines Execute a specified command related to engines. - Example: cortex engines llamacpp - engines list List all available engines. - Example: cortex engines list - -Options: - -h, --help Show this help message and exit. - - ``` ## Uninstall Cortex diff --git a/cortex-cpp/README.md b/cortex-cpp/README.md index 009c0254f..97564b910 100644 --- a/cortex-cpp/README.md +++ b/cortex-cpp/README.md @@ -1,39 +1,43 @@ # cortex-cpp - Embeddable AI

- nitrologo + cortex-cpplogo

- Documentation - API Reference - - Changelog - Bug reports - Discord + Documentation - API Reference + - Changelog - Bug reports - Discord

> ⚠️ **cortex-cpp is currently in Development**: Expect breaking changes and bugs! -## Features -- Fast Inference: Built on top of the cutting-edge inference library llama.cpp, modified to be production ready. -- Lightweight: Only 3MB, ideal for resource-sensitive environments. -- Easily Embeddable: Simple integration into existing applications, offering flexibility. -- Quick Setup: Approximately 10-second initialization for swift deployment. -- Enhanced Web Framework: Incorporates drogon cpp to boost web service efficiency. - ## About cortex-cpp -cortex-cpp is a high-efficiency C++ inference engine for edge computing, powering [Jan](https://jan.ai/). It is lightweight and embeddable, ideal for product integration. +Cortex.cpp is a stateless, C++ server that is 100% compatible with OpenAI API (stateless endpoints). It includes a Drogon server, with request queues, model orchestration logic, and hardware telemetry, and more, for prod environments. The binary of cortex-cpp after zipped is only ~3mb in size with none to minimal dependencies (if you use a GPU need CUDA for example) make it desirable for any edge/server deployment 👍. -> Read more about Nitro at https://nitro.jan.ai/ +> Read more about Cortex-cpp at https://jan.ai/cortex + +### Features +- Fast Inference: Built on top of the cutting-edge inference library llama.cpp, modified to be production ready. +- Lightweight: Only 3MB, ideal for resource-sensitive environments. +- Easily Embeddable: Simple integration into existing applications, offering flexibility. +- Quick Setup: Approximately 10-second initialization for swift deployment. +- Enhanced Web Framework: Incorporates drogon cpp to boost web service efficiency. ### Repo Structure ``` . -├── controllers -├── docs -├── llama.cpp -> Upstream llama C++ -├── cortex-cpp-deps -> Dependencies of the cortex-cpp project as a sub-project -└── utils +├── common # Common libraries or shared resources +├── controllers # Controller scripts or modules for managing interactions +├── cortex-common # Shared components across different cortex modules +├── cortex-cpp-deps # Dependencies specific to the cortex-cpp module +├── engines # Different processing or computational engines +├── examples # Example scripts or applications demonstrating usage +├── test # Test scripts and testing frameworks +└── utils # Utility scripts and helper functions + ``` ## Quickstart @@ -43,13 +47,13 @@ The binary of cortex-cpp after zipped is only ~3mb in size with none to minimal - For Linux and MacOS ```bash - curl -sfL https://raw.githubusercontent.com/janhq/nitro/main/install.sh | sudo /bin/bash - + curl -sfL https://raw.githubusercontent.com/janhq/cortex-cpp/main/install.sh | sudo /bin/bash - ``` - For Windows ```bash - powershell -Command "& { Invoke-WebRequest -Uri 'https://raw.githubusercontent.com/janhq/nitro/main/install.bat' -OutFile 'install.bat'; .\install.bat; Remove-Item -Path 'install.bat' }" + powershell -Command "& { Invoke-WebRequest -Uri 'https://raw.githubusercontent.com/janhq/cortex-cpp/main/install.bat' -OutFile 'install.bat'; .\install.bat; Remove-Item -Path 'install.bat' }" ``` **Step 2: Downloading a Model** @@ -116,7 +120,7 @@ Table of parameters |`grammar_file`| String |You can constrain the sampling using GBNF grammars by providing path to a grammar file| |`model_type` | String | Model type we want to use: llm or embedding, default value is llm| -***OPTIONAL***: You can run Nitro on a different port like 5000 instead of 3928 by running it manually in terminal +***OPTIONAL***: You can run Cortex-cpp on a different port like 5000 instead of 3928 by running it manually in terminal ```zsh ./cortex-cpp 1 127.0.0.1 5000 ([thread_num] [host] [port] [uploads_folder_path]) ``` @@ -142,37 +146,37 @@ To compile cortex-cpp please visit [Compile from source](docs/docs/new/build-sou Stable (Recommended) - + CPU - + CUDA - + Intel - + M1/M2 - + CPU - + CUDA @@ -181,28 +185,27 @@ To compile cortex-cpp please visit [Compile from source](docs/docs/new/build-sou Experimental (Nighlty Build) - + GitHub action artifactory -Download the latest version of Nitro at https://nitro.jan.ai/ or visit the **[GitHub Releases](https://github.com/janhq/cortex/releases)** to download any previous release. +Download the latest version of Cortex-cpp at https://jan.ai/ or visit the **[GitHub Releases](https://github.com/janhq/cortex/releases)** to download any previous release. ## Nightly Build -Nightly build is a process where the software is built automatically every night. This helps in detecting and fixing bugs early in the development cycle. The process for this project is defined in [`.github/workflows/build.yml`](.github/workflows/build.yml) +Nightly build is a process where the software is built automatically every night. This helps detect and fix bugs early in the development cycle. The process for this project is defined in [`.github/workflows/build.yml`](.github/workflows/build.yml) -You can join our Discord server [here](https://discord.gg/FTk2MvZwJH) and go to channel [github-nitro](https://discordapp.com/channels/1107178041848909847/1151022176019939328) to monitor the build process. +You can join our Discord server [here](https://discord.gg/FTk2MvZwJH) and go to channel [cortex](https://discordapp.com/channels/1107178041848909847/1151022176019939328) to monitor the build process. The nightly build is triggered at 2:00 AM UTC every day. -The nightly build can be downloaded from the url notified in the Discord channel. Please access the url from the browser and download the build artifacts from there. +The nightly build can be downloaded from the URL notified in the Discord channel. Please access the URL from the browser and download the build artifacts from there. ## Manual Build - -Manual build is a process where the software is built manually by the developers. This is usually done when a new feature is implemented or a bug is fixed. The process for this project is defined in [`.github/workflows/build.yml`](.github/workflows/build.yml) +Manual build is a process in which the developers build the software manually. This is usually done when a new feature is implemented or a bug is fixed. The process for this project is defined in [`.github/workflows/build.yml`](.github/workflows/build.yml) It is similar to the nightly build process, except that it is triggered manually by the developers. @@ -214,4 +217,4 @@ It is similar to the nightly build process, except that it is triggered manually ## Star History -[![Star History Chart](https://api.star-history.com/svg?repos=janhq/nitro&type=Date)](https://star-history.com/#janhq/nitro&Date) +[![Star History Chart](https://api.star-history.com/svg?repos=janhq/cortex-cpp&type=Date)](https://star-history.com/#janhq/cortex-cpp&Date) From 443f9087c7c646b038e808aa0b3a17c7b544347c Mon Sep 17 00:00:00 2001 From: irfanpena Date: Thu, 30 May 2024 14:06:35 +0700 Subject: [PATCH 03/11] docs: nits and fix the URL schema --- cortex-cpp/README.md | 22 ++++++---------------- cortex-js/src/main.ts | 2 +- 2 files changed, 7 insertions(+), 17 deletions(-) diff --git a/cortex-cpp/README.md b/cortex-cpp/README.md index 97564b910..903ad735f 100644 --- a/cortex-cpp/README.md +++ b/cortex-cpp/README.md @@ -12,14 +12,14 @@ ## About cortex-cpp -Cortex.cpp is a stateless, C++ server that is 100% compatible with OpenAI API (stateless endpoints). It includes a Drogon server, with request queues, model orchestration logic, and hardware telemetry, and more, for prod environments. +Cortex-cpp is a streamlined, stateless C++ server engineered to be fully compatible with OpenAI's API, particularly its stateless functionalities. It integrates a Drogon server framework to manage request handling and includes features like model orchestration and hardware telemetry, which are essential for production environments. -The binary of cortex-cpp after zipped is only ~3mb in size with none to minimal dependencies (if you use a GPU need CUDA for example) make it desirable for any edge/server deployment 👍. +Remarkably compact, the binary size of cortex-cpp is around 3 MB when compressed, with minimal dependencies. This lightweight and efficient design makes cortex-cpp an excellent choice for deployments in both edge computing and server contexts. -> Read more about Cortex-cpp at https://jan.ai/cortex +> Utilizing GPU capabilities does require CUDA. ### Features -- Fast Inference: Built on top of the cutting-edge inference library llama.cpp, modified to be production ready. +- Fast Inference: Built on top of the cutting-edge inference library llama.cpp, modified to be production-ready. - Lightweight: Only 3MB, ideal for resource-sensitive environments. - Easily Embeddable: Simple integration into existing applications, offering flexibility. - Quick Setup: Approximately 10-second initialization for swift deployment. @@ -44,17 +44,7 @@ The binary of cortex-cpp after zipped is only ~3mb in size with none to minimal **Step 1: Install cortex-cpp** -- For Linux and MacOS - - ```bash - curl -sfL https://raw.githubusercontent.com/janhq/cortex-cpp/main/install.sh | sudo /bin/bash - - ``` - -- For Windows - - ```bash - powershell -Command "& { Invoke-WebRequest -Uri 'https://raw.githubusercontent.com/janhq/cortex-cpp/main/install.bat' -OutFile 'install.bat'; .\install.bat; Remove-Item -Path 'install.bat' }" - ``` +Download cortex-cpp here: https://github.com/janhq/cortex/releases **Step 2: Downloading a Model** @@ -205,7 +195,7 @@ The nightly build is triggered at 2:00 AM UTC every day. The nightly build can be downloaded from the URL notified in the Discord channel. Please access the URL from the browser and download the build artifacts from there. ## Manual Build -Manual build is a process in which the developers build the software manually. This is usually done when a new feature is implemented or a bug is fixed. The process for this project is defined in [`.github/workflows/build.yml`](.github/workflows/build.yml) +Manual build is a process in which the developers build the software manually. This is usually done when a new feature is implemented, or a bug is fixed. The process for this project is defined in [`.github/workflows/build.yml`](.github/workflows/build.yml) It is similar to the nightly build process, except that it is triggered manually by the developers. diff --git a/cortex-js/src/main.ts b/cortex-js/src/main.ts index 3a722a5d5..3eaa8fc42 100644 --- a/cortex-js/src/main.ts +++ b/cortex-js/src/main.ts @@ -33,7 +33,7 @@ async function bootstrap() { .addTag('Models', 'These endpoints provide a list and descriptions of all available models within the Cortex framework.') .addTag('Messages', "These endpoints manage the retrieval and storage of conversation content, including responses from LLMs and other metadata related to chat interactions.") .addTag('Threads', 'These endpoints handle the creation, retrieval, updating, and deletion of conversation threads.') - .addServer('http://localhost:1337/') + .addServer('http://localhost:1337') .addServer('http://localhost:1337/v1') .build(); const document = SwaggerModule.createDocument(app, config); From 8c51521b7bf72a34249e7659c35d0a01ed689d00 Mon Sep 17 00:00:00 2001 From: irfanpena Date: Thu, 30 May 2024 14:51:48 +0700 Subject: [PATCH 04/11] docs: update the links --- cortex-cpp/README.md | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/cortex-cpp/README.md b/cortex-cpp/README.md index 903ad735f..ac1d3e5f8 100644 --- a/cortex-cpp/README.md +++ b/cortex-cpp/README.md @@ -136,50 +136,42 @@ To compile cortex-cpp please visit [Compile from source](docs/docs/new/build-sou Stable (Recommended) - + CPU - + CUDA - + Intel - + M1/M2 - + CPU - + CUDA - - Experimental (Nighlty Build) - - - GitHub action artifactory - - - Download the latest version of Cortex-cpp at https://jan.ai/ or visit the **[GitHub Releases](https://github.com/janhq/cortex/releases)** to download any previous release. From 1f57ed7374e4f3a062beb6b8c52517798ecbf4a5 Mon Sep 17 00:00:00 2001 From: irfanpena Date: Thu, 30 May 2024 15:27:34 +0700 Subject: [PATCH 05/11] docs: update the cortex-js readme --- cortex-js/README.md | 189 ++++++++++++++++++++++++++------------------ 1 file changed, 114 insertions(+), 75 deletions(-) diff --git a/cortex-js/README.md b/cortex-js/README.md index 4ec7ed411..707599773 100644 --- a/cortex-js/README.md +++ b/cortex-js/README.md @@ -1,4 +1,70 @@ -# Installation +# Cortex - CLI + +

+ Documentation - API Reference + - Changelog - Bug reports - Discord +

+ +> ⚠️ **Cortex is currently in Development**: Expect breaking changes and bugs! + +## About +Cortex is an openAI-compatible local AI server that developers can use to build LLM apps. It is packaged with a Docker-inspired command-line interface and a Typescript client library. It can be used as a standalone server, or imported as a library. + +Cortex currently supports two inference engines: + +- Llama.cpp +- TensorRT-LLM + +> Read more about Cortex at https://jan.ai/cortex + +### Repo Structure +``` +# Entity Definitions +domain/ # This is the core directory where the domains are defined. + abstracts/ # Abstract base classes for common attributes and methods. + models/ # Domain interface definitions, e.g. model, assistant. + repositories/ # Extensions abstract and interface + +# Business Rules +usecases/ # Application logic + assistants/ # CRUD logic (invokes dtos, entities). + chat/ # Logic for chat functionalities. + models/ # Logic for model operations. + +# Adapters & Implementations +infrastructure/ # Implementations for Cortex interactions + commanders/ # CLI handlers + models/ + questions/ # CLI installation UX + shortcuts/ # CLI chained syntax + types/ + usecases/ # Invokes UseCases + + controllers/ # Nest controllers and HTTP routes + assistants/ # Invokes UseCases + chat/ # Invokes UseCases + models/ # Invokes UseCases + + database/ # Database providers (mysql, sqlite) + + # Framework specific object definitions + dtos/ # DTO definitions (data transfer & validation) + entities/ # TypeORM entity definitions (db schema) + + # Providers + providers/cortex # Cortex [server] provider (a core extension) + repositories/extensions # Extension provider (core & external extensions) + +extensions/ # External extensions +command.module.ts # CLI Commands List +main.ts # Entrypoint +``` +## Quicklinks +Cortex +- [Website](https://jan.ai/) +- [GitHub](https://github.com/janhq/cortex) +- [User Guides](https://jan.ai/cortex) +- [API reference](https://jan.ai/api-reference) ## Prerequisites @@ -10,10 +76,9 @@ Before installation, ensure that you have installed the following: - **NPM**: Needed to manage packages. - **CPU Instruction Sets**: Available for download from the [Cortex GitHub Releases](https://github.com/janhq/cortex/releases) page. - +>💡 The **CPU instruction sets** are not required for the initial installation of Cortex. This dependency will be automatically installed during the Cortex initialization if they are not already on your system. + ### **Hardware** @@ -35,88 +100,34 @@ Ensure that your system meets the following requirements to run Cortex: - **Disk**: At least 10GB for app and model download. -## Cortex Installation - -To install Cortex, follow the steps below: - -### Step 1: Install Cortex - -Run the following command to install Cortex globally on your machine: - -```bash -# Install using NPM globally +## Quickstart +1. Install the NPM package: +``` bash npm i -g @janhq/cortex ``` -### Step 2: Verify the Installation - -After installation, you can verify that Cortex is installed correctly by getting help information. - -```bash -# Get the help information -cortex -h -``` - -### Step 3: Initialize Cortex - -Once verified, you need to initialize the Cortex engine. - -1. Initialize the Cortex engine: - -``` +2. Initialize a compatible engine: +``` bash cortex init ``` -1. Select between `CPU` and `GPU` modes. - -```bash -? Select run mode (Use arrow keys) -> CPU - GPU -``` - -2. Select between GPU types. - -```bash -? Select GPU types (Use arrow keys) -> Nvidia - Others (Vulkan) -``` - -3. Select CPU instructions (will be deprecated soon). - -```bash -? Select CPU instructions (Use arrow keys) -> AVX2 - AVX - AVX-512 +3. Download a GGUF model from Hugging Face +``` bash +cortex models pull janhq/TinyLlama-1.1B-Chat-v1.0-GGUF ``` - -1. Cortex will download the required CPU instruction sets if you choose `CPU` mode. If you choose `GPU` mode, Cortex will download the necessary dependencies to use your GPU. -2. Once downloaded, Cortex is ready to use! - -### Step 4: Pull a model - -From HuggingFace - -```bash -cortex pull janhq/phi-3-medium-128k-instruct-GGUF +4. Load the model +``` bash +cortex models start janhq/TinyLlama-1.1B-Chat-v1.0-GGUF ``` -From Jan Hub (TBD) - -```bash -cortex pull llama3 +5. Start chatting with the model +``` bash +cortex chat tell me a joke ``` -### Step 5: Chat - -```bash -cortex run janhq/phi-3-medium-128k-instruct-GGUF -``` ## Run as an API server - +To run Cortex as an API server: ```bash cortex serve ``` @@ -133,18 +144,42 @@ To install Cortex from the source, follow the steps below: npx nest build ``` -1. Make the `command.js` executable: +4. Make the `command.js` executable: ```bash chmod +x '[path-to]/cortex/cortex-js/dist/src/command.js' ``` -1. Link the package globally: +5. Link the package globally: ```bash npm link ``` +## Cortex CLI Command +The following CLI commands are currently available: +> ⚠️ **Cortex is currently in Development**: More commands will be added soon! + +```bash + + serve Providing API endpoint for Cortex backend + chat Send a chat request to a model + init|setup Init settings and download cortex's dependencies + ps Show running models and their status + kill Kill running cortex processes + pull|download Download a model. Working with HuggingFace model id. + run [options] EXPERIMENTAL: Shortcut to start a model and chat + models Subcommands for managing models + models list List all available models. + models pull Download a specified model. + models remove Delete a specified model. + models get Retrieve the configuration of a specified model. + models start Start a specified model. + models stop Stop a specified model. + models update Update the configuration of a specified model. + engines Execute a specified command related to engines. + engines list List all available engines. +``` ## Uninstall Cortex Run the following command to uninstall Cortex globally on your machine: @@ -153,3 +188,7 @@ Run the following command to uninstall Cortex globally on your machine: # Uninstall globally using NPM npm uninstall -g @janhq/cortex ``` +## Contact Support +- For support, please file a GitHub ticket. +- For questions, join our Discord [here](https://discord.gg/FTk2MvZwJH). +- For long-form inquiries, please email [hello@jan.ai](mailto:hello@jan.ai). \ No newline at end of file From e932e3d2dcc06cb8d40108743252c45a1f5b0004 Mon Sep 17 00:00:00 2001 From: irfanpena Date: Thu, 30 May 2024 15:40:12 +0700 Subject: [PATCH 06/11] docs: nits --- cortex-cpp/README.md | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-) diff --git a/cortex-cpp/README.md b/cortex-cpp/README.md index ac1d3e5f8..0fa86c151 100644 --- a/cortex-cpp/README.md +++ b/cortex-cpp/README.md @@ -121,9 +121,6 @@ Table of parameters cortex-cpp server is compatible with the OpenAI format, so you can expect the same output as the OpenAI ChatGPT API. -## Compile from source -To compile cortex-cpp please visit [Compile from source](docs/docs/new/build-source.md) - ## Download @@ -176,22 +173,11 @@ To compile cortex-cpp please visit [Compile from source](docs/docs/new/build-sou Download the latest version of Cortex-cpp at https://jan.ai/ or visit the **[GitHub Releases](https://github.com/janhq/cortex/releases)** to download any previous release. -## Nightly Build - -Nightly build is a process where the software is built automatically every night. This helps detect and fix bugs early in the development cycle. The process for this project is defined in [`.github/workflows/build.yml`](.github/workflows/build.yml) - -You can join our Discord server [here](https://discord.gg/FTk2MvZwJH) and go to channel [cortex](https://discordapp.com/channels/1107178041848909847/1151022176019939328) to monitor the build process. - -The nightly build is triggered at 2:00 AM UTC every day. - -The nightly build can be downloaded from the URL notified in the Discord channel. Please access the URL from the browser and download the build artifacts from there. ## Manual Build Manual build is a process in which the developers build the software manually. This is usually done when a new feature is implemented, or a bug is fixed. The process for this project is defined in [`.github/workflows/build.yml`](.github/workflows/build.yml) -It is similar to the nightly build process, except that it is triggered manually by the developers. - -### Contact +## Contact Support - For support, please file a GitHub ticket. - For questions, join our Discord [here](https://discord.gg/FTk2MvZwJH). From ff55fb71d6da4ea7458f9037437a1fd4c54b02aa Mon Sep 17 00:00:00 2001 From: irfanpena Date: Fri, 31 May 2024 12:00:02 +0700 Subject: [PATCH 07/11] docs: Update the Readme files --- CONTRIBUTING.md | 42 ++++++++++++++++++++++++ README.md | 55 ++++--------------------------- cortex-cpp/CONTRIBUTING.md | 14 ++++++++ cortex-cpp/README.md | 66 ++++++++++++++++---------------------- cortex-js/CONTRIBUTING.md | 42 ++++++++++++++++++++++++ cortex-js/README.md | 42 ------------------------ 6 files changed, 131 insertions(+), 130 deletions(-) create mode 100644 CONTRIBUTING.md create mode 100644 cortex-cpp/CONTRIBUTING.md create mode 100644 cortex-js/CONTRIBUTING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 000000000..53909c391 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,42 @@ +### Repo Structure +``` +# Entity Definitions +domain/ # This is the core directory where the domains are defined. + abstracts/ # Abstract base classes for common attributes and methods. + models/ # Domain interface definitions, e.g. model, assistant. + repositories/ # Extensions abstract and interface + +# Business Rules +usecases/ # Application logic + assistants/ # CRUD logic (invokes dtos, entities). + chat/ # Logic for chat functionalities. + models/ # Logic for model operations. + +# Adapters & Implementations +infrastructure/ # Implementations for Cortex interactions + commanders/ # CLI handlers + models/ + questions/ # CLI installation UX + shortcuts/ # CLI chained syntax + types/ + usecases/ # Invokes UseCases + + controllers/ # Nest controllers and HTTP routes + assistants/ # Invokes UseCases + chat/ # Invokes UseCases + models/ # Invokes UseCases + + database/ # Database providers (mysql, sqlite) + + # Framework specific object definitions + dtos/ # DTO definitions (data transfer & validation) + entities/ # TypeORM entity definitions (db schema) + + # Providers + providers/cortex # Cortex [server] provider (a core extension) + repositories/extensions # Extension provider (core & external extensions) + +extensions/ # External extensions +command.module.ts # CLI Commands List +main.ts # Entrypoint +``` \ No newline at end of file diff --git a/README.md b/README.md index 707599773..5ca8fd24c 100644 --- a/README.md +++ b/README.md @@ -17,50 +17,8 @@ Cortex currently supports two inference engines: > Read more about Cortex at https://jan.ai/cortex -### Repo Structure -``` -# Entity Definitions -domain/ # This is the core directory where the domains are defined. - abstracts/ # Abstract base classes for common attributes and methods. - models/ # Domain interface definitions, e.g. model, assistant. - repositories/ # Extensions abstract and interface - -# Business Rules -usecases/ # Application logic - assistants/ # CRUD logic (invokes dtos, entities). - chat/ # Logic for chat functionalities. - models/ # Logic for model operations. - -# Adapters & Implementations -infrastructure/ # Implementations for Cortex interactions - commanders/ # CLI handlers - models/ - questions/ # CLI installation UX - shortcuts/ # CLI chained syntax - types/ - usecases/ # Invokes UseCases - - controllers/ # Nest controllers and HTTP routes - assistants/ # Invokes UseCases - chat/ # Invokes UseCases - models/ # Invokes UseCases - - database/ # Database providers (mysql, sqlite) - - # Framework specific object definitions - dtos/ # DTO definitions (data transfer & validation) - entities/ # TypeORM entity definitions (db schema) - - # Providers - providers/cortex # Cortex [server] provider (a core extension) - repositories/extensions # Extension provider (core & external extensions) - -extensions/ # External extensions -command.module.ts # CLI Commands List -main.ts # Entrypoint -``` ## Quicklinks -Cortex +**Cortex**: - [Website](https://jan.ai/) - [GitHub](https://github.com/janhq/cortex) - [User Guides](https://jan.ai/cortex) @@ -101,7 +59,8 @@ Ensure that your system meets the following requirements to run Cortex: - **Disk**: At least 10GB for app and model download. ## Quickstart -1. Install the NPM package: +To install Cortex CLI, follow the steps below: +1. Install the Cortex NPM package: ``` bash npm i -g @janhq/cortex ``` @@ -111,16 +70,16 @@ npm i -g @janhq/cortex cortex init ``` -3. Download a GGUF model from Hugging Face +3. Download a GGUF model from Hugging Face: ``` bash cortex models pull janhq/TinyLlama-1.1B-Chat-v1.0-GGUF ``` -4. Load the model +4. Load the model: ``` bash cortex models start janhq/TinyLlama-1.1B-Chat-v1.0-GGUF ``` -5. Start chatting with the model +5. Start chatting with the model: ``` bash cortex chat tell me a joke ``` @@ -177,8 +136,6 @@ The following CLI commands are currently available: models start Start a specified model. models stop Stop a specified model. models update Update the configuration of a specified model. - engines Execute a specified command related to engines. - engines list List all available engines. ``` ## Uninstall Cortex diff --git a/cortex-cpp/CONTRIBUTING.md b/cortex-cpp/CONTRIBUTING.md new file mode 100644 index 000000000..c7ea6a1f1 --- /dev/null +++ b/cortex-cpp/CONTRIBUTING.md @@ -0,0 +1,14 @@ +### Repo Structure + +``` +. +├── common # Common libraries or shared resources +├── controllers # Controller scripts or modules for managing interactions +├── cortex-common # Shared components across different cortex modules +├── cortex-cpp-deps # Dependencies specific to the cortex-cpp module +├── engines # Different processing or computational engines +├── examples # Example scripts or applications demonstrating usage +├── test # Test scripts and testing frameworks +└── utils # Utility scripts and helper functions + +``` \ No newline at end of file diff --git a/cortex-cpp/README.md b/cortex-cpp/README.md index 0fa86c151..4019a5331 100644 --- a/cortex-cpp/README.md +++ b/cortex-cpp/README.md @@ -18,48 +18,46 @@ Remarkably compact, the binary size of cortex-cpp is around 3 MB when compressed > Utilizing GPU capabilities does require CUDA. -### Features -- Fast Inference: Built on top of the cutting-edge inference library llama.cpp, modified to be production-ready. -- Lightweight: Only 3MB, ideal for resource-sensitive environments. -- Easily Embeddable: Simple integration into existing applications, offering flexibility. -- Quick Setup: Approximately 10-second initialization for swift deployment. -- Enhanced Web Framework: Incorporates drogon cpp to boost web service efficiency. +## Prerequisites +### **Hardware** -### Repo Structure +Ensure that your system meets the following requirements to run Cortex: -``` -. -├── common # Common libraries or shared resources -├── controllers # Controller scripts or modules for managing interactions -├── cortex-common # Shared components across different cortex modules -├── cortex-cpp-deps # Dependencies specific to the cortex-cpp module -├── engines # Different processing or computational engines -├── examples # Example scripts or applications demonstrating usage -├── test # Test scripts and testing frameworks -└── utils # Utility scripts and helper functions - -``` +- **OS**: + - MacOSX 13.6 or higher. + - Windows 10 or higher. + - Ubuntu 18.04 and later. +- **RAM (CPU Mode):** + - 8GB for running up to 3B models. + - 16GB for running up to 7B models. + - 32GB for running up to 13B models. +- **VRAM (GPU Mode):** -## Quickstart + - 6GB can load the 3B model (int4) with `ngl` at 120 ~ full speed on CPU/ GPU. + - 8GB can load the 7B model (int4) with `ngl` at 120 ~ full speed on CPU/ GPU. + - 12GB can load the 13B model (int4) with `ngl` at 120 ~ full speed on CPU/ GPU. -**Step 1: Install cortex-cpp** +- **Disk**: At least 10GB for app and model download. -Download cortex-cpp here: https://github.com/janhq/cortex/releases +## Quickstart +To install Cortex CLI, follow the steps below: +1. Download cortex-cpp here: https://github.com/janhq/cortex/releases +2. Install cortex-cpp by running the downloaded file. -**Step 2: Downloading a Model** +3. Download a Model: ```bash mkdir model && cd model wget -O llama-2-7b-model.gguf https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q5_K_M.gguf?download=true ``` -**Step 3: Run cortex-cpp server** +4. Run cortex-cpp server: ```bash title="Run cortex-cpp server" cortex-cpp ``` -**Step 4: Load model** +5. Load a model: ```bash title="Load model" curl http://localhost:3928/inferences/server/loadmodel \ @@ -71,7 +69,7 @@ curl http://localhost:3928/inferences/server/loadmodel \ }' ``` -**Step 5: Making an Inference** +6. Make an Inference: ```bash title="cortex-cpp Inference" curl http://localhost:3928/v1/chat/completions \ @@ -86,7 +84,8 @@ curl http://localhost:3928/v1/chat/completions \ }' ``` -Table of parameters +## Table of parameters +Below is the available list of the model parameters you can set when loading a model in cortex-cpp: | Parameter | Type | Description | |------------------|---------|--------------------------------------------------------------| @@ -110,17 +109,6 @@ Table of parameters |`grammar_file`| String |You can constrain the sampling using GBNF grammars by providing path to a grammar file| |`model_type` | String | Model type we want to use: llm or embedding, default value is llm| -***OPTIONAL***: You can run Cortex-cpp on a different port like 5000 instead of 3928 by running it manually in terminal -```zsh -./cortex-cpp 1 127.0.0.1 5000 ([thread_num] [host] [port] [uploads_folder_path]) -``` -- thread_num : the number of thread that cortex-cpp webserver needs to have -- host : host value normally 127.0.0.1 or 0.0.0.0 -- port : the port that cortex-cpp got deployed onto -- uploads_folder_path: custom path for file uploads in Drogon. - -cortex-cpp server is compatible with the OpenAI format, so you can expect the same output as the OpenAI ChatGPT API. - ## Download
@@ -171,7 +159,7 @@ cortex-cpp server is compatible with the OpenAI format, so you can expect the sa
-Download the latest version of Cortex-cpp at https://jan.ai/ or visit the **[GitHub Releases](https://github.com/janhq/cortex/releases)** to download any previous release. +> Download the latest or older versions of Cortex-cpp at the **[GitHub Releases](https://github.com/janhq/cortex/releases)**. ## Manual Build diff --git a/cortex-js/CONTRIBUTING.md b/cortex-js/CONTRIBUTING.md new file mode 100644 index 000000000..53909c391 --- /dev/null +++ b/cortex-js/CONTRIBUTING.md @@ -0,0 +1,42 @@ +### Repo Structure +``` +# Entity Definitions +domain/ # This is the core directory where the domains are defined. + abstracts/ # Abstract base classes for common attributes and methods. + models/ # Domain interface definitions, e.g. model, assistant. + repositories/ # Extensions abstract and interface + +# Business Rules +usecases/ # Application logic + assistants/ # CRUD logic (invokes dtos, entities). + chat/ # Logic for chat functionalities. + models/ # Logic for model operations. + +# Adapters & Implementations +infrastructure/ # Implementations for Cortex interactions + commanders/ # CLI handlers + models/ + questions/ # CLI installation UX + shortcuts/ # CLI chained syntax + types/ + usecases/ # Invokes UseCases + + controllers/ # Nest controllers and HTTP routes + assistants/ # Invokes UseCases + chat/ # Invokes UseCases + models/ # Invokes UseCases + + database/ # Database providers (mysql, sqlite) + + # Framework specific object definitions + dtos/ # DTO definitions (data transfer & validation) + entities/ # TypeORM entity definitions (db schema) + + # Providers + providers/cortex # Cortex [server] provider (a core extension) + repositories/extensions # Extension provider (core & external extensions) + +extensions/ # External extensions +command.module.ts # CLI Commands List +main.ts # Entrypoint +``` \ No newline at end of file diff --git a/cortex-js/README.md b/cortex-js/README.md index 707599773..a84874a8c 100644 --- a/cortex-js/README.md +++ b/cortex-js/README.md @@ -17,48 +17,6 @@ Cortex currently supports two inference engines: > Read more about Cortex at https://jan.ai/cortex -### Repo Structure -``` -# Entity Definitions -domain/ # This is the core directory where the domains are defined. - abstracts/ # Abstract base classes for common attributes and methods. - models/ # Domain interface definitions, e.g. model, assistant. - repositories/ # Extensions abstract and interface - -# Business Rules -usecases/ # Application logic - assistants/ # CRUD logic (invokes dtos, entities). - chat/ # Logic for chat functionalities. - models/ # Logic for model operations. - -# Adapters & Implementations -infrastructure/ # Implementations for Cortex interactions - commanders/ # CLI handlers - models/ - questions/ # CLI installation UX - shortcuts/ # CLI chained syntax - types/ - usecases/ # Invokes UseCases - - controllers/ # Nest controllers and HTTP routes - assistants/ # Invokes UseCases - chat/ # Invokes UseCases - models/ # Invokes UseCases - - database/ # Database providers (mysql, sqlite) - - # Framework specific object definitions - dtos/ # DTO definitions (data transfer & validation) - entities/ # TypeORM entity definitions (db schema) - - # Providers - providers/cortex # Cortex [server] provider (a core extension) - repositories/extensions # Extension provider (core & external extensions) - -extensions/ # External extensions -command.module.ts # CLI Commands List -main.ts # Entrypoint -``` ## Quicklinks Cortex - [Website](https://jan.ai/) From cb75995d9f0f844b02cd14f58983fd037285e4dc Mon Sep 17 00:00:00 2001 From: irfanpena Date: Fri, 31 May 2024 12:55:01 +0700 Subject: [PATCH 08/11] add the banner image --- README.md | 3 +++ cortex-cpp/README.md | 2 +- cortex-js/README.md | 3 +++ 3 files changed, 7 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 5ca8fd24c..db9219115 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,7 @@ # Cortex - CLI +

+ cortex-cpplogo +

Documentation - API Reference diff --git a/cortex-cpp/README.md b/cortex-cpp/README.md index 4019a5331..61e9c2807 100644 --- a/cortex-cpp/README.md +++ b/cortex-cpp/README.md @@ -1,6 +1,6 @@ # cortex-cpp - Embeddable AI

- cortex-cpplogo + cortex-cpplogo

diff --git a/cortex-js/README.md b/cortex-js/README.md index a84874a8c..f02c0431d 100644 --- a/cortex-js/README.md +++ b/cortex-js/README.md @@ -1,4 +1,7 @@ # Cortex - CLI +

+ cortex-cpplogo +

Documentation - API Reference From e99945b4854c4c1f0cdd6dd6f77d4086e1b833d6 Mon Sep 17 00:00:00 2001 From: irfanpena Date: Fri, 31 May 2024 13:55:38 +0700 Subject: [PATCH 09/11] nits --- README.md | 6 +++--- cortex-js/README.md | 13 +++++++------ 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index db9219115..dd6749f39 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ Ensure that your system meets the following requirements to run Cortex: ## Quickstart To install Cortex CLI, follow the steps below: -1. Install the Cortex NPM package: +1. Install the Cortex NPM package globally: ``` bash npm i -g @janhq/cortex ``` @@ -75,11 +75,11 @@ cortex init 3. Download a GGUF model from Hugging Face: ``` bash -cortex models pull janhq/TinyLlama-1.1B-Chat-v1.0-GGUF +cortex pull tinyllama:1b ``` 4. Load the model: ``` bash -cortex models start janhq/TinyLlama-1.1B-Chat-v1.0-GGUF +cortex models start tinyllama:1b ``` 5. Start chatting with the model: diff --git a/cortex-js/README.md b/cortex-js/README.md index f02c0431d..a650c8b9d 100644 --- a/cortex-js/README.md +++ b/cortex-js/README.md @@ -62,7 +62,8 @@ Ensure that your system meets the following requirements to run Cortex: - **Disk**: At least 10GB for app and model download. ## Quickstart -1. Install the NPM package: +To install Cortex CLI, follow the steps below: +1. Install the Cortex NPM package globally: ``` bash npm i -g @janhq/cortex ``` @@ -72,16 +73,16 @@ npm i -g @janhq/cortex cortex init ``` -3. Download a GGUF model from Hugging Face +3. Download a GGUF model from Hugging Face: ``` bash -cortex models pull janhq/TinyLlama-1.1B-Chat-v1.0-GGUF +cortex pull tinyllama:1b ``` -4. Load the model +4. Load the model: ``` bash -cortex models start janhq/TinyLlama-1.1B-Chat-v1.0-GGUF +cortex models start tinyllama:1b ``` -5. Start chatting with the model +5. Start chatting with the model: ``` bash cortex chat tell me a joke ``` From d27d6dd9b5260ad8ff108ee23bc422abc411541f Mon Sep 17 00:00:00 2001 From: irfanpena Date: Fri, 31 May 2024 14:14:44 +0700 Subject: [PATCH 10/11] nits --- README.md | 11 +++++++++-- cortex-js/README.md | 11 +++++++++-- 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index dd6749f39..f81acab38 100644 --- a/README.md +++ b/README.md @@ -75,11 +75,18 @@ cortex init 3. Download a GGUF model from Hugging Face: ``` bash -cortex pull tinyllama:1b +# Pull a model most compatible with your hardware +cortex pull llama3 + +# Pull a specific variant with `repo_name:branch` +cortex pull llama3:7b + +# Pull a model with the HuggingFace `model_id` +cortex pull microsoft/Phi-3-mini-4k-instruct-gguf ``` 4. Load the model: ``` bash -cortex models start tinyllama:1b +cortex models start llama3:7b ``` 5. Start chatting with the model: diff --git a/cortex-js/README.md b/cortex-js/README.md index a650c8b9d..806c8c873 100644 --- a/cortex-js/README.md +++ b/cortex-js/README.md @@ -75,11 +75,18 @@ cortex init 3. Download a GGUF model from Hugging Face: ``` bash -cortex pull tinyllama:1b +# Pull a model most compatible with your hardware +cortex pull llama3 + +# Pull a specific variant with `repo_name:branch` +cortex pull llama3:7b + +# Pull a model with the HuggingFace `model_id` +cortex pull microsoft/Phi-3-mini-4k-instruct-gguf ``` 4. Load the model: ``` bash -cortex models start tinyllama:1b +cortex models start llama3:7b ``` 5. Start chatting with the model: From ab19475528cbb1e02d933f3b939eaf3c9dedab5f Mon Sep 17 00:00:00 2001 From: irfanpena <137022864+irfanpena@users.noreply.github.com> Date: Fri, 31 May 2024 14:35:58 +0700 Subject: [PATCH 11/11] Update CONTRIBUTING.md --- CONTRIBUTING.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 53909c391..607c3db7f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,9 +8,9 @@ domain/ # This is the core directory where the domains are de # Business Rules usecases/ # Application logic - assistants/ # CRUD logic (invokes dtos, entities). - chat/ # Logic for chat functionalities. - models/ # Logic for model operations. + assistants/ # CRUD logic (invokes dtos, entities). + chat/ # Logic for chat functionalities. + models/ # Logic for model operations. # Adapters & Implementations infrastructure/ # Implementations for Cortex interactions @@ -22,21 +22,21 @@ infrastructure/ # Implementations for Cortex interactions usecases/ # Invokes UseCases controllers/ # Nest controllers and HTTP routes - assistants/ # Invokes UseCases - chat/ # Invokes UseCases - models/ # Invokes UseCases + assistants/ # Invokes UseCases + chat/ # Invokes UseCases + models/ # Invokes UseCases database/ # Database providers (mysql, sqlite) - # Framework specific object definitions +# Framework specific object definitions dtos/ # DTO definitions (data transfer & validation) entities/ # TypeORM entity definitions (db schema) - # Providers +# Providers providers/cortex # Cortex [server] provider (a core extension) repositories/extensions # Extension provider (core & external extensions) extensions/ # External extensions command.module.ts # CLI Commands List main.ts # Entrypoint -``` \ No newline at end of file +```