From a5c0172020a4afd0d58145109c10e0c4ca8faf4c Mon Sep 17 00:00:00 2001 From: Gabrielle Ong Date: Mon, 4 Nov 2024 18:37:02 +0800 Subject: [PATCH] clean up models and engines, to refactor into separate pages --- docs/docs/cli/engines/index.mdx | 193 +++++------------------- docs/docs/cli/models/index.mdx | 254 +++----------------------------- 2 files changed, 59 insertions(+), 388 deletions(-) diff --git a/docs/docs/cli/engines/index.mdx b/docs/docs/cli/engines/index.mdx index 2949de810..2712e0af5 100644 --- a/docs/docs/cli/engines/index.mdx +++ b/docs/docs/cli/engines/index.mdx @@ -5,43 +5,21 @@ title: Cortex Engines import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; -:::warning -🚧 Cortex.cpp is currently under development. Our documentation outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase. -::: - # `cortex engines` This command allows you to manage various engines available within Cortex. - **Usage**: -:::info -You can use the `--verbose` flag to display more detailed output of the internal processes. To apply this flag, use the following format: `cortex --verbose [subcommand]`. -::: ```sh - # Stable cortex engines [options] [subcommand] - - # Beta - cortex-beta engines [options] [subcommand] - - # Nightly - cortex-nightly engines [options] [subcommand] ``` ```sh - # Stable cortex.exe engines [options] [subcommand] - - # Beta - cortex-beta.exe engines [options] [subcommand] - - # Nightly - cortex-nightly.exe engines [options] [subcommand] ``` @@ -54,127 +32,85 @@ You can use the `--verbose` flag to display more detailed output of the internal | `-h`, `--help` | Display help information for the command. | No | - | `-h` | {/* | `-vk`, `--vulkan` | Install Vulkan engine. | No | `false` | `-vk` | */} -## `cortex engines get` +--- +# Subcommands: +## `cortex engines list` :::info This CLI command calls the following API endpoint: -- [Get Engine](/api-reference#tag/engines/get/v1/engines/{name}) +- [List Engines](/api-reference#tag/engines/get/v1/engines) ::: -This command returns an engine detail defined by an engine `engine_name`. +This command lists all the Cortex's engines. **Usage**: -:::info -You can use the `--verbose` flag to display more detailed output of the internal processes. To apply this flag, use the following format: `cortex --verbose [subcommand]`. -::: ```sh - # Stable - cortex engines get - - # Beta - cortex-beta engines get - - # Nightly - cortex-nightly engines get + cortex engines list ``` ```sh - # Stable - cortex.exe engines get - - # Beta - cortex-beta.exe engines get - - # Nightly - cortex-nightly.exe engines get + cortex.exe engines list ``` For example, it returns the following: -```bash -┌─────────────┬────────────────────────────────────────────────────────────────────────────┐ -│ (index) │ Values │ -├─────────────┼────────────────────────────────────────────────────────────────────────────┤ -│ name │ 'onnx' │ -│ description │ 'This extension enables chat completion API calls using the Cortex engine' │ -│ version │ '0.0.1' │ -│ productName │ 'Cortex Inference Engine' │ -└─────────────┴────────────────────────────────────────────────────────────────────────────┘ ``` -:::info -To get an engine name, run the [`engines list`](/docs/cli/engines/list) command first. -::: - - -**Options**: - -| Option | Description | Required | Default value | Example | -|-------------------|-------------------------------------------------------|----------|---------------|-----------------| -| `engine_name` | The name of the engine that you want to retrieve. | Yes | - | `llama-cpp`| -| `-h`, `--help` | Display help information for the command. | No | - | `-h` | ++---+--------------+-------------------+---------+----------------------------+---------------+ +| # | Name | Supported Formats | Version | Variant | Status | ++---+--------------+-------------------+---------+----------------------------+---------------+ +| 1 | onnxruntime | ONNX | | | Incompatible | ++---+--------------+-------------------+---------+----------------------------+---------------+ +| 2 | llama-cpp | GGUF | 0.1.34 | linux-amd64-avx2-cuda-12-0 | Ready | ++---+--------------+-------------------+---------+----------------------------+---------------+ +| 3 | tensorrt-llm | TensorRT Engines | | | Not Installed | ++---+--------------+-------------------+---------+----------------------------+---------------+ +``` -## `cortex engines list` +## `cortex engines get` :::info This CLI command calls the following API endpoint: -- [List Engines](/api-reference#tag/engines/get/v1/engines) +- [Get Engine](/api-reference#tag/engines/get/v1/engines/{name}) ::: -This command lists all the Cortex's engines. - - +This command returns an engine detail defined by an engine `engine_name`. **Usage**: -:::info -You can use the `--verbose` flag to display more detailed output of the internal processes. To apply this flag, use the following format: `cortex --verbose [subcommand]`. -::: ```sh - # Stable - cortex engines list [options] - - # Beta - cortex-beta engines list [options] - - # Nightly - cortex-nightly engines list [options] + cortex engines get ``` ```sh - # Stable - cortex.exe engines list [options] - - # Beta - cortex-beta.exe engines list [options] - - # Nightly - cortex-nightly.exe engines list [options] + cortex.exe engines get ``` For example, it returns the following: -```bash -+---+--------------+-------------------+---------+----------------------------+---------------+ -| # | Name | Supported Formats | Version | Variant | Status | -+---+--------------+-------------------+---------+----------------------------+---------------+ -| 1 | onnxruntime | ONNX | | | Incompatible | -+---+--------------+-------------------+---------+----------------------------+---------------+ -| 2 | llama-cpp | GGUF | 0.1.34 | linux-amd64-avx2-cuda-12-0 | Ready | -+---+--------------+-------------------+---------+----------------------------+---------------+ -| 3 | tensorrt-llm | TensorRT Engines | | | Not Installed | -+---+--------------+-------------------+---------+----------------------------+---------------+ ``` ++-----------+-------------------+---------+-----------+--------+ +| Name | Supported Formats | Version | Variant | Status | ++-----------+-------------------+---------+-----------+--------+ +| llama-cpp | GGUF | 0.1.37 | mac-arm64 | Ready | ++-----------+-------------------+---------+-----------+--------+ +``` +:::info +To get an engine name, run the [`engines list`](/docs/cli/engines/list) command. +::: + **Options**: -| Option | Description | Required | Default value | Example | -|---------------------------|----------------------------------------------------|----------|---------------|----------------------| -| `-h`, `--help` | Display help for command. | No | - | `-h` | +| Option | Description | Required | Default value | Example | +|-------------------|-------------------------------------------------------|----------|---------------|-----------------| +| `engine_name` | The name of the engine that you want to retrieve. | Yes | - | `llama-cpp`| +| `-h`, `--help` | Display help information for the command. | No | - | `-h` | + ## `cortex engines install` @@ -188,54 +124,25 @@ This command downloads the required dependencies and installs the engine within - `tensorrt-llm` **Usage**: -:::info -You can use the `--verbose` flag to display more detailed output of the internal processes. To apply this flag, use the following format: `cortex --verbose [subcommand]`. -::: ```sh - # Stable cortex engines install [options] - - # Beta - cortex-beta engines install [options] - - # Nightly - cortex-nightly engines install [options] ``` ```sh - # Stable cortex.exe engines install [options] - - # Beta - cortex-beta.exe engines install [options] - # Nightly - cortex-nightly.exe engines install [options] ``` -For Example: -```bash -## Llama.cpp engine -cortex engines install llama-cpp - -## ONNX engine -cortex engines install onnxruntime - -## Tensorrt-LLM engine -cortex engines install tensorrt-llm - -``` - **Options**: | Option | Description | Required | Default value | Example | |---------------------------|----------------------------------------------------|----------|---------------|----------------------| -| `engine_name` | The name of the engine you want to install. | Yes | - | - | +| `engine_name` | The name of the engine you want to install. | Yes | `llama-cpp`, `onnxruntime`, `tensorrt-llm` | - | | `-h`, `--help` | Display help for command. | No | - | `-h` | ## `cortex engines uninstall` @@ -243,32 +150,15 @@ cortex engines install tensorrt-llm This command uninstalls the engine within Cortex. **Usage**: -:::info -You can use the `--verbose` flag to display more detailed output of the internal processes. To apply this flag, use the following format: `cortex --verbose [subcommand]`. -::: ```sh - # Stable cortex engines uninstall [options] - - # Beta - cortex-beta engines uninstall [options] - - # Nightly - cortex-nightly engines uninstall [options] ``` ```sh - # Stable cortex.exe engines uninstall [options] - - # Beta - cortex-beta.exe engines uninstall [options] - - # Nightly - cortex-nightly.exe engines uninstall [options] ``` @@ -277,13 +167,6 @@ For Example: ```bash ## Llama.cpp engine cortex engines uninstall llama-cpp - -## ONNX engine -cortex engines uninstall onnxruntime - -## Tensorrt-LLM engine -cortex engines uninstall tensorrt-llm - ``` **Options**: diff --git a/docs/docs/cli/models/index.mdx b/docs/docs/cli/models/index.mdx index 023ab412a..0445a9ba5 100644 --- a/docs/docs/cli/models/index.mdx +++ b/docs/docs/cli/models/index.mdx @@ -5,10 +5,6 @@ title: Cortex Models import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; -:::warning -🚧 Cortex.cpp is currently under development. Our documentation outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase. -::: - # `cortex models` This command allows you to start, stop, and manage various local or remote model operations within Cortex. @@ -21,26 +17,13 @@ You can use the `--verbose` flag to display more detailed output of the internal ```sh - # Stable cortex models [options] [subcommand] - - # Beta - cortex-beta models [options] [subcommand] - - # Nightly - cortex-nightly models [options] [subcommand] ``` ```sh - # Stable cortex.exe models [options] - - # Beta - cortex-beta.exe models [options] - # Nightly - cortex-nightly.exe models [options] ``` @@ -51,7 +34,8 @@ You can use the `--verbose` flag to display more detailed output of the internal |-------------------|-------------------------------------------------------|----------|---------------|-----------------| | `-h`, `--help` | Display help information for the command. | No | - | `-h` | - +--- +# Subcommands: ## `cortex models get` :::info @@ -63,91 +47,26 @@ This command returns a model detail defined by a `model_id`. **Usage**: -:::info -You can use the `--verbose` flag to display more detailed output of the internal processes. To apply this flag, use the following format: `cortex --verbose [subcommand]`. -::: ```sh - # Stable cortex models get - - # Beta - cortex-beta models get - - # Nightly - cortex-nightly models get ``` ```sh - # Stable cortex.exe models get - - # Beta - cortex-beta.exe models get - # Nightly - cortex-nightly.exe models get ``` For example, it returns the following: -```bash -ModelConfig Details: -------------------- -id: tinyllama -name: tinyllama 1B -model: tinyllama:1B -version: 1 -stop: [] -top_p: 0.95 -temperature: 0.7 -frequency_penalty: 0 -presence_penalty: 0 -max_tokens: 4096 -stream: true -ngl: 33 -ctx_len: 4096 -engine: llamacpp -prompt_template: - -<|system|> -{system_message} - - - - -<|user|> -{prompt} - - -<|assistant|> - - -system_template: - -<|system|> - -user_template: - - - - -<|user|> - -ai_template: - - -<|assistant|> - - -tp: 0 -text_model: false -files: [model_path] -created: 1725342964 +```yaml +{ + "ai_template":"<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n","created":9223372036854775888,"ctx_len":4096,"dynatemp_exponent":1.0,"dynatemp_range":0.0,"engine":"llama-cpp","files":["models/cortex.so/llama3.2/3b-gguf-q4-km/model.gguf"],"frequency_penalty":0.0,"gpu_arch":"","id":"Llama-3.2-3B-Instruct","ignore_eos":false,"max_tokens":4096,"min_keep":0,"min_p":0.05000000074505806,"mirostat":false,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"Llama-3.2-3B-Instruct","n_parallel":1,"n_probs":0,"name":"llama3.2:3b-gguf-q4-km","ngl":29,"object":"model","os":"","owned_by":"","penalize_nl":false,"precision":"","presence_penalty":0.0,"prompt_template":"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{system_message}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n","quantization_method":"","repeat_last_n":64,"repeat_penalty":1.0,"result":"OK","seed":-1,"stop":["<|eot_id|>"],"stream":true,"system_template":"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n","temperature":0.69999998807907104,"text_model":false,"tfs_z":1.0,"top_k":40,"top_p":0.89999997615814209,"typ_p":1.0,"user_template":"<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n","version":"2" +} ``` :::info This command uses a `model_id` from the model that you have downloaded or available in your file system. @@ -170,45 +89,30 @@ This command lists all the downloaded local and remote models. **Usage**: -:::info -You can use the `--verbose` flag to display more detailed output of the internal processes. To apply this flag, use the following format: `cortex --verbose [subcommand]`. -::: ```sh - # Stable cortex models list [options] - - # Beta - cortex-beta models list [options] - - # Nightly - cortex-nightly models list [options] ``` ```sh - # Stable cortex.exe models list [options] - - # Beta - cortex-beta.exe models list [options] - - # Nightly - cortex-nightly.exe models list [options] ``` -For example, it returns the following: +For example, it returns the following:w ```bash -+---------+----------------+-----------------+---------+ -| (Index) | ID | engine | version | -+---------+----------------+-----------------+---------+ -| 1 | tinyllama-gguf | llamacpp | 1 | -+---------+----------------+-----------------+---------+ -| 2 | tinyllama | llamacpp | 1 | -+---------+----------------+-----------------+---------+ ++---------+---------------------------------------------------------------------------+ +| (Index) | ID | ++---------+---------------------------------------------------------------------------+ +| 1 | llama3.2:3b-gguf-q4-km | ++---------+---------------------------------------------------------------------------+ +| 2 | tinyllama:1b-gguf | ++---------+---------------------------------------------------------------------------+ +| 3 | TheBloke:Mistral-7B-Instruct-v0.1-GGUF:mistral-7b-instruct-v0.1.Q2_K.gguf | ++---------+---------------------------------------------------------------------------+ ``` @@ -229,32 +133,16 @@ This command starts a model defined by a `model_id`. **Usage**: -:::info -You can use the `--verbose` flag to display more detailed output of the internal processes. To apply this flag, use the following format: `cortex --verbose [subcommand]`. -::: ```sh - # Stable cortex models start [options] - - # Beta - cortex-beta models start [options] - - # Nightly - cortex-nightly models start [options] ``` ```sh - # Stable cortex.exe models start [options] - - # Beta - cortex-beta.exe models start [options] - # Nightly - cortex-nightly.exe models start [options] ``` @@ -270,8 +158,6 @@ This command uses a `model_id` from the model that you have downloaded or availa |---------------------------|---------------------------------------------------------------------------|----------|----------------------------------------------|------------------------| | `model_id` | The identifier of the model you want to start. | Yes | `Prompt to select from the available models` | `mistral` | | `-h`, `--help` | Display help information for the command. | No | - | `-h` | - ## `cortex models stop` :::info @@ -283,32 +169,16 @@ This command stops a model defined by a `model_id`. **Usage**: -:::info -You can use the `--verbose` flag to display more detailed output of the internal processes. To apply this flag, use the following format: `cortex --verbose [subcommand]`. -::: ```sh - # Stable cortex models stop - - # Beta - cortex-beta models stop - - # Nightly - cortex-nightly models stop ``` ```sh - # Stable cortex.exe models stop - - # Beta - cortex-beta.exe models stop - # Nightly - cortex-nightly.exe models stop ``` @@ -333,32 +203,16 @@ This command deletes a local model defined by a `model_id`. **Usage**: -:::info -You can use the `--verbose` flag to display more detailed output of the internal processes. To apply this flag, use the following format: `cortex --verbose [subcommand]`. -::: ```sh - # Stable cortex models delete - - # Beta - cortex-beta models delete - - # Nightly - cortex-nightly models delete ``` ```sh - # Stable cortex.exe models delete - - # Beta - cortex-beta.exe models delete - # Nightly - cortex-nightly.exe models delete ``` @@ -373,79 +227,25 @@ This command uses a `model_id` from the model that you have downloaded or availa | `model_id` | The identifier of the model you want to delete. | Yes | - | `mistral` | | `-h`, `--help` | Display help for command. | No | - | `-h` | -## `cortex models alias` -This command adds an alias to a local model that function the same as `model_id`. - - -**Usage**: +## `cortex models update` :::info -You can use the `--verbose` flag to display more detailed output of the internal processes. To apply this flag, use the following format: `cortex --verbose [subcommand]`. +This CLI command calls the following API endpoint: +- [Update Model](/api-reference#tag/models/patch/v1/models/{modelId) ::: - - - ```sh - # Stable - cortex models alias --model_id --alias - - # Beta - cortex-beta models alias --model_id --alias - - # Nightly - cortex-nightly models alias --model_id --alias - ``` - - - ```sh - # Stable - cortex.exe models alias --model_id --alias - - # Beta - cortex-beta.exe models alias --model_id --alias - - # Nightly - cortex-nightly.exe models alias --model_id --alias - ``` - - - - -**Options**: -| Option | Description | Required | Default value | Example | -|---------------------------|-----------------------------------------------------------------------------|----------|----------------------|------------------------| -| `--model_id` | The identifier of the model. | Yes | - | `mistral` | -| `-alias` | The new identifier for the model. | Yes | - | `mistral_2` | - -## `cortex models update` This command updates the `model.yaml` file of a local model. **Usage**: -:::info -You can use the `--verbose` flag to display more detailed output of the internal processes. To apply this flag, use the following format: `cortex --verbose [subcommand]`. -::: ```sh - # Stable cortex models update [options] - - # Beta - cortex-beta models update [options] - - # Nightly - cortex-nightly models update [options] ``` ```sh - # Stable cortex.exe models update [options] - - # Beta - cortex-beta.exe models update [options] - # Nightly - cortex-nightly.exe models update [options] ``` @@ -506,31 +306,19 @@ This command imports the local model using the model's `gguf` file. **Usage**: :::info -You can use the `--verbose` flag to display more detailed output of the internal processes. To apply this flag, use the following format: `cortex --verbose [subcommand]`. +This CLI command calls the following API endpoint: +- [Import Model](/api-reference#tag/models/post/v1/models/import) ::: ```sh - # Stable cortex models import --model_id --model_path - - # Beta - cortex-beta models import --model_id --model_path - - # Nightly - cortex-nightly models import --model_id --model_path ``` ```sh - # Stable cortex.exe models import --model_id --model_path - - # Beta - cortex-beta.exe models import --model_id --model_path - # Nightly - cortex-nightly.exe models import --model_id --model_path ```