From b4169f210c51b9a330d42885e4c124e89c16d34a Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Mon, 28 Oct 2024 18:16:15 +0200 Subject: [PATCH 01/58] wip --- docs/cody/clients/model-configuration.mdx | 557 ++++++++++++++++------ 1 file changed, 418 insertions(+), 139 deletions(-) diff --git a/docs/cody/clients/model-configuration.mdx b/docs/cody/clients/model-configuration.mdx index 799f383b6..1f03e7383 100644 --- a/docs/cody/clients/model-configuration.mdx +++ b/docs/cody/clients/model-configuration.mdx @@ -1,31 +1,44 @@ # LLM Model Configuration -

This guide will walk you through the steps to customize the LLM models available from your Sourcegraph Enterprise instance.

+

+ This guide will walk you through the steps to customize the LLM models + available from your Sourcegraph Enterprise instance. +

-For Sourcegraph Cloud customers, configuring the available LLM models requires contacting your Sourcegraph account team representative + + For Sourcegraph Cloud customers, configuring the available LLM models + requires contacting your Sourcegraph account team representative + Cody Enterprise can be configured using one of two methods: + 1. "Completions" Configuration 2. Model Configuration (Early Access Program) -The Model Configuration method is in Early Access Program and only avaiable on Sourcegraph v5.6.0 or later. In the future when Model Configuration exits EAP, configuring Cody via your Sourcegraph Enterprise instance via the "Completions" Configuration site configuration section will be deprecated. For now, both methods remain supported. We recommend you continue to use the Completions Configuration unless you have specific reason to do otherwise. + + The Model Configuration method is in Early Access Program and only avaiable + on Sourcegraph v5.6.0 or later. In the future when Model Configuration exits + EAP, configuring Cody via your Sourcegraph Enterprise instance via the + "Completions" Configuration site configuration section will be deprecated. + For now, both methods remain supported. We recommend you continue to use the + Completions Configuration unless you have specific reason to do otherwise. + ## "Completions" Configuration - ## Setting up Cody Enterprise You can set up Cody for your Enterprise instance in one of the following ways: -- [Self-hosted Sourcegraph](#cody-on-self-hosted-sourcegraph-enterprise) -- [Sourcegraph Cloud](#cody-on-sourcegraph-cloud) +- [Self-hosted Sourcegraph](#cody-on-self-hosted-sourcegraph-enterprise) +- [Sourcegraph Cloud](#cody-on-sourcegraph-cloud) ## Cody on self-hosted Sourcegraph Enterprise ### Prerequisites -- You have Sourcegraph version 5.1.0 or above -- A Sourcegraph Enterprise subscription with [Cody Gateway access](/cody/core-concepts/cody-gateway) or [an account with a third-party LLM provider](#supported-models-and-model-providers) +- You have Sourcegraph version 5.1.0 or above +- A Sourcegraph Enterprise subscription with [Cody Gateway access](/cody/core-concepts/cody-gateway) or [an account with a third-party LLM provider](#supported-models-and-model-providers) ### Enable Cody on your Sourcegraph instance @@ -35,28 +48,32 @@ This requires site-admin privileges. To do so, 1. First, configure your desired LLM provider either by [Using Sourcegraph Cody Gateway](/cody/core-concepts/cody-gateway#using-cody-gateway-in-sourcegraph-enterprise) (recommended) or [Using a third-party LLM provider directly](#supported-models-and-model-providers) - If you are a Sourcegraph Cloud customer, skip directly to step 3. + + {' '} + If you are a Sourcegraph Cloud customer, skip directly to step 3. + +1 2. Next, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: ```json - { - // [...] - "cody.enabled": true, - "completions": { - "provider": "sourcegraph" - } - } +{ + // [...] + "cody.enabled": true, + "completions": { + "provider": "sourcegraph" + } +} ``` Cody is now fully enabled on your self-hosted Sourcegraph enterprise instance! ## Cody on Sourcegraph Cloud -- With [Sourcegraph Cloud](/cloud/), you get Cody as a managed service, and you **do not** need to [enable Cody as is required for self-hosted setup](#enable-cody-on-your-sourcegraph-instance) -- However, by contacting your account manager, Cody can still be enabled on-demand on your Sourcegraph instance. The Sourcegraph team will refer to the handbook -- Next, you can configure the [VS Code extension](#configure-the-vs-code-extension) by following the same steps as mentioned for the self-hosted environment -- After which, you are all set to use Cody with Sourcegraph Cloud +- With [Sourcegraph Cloud](/cloud/), you get Cody as a managed service, and you **do not** need to [enable Cody as is required for self-hosted setup](#enable-cody-on-your-sourcegraph-instance) +- However, by contacting your account manager, Cody can still be enabled on-demand on your Sourcegraph instance. The Sourcegraph team will refer to the handbook +- Next, you can configure the [VS Code extension](#configure-the-vs-code-extension) by following the same steps as mentioned for the self-hosted environment +- After which, you are all set to use Cody with Sourcegraph Cloud [Learn more about running Cody on Sourcegraph Cloud](/cloud/#cody). @@ -64,16 +81,16 @@ Cody is now fully enabled on your self-hosted Sourcegraph enterprise instance! To turn Cody off: -- Go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: +- Go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: ```json - { - // [...] - "cody.enabled": false - } +{ + // [...] + "cody.enabled": false +} ``` -- Next, remove `completions` configuration if they exist +- Next, remove `completions` configuration if they exist ## Enable Cody only for some users @@ -86,38 +103,43 @@ In Sourcegraph 5.3+, access to Cody is managed via user roles. By default, all u First, ensure Cody is enabled in your site configuration. Go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: ```json - { - // [...] - "cody.enabled": true, - // Make sure cody.restrictUsersFeatureFlag is not in your configuration! If it is, remove it. - } +{ + // [...] + "cody.enabled": true + // Make sure cody.restrictUsersFeatureFlag is not in your configuration! If it is, remove it. +} ``` - Ensure `cody.restrictUsersFeatureFlag` is **not** in your site configuration. If it is, remove it or else the old feature-flag approach from Sourcegraph 5.2 and earlier will be used. + + {' '} + Ensure `cody.restrictUsersFeatureFlag` is **not** in your site configuration. + If it is, remove it or else the old feature-flag approach from Sourcegraph 5.2 + and earlier will be used. + Next, go to **Site admin > Users & Auth > Roles** (`/site-admin/roles`) on your instance. On that page, you can: -- Control whether users _by default_ have access to Cody (expand `User [System]` and toggle **Cody** > **Access** as desired) -- Control whether groups of users have access to Cody (`+Create role` and enable the **Cody** > **Access** toggle as desired) +- Control whether users _by default_ have access to Cody (expand `User [System]` and toggle **Cody** > **Access** as desired) +- Control whether groups of users have access to Cody (`+Create role` and enable the **Cody** > **Access** toggle as desired) ### Sourcegraph 5.2 and earlier In Sourcegraph 5.2 and earlier, you should use the feature flag `cody` to turn Cody on selectively for some users. To do so: -- Go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: +- Go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: ```json - { - // [...] - "cody.enabled": true, - "cody.restrictUsersFeatureFlag": true - } +{ + // [...] + "cody.enabled": true, + "cody.restrictUsersFeatureFlag": true +} ``` -- Next, go to **Site admin > Feature flags** (`/site-admin/feature-flags`) -- Add a feature flag called `cody` -- Select the `boolean` type and set it to `false` -- Once added, click on the feature flag and use **add overrides** to pick users that will have access to Cody +- Next, go to **Site admin > Feature flags** (`/site-admin/feature-flags`) +- Add a feature flag called `cody` +- Select the `boolean` type and set it to `false` +- Once added, click on the feature flag and use **add overrides** to pick users that will have access to Cody ![add-overrides](https://user-images.githubusercontent.com/25070988/235454594-9f1a6b27-6882-44d9-be32-258d6c244880.png) @@ -125,16 +147,16 @@ In Sourcegraph 5.2 and earlier, you should use the feature flag `cody` to turn C [Cody Enterprise](https://sourcegraph.com/enterprise) supports many models and model providers. You can configure Cody Enterprise to access models via Sourcegraph Cody Gateway or directly using your own model provider account or infrastructure. -- Using [Sourcegraph Cody Gateway](/cody/core-concepts/cody-gateway): - - Recommended for most organizations. - - Supports [state-of-the-art models](/cody/capabilities/supported-models) from Anthropic, OpenAI, and more, without needing a separate account or incurring separate charges. -- Using your organization's account with a model provider: - - [Use your organization's Anthropic account](#use-your-organizations-anthropic-account) - - [Use your organization's OpenAI account](#use-your-organizations-openai-account) -- Using your organization's public cloud infrastructure: - - [Use Amazon Bedrock (AWS)](#use-amazon-bedrock-aws) - - [Use Azure OpenAI Service](#use-azure-openai-service) - - *Use Vertex AI on Google Cloud (coming soon)* +- Using [Sourcegraph Cody Gateway](/cody/core-concepts/cody-gateway): + - Recommended for most organizations. + - Supports [state-of-the-art models](/cody/capabilities/supported-models) from Anthropic, OpenAI, and more, without needing a separate account or incurring separate charges. +- Using your organization's account with a model provider: + - [Use your organization's Anthropic account](#use-your-organizations-anthropic-account) + - [Use your organization's OpenAI account](#use-your-organizations-openai-account) +- Using your organization's public cloud infrastructure: + - [Use Amazon Bedrock (AWS)](#use-amazon-bedrock-aws) + - [Use Azure OpenAI Service](#use-azure-openai-service) + - _Use Vertex AI on Google Cloud (coming soon)_ ### Use your organization's Anthropic account @@ -142,15 +164,15 @@ First, [create your own key with Anthropic](https://console.anthropic.com/accoun ```json { - // [...] - "cody.enabled": true, - "completions": { - "provider": "anthropic", - "chatModel": "claude-2.0", // Or any other model you would like to use - "fastChatModel": "claude-instant-1.2", // Or any other model you would like to use - "completionModel": "claude-instant-1.2", // Or any other model you would like to use - "accessToken": "" - } + // [...] + "cody.enabled": true, + "completions": { + "provider": "anthropic", + "chatModel": "claude-2.0", // Or any other model you would like to use + "fastChatModel": "claude-instant-1.2", // Or any other model you would like to use + "completionModel": "claude-instant-1.2", // Or any other model you would like to use + "accessToken": "" + } } ``` @@ -160,15 +182,15 @@ First, [create your own key with OpenAI](https://beta.openai.com/account/api-key ```json { - // [...] - "cody.enabled": true, - "completions": { - "provider": "openai", - "chatModel": "gpt-4", // Or any other model you would like to use - "fastChatModel": "gpt-3.5-turbo", // Or any other model you would like to use - "completionModel": "gpt-3.5-turbo-instruct", // Or any other model that supports the legacy completions endpoint - "accessToken": "" - } + // [...] + "cody.enabled": true, + "completions": { + "provider": "openai", + "chatModel": "gpt-4", // Or any other model you would like to use + "fastChatModel": "gpt-3.5-turbo", // Or any other model you would like to use + "completionModel": "gpt-3.5-turbo-instruct", // Or any other model that supports the legacy completions endpoint + "accessToken": "" + } } ``` @@ -187,15 +209,15 @@ Once ready, go to **Site admin > Site configuration** (`/site-admin/configuratio ```json { - // [...] - "cody.enabled": true, - "completions": { - "provider": "aws-bedrock", - "chatModel": "anthropic.claude-3-opus-20240229-v1:0", - "completionModel": "anthropic.claude-instant-v1", - "endpoint": "", - "accessToken": "" - } + // [...] + "cody.enabled": true, + "completions": { + "provider": "aws-bedrock", + "chatModel": "anthropic.claude-3-opus-20240229-v1:0", + "completionModel": "anthropic.claude-instant-v1", + "endpoint": "", + "accessToken": "" + } } ``` @@ -203,15 +225,14 @@ For the `chatModel` and `completionModel` fields, see [Amazon's Bedrock document For `endpoint`, you can either: -- For Pay-as-you-go, set it to an AWS region code (e.g., `us-west-2`) when using a public Amazon Bedrock endpoint -- For Provisioned Throughput, set it to the provisioned VPC endpoint for the `bedrock-runtime` API (e.g., `"https://vpce-0a10b2345cd67e89f-abc0defg.bedrock-runtime.us-west-2.vpce.amazonaws.com"`) +- For Pay-as-you-go, set it to an AWS region code (e.g., `us-west-2`) when using a public Amazon Bedrock endpoint +- For Provisioned Throughput, set it to the provisioned VPC endpoint for the `bedrock-runtime` API (e.g., `"https://vpce-0a10b2345cd67e89f-abc0defg.bedrock-runtime.us-west-2.vpce.amazonaws.com"`) For `accessToken`, you can either: -- Leave it empty and rely on instance role bindings or other AWS configurations in the `frontend` service -- Set it to `:` if directly configuring the credentials -- Set it to `::` if a session token is also required - +- Leave it empty and rely on instance role bindings or other AWS configurations in the `frontend` service +- Set it to `:` if directly configuring the credentials +- Set it to `::` if a session token is also required ### Using GCP Vertex AI @@ -219,46 +240,49 @@ Right now, We only support Anthropic Claude models on [GCP Vertex](https://cloud 1. Enable the [Vertex AI API](https://console.cloud.google.com/marketplace/product/google/aiplatform.googleapis.com) in the GCP console. Once Vertex has been enabled in your project, navigate to the [Vertex Model Garden](https://console.cloud.google.com/vertex-ai/model-garden) to select & enable the Anthropic Claude model(s) which you wish to use with Cody. See [Supported LLM Models](../capabilities/supported-models) for an up-to-date list of Anthropic Claude models supported by Cody. -It may take some time to enable Vertex and provision access to the models you plan to use + + It may take some time to enable Vertex and provision access to the models + you plan to use + 2. **Create a Service Account**: - - Create a [service account](https://cloud.google.com/iam/docs/service-account-overview). - - Assign the `Vertex AI User` role to the service account. - - Generate a JSON key for the service account and download it. + + - Create a [service account](https://cloud.google.com/iam/docs/service-account-overview). + - Assign the `Vertex AI User` role to the service account. + - Generate a JSON key for the service account and download it. 3. **Convert JSON Key to Base64** by doing: + ```python cat | base64 ``` Once ready, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: - ```json { - // [...] - "cody.enabled": true, - "completions": { - "chatModel": "claude-3-opus@20240229", - "completionModel": "claude-3-haiku@20240307", - "provider": "google", - "endpoint": "", - "accessToken": "" - } + // [...] + "cody.enabled": true, + "completions": { + "chatModel": "claude-3-opus@20240229", + "completionModel": "claude-3-haiku@20240307", + "provider": "google", + "endpoint": "", + "accessToken": "" + } } - ``` For the `Endpoint`, you can + 1. Navigate to the Documentation Page: Go to the Claude 3 Haiku Documentation on the GCP Console Model garden 2. Locate the Example: if you scroll enough through the page to find the example that shows how to use the cURL command with the Claude 3 Haiku model. The example will include a sample request JSON body and the necessary endpoint URL. Copy the URL in the site-admin config: - The endpoint URL will look something like this: + The endpoint URL will look something like this: `https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/anthropic/models/` 3. Example URL: -`https://us-east5-aiplatform.googleapis.com/v1/projects/sourcegraph-vertex-staging/locations/us-east5/publishers/anthropic/models` - + `https://us-east5-aiplatform.googleapis.com/v1/projects/sourcegraph-vertex-staging/locations/us-east5/publishers/anthropic/models` ### Use Azure OpenAI Service @@ -270,23 +294,23 @@ Once done, go to **Site admin > Site configuration** (`/site-admin/configuration ```json { - // [...] - "cody.enabled": true, - "completions": { - "provider": "azure-openai", - "chatModel": "", - "fastChatModel": "", - "completionModel": "", // the model must support the legacy completions endpoint such as gpt-3.5-turbo-instruct - "endpoint": "", - "accessToken": "" - } + // [...] + "cody.enabled": true, + "completions": { + "provider": "azure-openai", + "chatModel": "", + "fastChatModel": "", + "completionModel": "", // the model must support the legacy completions endpoint such as gpt-3.5-turbo-instruct + "endpoint": "", + "accessToken": "" + } } ``` For the access token, you can either: -- As of 5.2.4 the access token can be left empty and it will rely on Environmental, Workload Identity or Managed Identity credentials configured for the `frontend` and `worker` services -- Set it to `` if directly configuring the credentials using the API key specified in the Azure portal +- As of 5.2.4 the access token can be left empty and it will rely on Environmental, Workload Identity or Managed Identity credentials configured for the `frontend` and `worker` services +- Set it to `` if directly configuring the credentials using the API key specified in the Azure portal ### Use StarCoder for Autocomplete @@ -298,30 +322,42 @@ To enable StarCoder go to **Site admin > Site configuration** (`/site-admin/conf ```json { - // [...] - "cody.enabled": true, - "completions": { - "provider": "sourcegraph", - "completionModel": "fireworks/starcoder" - } + // [...] + "cody.enabled": true, + "completions": { + "provider": "sourcegraph", + "completionModel": "fireworks/starcoder" + } } ``` Users of the Cody Extensions will automatically pick up this change when connected to your Enterprise instance. - # Model Configuration -Sourcegraph v5.6.0 or later supports the ability to choose between different LLM models, allowing developers to use the best model for Cody Chat as needed. This is accomplished exposing much more flexible configuration options for Cody when using Sourcegraph Enterprise. The newer style of configuration is described next. However, you can still use the [Older style "Completions" Configuration](#legacy-completions-configuration). +Sourcegraph v5.6.0 or later supports the ability to choose between different LLM models, allowing developers to use the +best model for Cody Chat as needed. This is accomplished exposing much more flexible configuration options for Cody when +using Sourcegraph Enterprise. The newer style of configuration is described next. +However, you can still use the [Older style "Completions" Configuration](#legacy-completions-configuration). -## Quickstart + + Before you start, please note that the model configuration is an early + access program (EAP) and we are working towards improving on its coverage of + supported providers. If you are having any issues with this configuration, + please reach out to your Sourcegraph Account Representative or roll back + your configuration to the Legacy "Completions" configuration{' '} +{' '} + +The LLM models available for use from a Sourcegraph Enterprise instance are the union of "Sourcegraph-supplied models" +and any custom models providers that you explicitly add to your Sourcegraph instance's site configuration. +For most administrators, just relying on Sourcegraph-supplied models will ensure that you are using quality models +without needing to worry about the specifics. -Before you start, please note that the model configuration is an early access program (EAP) and we are working towards improving on its coverage of supported providers. If you are having any issues with this configuration, please reach out to your Sourcegraph Account Representative or roll back your configuration to the Legacy "Completions" configuration +## Quickstart -The simplest way to configure your Sourcegraph Enterprise would be to add the following configuration section to your instance's [site configuration](/admin/config/site_config): +The simplest way to configure your Sourcegraph Enterprise would be to [use the default model configuration via the Cody Gateway](/link): ```json - ... "cody.enabled": true, "modelConfiguration": { "sourcegraph": {} @@ -329,10 +365,251 @@ The simplest way to configure your Sourcegraph Enterprise would be to add the fo ... ``` -The `"modelConfiguration"` section defines which LLM models are supported by the Sourcegraph instance, and how to invoke them. The `"sourcegraph"` section defines how Sourcegraph-supplied LLM models should be configured. (That is, LLM models made available by the [Cody Gateway](/cody/core-concepts/cody-gateway) service.) The default settings will expose all current Cody Gateway models from your Sourcegraph instance, and make them available to users. +The `"sourcegraph"` section defines how Sourcegraph-supplied LLM models should be configured. (That is, LLM models made +available by the [Cody Gateway](/cody/core-concepts/cody-gateway) service.) +The default settings will expose all current Cody Gateway models from your Sourcegraph instance, and make them available to users. However, if you are seeking more control and wish to restrict which LLM models are available, or if you wish to use your own API access key, you can expand upon the `"modelConfiguration"` section as needed. +## Sourcegraph-supplied Models + +The Sourcegraph-supplied models are those that are available from [Cody Gateway](/cody/core-concepts/cody-gateway), +and your site configuration controls which of those models can be used. +There are three top-level settings for configuring Sourcegraph-supplied LLM models: + +| Field | Description | +| ------------------------- | ---------------------------------------------------------------------------------------- | +| `endpoint` (optional) | The URL for connecting to Cody Gateway, defaults to the production instance. | +| `accessToken` (optional) | The access token used to connect to Cody Gateway, defaulting to the current license key. | +| `modelFilters` (optional) | Filters for which models to include from Cody Gateway. | + +### Configure Cody to use the default model configuration via the Cody Gateway + +The easiest way to set up Cody on your Sourcegraph instance is by using the default model configuration, +which routes requests to LLM providers through the Cody Gateway. +To enable this, add the following section to your site configuration: + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": {} +} +``` + +This configuration establishes the following: + +- Your instance uses Sourcegraph-defined default models. +- Your instance can access all models provided by Sourcegraph. +- Your instance communicates with LLM providers through the Cody Gateway service. + +To view the default and available models, run the following command: + +```bash +curl curl --location 'https://your-instance.com/.api/modelconfig/supported-models.json' \ + --header 'Authorization: token YOUR_ACCESS_TOKEN' +``` + +This will return a JSON response with the following structure: + +```json +{ + "providers": [ + { + "id": "anthropic", + "displayName": "Anthropic" + }, + { + "id": "fireworks", + "displayName": "Fireworks" + }, + { + "id": "google", + "displayName": "Google" + }, + { + "id": "openai", + "displayName": "OpenAI" + } + // ...other providers + ], + "models": [ + { + "modelRef": "anthropic::2024-10-22::claude-3-5-sonnet-latest", + "displayName": "Claude 3.5 Sonnet (Latest)", + "modelName": "claude-3-5-sonnet-latest", + "capabilities": ["edit", "chat", "vision"], + "category": "accuracy", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "anthropic::2023-06-01::claude-3-opus", + "displayName": "Claude 3 Opus", + "modelName": "claude-3-opus-20240229", + "capabilities": ["edit", "chat"], + "category": "other", + "status": "stable", + "tier": "pro", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "anthropic::2023-06-01::claude-3-haiku", + "displayName": "Claude 3 Haiku", + "modelName": "claude-3-haiku-20240307", + "capabilities": ["edit", "chat"], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "fireworks::v1::starcoder", + "displayName": "StarCoder", + "modelName": "starcoder", + "capabilities": ["autocomplete"], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 2048, + "maxOutputTokens": 256 + } + }, + { + "modelRef": "fireworks::v1::deepseek-coder-v2-lite-base", + "displayName": "DeepSeek V2 Lite Base", + "modelName": "accounts/sourcegraph/models/deepseek-coder-v2-lite-base", + "capabilities": ["autocomplete"], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 2048, + "maxOutputTokens": 256 + } + }, + { + "modelRef": "google::v1::gemini-1.5-pro", + "displayName": "Gemini 1.5 Pro", + "modelName": "gemini-1.5-pro", + "capabilities": ["edit", "chat"], + "category": "balanced", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "openai::2024-02-01::gpt-4o", + "displayName": "GPT-4o", + "modelName": "gpt-4o", + "capabilities": ["edit", "chat", "vision"], + "category": "accuracy", + "status": "stable", + "tier": "pro", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + } + // ...other models + ], + "defaultModels": { + "chat": "anthropic::2024-10-22::claude-3-5-sonnet-latest", + "fastChat": "anthropic::2023-06-01::claude-3-haiku", + "codeCompletion": "fireworks::v1::starcoder" + } +} +``` + +Most of the configuration fields are self-explanatory, such as labeling the model's category ("stable", "beta") or category ("accuracy", "speed"). The more important fields are described below: + +**modelRef**. Each model is given a unique identifier, referred to as a model reference or "mref". This is a string of the form `${providerId}::${apiVersionId}::${modelId}`. + +In order to associate a model with your provider, the `${providerId}` must match. The `${modelId}` can be almost any URL-safe string. + +The `${apiVersionId}` is required in order to detect compatibility issues between newer models and Sourcegraph instances. In the following example, the string "2023-06-01" is used to clarify that this LLM model should formulate API requests using that version of the Anthropic API. If you are unsure, when defining your own models you can leave this as `"unknown"`. + +**contextWindow**. The context window is the number of "tokens" sent to an LLM. Either in the amount of contextual data sent in the LLM prompt (e.g. the question, relevant snippets, etc.) and the maximum size of the output allowed in the response. These values directly control factors such as the time it takes to respond to a prompt and the cost of the LLM request. And each LLM model or provider may have their own limits as well. + +**modelName**. The model _name_, is the value required by the LLM model's API provider. In this example, the `modelRef` defined the model's ID as `claude-3-sonnet` but the `modelName` was the more specific "claude-3-sonnet-20240229". + +**capabilities**. The capabilities of a model determine which situations the model can be used. For example, models only supported for "autocomplete" will not be available for Cody chats. + +### Model filters + +The `"modelFilters"` section is how you restrict which Cody Gateway models are made available to your Sourcegraph Enterprise instance's users. + +The first field is the `"statusFilter"`. Each LLM model is given a label by Sourcegraph as per its release, such as "stable", beta", or "experimental". By default, all models available on +Cody Gateway are exposed. Using the category filter ensures that only models with a particular category are made available to your users. + +The `"allow"` and `"deny"` fields, are arrays of `modelRef`s for what models should or should not be included. These values accept wild cards. + +The following examples illustrate how to use all these settings in conjunction: + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": { + "modelFilters": { + // Only allow "beta" and "stable" models. + // Not "experimental" or "deprecated". + "statusFilter": ["beta", "stable"], + + // Allow any models provided by Anthropic, OpenAI, Google and Fireworks. + "allow": [ + "anthropic::*", // Anthropic models + "openai::*", // OpenAI models + "google::*", // Google Gemini models + "fireworks::*", // Autocomplete models like StarCoder and DeepSeek-V2-Coder hosted on Fireworks + ], + + // Do not include any models with the Model ID containing "turbo", + // or any from AcmeCo. + "deny": [ + "*turbo*", + "acmeco::*" + ] + } + } +} +``` + +## Change default models + +The response from the request described in the section above includes a `defaultModels` section, indicating the default models for each of the Cody features. +To override these default models, add a `defaultModels` section within the `modelConfiguration` section in your site configuration. + +```json +"cody.enabled": true, +"modelConfiguration": { + // allow all Sourcegraph-supplied models via Cody Gateway + "sourcegraph": {}, + // override the default model for chat + "defaultModels": { + "chat": "google::v1::gemini-1.5-pro", + } +} +``` + +The configuration snippet above changes the default model for chat while retaining the default values for other features, such as "fastChat" and "autocomplete." + +Note: The values in the `defaultModels` section should match the `modelRef` of one of the available models (found in the `"models"` field of the `.api/modelconfig/supported-models.json` response). + +## + ## Concepts The LLM models available for use from a Sourcegraph Enterprise instance are the union of "Sourcegraph-supplied models" and any custom models providers that you explicitly add to your Sourcegraph instance's site configuration. For most administrators, just relying on Sourcegraph-supplied models will ensure that you are using quality models without needing to worry about the specifics. @@ -346,8 +623,8 @@ If you wish to not use _any_ Sourcegraph-supplied models, and instead _only_ rel There are three top-level settings for configuring Sourcegraph-supplied LLM models: | Field | Description | -| ----------- | ---------------------------------------------------------------------------------------- | -| `endpoint` (optional) | The URL for connecting to Cody Gateway, defaults to the production instance. | +| ------------------------- | ---------------------------------------------------------------------------------------- | +| `endpoint` (optional) | The URL for connecting to Cody Gateway, defaults to the production instance. | | `accessToken` (optional) | The access token used to connect to Cody Gateway, defaulting to the current license key. | | `modelFilters` (optional) | Filters for which models to include from Cody Gateway. | @@ -360,7 +637,6 @@ Cody Gateway are exposed. Using the category filter ensures that only models wit The `"allow"` and `"deny"` fields, are arrays of [model references](#model-configuration) for what models should or should not be included. These values accept wild cards. - The following examples illustrate how to use all these settings in conjunction: ```json @@ -414,7 +690,12 @@ The format of these strings is a "Model Reference", which is a format for unique For most administrators, relying on the LLM models made available by Cody Gateway is sufficient. However, if even more customization is required, you can configure your own LLM providers and models. -Defining your own LLM providers and models is an advanced use-case and requires care to get the correct results. It also may bypass protections to ensure compatibility between your Sourcegraph instance and LLMs. If you need help contact your Sourcegraph account executive. + + Defining your own LLM providers and models is an advanced use-case and + requires care to get the correct results. It also may bypass protections to + ensure compatibility between your Sourcegraph instance and LLMs. If you need + help contact your Sourcegraph account executive. + ### Overview @@ -426,7 +707,7 @@ A "provider" is a way to organize LLM models. Typically a provider would be refe By defining a provider override in your Sourcegraph site configuration, you are introducing a new namespace to contain models. Or customize the existing provider namespace supplied by Sourcegraph. (e.g. all `"anthropic"` models.) -The following configuration shippet defines a single provider override with the ID `"anthropic"`. +The following configuration snippet defines a single provider override with the ID `"anthropic"`. ```json "cody.enabled": true, @@ -475,8 +756,8 @@ In the example, the `"type"` field was `"anthropic"`. Meaning that any interacti However, Sourcegraph supports several different types of LLM API providers natively. The current set of supported LLM API providers is: -| Provider type | Description | -| -------------------- | ------------ | +| Provider type | Description | +| -------------------- | ------------------------------------------------------------------------------------------------------------ | | `"sourcegraph"` | [Cody Gateway](/cody/core-concepts/cody-gateway), which supports many different models from various services | | `"openaicompatible"` | Any OpenAI-compatible API implementation | | `"awsBedrock"` | [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/) | @@ -531,6 +812,4 @@ The `${apiVersionId}` is required in order to detect compatibility issues betwee **capabilities**. The capabilities of a model determine which situations the model can be used. For example, models only supported for "autocomplete" will not be available for Cody chats. - - It's recommended that every instance admin not using a third-party LLM provider makes this change and we are planning to make this the default in a future release. From f651e4feb878ef3db7ae39b90dd54968252a3137 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Mon, 28 Oct 2024 18:34:06 +0200 Subject: [PATCH 02/58] wip --- docs/cody/clients/model-configuration.mdx | 78 ----------------------- 1 file changed, 78 deletions(-) diff --git a/docs/cody/clients/model-configuration.mdx b/docs/cody/clients/model-configuration.mdx index 1f03e7383..d22dc7c4a 100644 --- a/docs/cody/clients/model-configuration.mdx +++ b/docs/cody/clients/model-configuration.mdx @@ -608,84 +608,6 @@ The configuration snippet above changes the default model for chat while retaini Note: The values in the `defaultModels` section should match the `modelRef` of one of the available models (found in the `"models"` field of the `.api/modelconfig/supported-models.json` response). -## - -## Concepts - -The LLM models available for use from a Sourcegraph Enterprise instance are the union of "Sourcegraph-supplied models" and any custom models providers that you explicitly add to your Sourcegraph instance's site configuration. For most administrators, just relying on Sourcegraph-supplied models will ensure that you are using quality models without needing to worry about the specifics. - -### Sourcegraph-supplied Models - -The Sourcegraph-supplied models are those that are available from [Cody Gateway](/cody/core-concepts/cody-gateway), and your site configuration controls which of those models can be used. - -If you wish to not use _any_ Sourcegraph-supplied models, and instead _only_ rely on those you have explicitly defined in your site configuration, you can set the `"sourcegraph"` field to `null`. - -There are three top-level settings for configuring Sourcegraph-supplied LLM models: - -| Field | Description | -| ------------------------- | ---------------------------------------------------------------------------------------- | -| `endpoint` (optional) | The URL for connecting to Cody Gateway, defaults to the production instance. | -| `accessToken` (optional) | The access token used to connect to Cody Gateway, defaulting to the current license key. | -| `modelFilters` (optional) | Filters for which models to include from Cody Gateway. | - -**Model Filters** - -The `"modelFilters"` section is how you restrict which Cody Gateway models are made available to your Sourcegraph Enterprise instance's users. - -The first field is the `"statusFilter"`. Each LLM model is given a label by Sourcegraph as per its release, such as "stable", beta", or "experimental". By default, all models available on -Cody Gateway are exposed. Using the category filter ensures that only models with a particular category are made available to your users. - -The `"allow"` and `"deny"` fields, are arrays of [model references](#model-configuration) for what models should or should not be included. These values accept wild cards. - -The following examples illustrate how to use all these settings in conjunction: - -```json -"cody.enabled": true, -"modelConfiguration": { - "sourcegraph": { - "modelFilters": { - // Only allow "beta" and "stable" models. - // Not "experimental" or "deprecated". - "statusFilter": ["beta", "stable"], - - // Allow any models provided by Anthropic, OpenAI, Google and Fireworks. - "allow": [ - "anthropic::*", // Anthropic models - "openai::*", // OpenAI models - "google::*", // Google Gemini models - "fireworks::*", // Autocomplete models like StarCoder and DeepSeek-V2-Coder hosted on Fireworks - ], - - // Do not include any models with the Model ID containing "turbo", - // or any from AcmeCo. - "deny": [ - "*turbo*", - "acmeco::*" - ] - } - } -} -``` - -## Default Models - -The `"modelConfiguration"` setting also contains a `"defaultModels"` field that allows you to specify the LLM model used depending on the situation. If no default is specified, or refers to a model that isn't found, it will silently fallback to a suitable alternative. - -```json - ... - "cody.enabled": true, - "modelConfiguration": { - "defaultModels": { - "chat": "anthropic::2023-06-01::claude-3.5-sonnet", - "codeCompletion": "anthropic::2023-06-01::claude-3.5-sonnet", - "fastChat": "anthropic::2023-06-01::claude-3-haiku" - } - } - ... -``` - -The format of these strings is a "Model Reference", which is a format for uniquely identifying each LLM model exposed from your Sourcegraph instance. - ## Advanced Configuration For most administrators, relying on the LLM models made available by Cody Gateway is sufficient. However, if even more customization is required, you can configure your own LLM providers and models. From f4533745ae2b5ae48f7fdab626e1c72ae26f38f3 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Mon, 28 Oct 2024 19:09:23 +0200 Subject: [PATCH 03/58] wip --- docs/cody/clients/model-configuration.mdx | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/docs/cody/clients/model-configuration.mdx b/docs/cody/clients/model-configuration.mdx index d22dc7c4a..7b6db8620 100644 --- a/docs/cody/clients/model-configuration.mdx +++ b/docs/cody/clients/model-configuration.mdx @@ -345,7 +345,7 @@ However, you can still use the [Older style "Completions" Configuration](#legacy access program (EAP) and we are working towards improving on its coverage of supported providers. If you are having any issues with this configuration, please reach out to your Sourcegraph Account Representative or roll back - your configuration to the Legacy "Completions" configuration{' '} + your configuration to the Legacy "Completions" configuration. {' '} The LLM models available for use from a Sourcegraph Enterprise instance are the union of "Sourcegraph-supplied models" @@ -355,15 +355,7 @@ without needing to worry about the specifics. ## Quickstart -The simplest way to configure your Sourcegraph Enterprise would be to [use the default model configuration via the Cody Gateway](/link): - -```json - "cody.enabled": true, - "modelConfiguration": { - "sourcegraph": {} - }, - ... -``` +The simplest way to configure your Sourcegraph Enterprise would be to [use the default model configuration](/link): The `"sourcegraph"` section defines how Sourcegraph-supplied LLM models should be configured. (That is, LLM models made available by the [Cody Gateway](/cody/core-concepts/cody-gateway) service.) @@ -383,7 +375,7 @@ There are three top-level settings for configuring Sourcegraph-supplied LLM mode | `accessToken` (optional) | The access token used to connect to Cody Gateway, defaulting to the current license key. | | `modelFilters` (optional) | Filters for which models to include from Cody Gateway. | -### Configure Cody to use the default model configuration via the Cody Gateway +### Use the default model configuration The easiest way to set up Cody on your Sourcegraph instance is by using the default model configuration, which routes requests to LLM providers through the Cody Gateway. From 44bf2d9647e6bdeb76fac465732d564ae69e1f4a Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Tue, 29 Oct 2024 13:42:11 +0200 Subject: [PATCH 04/58] revert changes --- docs/cody/clients/model-configuration.mdx | 527 +++++++--------------- 1 file changed, 167 insertions(+), 360 deletions(-) diff --git a/docs/cody/clients/model-configuration.mdx b/docs/cody/clients/model-configuration.mdx index 7b6db8620..799f383b6 100644 --- a/docs/cody/clients/model-configuration.mdx +++ b/docs/cody/clients/model-configuration.mdx @@ -1,44 +1,31 @@ # LLM Model Configuration -

- This guide will walk you through the steps to customize the LLM models - available from your Sourcegraph Enterprise instance. -

+

This guide will walk you through the steps to customize the LLM models available from your Sourcegraph Enterprise instance.

- - For Sourcegraph Cloud customers, configuring the available LLM models - requires contacting your Sourcegraph account team representative - +For Sourcegraph Cloud customers, configuring the available LLM models requires contacting your Sourcegraph account team representative Cody Enterprise can be configured using one of two methods: - 1. "Completions" Configuration 2. Model Configuration (Early Access Program) - - The Model Configuration method is in Early Access Program and only avaiable - on Sourcegraph v5.6.0 or later. In the future when Model Configuration exits - EAP, configuring Cody via your Sourcegraph Enterprise instance via the - "Completions" Configuration site configuration section will be deprecated. - For now, both methods remain supported. We recommend you continue to use the - Completions Configuration unless you have specific reason to do otherwise. - +The Model Configuration method is in Early Access Program and only avaiable on Sourcegraph v5.6.0 or later. In the future when Model Configuration exits EAP, configuring Cody via your Sourcegraph Enterprise instance via the "Completions" Configuration site configuration section will be deprecated. For now, both methods remain supported. We recommend you continue to use the Completions Configuration unless you have specific reason to do otherwise. ## "Completions" Configuration + ## Setting up Cody Enterprise You can set up Cody for your Enterprise instance in one of the following ways: -- [Self-hosted Sourcegraph](#cody-on-self-hosted-sourcegraph-enterprise) -- [Sourcegraph Cloud](#cody-on-sourcegraph-cloud) +- [Self-hosted Sourcegraph](#cody-on-self-hosted-sourcegraph-enterprise) +- [Sourcegraph Cloud](#cody-on-sourcegraph-cloud) ## Cody on self-hosted Sourcegraph Enterprise ### Prerequisites -- You have Sourcegraph version 5.1.0 or above -- A Sourcegraph Enterprise subscription with [Cody Gateway access](/cody/core-concepts/cody-gateway) or [an account with a third-party LLM provider](#supported-models-and-model-providers) +- You have Sourcegraph version 5.1.0 or above +- A Sourcegraph Enterprise subscription with [Cody Gateway access](/cody/core-concepts/cody-gateway) or [an account with a third-party LLM provider](#supported-models-and-model-providers) ### Enable Cody on your Sourcegraph instance @@ -48,32 +35,28 @@ This requires site-admin privileges. To do so, 1. First, configure your desired LLM provider either by [Using Sourcegraph Cody Gateway](/cody/core-concepts/cody-gateway#using-cody-gateway-in-sourcegraph-enterprise) (recommended) or [Using a third-party LLM provider directly](#supported-models-and-model-providers) - - {' '} - If you are a Sourcegraph Cloud customer, skip directly to step 3. - -1 + If you are a Sourcegraph Cloud customer, skip directly to step 3. 2. Next, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: ```json -{ - // [...] - "cody.enabled": true, - "completions": { - "provider": "sourcegraph" - } -} + { + // [...] + "cody.enabled": true, + "completions": { + "provider": "sourcegraph" + } + } ``` Cody is now fully enabled on your self-hosted Sourcegraph enterprise instance! ## Cody on Sourcegraph Cloud -- With [Sourcegraph Cloud](/cloud/), you get Cody as a managed service, and you **do not** need to [enable Cody as is required for self-hosted setup](#enable-cody-on-your-sourcegraph-instance) -- However, by contacting your account manager, Cody can still be enabled on-demand on your Sourcegraph instance. The Sourcegraph team will refer to the handbook -- Next, you can configure the [VS Code extension](#configure-the-vs-code-extension) by following the same steps as mentioned for the self-hosted environment -- After which, you are all set to use Cody with Sourcegraph Cloud +- With [Sourcegraph Cloud](/cloud/), you get Cody as a managed service, and you **do not** need to [enable Cody as is required for self-hosted setup](#enable-cody-on-your-sourcegraph-instance) +- However, by contacting your account manager, Cody can still be enabled on-demand on your Sourcegraph instance. The Sourcegraph team will refer to the handbook +- Next, you can configure the [VS Code extension](#configure-the-vs-code-extension) by following the same steps as mentioned for the self-hosted environment +- After which, you are all set to use Cody with Sourcegraph Cloud [Learn more about running Cody on Sourcegraph Cloud](/cloud/#cody). @@ -81,16 +64,16 @@ Cody is now fully enabled on your self-hosted Sourcegraph enterprise instance! To turn Cody off: -- Go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: +- Go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: ```json -{ - // [...] - "cody.enabled": false -} + { + // [...] + "cody.enabled": false + } ``` -- Next, remove `completions` configuration if they exist +- Next, remove `completions` configuration if they exist ## Enable Cody only for some users @@ -103,43 +86,38 @@ In Sourcegraph 5.3+, access to Cody is managed via user roles. By default, all u First, ensure Cody is enabled in your site configuration. Go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: ```json -{ - // [...] - "cody.enabled": true - // Make sure cody.restrictUsersFeatureFlag is not in your configuration! If it is, remove it. -} + { + // [...] + "cody.enabled": true, + // Make sure cody.restrictUsersFeatureFlag is not in your configuration! If it is, remove it. + } ``` - - {' '} - Ensure `cody.restrictUsersFeatureFlag` is **not** in your site configuration. - If it is, remove it or else the old feature-flag approach from Sourcegraph 5.2 - and earlier will be used. - + Ensure `cody.restrictUsersFeatureFlag` is **not** in your site configuration. If it is, remove it or else the old feature-flag approach from Sourcegraph 5.2 and earlier will be used. Next, go to **Site admin > Users & Auth > Roles** (`/site-admin/roles`) on your instance. On that page, you can: -- Control whether users _by default_ have access to Cody (expand `User [System]` and toggle **Cody** > **Access** as desired) -- Control whether groups of users have access to Cody (`+Create role` and enable the **Cody** > **Access** toggle as desired) +- Control whether users _by default_ have access to Cody (expand `User [System]` and toggle **Cody** > **Access** as desired) +- Control whether groups of users have access to Cody (`+Create role` and enable the **Cody** > **Access** toggle as desired) ### Sourcegraph 5.2 and earlier In Sourcegraph 5.2 and earlier, you should use the feature flag `cody` to turn Cody on selectively for some users. To do so: -- Go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: +- Go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: ```json -{ - // [...] - "cody.enabled": true, - "cody.restrictUsersFeatureFlag": true -} + { + // [...] + "cody.enabled": true, + "cody.restrictUsersFeatureFlag": true + } ``` -- Next, go to **Site admin > Feature flags** (`/site-admin/feature-flags`) -- Add a feature flag called `cody` -- Select the `boolean` type and set it to `false` -- Once added, click on the feature flag and use **add overrides** to pick users that will have access to Cody +- Next, go to **Site admin > Feature flags** (`/site-admin/feature-flags`) +- Add a feature flag called `cody` +- Select the `boolean` type and set it to `false` +- Once added, click on the feature flag and use **add overrides** to pick users that will have access to Cody ![add-overrides](https://user-images.githubusercontent.com/25070988/235454594-9f1a6b27-6882-44d9-be32-258d6c244880.png) @@ -147,16 +125,16 @@ In Sourcegraph 5.2 and earlier, you should use the feature flag `cody` to turn C [Cody Enterprise](https://sourcegraph.com/enterprise) supports many models and model providers. You can configure Cody Enterprise to access models via Sourcegraph Cody Gateway or directly using your own model provider account or infrastructure. -- Using [Sourcegraph Cody Gateway](/cody/core-concepts/cody-gateway): - - Recommended for most organizations. - - Supports [state-of-the-art models](/cody/capabilities/supported-models) from Anthropic, OpenAI, and more, without needing a separate account or incurring separate charges. -- Using your organization's account with a model provider: - - [Use your organization's Anthropic account](#use-your-organizations-anthropic-account) - - [Use your organization's OpenAI account](#use-your-organizations-openai-account) -- Using your organization's public cloud infrastructure: - - [Use Amazon Bedrock (AWS)](#use-amazon-bedrock-aws) - - [Use Azure OpenAI Service](#use-azure-openai-service) - - _Use Vertex AI on Google Cloud (coming soon)_ +- Using [Sourcegraph Cody Gateway](/cody/core-concepts/cody-gateway): + - Recommended for most organizations. + - Supports [state-of-the-art models](/cody/capabilities/supported-models) from Anthropic, OpenAI, and more, without needing a separate account or incurring separate charges. +- Using your organization's account with a model provider: + - [Use your organization's Anthropic account](#use-your-organizations-anthropic-account) + - [Use your organization's OpenAI account](#use-your-organizations-openai-account) +- Using your organization's public cloud infrastructure: + - [Use Amazon Bedrock (AWS)](#use-amazon-bedrock-aws) + - [Use Azure OpenAI Service](#use-azure-openai-service) + - *Use Vertex AI on Google Cloud (coming soon)* ### Use your organization's Anthropic account @@ -164,15 +142,15 @@ First, [create your own key with Anthropic](https://console.anthropic.com/accoun ```json { - // [...] - "cody.enabled": true, - "completions": { - "provider": "anthropic", - "chatModel": "claude-2.0", // Or any other model you would like to use - "fastChatModel": "claude-instant-1.2", // Or any other model you would like to use - "completionModel": "claude-instant-1.2", // Or any other model you would like to use - "accessToken": "" - } + // [...] + "cody.enabled": true, + "completions": { + "provider": "anthropic", + "chatModel": "claude-2.0", // Or any other model you would like to use + "fastChatModel": "claude-instant-1.2", // Or any other model you would like to use + "completionModel": "claude-instant-1.2", // Or any other model you would like to use + "accessToken": "" + } } ``` @@ -182,15 +160,15 @@ First, [create your own key with OpenAI](https://beta.openai.com/account/api-key ```json { - // [...] - "cody.enabled": true, - "completions": { - "provider": "openai", - "chatModel": "gpt-4", // Or any other model you would like to use - "fastChatModel": "gpt-3.5-turbo", // Or any other model you would like to use - "completionModel": "gpt-3.5-turbo-instruct", // Or any other model that supports the legacy completions endpoint - "accessToken": "" - } + // [...] + "cody.enabled": true, + "completions": { + "provider": "openai", + "chatModel": "gpt-4", // Or any other model you would like to use + "fastChatModel": "gpt-3.5-turbo", // Or any other model you would like to use + "completionModel": "gpt-3.5-turbo-instruct", // Or any other model that supports the legacy completions endpoint + "accessToken": "" + } } ``` @@ -209,15 +187,15 @@ Once ready, go to **Site admin > Site configuration** (`/site-admin/configuratio ```json { - // [...] - "cody.enabled": true, - "completions": { - "provider": "aws-bedrock", - "chatModel": "anthropic.claude-3-opus-20240229-v1:0", - "completionModel": "anthropic.claude-instant-v1", - "endpoint": "", - "accessToken": "" - } + // [...] + "cody.enabled": true, + "completions": { + "provider": "aws-bedrock", + "chatModel": "anthropic.claude-3-opus-20240229-v1:0", + "completionModel": "anthropic.claude-instant-v1", + "endpoint": "", + "accessToken": "" + } } ``` @@ -225,14 +203,15 @@ For the `chatModel` and `completionModel` fields, see [Amazon's Bedrock document For `endpoint`, you can either: -- For Pay-as-you-go, set it to an AWS region code (e.g., `us-west-2`) when using a public Amazon Bedrock endpoint -- For Provisioned Throughput, set it to the provisioned VPC endpoint for the `bedrock-runtime` API (e.g., `"https://vpce-0a10b2345cd67e89f-abc0defg.bedrock-runtime.us-west-2.vpce.amazonaws.com"`) +- For Pay-as-you-go, set it to an AWS region code (e.g., `us-west-2`) when using a public Amazon Bedrock endpoint +- For Provisioned Throughput, set it to the provisioned VPC endpoint for the `bedrock-runtime` API (e.g., `"https://vpce-0a10b2345cd67e89f-abc0defg.bedrock-runtime.us-west-2.vpce.amazonaws.com"`) For `accessToken`, you can either: -- Leave it empty and rely on instance role bindings or other AWS configurations in the `frontend` service -- Set it to `:` if directly configuring the credentials -- Set it to `::` if a session token is also required +- Leave it empty and rely on instance role bindings or other AWS configurations in the `frontend` service +- Set it to `:` if directly configuring the credentials +- Set it to `::` if a session token is also required + ### Using GCP Vertex AI @@ -240,49 +219,46 @@ Right now, We only support Anthropic Claude models on [GCP Vertex](https://cloud 1. Enable the [Vertex AI API](https://console.cloud.google.com/marketplace/product/google/aiplatform.googleapis.com) in the GCP console. Once Vertex has been enabled in your project, navigate to the [Vertex Model Garden](https://console.cloud.google.com/vertex-ai/model-garden) to select & enable the Anthropic Claude model(s) which you wish to use with Cody. See [Supported LLM Models](../capabilities/supported-models) for an up-to-date list of Anthropic Claude models supported by Cody. - - It may take some time to enable Vertex and provision access to the models - you plan to use - +It may take some time to enable Vertex and provision access to the models you plan to use 2. **Create a Service Account**: - - - Create a [service account](https://cloud.google.com/iam/docs/service-account-overview). - - Assign the `Vertex AI User` role to the service account. - - Generate a JSON key for the service account and download it. + - Create a [service account](https://cloud.google.com/iam/docs/service-account-overview). + - Assign the `Vertex AI User` role to the service account. + - Generate a JSON key for the service account and download it. 3. **Convert JSON Key to Base64** by doing: - ```python cat | base64 ``` Once ready, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: + ```json { - // [...] - "cody.enabled": true, - "completions": { - "chatModel": "claude-3-opus@20240229", - "completionModel": "claude-3-haiku@20240307", - "provider": "google", - "endpoint": "", - "accessToken": "" - } + // [...] + "cody.enabled": true, + "completions": { + "chatModel": "claude-3-opus@20240229", + "completionModel": "claude-3-haiku@20240307", + "provider": "google", + "endpoint": "", + "accessToken": "" + } } + ``` For the `Endpoint`, you can - 1. Navigate to the Documentation Page: Go to the Claude 3 Haiku Documentation on the GCP Console Model garden 2. Locate the Example: if you scroll enough through the page to find the example that shows how to use the cURL command with the Claude 3 Haiku model. The example will include a sample request JSON body and the necessary endpoint URL. Copy the URL in the site-admin config: - The endpoint URL will look something like this: + The endpoint URL will look something like this: `https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/anthropic/models/` 3. Example URL: - `https://us-east5-aiplatform.googleapis.com/v1/projects/sourcegraph-vertex-staging/locations/us-east5/publishers/anthropic/models` +`https://us-east5-aiplatform.googleapis.com/v1/projects/sourcegraph-vertex-staging/locations/us-east5/publishers/anthropic/models` + ### Use Azure OpenAI Service @@ -294,23 +270,23 @@ Once done, go to **Site admin > Site configuration** (`/site-admin/configuration ```json { - // [...] - "cody.enabled": true, - "completions": { - "provider": "azure-openai", - "chatModel": "", - "fastChatModel": "", - "completionModel": "", // the model must support the legacy completions endpoint such as gpt-3.5-turbo-instruct - "endpoint": "", - "accessToken": "" - } + // [...] + "cody.enabled": true, + "completions": { + "provider": "azure-openai", + "chatModel": "", + "fastChatModel": "", + "completionModel": "", // the model must support the legacy completions endpoint such as gpt-3.5-turbo-instruct + "endpoint": "", + "accessToken": "" + } } ``` For the access token, you can either: -- As of 5.2.4 the access token can be left empty and it will rely on Environmental, Workload Identity or Managed Identity credentials configured for the `frontend` and `worker` services -- Set it to `` if directly configuring the credentials using the API key specified in the Azure portal +- As of 5.2.4 the access token can be left empty and it will rely on Environmental, Workload Identity or Managed Identity credentials configured for the `frontend` and `worker` services +- Set it to `` if directly configuring the credentials using the API key specified in the Azure portal ### Use StarCoder for Autocomplete @@ -322,232 +298,68 @@ To enable StarCoder go to **Site admin > Site configuration** (`/site-admin/conf ```json { - // [...] - "cody.enabled": true, - "completions": { - "provider": "sourcegraph", - "completionModel": "fireworks/starcoder" - } + // [...] + "cody.enabled": true, + "completions": { + "provider": "sourcegraph", + "completionModel": "fireworks/starcoder" + } } ``` Users of the Cody Extensions will automatically pick up this change when connected to your Enterprise instance. -# Model Configuration - -Sourcegraph v5.6.0 or later supports the ability to choose between different LLM models, allowing developers to use the -best model for Cody Chat as needed. This is accomplished exposing much more flexible configuration options for Cody when -using Sourcegraph Enterprise. The newer style of configuration is described next. -However, you can still use the [Older style "Completions" Configuration](#legacy-completions-configuration). - - Before you start, please note that the model configuration is an early - access program (EAP) and we are working towards improving on its coverage of - supported providers. If you are having any issues with this configuration, - please reach out to your Sourcegraph Account Representative or roll back - your configuration to the Legacy "Completions" configuration. -{' '} +# Model Configuration -The LLM models available for use from a Sourcegraph Enterprise instance are the union of "Sourcegraph-supplied models" -and any custom models providers that you explicitly add to your Sourcegraph instance's site configuration. -For most administrators, just relying on Sourcegraph-supplied models will ensure that you are using quality models -without needing to worry about the specifics. +Sourcegraph v5.6.0 or later supports the ability to choose between different LLM models, allowing developers to use the best model for Cody Chat as needed. This is accomplished exposing much more flexible configuration options for Cody when using Sourcegraph Enterprise. The newer style of configuration is described next. However, you can still use the [Older style "Completions" Configuration](#legacy-completions-configuration). ## Quickstart -The simplest way to configure your Sourcegraph Enterprise would be to [use the default model configuration](/link): - -The `"sourcegraph"` section defines how Sourcegraph-supplied LLM models should be configured. (That is, LLM models made -available by the [Cody Gateway](/cody/core-concepts/cody-gateway) service.) -The default settings will expose all current Cody Gateway models from your Sourcegraph instance, and make them available to users. - -However, if you are seeking more control and wish to restrict which LLM models are available, or if you wish to use your own API access key, you can expand upon the `"modelConfiguration"` section as needed. - -## Sourcegraph-supplied Models - -The Sourcegraph-supplied models are those that are available from [Cody Gateway](/cody/core-concepts/cody-gateway), -and your site configuration controls which of those models can be used. -There are three top-level settings for configuring Sourcegraph-supplied LLM models: - -| Field | Description | -| ------------------------- | ---------------------------------------------------------------------------------------- | -| `endpoint` (optional) | The URL for connecting to Cody Gateway, defaults to the production instance. | -| `accessToken` (optional) | The access token used to connect to Cody Gateway, defaulting to the current license key. | -| `modelFilters` (optional) | Filters for which models to include from Cody Gateway. | - -### Use the default model configuration +Before you start, please note that the model configuration is an early access program (EAP) and we are working towards improving on its coverage of supported providers. If you are having any issues with this configuration, please reach out to your Sourcegraph Account Representative or roll back your configuration to the Legacy "Completions" configuration -The easiest way to set up Cody on your Sourcegraph instance is by using the default model configuration, -which routes requests to LLM providers through the Cody Gateway. -To enable this, add the following section to your site configuration: +The simplest way to configure your Sourcegraph Enterprise would be to add the following configuration section to your instance's [site configuration](/admin/config/site_config): ```json -"cody.enabled": true, -"modelConfiguration": { - "sourcegraph": {} -} + ... + "cody.enabled": true, + "modelConfiguration": { + "sourcegraph": {} + }, + ... ``` -This configuration establishes the following: - -- Your instance uses Sourcegraph-defined default models. -- Your instance can access all models provided by Sourcegraph. -- Your instance communicates with LLM providers through the Cody Gateway service. +The `"modelConfiguration"` section defines which LLM models are supported by the Sourcegraph instance, and how to invoke them. The `"sourcegraph"` section defines how Sourcegraph-supplied LLM models should be configured. (That is, LLM models made available by the [Cody Gateway](/cody/core-concepts/cody-gateway) service.) The default settings will expose all current Cody Gateway models from your Sourcegraph instance, and make them available to users. -To view the default and available models, run the following command: - -```bash -curl curl --location 'https://your-instance.com/.api/modelconfig/supported-models.json' \ - --header 'Authorization: token YOUR_ACCESS_TOKEN' -``` +However, if you are seeking more control and wish to restrict which LLM models are available, or if you wish to use your own API access key, you can expand upon the `"modelConfiguration"` section as needed. -This will return a JSON response with the following structure: +## Concepts -```json -{ - "providers": [ - { - "id": "anthropic", - "displayName": "Anthropic" - }, - { - "id": "fireworks", - "displayName": "Fireworks" - }, - { - "id": "google", - "displayName": "Google" - }, - { - "id": "openai", - "displayName": "OpenAI" - } - // ...other providers - ], - "models": [ - { - "modelRef": "anthropic::2024-10-22::claude-3-5-sonnet-latest", - "displayName": "Claude 3.5 Sonnet (Latest)", - "modelName": "claude-3-5-sonnet-latest", - "capabilities": ["edit", "chat", "vision"], - "category": "accuracy", - "status": "stable", - "tier": "free", - "contextWindow": { - "maxInputTokens": 45000, - "maxOutputTokens": 4000 - } - }, - { - "modelRef": "anthropic::2023-06-01::claude-3-opus", - "displayName": "Claude 3 Opus", - "modelName": "claude-3-opus-20240229", - "capabilities": ["edit", "chat"], - "category": "other", - "status": "stable", - "tier": "pro", - "contextWindow": { - "maxInputTokens": 45000, - "maxOutputTokens": 4000 - } - }, - { - "modelRef": "anthropic::2023-06-01::claude-3-haiku", - "displayName": "Claude 3 Haiku", - "modelName": "claude-3-haiku-20240307", - "capabilities": ["edit", "chat"], - "category": "speed", - "status": "stable", - "tier": "free", - "contextWindow": { - "maxInputTokens": 7000, - "maxOutputTokens": 4000 - } - }, - { - "modelRef": "fireworks::v1::starcoder", - "displayName": "StarCoder", - "modelName": "starcoder", - "capabilities": ["autocomplete"], - "category": "speed", - "status": "stable", - "tier": "free", - "contextWindow": { - "maxInputTokens": 2048, - "maxOutputTokens": 256 - } - }, - { - "modelRef": "fireworks::v1::deepseek-coder-v2-lite-base", - "displayName": "DeepSeek V2 Lite Base", - "modelName": "accounts/sourcegraph/models/deepseek-coder-v2-lite-base", - "capabilities": ["autocomplete"], - "category": "speed", - "status": "stable", - "tier": "free", - "contextWindow": { - "maxInputTokens": 2048, - "maxOutputTokens": 256 - } - }, - { - "modelRef": "google::v1::gemini-1.5-pro", - "displayName": "Gemini 1.5 Pro", - "modelName": "gemini-1.5-pro", - "capabilities": ["edit", "chat"], - "category": "balanced", - "status": "stable", - "tier": "free", - "contextWindow": { - "maxInputTokens": 45000, - "maxOutputTokens": 4000 - } - }, - { - "modelRef": "openai::2024-02-01::gpt-4o", - "displayName": "GPT-4o", - "modelName": "gpt-4o", - "capabilities": ["edit", "chat", "vision"], - "category": "accuracy", - "status": "stable", - "tier": "pro", - "contextWindow": { - "maxInputTokens": 45000, - "maxOutputTokens": 4000 - } - } - // ...other models - ], - "defaultModels": { - "chat": "anthropic::2024-10-22::claude-3-5-sonnet-latest", - "fastChat": "anthropic::2023-06-01::claude-3-haiku", - "codeCompletion": "fireworks::v1::starcoder" - } -} -``` +The LLM models available for use from a Sourcegraph Enterprise instance are the union of "Sourcegraph-supplied models" and any custom models providers that you explicitly add to your Sourcegraph instance's site configuration. For most administrators, just relying on Sourcegraph-supplied models will ensure that you are using quality models without needing to worry about the specifics. -Most of the configuration fields are self-explanatory, such as labeling the model's category ("stable", "beta") or category ("accuracy", "speed"). The more important fields are described below: +### Sourcegraph-supplied Models -**modelRef**. Each model is given a unique identifier, referred to as a model reference or "mref". This is a string of the form `${providerId}::${apiVersionId}::${modelId}`. +The Sourcegraph-supplied models are those that are available from [Cody Gateway](/cody/core-concepts/cody-gateway), and your site configuration controls which of those models can be used. -In order to associate a model with your provider, the `${providerId}` must match. The `${modelId}` can be almost any URL-safe string. +If you wish to not use _any_ Sourcegraph-supplied models, and instead _only_ rely on those you have explicitly defined in your site configuration, you can set the `"sourcegraph"` field to `null`. -The `${apiVersionId}` is required in order to detect compatibility issues between newer models and Sourcegraph instances. In the following example, the string "2023-06-01" is used to clarify that this LLM model should formulate API requests using that version of the Anthropic API. If you are unsure, when defining your own models you can leave this as `"unknown"`. - -**contextWindow**. The context window is the number of "tokens" sent to an LLM. Either in the amount of contextual data sent in the LLM prompt (e.g. the question, relevant snippets, etc.) and the maximum size of the output allowed in the response. These values directly control factors such as the time it takes to respond to a prompt and the cost of the LLM request. And each LLM model or provider may have their own limits as well. - -**modelName**. The model _name_, is the value required by the LLM model's API provider. In this example, the `modelRef` defined the model's ID as `claude-3-sonnet` but the `modelName` was the more specific "claude-3-sonnet-20240229". +There are three top-level settings for configuring Sourcegraph-supplied LLM models: -**capabilities**. The capabilities of a model determine which situations the model can be used. For example, models only supported for "autocomplete" will not be available for Cody chats. +| Field | Description | +| ----------- | ---------------------------------------------------------------------------------------- | +| `endpoint` (optional) | The URL for connecting to Cody Gateway, defaults to the production instance. | +| `accessToken` (optional) | The access token used to connect to Cody Gateway, defaulting to the current license key. | +| `modelFilters` (optional) | Filters for which models to include from Cody Gateway. | -### Model filters +**Model Filters** The `"modelFilters"` section is how you restrict which Cody Gateway models are made available to your Sourcegraph Enterprise instance's users. The first field is the `"statusFilter"`. Each LLM model is given a label by Sourcegraph as per its release, such as "stable", beta", or "experimental". By default, all models available on Cody Gateway are exposed. Using the category filter ensures that only models with a particular category are made available to your users. -The `"allow"` and `"deny"` fields, are arrays of `modelRef`s for what models should or should not be included. These values accept wild cards. +The `"allow"` and `"deny"` fields, are arrays of [model references](#model-configuration) for what models should or should not be included. These values accept wild cards. + The following examples illustrate how to use all these settings in conjunction: @@ -579,37 +391,30 @@ The following examples illustrate how to use all these settings in conjunction: } ``` -## Change default models +## Default Models -The response from the request described in the section above includes a `defaultModels` section, indicating the default models for each of the Cody features. -To override these default models, add a `defaultModels` section within the `modelConfiguration` section in your site configuration. +The `"modelConfiguration"` setting also contains a `"defaultModels"` field that allows you to specify the LLM model used depending on the situation. If no default is specified, or refers to a model that isn't found, it will silently fallback to a suitable alternative. ```json -"cody.enabled": true, -"modelConfiguration": { - // allow all Sourcegraph-supplied models via Cody Gateway - "sourcegraph": {}, - // override the default model for chat - "defaultModels": { - "chat": "google::v1::gemini-1.5-pro", + ... + "cody.enabled": true, + "modelConfiguration": { + "defaultModels": { + "chat": "anthropic::2023-06-01::claude-3.5-sonnet", + "codeCompletion": "anthropic::2023-06-01::claude-3.5-sonnet", + "fastChat": "anthropic::2023-06-01::claude-3-haiku" + } } -} + ... ``` -The configuration snippet above changes the default model for chat while retaining the default values for other features, such as "fastChat" and "autocomplete." - -Note: The values in the `defaultModels` section should match the `modelRef` of one of the available models (found in the `"models"` field of the `.api/modelconfig/supported-models.json` response). +The format of these strings is a "Model Reference", which is a format for uniquely identifying each LLM model exposed from your Sourcegraph instance. ## Advanced Configuration For most administrators, relying on the LLM models made available by Cody Gateway is sufficient. However, if even more customization is required, you can configure your own LLM providers and models. - - Defining your own LLM providers and models is an advanced use-case and - requires care to get the correct results. It also may bypass protections to - ensure compatibility between your Sourcegraph instance and LLMs. If you need - help contact your Sourcegraph account executive. - +Defining your own LLM providers and models is an advanced use-case and requires care to get the correct results. It also may bypass protections to ensure compatibility between your Sourcegraph instance and LLMs. If you need help contact your Sourcegraph account executive. ### Overview @@ -621,7 +426,7 @@ A "provider" is a way to organize LLM models. Typically a provider would be refe By defining a provider override in your Sourcegraph site configuration, you are introducing a new namespace to contain models. Or customize the existing provider namespace supplied by Sourcegraph. (e.g. all `"anthropic"` models.) -The following configuration snippet defines a single provider override with the ID `"anthropic"`. +The following configuration shippet defines a single provider override with the ID `"anthropic"`. ```json "cody.enabled": true, @@ -670,8 +475,8 @@ In the example, the `"type"` field was `"anthropic"`. Meaning that any interacti However, Sourcegraph supports several different types of LLM API providers natively. The current set of supported LLM API providers is: -| Provider type | Description | -| -------------------- | ------------------------------------------------------------------------------------------------------------ | +| Provider type | Description | +| -------------------- | ------------ | | `"sourcegraph"` | [Cody Gateway](/cody/core-concepts/cody-gateway), which supports many different models from various services | | `"openaicompatible"` | Any OpenAI-compatible API implementation | | `"awsBedrock"` | [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/) | @@ -726,4 +531,6 @@ The `${apiVersionId}` is required in order to detect compatibility issues betwee **capabilities**. The capabilities of a model determine which situations the model can be used. For example, models only supported for "autocomplete" will not be available for Cody chats. + + It's recommended that every instance admin not using a third-party LLM provider makes this change and we are planning to make this the default in a future release. From 63274ec92ff886ed7c7987bb325b3e1568c6308c Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Tue, 29 Oct 2024 18:13:37 +0200 Subject: [PATCH 05/58] wip --- docs/cody/model-configuration/index.mdx | 222 ++++++++++++++++++++ docs/cody/model-configuration/reference.mdx | 70 ++++++ src/data/navigation.ts | 10 + 3 files changed, 302 insertions(+) create mode 100644 docs/cody/model-configuration/index.mdx create mode 100644 docs/cody/model-configuration/reference.mdx diff --git a/docs/cody/model-configuration/index.mdx b/docs/cody/model-configuration/index.mdx new file mode 100644 index 000000000..360e55425 --- /dev/null +++ b/docs/cody/model-configuration/index.mdx @@ -0,0 +1,222 @@ +# Model Configuration + +Sourcegraph v5.6.0 or later supports the ability to choose between different LLM models, allowing developers to use the +best model for Cody Chat as needed. This is accomplished exposing much more flexible configuration options for Cody when +using Sourcegraph Enterprise. The newer style of configuration is described next. However, you can still use the +[Older style "Completions" Configuration](#legacy-completions-configuration). + + + Before you start, please note that the model configuration is an early + access program (EAP) and we are working towards improving on its coverage of + supported providers. If you are having any issues with this configuration, + please reach out to your Sourcegraph Account Representative or roll back + your configuration to the Legacy "Completions" configuration{' '} + + +The LLM models available for use from a Sourcegraph Enterprise instance are the union of [Sourcegraph-supplied models](/cody/model-configuration#sourcegraph-supplied-models) +and any custom models providers that you explicitly add to your Sourcegraph instance's site configuration. + +The model configuration for Cody is managed through the `"modelConfiguration"` field in the site config. +It includes the fields listed below, with a brief overview of each in the table. +For more detailed information, refer to the documentation for each field. + +| Field | Description | +| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| [`sourcegraph`](/cody/model-configuration#sourcegraph-supplied-models) | Configures access to Sourcegraph-supplied models available through Cody Gateway. | +| `providerOverrides` | Configures model providers, allowing you to customize Cody's connection to model providers, such as using your own API keys or self-hosted models. | +| `modelOverrides` | Extends or modifies the list of models Cody recognizes, along with their configurations. | +| `selfHostedModels` | Adds models to Cody’s recognized models list with default configurations provided by Sourcegraph. Only available for certain models; general models can be configured in `modelOverrides`. | +| `defaultModels` | Specifies the models assigned to each Cody feature (chat, fast edit, autocomplete). | + +## Quickstart + +The quickest and recommended way to set up model configuration is by using Sourcegraph-supplied models through the Cody Gateway. + +For a minimal configuration example, see [Configure Sourcegraph-supplied models](/cody/model-configuration#configure-sourcegraph-supplied-models). + +## Sourcegraph-supplied models + +Sourcegraph-supplied models, accessible through the [Cody Gateway](/cody/core-concepts/cody-gateway), are managed via your site configuration. + +For most administrators, relying on these models alone ensures access to high-quality models without needing to manage specific configurations. + +Usage of these models is controlled through the `"modelConfiguration.sourcegraph"` field in the site config. + +### Disable Sourcegraph-supplied models + +To disable all Sourcegraph-supplied models and use only the models explicitly defined in your site configuration, set the `"sourcegraph"` field to `null` (see example below). + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": null, // ignore Sourcegraph-supplied models + "providerOverrides": { + // define access to the LLM providers + }, + "modelOverrides": { + // define models available via providers defined in the providerOverrides + }, + "defaultModels": { + // set default models per Cody feature from the list of models defined in modelOverrides + } +} +``` + +### Configure Sourcegraph-supplied models + +The minimal configuration for Sourcegraph-supplied models is: + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": {} +} +``` + +The above configuration sets up the following: + +- Sourcegraph-supplied models are enabled (`sourcegraph` is not set to `null`). +- Requests to LLM providers are routed through the Cody Gateway (no `providerOverrides` field specified). +- Sourcegraph-defined default models are used for Cody features (no `defaultModels` field specified). + +There are three main settings for configuring Sourcegraph-supplied LLM models: + +| Field | Description | +| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | +| `endpoint` | (Optional) The URL for connecting to Cody Gateway, defaulting to the production instance. | +| `accessToken` | (Optional) The access token for connecting to Cody Gateway, which defaults to the current license key. | +| [`modelFilters`](/cody/model-configuration#model-filters) | (Optional) Filters specifying which models to include from Cody Gateway. | + +### Model Filters + +The `"modelFilters"` section allows you to control which Cody Gateway models are available to users of your Sourcegraph Enterprise instance. The following table describes each field: + +| Field | Description | +| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `statusFilter` | Filters models by their release status, such as "stable", "beta", "deprecated" or "experimental." By default, all models available on Cody Gateway are accessible. | +| `allow` | An array of `modelRef`s specifying which models to include. Supports wildcards. | +| `deny` | An array of `modelRef`s specifying which models to exclude. Supports wildcards. | + +The following examples demonstrate how to use each of these settings together: + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": { + "modelFilters": { + // Only allow "beta" and "stable" models. + // Not "experimental" or "deprecated". + "statusFilter": ["beta", "stable"], + + // Allow any models provided by Anthropic, OpenAI, Google and Fireworks. + "allow": [ + "anthropic::*", // Anthropic models + "openai::*", // OpenAI models + "google::*", // Google Gemini models + "fireworks::*", // Autocomplete models like StarCoder and DeepSeek-V2-Coder hosted on Fireworks + ], + + // Do not include any models with the Model ID containing "turbo", + // or any from AcmeCo. + "deny": [ + "*turbo*", + "acmeco::*" + ] + } + } +} +``` + +## Provider Overrides + +A "provider" is an organizational concept for grouping LLM models. Typically, a provider refers to the company that +produced the model or the specific API/service used to access it, serving as a namespace. + +By defining a provider override in your Sourcegraph site configuration, you can introduce a new namespace to organize +models or customize the existing provider namespace supplied by Sourcegraph (e.g., for all `"anthropic"` models). + +Provider overrides are configured via the `"modelConfiguration.providerOverrides"` field in the site configuration. +This field is an array of items, each containing the following fields: + +| Field | Description | +| ------------------ | --------------------------------------------------------------------------------- | +| `id` | The namespace for models accessed via the provider. | +| `displayName` | A human-readable name for the provider. | +| `serverSideConfig` | Defines how to access the provider. See the section below for additional details. | + +TODO: add configuration examples. + +### Server-side Config + +The most important part of a provider's configuration is the `"serverSideConfig"` field, which defines how the LLM models +should be invoked—that is, which external service or API will handle the LLM requests. + +Sourcegraph natively supports several types of LLM API providers. The current set of supported providers includes: + +| Provider type | Description | +| -------------------- | ----------------------------------------------------------------------------------------------------------- | +| `"sourcegraph"` | [Cody Gateway](/cody/core-concepts/cody-gateway), offering access to various models from multiple services. | +| `"openaicompatible"` | Any OpenAI-compatible API implementation. | +| `"awsBedrock"` | [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/) | +| `"azureOpenAI"` | [Microsoft Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service/) | +| `"anthropic"` | [Anthropic](https://www.anthropic.com) | +| `"fireworks"` | [Fireworks AI](https://fireworks.ai) | +| `"google"` | [Google Gemini](http://cloud.google.com/gemini) and [Vertex](https://cloud.google.com/vertex-ai/) | +| `"openai"` | [OpenAI](http://platform.openai.com) | +| `"huggingface-tgi"` | [Hugging Face Text Generation Interface](https://huggingface.co/docs/text-generation-inference/en/index) | + +The configuration for `serverSideConfig` varies by provider type. + +TODO: add provider-specific examples here. + +## Model Overrides + +With a provider defined (either a Sourcegraph-supplied provider or a custom provider configured via the `providerOverrides` +field), custom models can be specified for that provider by adding them to the `"modelConfiguration.modelOverrides"` section. + +This field is an array of items, each with the following fields: + +- `modelRef` - Uniquely identifies the model within the provider namespace. + - A string in the format `${providerId}::${apiVersionId}::${modelId}`. + - To associate a model with your provider, `${providerId}` must match the provider’s ID. + - `${modelId}` can be any URL-safe string. + - `${apiVersionId}` specifies the API version, which helps detect compatibility issues between models and Sourcegraph instances. + For example, `"2023-06-01"` can indicate that the model uses that version of the Anthropic API. If unsure, you may set this to `"unknown"` when defining custom models. +- `displayName` - An optional, user-friendly name for the model. If not set, clients should display the `ModelID` part of the `modelRef` instead (not the `modelName`). +- `modelName` - A unique identifier used by the API provider to specify which model is being invoked. + This is the identifier that the LLM provider recognizes to determine the model you are calling. +- `capabilities` - A list of capabilities that the model supports. Supported values: "autocomplete", "chat", "edit". +- `category` - Specifies the model's category, with the following options: + + - `"balanced"` - Typically the best default choice for most users. This category is suited for state-of-the-art models, like Sonnet 3.5 (as of October 2024). + - `"speed"` - Ideal for low-parameter models that may not be suited for general-purpose chat but are beneficial for specialized tasks, such as query rewriting. + - `"accuracy"` - Reserved for models, like OpenAI o1, that use advanced reasoning techniques to improve response accuracy, though with slower latency. + - `"other"` - Used for older models without distinct advantages in reasoning or speed. Select this category if uncertain about which category to choose. + +- `contextWindow` - An object that defines the number of "tokens" (units of text) that can be sent to the LLM. + This setting influences response time and request cost, and may vary according to the limits set by each LLM model or provider. + It includes two fields: + - `maxInputTokens` - Specifies the maximum number of tokens for the contextual data in the prompt (e.g., question, relevant snippets). + - `maxOutputTokens` - Specifies the maximum number of tokens allowed in the response. +- `serverSideConfig` - Additional configuration for the model. The available fields include: + + - `awsBedrockProvisionedThroughput` - Specifies provisioned throughput settings for AWS Bedrock models, with the following fields: + + - `type` - Must be `"awsBedrockProvisionedThroughput"`. + - `arn` - The ARN (Amazon Resource Name) for provisioned throughput to use when sending requests to AWS Bedrock. + + - `openaicompatible` - Configuration specific to models provided by an OpenAI-compatible provider, with the following fields: + - `type` - Must be `"openaicompatible"`. + - `apiModel` - The literal string value of the `model` field to be sent to the `/chat/completions` API. + If set, Sourcegraph treats this as an opaque string and sends it directly to the API without inferring any additional information. + By default, the configured model name is sent. + +TODO: add examples here. + +## Self-hosted Models + +TODO + +## Default Models + +TODO diff --git a/docs/cody/model-configuration/reference.mdx b/docs/cody/model-configuration/reference.mdx new file mode 100644 index 000000000..3019a2caf --- /dev/null +++ b/docs/cody/model-configuration/reference.mdx @@ -0,0 +1,70 @@ +# Reference + +The model configuration includes the following fields, providing an overview of each in the table below. +For detailed information, refer to the documentation for each field. + +| Field | Description | +|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `sourcegraph` | Configures access to Sourcegraph-supplied models available through Cody Gateway. | +| `defaultModels` | Specifies the models assigned to each Cody feature (chat, fast edit, autocomplete). | +| `providerOverrides` | Configures model providers, allowing you to customize Cody's connection to model providers, such as using your own API keys or self-hosted models. | +| `modelOverrides` | Extends or modifies the list of models Cody recognizes, along with their configurations. | +| `selfHostedModels` | Adds models to Cody’s recognized models list with default configurations provided by Sourcegraph. Only available for certain models; general models can be configured in `modelOverrides`. | + +## `sourcegraph` + +The Sourcegraph-supplied models are those that are available from [Cody Gateway](/cody/core-concepts/cody-gateway), and your site configuration controls which of those models can be used. + +If you wish to not use _any_ Sourcegraph-supplied models, and instead _only_ rely on those you have explicitly defined in your site configuration, you can set the `"sourcegraph"` field to `null`. + +There are three top-level settings for configuring Sourcegraph-supplied LLM models: + +| Field | Description | +|---------------------------|------------------------------------------------------------------------------------------| +| `endpoint` (optional) | The URL for connecting to Cody Gateway, defaults to the production instance. | +| `accessToken` (optional) | The access token used to connect to Cody Gateway, defaulting to the current license key. | +| `modelFilters` (optional) | Filters for which models to include from Cody Gateway. | + +**Model Filters** + +The `"modelFilters"` section is how you restrict which Cody Gateway models are made available to your Sourcegraph Enterprise instance's users. + +The first field is the `"statusFilter"`. Each LLM model is given a label by Sourcegraph as per its release, such as "stable", beta", or "experimental". By default, all models available on +Cody Gateway are exposed. Using the category filter ensures that only models with a particular category are made available to your users. + +The `"allow"` and `"deny"` fields, are arrays of [model references](#model-configuration) for what models should or should not be included. These values accept wild cards. + + +The following examples illustrate how to use all these settings in conjunction: + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": { + "modelFilters": { + // Only allow "beta" and "stable" models. + // Not "experimental" or "deprecated". + "statusFilter": ["beta", "stable"], + + // Allow any models provided by Anthropic, OpenAI, Google and Fireworks. + "allow": [ + "anthropic::*", // Anthropic models + "openai::*", // OpenAI models + "google::*", // Google Gemini models + "fireworks::*", // Autocomplete models like StarCoder and DeepSeek-V2-Coder hosted on Fireworks + ], + + // Do not include any models with the Model ID containing "turbo", + // or any from AcmeCo. + "deny": [ + "*turbo*", + "acmeco::*" + ] + } + } +} +``` + +| Field | Description | +| ----- | ----------- | +| \ No newline at end of file diff --git a/src/data/navigation.ts b/src/data/navigation.ts index 0a70c1909..46023bd0d 100644 --- a/src/data/navigation.ts +++ b/src/data/navigation.ts @@ -41,6 +41,16 @@ export const navigation: NavigationItem[] = [ { title: "Model Configuration", href: "/cody/clients/model-configuration", }, ] }, + { + title: "Model configuration", + href: "/cody/model-configuration", + subsections: [ + { + title: "Reference", + href: "/cody/model-configuration/reference", + } + ], + }, { title: "Capabilities", href: "/cody/capabilities", subsections: [ From 0b6962cf07be2ac51fac9979f5206594e6685594 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Wed, 30 Oct 2024 15:52:43 +0200 Subject: [PATCH 06/58] wip --- docs/cody/model-configuration/examples.mdx | 1 + docs/cody/model-configuration/index.mdx | 372 ++- docs/cody/model-configuration/reference.mdx | 70 - src/data/navigation.ts | 2257 ++++++++++++------- 4 files changed, 1750 insertions(+), 950 deletions(-) create mode 100644 docs/cody/model-configuration/examples.mdx delete mode 100644 docs/cody/model-configuration/reference.mdx diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx new file mode 100644 index 000000000..df635b4e6 --- /dev/null +++ b/docs/cody/model-configuration/examples.mdx @@ -0,0 +1 @@ +# Examples diff --git a/docs/cody/model-configuration/index.mdx b/docs/cody/model-configuration/index.mdx index 360e55425..1ef1994b0 100644 --- a/docs/cody/model-configuration/index.mdx +++ b/docs/cody/model-configuration/index.mdx @@ -23,10 +23,10 @@ For more detailed information, refer to the documentation for each field. | Field | Description | | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [`sourcegraph`](/cody/model-configuration#sourcegraph-supplied-models) | Configures access to Sourcegraph-supplied models available through Cody Gateway. | -| `providerOverrides` | Configures model providers, allowing you to customize Cody's connection to model providers, such as using your own API keys or self-hosted models. | -| `modelOverrides` | Extends or modifies the list of models Cody recognizes, along with their configurations. | -| `selfHostedModels` | Adds models to Cody’s recognized models list with default configurations provided by Sourcegraph. Only available for certain models; general models can be configured in `modelOverrides`. | -| `defaultModels` | Specifies the models assigned to each Cody feature (chat, fast edit, autocomplete). | +| [`providerOverrides`](/cody/model-configuration#provider-overrides) | Configures model providers, allowing you to customize Cody's connection to model providers, such as using your own API keys or self-hosted models. | +| [`modelOverrides`](/cody/model-configuration#model-overrides) | Extends or modifies the list of models Cody recognizes, along with their configurations. | +| [`selfHostedModels`](/cody/model-configuration#self-hosted-models) | Adds models to Cody’s recognized models list with default configurations provided by Sourcegraph. Only available for certain models; general models can be configured in `modelOverrides`. | +| [`defaultModels`](/cody/model-configuration#default-models) | Specifies the models assigned to each Cody feature (chat, fast edit, autocomplete). | ## Quickstart @@ -144,7 +144,38 @@ This field is an array of items, each containing the following fields: | `displayName` | A human-readable name for the provider. | | `serverSideConfig` | Defines how to access the provider. See the section below for additional details. | -TODO: add configuration examples. +Example configuration: + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": {}, + "providerOverrides": [ + { + "id": "openai", + "displayName": "OpenAI (via BYOK)", + "serverSideConfig": { + "type": "openai", + "accessToken": "sk-token", + "endpoint": "https://api.openai.com" + } + }, + ], + "defaultModels": { + "chat": "openai::v1::google::v1::gemini-1.5-pro", + "fastChat": "anthropic::2023-06-01::claude-3-haiku", + "autocomplete": "fireworks::v1::deepseek-coder-v2-lite-base" + } +} +``` + +In the example above: + +- Sourcegraph-supplied models are enabled (`sourcegraph` is not set to `null`). +- The `"openai"` provider configuration is overridden to be accessed using your own key, with the provider API accessed + directly via the specified `endpoint` URL. In contrast, models from the `"anthropic"` provider are accessed through the Cody Gateway. + +For additional examples, refer to the [examples page](/cody/model-configuration/examples). ### Server-side Config @@ -167,7 +198,7 @@ Sourcegraph natively supports several types of LLM API providers. The current se The configuration for `serverSideConfig` varies by provider type. -TODO: add provider-specific examples here. +For examples, refer to the [examples page](/cody/model-configuration/examples). ## Model Overrides @@ -211,12 +242,335 @@ This field is an array of items, each with the following fields: If set, Sourcegraph treats this as an opaque string and sends it directly to the API without inferring any additional information. By default, the configured model name is sent. -TODO: add examples here. +- `clientSideConfig` - Advanced configuration options that are only respected if the model is provided by the `"openaicompatible"` provider. + TODO: add more details. + +Example configuration: + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": {}, + "providerOverrides": [ + { + "id": "huggingface-codellama", + "displayName": "huggingface", + "serverSideConfig": { + "type": "openaicompatible", + "endpoints": [ + { + "url": "https://api-inference.huggingface.co/models/meta-llama/CodeLlama-7b-hf/v1/", + "accessToken": "hf_token" + } + ] + } + } + ], + "modelOverrides": [ + { + "modelRef": "huggingface-codellama::v1::CodeLlama-7b-hf", + "modelName": "CodeLlama-7b-hf", + "displayName": "(HuggingFace) CodeLlama-7b-hf", + "contextWindow": { + "maxInputTokens": 8192, + "maxOutputTokens": 4096 + }, + "serverSideConfig": { + "type": "openaicompatible", + "apiModel": "meta-llama/CodeLlama-7b-hf" + }, + "clientSideConfig": { + "openaicompatible": {} + }, + "capabilities": ["autocomplete", "chat"], + "category": "balanced", + "status": "stable" + } + ], + "defaultModels": { + "chat": "openai::v1::google::v1::gemini-1.5-pro", + "fastChat": "anthropic::2023-06-01::claude-3-haiku", + "autocomplete": "huggingface-codellama::v1::CodeLlama-7b-hf" + } +} +``` + +In the example above: + +- Sourcegraph-supplied models are enabled (`sourcegraph` is not set to `null`). +- An additional provider, `"huggingface-codellama"`, is configured to access Hugging Face’s OpenAI-compatible API directly. +- A custom model, `"CodeLlama-7b-hf"`, is added using the `"huggingface-codellama"` provider. +- Default models are set up as follows: + - Sourcegraph-supplied models are used for `"chat"` and `"fastChat"` (accessed via Cody Gateway). + - The newly configured model, `"huggingface-codellama::v1::CodeLlama-7b-hf"`, is used for `"autocomplete"` + (connecting directly to Hugging Face’s OpenAI-compatible API). + +For additional examples, refer to the [examples page](/cody/model-configuration/examples). ## Self-hosted Models -TODO +With a provider defined (either a Sourcegraph-supplied provider or a custom provider configured via the `providerOverrides` field), +self-hosted models can be added to Cody’s recognized models list by configuring the `"modelConfiguration.selfHostedModels"` section. +These models use Sourcegraph’s default configuration settings and are only available for select models. +Generic models should instead be configured in the `"modelOverrides"` section. + +Specifying the same model in both `"selfHostedModels"` and `"modelOverrides"` is not allowed. + +This field is an array of items, each with the following fields: + +| Field | Description | +| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `provider` | (Required) The provider ID of the model, configured in the `providerOverrides` field. | +| `apiVersion` | (Optional) The API version to use. Defaults to `"v1"` if not specified. | +| `model` | (Optional) Specifies which default model configuration to use. Sourcegraph provides default configurations for select models. For arbitrary models, use the `modelOverrides` section. Available options include: `"starcoder2-7b@v1"`, `"starcoder2-15b@v1"`, `"mistral-7b-instruct@v1"`, `"mixtral-8x7b-instruct@v1"`, `"mixtral-8x22b-instruct@v1"` | + +TODO: add examples here. ## Default Models -TODO +The `"modelConfiguration"` setting includes a `"defaultModels"` field, which allows you to specify the LLM model used for +each Cody feature (`"chat"`, `"fastChat"`, and `"autocomplete"`). The values for each feature should be `modelRef`s of +either Sourcegraph-supplied models or models configured in the `modelOverrides` section. + +If no default is specified or if the specified model is not found, the configuration will silently fall back to a suitable alternative. + +Example configuration: + +```json +"cody.enabled": true, + +// allow Sourcegraph-supplied models +"sourcegraph": {}, + +"defaultModels": { + "chat": "openai::2024-02-01::gpt-4o", + "fastChat": "google::v1::gemini-1.5-flash", + "codeCompletion": "fireworks::v1::deepseek-coder-v2-lite-base" +} +``` + +## View Configuration + +To view the current model configuration, run the following command: + +```bash +export INSTANCE_URL="https://sourcegraph.test:3443" # Replace with your Sourcegraph instance URL +export ACCESS_TOKEN="your access token" + +curl --location "${INSTANCE_URL}/.api/modelconfig/supported-models.json" \ +--header "Authorization: token $ACCESS_TOKEN" +``` + +The response includes: + +- Configured providers and models—both Sourcegraph-supplied (if enabled, with any applied filters) and any overrides. +- Default models for Cody features. + +Example response: + +```json +"schemaVersion": "1.0", +"revision": "0.0.0+dev", +"providers": [ + { + "id": "anthropic", + "displayName": "Anthropic" + }, + { + "id": "fireworks", + "displayName": "Fireworks" + }, + { + "id": "google", + "displayName": "Google" + }, + { + "id": "openai", + "displayName": "OpenAI" + }, + { + "id": "mistral", + "displayName": "Mistral" + } +], +"models": [ + { + "modelRef": "anthropic::2024-10-22::claude-3-5-sonnet-latest", + "displayName": "Claude 3.5 Sonnet (Latest)", + "modelName": "claude-3-5-sonnet-latest", + "capabilities": [ + "edit", + "chat", + "vision" + ], + "category": "accuracy", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "anthropic::2023-06-01::claude-3-opus", + "displayName": "Claude 3 Opus", + "modelName": "claude-3-opus-20240229", + "capabilities": [ + "edit", + "chat" + ], + "category": "other", + "status": "stable", + "tier": "pro", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "anthropic::2023-06-01::claude-3-haiku", + "displayName": "Claude 3 Haiku", + "modelName": "claude-3-haiku-20240307", + "capabilities": [ + "edit", + "chat" + ], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "fireworks::v1::starcoder", + "displayName": "StarCoder", + "modelName": "starcoder", + "capabilities": [ + "autocomplete" + ], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 2048, + "maxOutputTokens": 256 + } + }, + { + "modelRef": "fireworks::v1::deepseek-coder-v2-lite-base", + "displayName": "DeepSeek V2 Lite Base", + "modelName": "accounts/sourcegraph/models/deepseek-coder-v2-lite-base", + "capabilities": [ + "autocomplete" + ], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 2048, + "maxOutputTokens": 256 + } + }, + { + "modelRef": "google::v1::gemini-1.5-pro", + "displayName": "Gemini 1.5 Pro", + "modelName": "gemini-1.5-pro", + "capabilities": [ + "edit", + "chat" + ], + "category": "balanced", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "google::v1::gemini-1.5-flash", + "displayName": "Gemini 1.5 Flash", + "modelName": "gemini-1.5-flash", + "capabilities": [ + "edit", + "chat" + ], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "mistral::v1::mixtral-8x7b-instruct", + "displayName": "Mixtral 8x7B", + "modelName": "accounts/fireworks/models/mixtral-8x7b-instruct", + "capabilities": [ + "chat" + ], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "openai::2024-02-01::gpt-4o", + "displayName": "GPT-4o", + "modelName": "gpt-4o", + "capabilities": [ + "edit", + "chat", + "vision" + ], + "category": "accuracy", + "status": "stable", + "tier": "pro", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "openai::2024-02-01::cody-chat-preview-001", + "displayName": "OpenAI o1-preview", + "modelName": "cody-chat-preview-001", + "capabilities": [ + "chat" + ], + "category": "accuracy", + "status": "waitlist", + "tier": "pro", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "openai::2024-02-01::cody-chat-preview-002", + "displayName": "OpenAI o1-mini", + "modelName": "cody-chat-preview-002", + "capabilities": [ + "chat" + ], + "category": "accuracy", + "status": "waitlist", + "tier": "pro", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + } +], +"defaultModels": { + "chat": "anthropic::2024-10-22::claude-3-5-sonnet-latest", + "fastChat": "anthropic::2023-06-01::claude-3-haiku", + "codeCompletion": "fireworks::v1::deepseek-coder-v2-lite-base" +} +``` diff --git a/docs/cody/model-configuration/reference.mdx b/docs/cody/model-configuration/reference.mdx deleted file mode 100644 index 3019a2caf..000000000 --- a/docs/cody/model-configuration/reference.mdx +++ /dev/null @@ -1,70 +0,0 @@ -# Reference - -The model configuration includes the following fields, providing an overview of each in the table below. -For detailed information, refer to the documentation for each field. - -| Field | Description | -|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `sourcegraph` | Configures access to Sourcegraph-supplied models available through Cody Gateway. | -| `defaultModels` | Specifies the models assigned to each Cody feature (chat, fast edit, autocomplete). | -| `providerOverrides` | Configures model providers, allowing you to customize Cody's connection to model providers, such as using your own API keys or self-hosted models. | -| `modelOverrides` | Extends or modifies the list of models Cody recognizes, along with their configurations. | -| `selfHostedModels` | Adds models to Cody’s recognized models list with default configurations provided by Sourcegraph. Only available for certain models; general models can be configured in `modelOverrides`. | - -## `sourcegraph` - -The Sourcegraph-supplied models are those that are available from [Cody Gateway](/cody/core-concepts/cody-gateway), and your site configuration controls which of those models can be used. - -If you wish to not use _any_ Sourcegraph-supplied models, and instead _only_ rely on those you have explicitly defined in your site configuration, you can set the `"sourcegraph"` field to `null`. - -There are three top-level settings for configuring Sourcegraph-supplied LLM models: - -| Field | Description | -|---------------------------|------------------------------------------------------------------------------------------| -| `endpoint` (optional) | The URL for connecting to Cody Gateway, defaults to the production instance. | -| `accessToken` (optional) | The access token used to connect to Cody Gateway, defaulting to the current license key. | -| `modelFilters` (optional) | Filters for which models to include from Cody Gateway. | - -**Model Filters** - -The `"modelFilters"` section is how you restrict which Cody Gateway models are made available to your Sourcegraph Enterprise instance's users. - -The first field is the `"statusFilter"`. Each LLM model is given a label by Sourcegraph as per its release, such as "stable", beta", or "experimental". By default, all models available on -Cody Gateway are exposed. Using the category filter ensures that only models with a particular category are made available to your users. - -The `"allow"` and `"deny"` fields, are arrays of [model references](#model-configuration) for what models should or should not be included. These values accept wild cards. - - -The following examples illustrate how to use all these settings in conjunction: - -```json -"cody.enabled": true, -"modelConfiguration": { - "sourcegraph": { - "modelFilters": { - // Only allow "beta" and "stable" models. - // Not "experimental" or "deprecated". - "statusFilter": ["beta", "stable"], - - // Allow any models provided by Anthropic, OpenAI, Google and Fireworks. - "allow": [ - "anthropic::*", // Anthropic models - "openai::*", // OpenAI models - "google::*", // Google Gemini models - "fireworks::*", // Autocomplete models like StarCoder and DeepSeek-V2-Coder hosted on Fireworks - ], - - // Do not include any models with the Model ID containing "turbo", - // or any from AcmeCo. - "deny": [ - "*turbo*", - "acmeco::*" - ] - } - } -} -``` - -| Field | Description | -| ----- | ----------- | -| \ No newline at end of file diff --git a/src/data/navigation.ts b/src/data/navigation.ts index 46023bd0d..caa29a77f 100644 --- a/src/data/navigation.ts +++ b/src/data/navigation.ts @@ -1,895 +1,1410 @@ type Subsection = { - title: string; - href: string; + title: string; + href: string; }; type Section = { - title: string; - href: string; - subsections?: Subsection[]; + title: string; + href: string; + subsections?: Subsection[]; }; export type Topic = { - title: string; - href: string; - sections?: Section[]; + title: string; + href: string; + sections?: Section[]; }; export type NavigationItem = { - separator: string; - topics: Topic[]; + separator: string; + topics: Topic[]; }; export type VersionNavigations = Record; export const navigation: NavigationItem[] = [ - { - separator: "Code Intelligence", - topics: [ - { - title: "Cody", - href: "/cody", - sections: [ - { title: "Quickstart", href: "/cody/quickstart" }, - { - title: "Installation", href: "/cody/clients", - subsections: [ - { title: "Cody for VS Code", href: "/cody/clients/install-vscode", }, - { title: "Cody for JetBrains", href: "/cody/clients/install-jetbrains", }, - { title: "Cody for Web", href: "/cody/clients/cody-with-sourcegraph", }, - { title: "Cody for Enterprise", href: "/cody/clients/enable-cody-enterprise", }, - { title: "Model Configuration", href: "/cody/clients/model-configuration", }, - ] - }, - { - title: "Model configuration", - href: "/cody/model-configuration", - subsections: [ - { - title: "Reference", - href: "/cody/model-configuration/reference", - } - ], - }, - { - title: "Capabilities", href: "/cody/capabilities", - subsections: [ - { title: "Chat", href: "/cody/capabilities/chat", }, - { title: "Autocomplete", href: "/cody/capabilities/autocomplete", }, - { title: "Prompts & Commands", href: "/cody/capabilities/commands", }, - { title: "OpenCtx", href: "/cody/capabilities/openctx", }, - { title: "Debug Code", href: "/cody/capabilities/debug-code", }, - { title: "Context Filters", href: "/cody/capabilities/ignore-context", }, - { title: "Proxy Setup", href: "/cody/capabilities/proxy-setup", }, - { title: "Supported Models", href: "/cody/capabilities/supported-models", }, - { title: "Feature Parity Reference", href: "/cody/clients/feature-reference", }, - ] - }, - { - title: "Core Concepts", href: "/cody/core-concepts/context", - subsections: [ - { title: "Context", href: "/cody/core-concepts/context", }, - { title: "Token Limits", href: "/cody/core-concepts/token-limits", }, - // { title: "Embeddings", href: "/cody/core-concepts/embeddings", }, - { title: "Keyword Search", href: "/cody/core-concepts/keyword-search", }, - // { title: "Code Graph", href: "/cody/core-concepts/code-graph", }, - { title: "Cody Gateway", href: "/cody/core-concepts/cody-gateway", }, - { title: "Enterprise Architecture", href: "/cody/core-concepts/enterprise-architecture", }, - ] - }, - // { - // title: "Use Cases", href: "/cody/use-cases/generate-unit-tests", - // subsections: [ - // { title: "Generate Unit Tests", href: "/cody/use-cases/generate-unit-tests", }, - // // { title: "Build UI", href: "/cody/use-cases/build-ui", }, - // ] - // }, - { title: "Usage and Pricing", href: "/cody/usage-and-pricing" }, - { title: "Troubleshooting", href: "/cody/troubleshooting" }, - { title: "FAQs", href: "/cody/faq" }, - ], - }, - { - title: "Code Search", - href: "/code-search", - sections: [ - { title: "Features", href: "/code-search/features" }, - { - title: "Search Query Syntax", href: "/code-search/queries", - subsections: [ - { title: "Search Filters Panel", href: "/code-search/working/search_filters", }, - { title: "Search Examples", href: "/code-search/queries/examples", }, - { title: "Symbol Search", href: "/code-search/types/symbol", }, - { title: "Language Reference", href: "/code-search/queries/language", }, - ] - }, - { - title: "Code Navigation", href: "/code-search/code-navigation", - subsections: [ - { title: "Features", href: "/code-search/code-navigation/features", }, - { title: "Search-based code navigation", href: "/code-search/code-navigation/search_based_code_navigation", }, - { title: "Precise code navigation", href: "/code-search/code-navigation/precise_code_navigation", }, - { title: "Indexers", href: "/code-search/code-navigation/writing_an_indexer", }, - { title: "Auto-indexing", href: "/code-search/code-navigation/auto_indexing", }, - { title: "Environment Variables", href: "/code-search/code-navigation/envvars", }, - { title: "Troubleshooting", href: "/code-search/code-navigation/troubleshooting", }, - ] - }, - { - title: "Advanced Features", href: "/code-search/working/saved_searches", - subsections: [ - { title: "Fuzzy Finder", href: "/code-search/types/fuzzy", }, - { title: "Search Contexts", href: "/code-search/working/search_contexts", }, - { title: "Search Jobs", href: "/code-search/types/search-jobs", }, - { title: "Search Snippets", href: "/code-search/working/snippets", }, - { title: "Search Subexpressions", href: "/code-search/working/search_subexpressions", }, - { title: "Saved Searches", href: "/code-search/working/saved_searches", }, - { title: "Structural Search", href: "/code-search/types/structural", } - ] - }, - { - title: "FAQs", href: "/code-search/faq", - }, - ], - }, - ], - }, - { - separator: "Code Management", - topics: [ - { - title: "Batch Changes", - href: "/batch-changes", - sections: [ - { title: "Quickstart", href: "/batch-changes/quickstart" }, - { - title: "Create a Batch Change", href: "/batch-changes/create-a-batch-change", - subsections: [ - { title: "Update a Batch Change", href: "/batch-changes/update-a-batch-change", }, - { title: "Close a Batch Change", href: "/batch-changes/delete-a-batch-change", }, - { title: "Run Server Side", href: "/batch-changes/server-side", }, - - ] - }, - { - title: "Bulk Operations", href: "/batch-changes/bulk-operations-on-changesets", - subsections: [ - { title: "Publishing Changesets", href: "/batch-changes/publishing-changesets", }, - { title: "Tracking Changesets", href: "/batch-changes/tracking-existing-changesets", }, - ] - }, - { - title: "Configuring Credentials", href: "/batch-changes/configuring-credentials", - subsections: [ - { title: "Site Admin Config", href: "/batch-changes/site-admin-configuration", }, - ] - }, - { title: "Access Permissions", href: "/batch-changes/permissions-in-batch-changes" }, - { - title: "Changesets in Monorepo", href: "/batch-changes/creating-changesets-per-project-in-monorepos", - subsections: [ - { title: "Changesets in Large Repos", href: "/batch-changes/creating-multiple-changesets-in-large-repositories", }, - ] - }, - { title: "Error Handling", href: "/batch-changes/handling-errored-changesets" }, - { - title: "Examples", href: "/batch-changes/examples", - subsections: [ - { title: "Refactor Go Code via Comby", href: "/batch-changes/refactor-go-comby", }, - { title: "Update Go Import Statements via Comby", href: "/batch-changes/updating-go-import-statements", }, - { title: "Update Base Images in Dockerfiles", href: "/batch-changes/update-base-images-in-dockerfiles", }, - { title: "Search and Replace Specific Terms", href: "/batch-changes/search-and-replace-specific-terms", }, - ] - }, - { - title: "Batch Spec Reference", href: "/batch-changes/batch-spec-yaml-reference", - subsections: [ - { title: "Batch Spec Templating", href: "/batch-changes/batch-spec-templating", }, - { title: "Batch Spec Cheatsheet", href: "/batch-changes/batch-spec-cheat-sheet", }, - ] - }, - { - title: "Working with CLI", href: "/batch-changes/how-src-executes-a-batch-spec", - subsections: [ - { title: "Re-executing Batch Specs", href: "/batch-changes/reexecuting-batch-specs-multiple-times", }, - { title: "CLI Subcommands", href: "/cli/references/batch", }, - ] - }, - { title: "Troubleshooting", href: "/batch-changes/troubleshooting" }, - { title: "FAQs", href: "/batch-changes/faq" }, - ], - }, - { - title: "Code Monitoring", - href: "/code_monitoring", - sections: [ - { title: "Quickstart", href: "/code_monitoring/quickstart" }, - { title: "Explanations", href: "/code_monitoring/explanations" }, - { title: "How-to Guides", href: "/code_monitoring/how-tos" }, - ], - }, - { - title: "Code Ownership", - href: "/own", - sections: [ - { title: "CODEOWNERS Format", href: "/own/codeowners_format" }, - { title: "CODEOWNERS Ingestion", href: "/own/codeowners_ingestion" }, - { title: "Configuration Reference", href: "/own/configuration_reference" }, - { title: "Assigned Ownership", href: "/own/assigned_ownership" }, - - ], - }, - { - title: "Code Insights", - href: "/code_insights", - sections: [ - { title: "Quickstart", href: "/code_insights/quickstart" }, - { title: "Explanations", href: "/code_insights/explanations" }, - { title: "How-to Guides", href: "/code_insights/how-tos" }, - { title: "References", href: "/code_insights/references" }, - ], - }, - { - title: "Notebooks", - href: "/notebooks", - sections: [{ title: "Quickstart", href: "/notebooks/quickstart" }], - }, - ], - }, - { - separator: "Platform", - topics: [ - { - title: "Sourcegraph Admin", - href: "/admin", - sections: [ - { title: "Deploy", href: "/admin/deploy" }, - { title: "Upgrade", href: "/admin/updates" }, - { title: "Configuration", href: "/admin/config" }, - { title: "Licensing", href: "/admin/licensing" }, - { title: "Codehosts", href: "/admin/code_hosts" }, - { title: "User Authentication", href: "/admin/auth" }, - { title: "Access Control", href: "/admin/access_control" }, - { title: "Repository Permissions", href: "/admin/permissions" }, - { title: "Observability", href: "/admin/observability" }, - { title: "Analytics", href: "/admin/analytics" }, - { title: "Executors", href: "/admin/executors" }, - { title: "FAQs", href: "/admin/faq" }, - { title: "Troubleshooting", href: "/admin/troubleshooting" }, - { title: "How-to Guides", href: "/admin/how-to" }, - { - title: "Enterprise Getting Started", - href: "/admin/enterprise_getting_started_guide", - }, - { title: "Pricing", href: "/admin/pricing" }, - ], - }, - { - title: "Sourcegraph Cloud", - href: "/cloud", - }, - { - title: "Integrations", - href: "/integration", - sections: [ - { - title: "Browser Extension", - href: "/integration/browser_extension", - }, - { title: "Editors", href: "/integration/editor" }, - { - title: "Browser Search Engine", - href: "/integration/browser_extension/how-tos/browser_search_engine", - }, - ], - }, - ], - }, - { - separator: "CLI & API", - topics: [ - { - title: "Sourcegraph & Cody CLI", - href: "/cli", - sections: [ - { title: "Quickstart", href: "/cli/quickstart" }, - { title: "Explanations", href: "/cli/explanations" }, - { title: "How-to Guides", href: "/cli/how-tos" }, - { title: "References", href: "/cli/references" }, - { title: "Cody CLI", href: "/cody/clients/install-cli", }, - - ], - }, - { - title: "Sourcegraph GraphQL API", - href: "/api/graphql", - }, - { - title: "Sourcegraph Stream API", - href: "/api/stream_api", - }, - ], - }, - { - separator: "Help & Support", - topics: [ - { - title: "SLAs & Premium Support", - href: "/sla", - }, - { - title: "How to videos", - href: "/how-to-videos", - sections: [ - { title: "Code Search", href: "/how-to-videos/code-search" }, - { title: "Cody", href: "/how-to-videos/cody" }, - ], - }, - { - title: "Tutorials", - href: "/tutorials", - }, - { - title: "Sourcegraph Accounts", - href: "/sourcegraph-accounts" - }, - { - title: "Changelog", - href: "https://sourcegraph.com/changelog", - }, - { - title: "Technical changelog", - href: "/technical-changelog", - }, - { - title: "Releases", - href: "/releases", - }, - ], - }, + { + separator: 'Code Intelligence', + topics: [ + { + title: 'Cody', + href: '/cody', + sections: [ + {title: 'Quickstart', href: '/cody/quickstart'}, + { + title: 'Installation', + href: '/cody/clients', + subsections: [ + { + title: 'Cody for VS Code', + href: '/cody/clients/install-vscode' + }, + { + title: 'Cody for JetBrains', + href: '/cody/clients/install-jetbrains' + }, + { + title: 'Cody for Web', + href: '/cody/clients/cody-with-sourcegraph' + }, + { + title: 'Cody for Enterprise', + href: '/cody/clients/enable-cody-enterprise' + }, + { + title: 'Model Configuration', + href: '/cody/clients/model-configuration' + } + ] + }, + { + title: 'Model configuration', + href: '/cody/model-configuration', + subsections: [ + { + title: 'Examples', + href: '/cody/model-configuration/examples' + } + ] + }, + { + title: 'Capabilities', + href: '/cody/capabilities', + subsections: [ + {title: 'Chat', href: '/cody/capabilities/chat'}, + { + title: 'Autocomplete', + href: '/cody/capabilities/autocomplete' + }, + { + title: 'Prompts & Commands', + href: '/cody/capabilities/commands' + }, + { + title: 'OpenCtx', + href: '/cody/capabilities/openctx' + }, + { + title: 'Debug Code', + href: '/cody/capabilities/debug-code' + }, + { + title: 'Context Filters', + href: '/cody/capabilities/ignore-context' + }, + { + title: 'Proxy Setup', + href: '/cody/capabilities/proxy-setup' + }, + { + title: 'Supported Models', + href: '/cody/capabilities/supported-models' + }, + { + title: 'Feature Parity Reference', + href: '/cody/clients/feature-reference' + } + ] + }, + { + title: 'Core Concepts', + href: '/cody/core-concepts/context', + subsections: [ + { + title: 'Context', + href: '/cody/core-concepts/context' + }, + { + title: 'Token Limits', + href: '/cody/core-concepts/token-limits' + }, + // { title: "Embeddings", href: "/cody/core-concepts/embeddings", }, + { + title: 'Keyword Search', + href: '/cody/core-concepts/keyword-search' + }, + // { title: "Code Graph", href: "/cody/core-concepts/code-graph", }, + { + title: 'Cody Gateway', + href: '/cody/core-concepts/cody-gateway' + }, + { + title: 'Enterprise Architecture', + href: '/cody/core-concepts/enterprise-architecture' + } + ] + }, + // { + // title: "Use Cases", href: "/cody/use-cases/generate-unit-tests", + // subsections: [ + // { title: "Generate Unit Tests", href: "/cody/use-cases/generate-unit-tests", }, + // // { title: "Build UI", href: "/cody/use-cases/build-ui", }, + // ] + // }, + { + title: 'Usage and Pricing', + href: '/cody/usage-and-pricing' + }, + {title: 'Troubleshooting', href: '/cody/troubleshooting'}, + {title: 'FAQs', href: '/cody/faq'} + ] + }, + { + title: 'Code Search', + href: '/code-search', + sections: [ + {title: 'Features', href: '/code-search/features'}, + { + title: 'Search Query Syntax', + href: '/code-search/queries', + subsections: [ + { + title: 'Search Filters Panel', + href: '/code-search/working/search_filters' + }, + { + title: 'Search Examples', + href: '/code-search/queries/examples' + }, + { + title: 'Symbol Search', + href: '/code-search/types/symbol' + }, + { + title: 'Language Reference', + href: '/code-search/queries/language' + } + ] + }, + { + title: 'Code Navigation', + href: '/code-search/code-navigation', + subsections: [ + { + title: 'Features', + href: '/code-search/code-navigation/features' + }, + { + title: 'Search-based code navigation', + href: '/code-search/code-navigation/search_based_code_navigation' + }, + { + title: 'Precise code navigation', + href: '/code-search/code-navigation/precise_code_navigation' + }, + { + title: 'Indexers', + href: '/code-search/code-navigation/writing_an_indexer' + }, + { + title: 'Auto-indexing', + href: '/code-search/code-navigation/auto_indexing' + }, + { + title: 'Environment Variables', + href: '/code-search/code-navigation/envvars' + }, + { + title: 'Troubleshooting', + href: '/code-search/code-navigation/troubleshooting' + } + ] + }, + { + title: 'Advanced Features', + href: '/code-search/working/saved_searches', + subsections: [ + { + title: 'Fuzzy Finder', + href: '/code-search/types/fuzzy' + }, + { + title: 'Search Contexts', + href: '/code-search/working/search_contexts' + }, + { + title: 'Search Jobs', + href: '/code-search/types/search-jobs' + }, + { + title: 'Search Snippets', + href: '/code-search/working/snippets' + }, + { + title: 'Search Subexpressions', + href: '/code-search/working/search_subexpressions' + }, + { + title: 'Saved Searches', + href: '/code-search/working/saved_searches' + }, + { + title: 'Structural Search', + href: '/code-search/types/structural' + } + ] + }, + { + title: 'FAQs', + href: '/code-search/faq' + } + ] + } + ] + }, + { + separator: 'Code Management', + topics: [ + { + title: 'Batch Changes', + href: '/batch-changes', + sections: [ + {title: 'Quickstart', href: '/batch-changes/quickstart'}, + { + title: 'Create a Batch Change', + href: '/batch-changes/create-a-batch-change', + subsections: [ + { + title: 'Update a Batch Change', + href: '/batch-changes/update-a-batch-change' + }, + { + title: 'Close a Batch Change', + href: '/batch-changes/delete-a-batch-change' + }, + { + title: 'Run Server Side', + href: '/batch-changes/server-side' + } + ] + }, + { + title: 'Bulk Operations', + href: '/batch-changes/bulk-operations-on-changesets', + subsections: [ + { + title: 'Publishing Changesets', + href: '/batch-changes/publishing-changesets' + }, + { + title: 'Tracking Changesets', + href: '/batch-changes/tracking-existing-changesets' + } + ] + }, + { + title: 'Configuring Credentials', + href: '/batch-changes/configuring-credentials', + subsections: [ + { + title: 'Site Admin Config', + href: '/batch-changes/site-admin-configuration' + } + ] + }, + { + title: 'Access Permissions', + href: '/batch-changes/permissions-in-batch-changes' + }, + { + title: 'Changesets in Monorepo', + href: '/batch-changes/creating-changesets-per-project-in-monorepos', + subsections: [ + { + title: 'Changesets in Large Repos', + href: '/batch-changes/creating-multiple-changesets-in-large-repositories' + } + ] + }, + { + title: 'Error Handling', + href: '/batch-changes/handling-errored-changesets' + }, + { + title: 'Examples', + href: '/batch-changes/examples', + subsections: [ + { + title: 'Refactor Go Code via Comby', + href: '/batch-changes/refactor-go-comby' + }, + { + title: 'Update Go Import Statements via Comby', + href: '/batch-changes/updating-go-import-statements' + }, + { + title: 'Update Base Images in Dockerfiles', + href: '/batch-changes/update-base-images-in-dockerfiles' + }, + { + title: 'Search and Replace Specific Terms', + href: '/batch-changes/search-and-replace-specific-terms' + } + ] + }, + { + title: 'Batch Spec Reference', + href: '/batch-changes/batch-spec-yaml-reference', + subsections: [ + { + title: 'Batch Spec Templating', + href: '/batch-changes/batch-spec-templating' + }, + { + title: 'Batch Spec Cheatsheet', + href: '/batch-changes/batch-spec-cheat-sheet' + } + ] + }, + { + title: 'Working with CLI', + href: '/batch-changes/how-src-executes-a-batch-spec', + subsections: [ + { + title: 'Re-executing Batch Specs', + href: '/batch-changes/reexecuting-batch-specs-multiple-times' + }, + { + title: 'CLI Subcommands', + href: '/cli/references/batch' + } + ] + }, + { + title: 'Troubleshooting', + href: '/batch-changes/troubleshooting' + }, + {title: 'FAQs', href: '/batch-changes/faq'} + ] + }, + { + title: 'Code Monitoring', + href: '/code_monitoring', + sections: [ + {title: 'Quickstart', href: '/code_monitoring/quickstart'}, + { + title: 'Explanations', + href: '/code_monitoring/explanations' + }, + {title: 'How-to Guides', href: '/code_monitoring/how-tos'} + ] + }, + { + title: 'Code Ownership', + href: '/own', + sections: [ + { + title: 'CODEOWNERS Format', + href: '/own/codeowners_format' + }, + { + title: 'CODEOWNERS Ingestion', + href: '/own/codeowners_ingestion' + }, + { + title: 'Configuration Reference', + href: '/own/configuration_reference' + }, + { + title: 'Assigned Ownership', + href: '/own/assigned_ownership' + } + ] + }, + { + title: 'Code Insights', + href: '/code_insights', + sections: [ + {title: 'Quickstart', href: '/code_insights/quickstart'}, + { + title: 'Explanations', + href: '/code_insights/explanations' + }, + {title: 'How-to Guides', href: '/code_insights/how-tos'}, + {title: 'References', href: '/code_insights/references'} + ] + }, + { + title: 'Notebooks', + href: '/notebooks', + sections: [{title: 'Quickstart', href: '/notebooks/quickstart'}] + } + ] + }, + { + separator: 'Platform', + topics: [ + { + title: 'Sourcegraph Admin', + href: '/admin', + sections: [ + {title: 'Deploy', href: '/admin/deploy'}, + {title: 'Upgrade', href: '/admin/updates'}, + {title: 'Configuration', href: '/admin/config'}, + {title: 'Licensing', href: '/admin/licensing'}, + {title: 'Codehosts', href: '/admin/code_hosts'}, + {title: 'User Authentication', href: '/admin/auth'}, + {title: 'Access Control', href: '/admin/access_control'}, + { + title: 'Repository Permissions', + href: '/admin/permissions' + }, + {title: 'Observability', href: '/admin/observability'}, + {title: 'Analytics', href: '/admin/analytics'}, + {title: 'Executors', href: '/admin/executors'}, + {title: 'FAQs', href: '/admin/faq'}, + {title: 'Troubleshooting', href: '/admin/troubleshooting'}, + {title: 'How-to Guides', href: '/admin/how-to'}, + { + title: 'Enterprise Getting Started', + href: '/admin/enterprise_getting_started_guide' + }, + {title: 'Pricing', href: '/admin/pricing'} + ] + }, + { + title: 'Sourcegraph Cloud', + href: '/cloud' + }, + { + title: 'Integrations', + href: '/integration', + sections: [ + { + title: 'Browser Extension', + href: '/integration/browser_extension' + }, + {title: 'Editors', href: '/integration/editor'}, + { + title: 'Browser Search Engine', + href: '/integration/browser_extension/how-tos/browser_search_engine' + } + ] + } + ] + }, + { + separator: 'CLI & API', + topics: [ + { + title: 'Sourcegraph & Cody CLI', + href: '/cli', + sections: [ + {title: 'Quickstart', href: '/cli/quickstart'}, + {title: 'Explanations', href: '/cli/explanations'}, + {title: 'How-to Guides', href: '/cli/how-tos'}, + {title: 'References', href: '/cli/references'}, + {title: 'Cody CLI', href: '/cody/clients/install-cli'} + ] + }, + { + title: 'Sourcegraph GraphQL API', + href: '/api/graphql' + }, + { + title: 'Sourcegraph Stream API', + href: '/api/stream_api' + } + ] + }, + { + separator: 'Help & Support', + topics: [ + { + title: 'SLAs & Premium Support', + href: '/sla' + }, + { + title: 'How to videos', + href: '/how-to-videos', + sections: [ + {title: 'Code Search', href: '/how-to-videos/code-search'}, + {title: 'Cody', href: '/how-to-videos/cody'} + ] + }, + { + title: 'Tutorials', + href: '/tutorials' + }, + { + title: 'Sourcegraph Accounts', + href: '/sourcegraph-accounts' + }, + { + title: 'Changelog', + href: 'https://sourcegraph.com/changelog' + }, + { + title: 'Technical changelog', + href: '/technical-changelog' + }, + { + title: 'Releases', + href: '/releases' + } + ] + } ]; const navigation_5_2: NavigationItem[] = [ - { - separator: "Code Intelligence", - topics: [ - { - title: "Cody", - href: "/cody", - sections: [ - { title: "Quickstart", href: "/cody/quickstart" }, - { - title: "Installation", href: "/cody/clients", - subsections: [ - { title: "Cody for VS Code", href: "/cody/clients/install-vscode", }, - { title: "Cody for JetBrains", href: "/cody/clients/install-jetbrains", }, - { title: "Cody for Neovim", href: "/cody/clients/install-neovim", }, - { title: "Cody for Web", href: "/cody/clients/cody-with-sourcegraph", }, - { title: "Cody for Enterprise", href: "/cody/clients/enable-cody-enterprise", }, - { title: "LLM Model Configuration", href: "/cody/clients/model-configuration", }, - ] - }, - { - title: "Capabilities", href: "/cody/capabilities", - subsections: [ - { title: "Chat", href: "/cody/capabilities/chat", }, - { title: "Autocomplete", href: "/cody/capabilities/autocomplete", }, - { title: "Commands", href: "/cody/capabilities/commands", }, - { title: "Debug Code", href: "/cody/capabilities/debug-code", }, - { title: "Manage Context", href: "/cody/capabilities/ignore-context", }, - { title: "Feature Parity Reference", href: "/cody/clients/feature-reference", }, - ] - }, - { - title: "Core Concepts", href: "/cody/core-concepts/context", - subsections: [ - { title: "Context", href: "/cody/core-concepts/context", }, - { title: "Keyword Search", href: "/cody/core-concepts/keyword-search", }, - { title: "Cody Gateway", href: "/cody/core-concepts/cody-gateway", }, - ] - }, - { title: "Usage and Pricing", href: "/cody/usage-and-pricing" }, - { title: "Troubleshooting", href: "/cody/troubleshooting" }, - { title: "FAQs", href: "/cody/faq" }, - ], - }, - { - title: "Code Search", - href: "/code-search", - sections: [ - { title: "Features", href: "/code-search/features" }, - { - title: "Search Query Syntax", href: "/code-search/queries", - subsections: [ - { title: "Search Filters Panel", href: "/code-search/working/search_filters", }, - { title: "Search Examples", href: "/code-search/queries/examples", }, - { title: "Symbol Search", href: "/code-search/types/symbol", }, - { title: "Language Reference", href: "/code-search/queries/language", }, - ] - }, - { - title: "Advanced Features", href: "/code-search/working/saved_searches", - subsections: [ - { title: "Fuzzy Finder", href: "/code-search/types/fuzzy", }, - { title: "Search Contexts", href: "/code-search/working/search_contexts", }, - { title: "Search Jobs", href: "/code-search/types/search-jobs", }, - { title: "Search Snippets", href: "/code-search/working/snippets", }, - { title: "Search Subexpressions", href: "/code-search/working/search_subexpressions", }, - { title: "Saved Searches", href: "/code-search/working/saved_searches", }, - { title: "Structural Search", href: "/code-search/types/structural", } - ] - }, - { - title: "FAQs", href: "/code-search/faq", - }, - ], - }, - { - title: "Code Navigation", href: "/code_navigation", - sections: [ - { title: "Features", href: "/code_navigation/explanations/features", }, - { title: "Search-based code navigation", href: "/code_navigation/explanations/search_based_code_navigation", }, - { title: "Precise code navigation", href: "/code_navigation/explanations/precise_code_navigation", }, - { title: "Indexers", href: "/code_navigation/explanations/writing_an_indexer", }, - { title: "Auto-indexing", href: "/code_navigation/explanations/auto_indexing", }, - { title: "Environment Variables", href: "/code_navigation/references/envvars", }, - { title: "Troubleshooting", href: "/code_navigation/references/troubleshooting", }, - ] - }, - ], - }, - { - separator: "Code Management", - topics: [ - { - title: "Batch Changes", - href: "/batch_changes", - sections: [ - { title: "Quickstart", href: "/batch_changes/quickstart" }, - { title: "Explanations", href: "/batch_changes/explanations" }, - { title: "Tutorials", href: "/batch_changes/tutorials" }, - { title: "How-to Guides", href: "/batch_changes/how-tos" }, - { title: "Reference", href: "/batch_changes/references" }, - ], - }, - { - title: "Code Monitoring", - href: "/code_monitoring", - sections: [ - { title: "Quickstart", href: "/code_monitoring/quickstart" }, - { title: "Explanations", href: "/code_monitoring/explanations" }, - { title: "How-to Guides", href: "/code_monitoring/how-tos" }, - ], - }, - { - title: "Code Ownership", - href: "/own", - sections: [ - { title: "CODEOWNERS Format", href: "/own/codeowners_format" }, - { title: "CODEOWNERS Ingestion", href: "/own/codeowners_ingestion" }, - { title: "Configuration Reference", href: "/own/configuration_reference" }, - { title: "Assigned Ownership", href: "/own/assigned_ownership" }, - - ], - }, - { - title: "Code Insights", - href: "/code_insights", - sections: [ - { title: "Quickstart", href: "/code_insights/quickstart" }, - { title: "Explanations", href: "/code_insights/explanations" }, - { title: "How-to Guides", href: "/code_insights/how-tos" }, - { title: "References", href: "/code_insights/references" }, - ], - }, - { - title: "Notebooks", - href: "/notebooks", - sections: [{ title: "Quickstart", href: "/notebooks/quickstart" }], - }, - ], - }, - { - separator: "Platform", - topics: [ - { - title: "Sourcegraph Admin", - href: "/admin", - sections: [ - { title: "Deploy", href: "/admin/deploy" }, - { title: "Upgrade", href: "/admin/updates" }, - { title: "Configuration", href: "/admin/config" }, - { title: "Licensing", href: "/admin/licensing" }, - { title: "Codehosts", href: "/admin/code_hosts" }, - { title: "User Authentication", href: "/admin/auth" }, - { title: "Access Control", href: "/admin/access_control" }, - { title: "Repository Permissions", href: "/admin/permissions" }, - { title: "Observability", href: "/admin/observability" }, - { title: "Analytics", href: "/admin/analytics" }, - { title: "Executors", href: "/admin/executors" }, - { title: "FAQs", href: "/admin/faq" }, - { title: "Troubleshooting", href: "/admin/troubleshooting" }, - { title: "How-to Guides", href: "/admin/how-to" }, - { - title: "Enterprise Getting Started", - href: "/admin/enterprise_getting_started_guide", - }, - { title: "Pricing", href: "/admin/pricing" }, - ], - }, - { - title: "Sourcegraph Cloud", - href: "/cloud", - }, - { - title: "Integrations", - href: "/integration", - sections: [ - { - title: "Browser Extension", - href: "/integration/browser_extension", - }, - { title: "Editors", href: "/integration/editor" }, - { - title: "Browser Search Engine", - href: "/integration/browser_extension/how-tos/browser_search_engine", - }, - ], - }, - ], - }, - { - separator: "CLI & API", - topics: [ - { - title: "Sourcegraph CLI", - href: "/cli", - sections: [ - { title: "Quickstart", href: "/cli/quickstart" }, - { title: "Explanations", href: "/cli/explanations" }, - { title: "How-to Guides", href: "/cli/how-tos" }, - { title: "References", href: "/cli/references" }, - ], - }, - { - title: "Sourcegraph GraphQL API", - href: "/api/graphql", - }, - { - title: "Sourcegraph Stream API", - href: "/api/stream_api", - }, - ], - }, - { - separator: "Help & Support", - topics: [ - { - title: "SLAs & Premium Support", - href: "/sla", - }, - { - title: "Tutorials", - href: "/tutorials", - }, - { - title: "Sourcegraph Accounts", - href: "/sourcegraph-accounts" - }, - { - title: "Changelog", - href: "https://sourcegraph.com/changelog", - }, - { - title: "Technical changelog", - href: "/technical-changelog", - }, - ], - }, + { + separator: 'Code Intelligence', + topics: [ + { + title: 'Cody', + href: '/cody', + sections: [ + {title: 'Quickstart', href: '/cody/quickstart'}, + { + title: 'Installation', + href: '/cody/clients', + subsections: [ + { + title: 'Cody for VS Code', + href: '/cody/clients/install-vscode' + }, + { + title: 'Cody for JetBrains', + href: '/cody/clients/install-jetbrains' + }, + { + title: 'Cody for Neovim', + href: '/cody/clients/install-neovim' + }, + { + title: 'Cody for Web', + href: '/cody/clients/cody-with-sourcegraph' + }, + { + title: 'Cody for Enterprise', + href: '/cody/clients/enable-cody-enterprise' + }, + { + title: 'LLM Model Configuration', + href: '/cody/clients/model-configuration' + } + ] + }, + { + title: 'Capabilities', + href: '/cody/capabilities', + subsections: [ + {title: 'Chat', href: '/cody/capabilities/chat'}, + { + title: 'Autocomplete', + href: '/cody/capabilities/autocomplete' + }, + { + title: 'Commands', + href: '/cody/capabilities/commands' + }, + { + title: 'Debug Code', + href: '/cody/capabilities/debug-code' + }, + { + title: 'Manage Context', + href: '/cody/capabilities/ignore-context' + }, + { + title: 'Feature Parity Reference', + href: '/cody/clients/feature-reference' + } + ] + }, + { + title: 'Core Concepts', + href: '/cody/core-concepts/context', + subsections: [ + { + title: 'Context', + href: '/cody/core-concepts/context' + }, + { + title: 'Keyword Search', + href: '/cody/core-concepts/keyword-search' + }, + { + title: 'Cody Gateway', + href: '/cody/core-concepts/cody-gateway' + } + ] + }, + { + title: 'Usage and Pricing', + href: '/cody/usage-and-pricing' + }, + {title: 'Troubleshooting', href: '/cody/troubleshooting'}, + {title: 'FAQs', href: '/cody/faq'} + ] + }, + { + title: 'Code Search', + href: '/code-search', + sections: [ + {title: 'Features', href: '/code-search/features'}, + { + title: 'Search Query Syntax', + href: '/code-search/queries', + subsections: [ + { + title: 'Search Filters Panel', + href: '/code-search/working/search_filters' + }, + { + title: 'Search Examples', + href: '/code-search/queries/examples' + }, + { + title: 'Symbol Search', + href: '/code-search/types/symbol' + }, + { + title: 'Language Reference', + href: '/code-search/queries/language' + } + ] + }, + { + title: 'Advanced Features', + href: '/code-search/working/saved_searches', + subsections: [ + { + title: 'Fuzzy Finder', + href: '/code-search/types/fuzzy' + }, + { + title: 'Search Contexts', + href: '/code-search/working/search_contexts' + }, + { + title: 'Search Jobs', + href: '/code-search/types/search-jobs' + }, + { + title: 'Search Snippets', + href: '/code-search/working/snippets' + }, + { + title: 'Search Subexpressions', + href: '/code-search/working/search_subexpressions' + }, + { + title: 'Saved Searches', + href: '/code-search/working/saved_searches' + }, + { + title: 'Structural Search', + href: '/code-search/types/structural' + } + ] + }, + { + title: 'FAQs', + href: '/code-search/faq' + } + ] + }, + { + title: 'Code Navigation', + href: '/code_navigation', + sections: [ + { + title: 'Features', + href: '/code_navigation/explanations/features' + }, + { + title: 'Search-based code navigation', + href: '/code_navigation/explanations/search_based_code_navigation' + }, + { + title: 'Precise code navigation', + href: '/code_navigation/explanations/precise_code_navigation' + }, + { + title: 'Indexers', + href: '/code_navigation/explanations/writing_an_indexer' + }, + { + title: 'Auto-indexing', + href: '/code_navigation/explanations/auto_indexing' + }, + { + title: 'Environment Variables', + href: '/code_navigation/references/envvars' + }, + { + title: 'Troubleshooting', + href: '/code_navigation/references/troubleshooting' + } + ] + } + ] + }, + { + separator: 'Code Management', + topics: [ + { + title: 'Batch Changes', + href: '/batch_changes', + sections: [ + {title: 'Quickstart', href: '/batch_changes/quickstart'}, + { + title: 'Explanations', + href: '/batch_changes/explanations' + }, + {title: 'Tutorials', href: '/batch_changes/tutorials'}, + {title: 'How-to Guides', href: '/batch_changes/how-tos'}, + {title: 'Reference', href: '/batch_changes/references'} + ] + }, + { + title: 'Code Monitoring', + href: '/code_monitoring', + sections: [ + {title: 'Quickstart', href: '/code_monitoring/quickstart'}, + { + title: 'Explanations', + href: '/code_monitoring/explanations' + }, + {title: 'How-to Guides', href: '/code_monitoring/how-tos'} + ] + }, + { + title: 'Code Ownership', + href: '/own', + sections: [ + { + title: 'CODEOWNERS Format', + href: '/own/codeowners_format' + }, + { + title: 'CODEOWNERS Ingestion', + href: '/own/codeowners_ingestion' + }, + { + title: 'Configuration Reference', + href: '/own/configuration_reference' + }, + { + title: 'Assigned Ownership', + href: '/own/assigned_ownership' + } + ] + }, + { + title: 'Code Insights', + href: '/code_insights', + sections: [ + {title: 'Quickstart', href: '/code_insights/quickstart'}, + { + title: 'Explanations', + href: '/code_insights/explanations' + }, + {title: 'How-to Guides', href: '/code_insights/how-tos'}, + {title: 'References', href: '/code_insights/references'} + ] + }, + { + title: 'Notebooks', + href: '/notebooks', + sections: [{title: 'Quickstart', href: '/notebooks/quickstart'}] + } + ] + }, + { + separator: 'Platform', + topics: [ + { + title: 'Sourcegraph Admin', + href: '/admin', + sections: [ + {title: 'Deploy', href: '/admin/deploy'}, + {title: 'Upgrade', href: '/admin/updates'}, + {title: 'Configuration', href: '/admin/config'}, + {title: 'Licensing', href: '/admin/licensing'}, + {title: 'Codehosts', href: '/admin/code_hosts'}, + {title: 'User Authentication', href: '/admin/auth'}, + {title: 'Access Control', href: '/admin/access_control'}, + { + title: 'Repository Permissions', + href: '/admin/permissions' + }, + {title: 'Observability', href: '/admin/observability'}, + {title: 'Analytics', href: '/admin/analytics'}, + {title: 'Executors', href: '/admin/executors'}, + {title: 'FAQs', href: '/admin/faq'}, + {title: 'Troubleshooting', href: '/admin/troubleshooting'}, + {title: 'How-to Guides', href: '/admin/how-to'}, + { + title: 'Enterprise Getting Started', + href: '/admin/enterprise_getting_started_guide' + }, + {title: 'Pricing', href: '/admin/pricing'} + ] + }, + { + title: 'Sourcegraph Cloud', + href: '/cloud' + }, + { + title: 'Integrations', + href: '/integration', + sections: [ + { + title: 'Browser Extension', + href: '/integration/browser_extension' + }, + {title: 'Editors', href: '/integration/editor'}, + { + title: 'Browser Search Engine', + href: '/integration/browser_extension/how-tos/browser_search_engine' + } + ] + } + ] + }, + { + separator: 'CLI & API', + topics: [ + { + title: 'Sourcegraph CLI', + href: '/cli', + sections: [ + {title: 'Quickstart', href: '/cli/quickstart'}, + {title: 'Explanations', href: '/cli/explanations'}, + {title: 'How-to Guides', href: '/cli/how-tos'}, + {title: 'References', href: '/cli/references'} + ] + }, + { + title: 'Sourcegraph GraphQL API', + href: '/api/graphql' + }, + { + title: 'Sourcegraph Stream API', + href: '/api/stream_api' + } + ] + }, + { + separator: 'Help & Support', + topics: [ + { + title: 'SLAs & Premium Support', + href: '/sla' + }, + { + title: 'Tutorials', + href: '/tutorials' + }, + { + title: 'Sourcegraph Accounts', + href: '/sourcegraph-accounts' + }, + { + title: 'Changelog', + href: 'https://sourcegraph.com/changelog' + }, + { + title: 'Technical changelog', + href: '/technical-changelog' + } + ] + } ]; const navigation_5_3: NavigationItem[] = [ - { - separator: "Code Intelligence", - topics: [ - { - title: "Cody", - href: "/cody", - sections: [ - { title: "Quickstart", href: "/cody/quickstart" }, - { - title: "Installation", href: "/cody/clients", - subsections: [ - { title: "Cody for VS Code", href: "/cody/clients/install-vscode", }, - { title: "Cody for JetBrains", href: "/cody/clients/install-jetbrains", }, - { title: "Cody for Neovim", href: "/cody/clients/install-neovim", }, - { title: "Cody for Web", href: "/cody/clients/cody-with-sourcegraph", }, - { title: "Cody for Enterprise", href: "/cody/clients/enable-cody-enterprise", }, - ] - }, - { - title: "Capabilities", href: "/cody/capabilities", - subsections: [ - { title: "Chat", href: "/cody/capabilities/chat", }, - { title: "Autocomplete", href: "/cody/capabilities/autocomplete", }, - { title: "Commands", href: "/cody/capabilities/commands", }, - { title: "Debug Code", href: "/cody/capabilities/debug-code", }, - { title: "Cody Ignore", href: "/cody/capabilities/ignore-context", }, - { title: "Supported Models", href: "/cody/capabilities/supported-models", }, - { title: "Feature Parity Reference", href: "/cody/clients/feature-reference", }, - ] - }, - { - title: "Core Concepts", href: "/cody/core-concepts/context", - subsections: [ - { title: "Context", href: "/cody/core-concepts/context", }, - { title: "Token Limits", href: "/cody/core-concepts/token-limits", }, - // { title: "Embeddings", href: "/cody/core-concepts/embeddings", }, - { title: "Keyword Search", href: "/cody/core-concepts/keyword-search", }, - // { title: "Code Graph", href: "/cody/core-concepts/code-graph", }, - { title: "Cody Gateway", href: "/cody/core-concepts/cody-gateway", }, - { title: "Enterprise Architecture", href: "/cody/core-concepts/enterprise-architecture", }, - ] - }, - // { - // title: "Use Cases", href: "/cody/use-cases/generate-unit-tests", - // subsections: [ - // { title: "Generate Unit Tests", href: "/cody/use-cases/generate-unit-tests", }, - // // { title: "Build UI", href: "/cody/use-cases/build-ui", }, - // ] - // }, - { title: "Usage and Pricing", href: "/cody/usage-and-pricing" }, - { title: "Troubleshooting", href: "/cody/troubleshooting" }, - { title: "FAQs", href: "/cody/faq" }, - ], - }, - { - title: "Code Search", - href: "/code-search", - sections: [ - { title: "Features", href: "/code-search/features" }, - { - title: "Search Query Syntax", href: "/code-search/queries", - subsections: [ - { title: "Search Filters Panel", href: "/code-search/working/search_filters", }, - { title: "Search Examples", href: "/code-search/queries/examples", }, - { title: "Symbol Search", href: "/code-search/types/symbol", }, - { title: "Language Reference", href: "/code-search/queries/language", }, - ] - }, - { - title: "Code Navigation", href: "/code-search/code-navigation", - subsections: [ - { title: "Features", href: "/code-search/code-navigation/features", }, - { title: "Search-based code navigation", href: "/code-search/code-navigation/search_based_code_navigation", }, - { title: "Precise code navigation", href: "/code-search/code-navigation/precise_code_navigation", }, - { title: "Indexers", href: "/code-search/code-navigation/writing_an_indexer", }, - { title: "Auto-indexing", href: "/code-search/code-navigation/auto_indexing", }, - { title: "Environment Variables", href: "/code-search/code-navigation/envvars", }, - { title: "Troubleshooting", href: "/code-search/code-navigation/troubleshooting", }, - ] - }, - { - title: "Advanced Features", href: "/code-search/working/saved_searches", - subsections: [ - { title: "Fuzzy Finder", href: "/code-search/types/fuzzy", }, - { title: "Search Contexts", href: "/code-search/working/search_contexts", }, - { title: "Search Subexpressions", href: "/code-search/working/search_subexpressions", }, - { title: "Search Jobs", href: "/code-search/types/search-jobs", }, - { title: "Search Snippets", href: "/code-search/working/snippets", }, - { title: "Saved Searches", href: "/code-search/working/saved_searches", }, - { title: "Structural Search", href: "/code-search/types/structural", } - ] - }, - { - title: "FAQs", href: "/code-search/faq", - }, - ], - }, - ], - }, - { - separator: "Code Management", - topics: [ - { - title: "Batch Changes", - href: "/batch-changes", - sections: [ - { title: "Quickstart", href: "/batch-changes/quickstart" }, - { - title: "Create a Batch Change", href: "/batch-changes/create-a-batch-change", - subsections: [ - { title: "Update a Batch Change", href: "/batch-changes/update-a-batch-change", }, - { title: "Close a Batch Change", href: "/batch-changes/delete-a-batch-change", }, - { title: "Run Server Side", href: "/batch-changes/server-side", }, - - ] - }, - { - title: "Bulk Operations", href: "/batch-changes/bulk-operations-on-changesets", - subsections: [ - { title: "Publishing Changesets", href: "/batch-changes/publishing-changesets", }, - { title: "Tracking Changesets", href: "/batch-changes/tracking-existing-changesets", }, - ] - }, - { - title: "Configuring Credentials", href: "/batch-changes/configuring-credentials", - subsections: [ - { title: "Site Admin Config", href: "/batch-changes/site-admin-configuration", }, - ] - }, - { title: "Access Permissions", href: "/batch-changes/permissions-in-batch-changes" }, - { - title: "Changesets in Monorepo", href: "/batch-changes/creating-changesets-per-project-in-monorepos", - subsections: [ - { title: "Changesets in Large Repos", href: "/batch-changes/creating-multiple-changesets-in-large-repositories", }, - ] - }, - { title: "Error Handling", href: "/batch-changes/handling-errored-changesets" }, - { - title: "Examples", href: "/batch-changes/examples", - subsections: [ - { title: "Refactor Go Code via Comby", href: "/batch-changes/refactor-go-comby", }, - { title: "Update Go Import Statements via Comby", href: "/batch-changes/updating-go-import-statements", }, - { title: "Update Base Images in Dockerfiles", href: "/batch-changes/update-base-images-in-dockerfiles", }, - { title: "Search and Replace Specific Terms", href: "/batch-changes/search-and-replace-specific-terms", }, - ] - }, - { - title: "Batch Spec Reference", href: "/batch-changes/batch-spec-yaml-reference", - subsections: [ - { title: "Batch Spec Templating", href: "/batch-changes/batch-spec-templating", }, - { title: "Batch Spec Cheatsheet", href: "/batch-changes/batch-spec-cheat-sheet", }, - ] - }, - { - title: "Working with CLI", href: "/batch-changes/how-src-executes-a-batch-spec", - subsections: [ - { title: "Re-executing Batch Specs", href: "/batch-changes/reexecuting-batch-specs-multiple-times", }, - ] - }, - { title: "Troubleshooting", href: "/batch-changes/troubleshooting" }, - { title: "FAQs", href: "/batch-changes/faq" }, - ], - }, - { - title: "Code Monitoring", - href: "/code_monitoring", - sections: [ - { title: "Quickstart", href: "/code_monitoring/quickstart" }, - { title: "Explanations", href: "/code_monitoring/explanations" }, - { title: "How-to Guides", href: "/code_monitoring/how-tos" }, - ], - }, - { - title: "Code Ownership", - href: "/own", - sections: [ - { title: "CODEOWNERS Format", href: "/own/codeowners_format" }, - { title: "CODEOWNERS Ingestion", href: "/own/codeowners_ingestion" }, - { title: "Configuration Reference", href: "/own/configuration_reference" }, - { title: "Assigned Ownership", href: "/own/assigned_ownership" }, - - ], - }, - { - title: "Code Insights", - href: "/code_insights", - sections: [ - { title: "Quickstart", href: "/code_insights/quickstart" }, - { title: "Explanations", href: "/code_insights/explanations" }, - { title: "How-to Guides", href: "/code_insights/how-tos" }, - { title: "References", href: "/code_insights/references" }, - ], - }, - { - title: "Notebooks", - href: "/notebooks", - sections: [{ title: "Quickstart", href: "/notebooks/quickstart" }], - }, - ], - }, - { - separator: "Platform", - topics: [ - { - title: "Sourcegraph Admin", - href: "/admin", - sections: [ - { title: "Deploy", href: "/admin/deploy" }, - { title: "Upgrade", href: "/admin/updates" }, - { title: "Configuration", href: "/admin/config" }, - { title: "Licensing", href: "/admin/licensing" }, - { title: "Codehosts", href: "/admin/code_hosts" }, - { title: "User Authentication", href: "/admin/auth" }, - { title: "Access Control", href: "/admin/access_control" }, - { title: "Repository Permissions", href: "/admin/permissions" }, - { title: "Observability", href: "/admin/observability" }, - { title: "Analytics", href: "/admin/analytics" }, - { title: "Executors", href: "/admin/executors" }, - { title: "FAQs", href: "/admin/faq" }, - { title: "Troubleshooting", href: "/admin/troubleshooting" }, - { title: "How-to Guides", href: "/admin/how-to" }, - { - title: "Enterprise Getting Started", - href: "/admin/enterprise_getting_started_guide", - }, - { title: "Pricing", href: "/admin/pricing" }, - ], - }, - { - title: "Sourcegraph Cloud", - href: "/cloud", - }, - { - title: "Integrations", - href: "/integration", - sections: [ - { - title: "Browser Extension", - href: "/integration/browser_extension", - }, - { title: "Editors", href: "/integration/editor" }, - { - title: "Browser Search Engine", - href: "/integration/browser_extension/how-tos/browser_search_engine", - }, - ], - }, - ], - }, - { - separator: "CLI & API", - topics: [ - { - title: "Sourcegraph CLI", - href: "/cli", - sections: [ - { title: "Quickstart", href: "/cli/quickstart" }, - { title: "Explanations", href: "/cli/explanations" }, - { title: "How-to Guides", href: "/cli/how-tos" }, - { title: "References", href: "/cli/references" }, - ], - }, - { - title: "Sourcegraph GraphQL API", - href: "/api/graphql", - }, - { - title: "Sourcegraph Stream API", - href: "/api/stream_api", - }, - ], - }, - { - separator: "Help & Support", - topics: [ - { - title: "SLAs & Premium Support", - href: "/sla", - }, - { - title: "Tutorials", - href: "/tutorials", - }, - { - title: "Sourcegraph Accounts", - href: "/sourcegraph-accounts" - }, - { - title: "Changelog", - href: "https://sourcegraph.com/changelog", - }, - { - title: "Technical changelog", - href: "/technical-changelog", - }, - ], - }, + { + separator: 'Code Intelligence', + topics: [ + { + title: 'Cody', + href: '/cody', + sections: [ + {title: 'Quickstart', href: '/cody/quickstart'}, + { + title: 'Installation', + href: '/cody/clients', + subsections: [ + { + title: 'Cody for VS Code', + href: '/cody/clients/install-vscode' + }, + { + title: 'Cody for JetBrains', + href: '/cody/clients/install-jetbrains' + }, + { + title: 'Cody for Neovim', + href: '/cody/clients/install-neovim' + }, + { + title: 'Cody for Web', + href: '/cody/clients/cody-with-sourcegraph' + }, + { + title: 'Cody for Enterprise', + href: '/cody/clients/enable-cody-enterprise' + } + ] + }, + { + title: 'Capabilities', + href: '/cody/capabilities', + subsections: [ + {title: 'Chat', href: '/cody/capabilities/chat'}, + { + title: 'Autocomplete', + href: '/cody/capabilities/autocomplete' + }, + { + title: 'Commands', + href: '/cody/capabilities/commands' + }, + { + title: 'Debug Code', + href: '/cody/capabilities/debug-code' + }, + { + title: 'Cody Ignore', + href: '/cody/capabilities/ignore-context' + }, + { + title: 'Supported Models', + href: '/cody/capabilities/supported-models' + }, + { + title: 'Feature Parity Reference', + href: '/cody/clients/feature-reference' + } + ] + }, + { + title: 'Core Concepts', + href: '/cody/core-concepts/context', + subsections: [ + { + title: 'Context', + href: '/cody/core-concepts/context' + }, + { + title: 'Token Limits', + href: '/cody/core-concepts/token-limits' + }, + // { title: "Embeddings", href: "/cody/core-concepts/embeddings", }, + { + title: 'Keyword Search', + href: '/cody/core-concepts/keyword-search' + }, + // { title: "Code Graph", href: "/cody/core-concepts/code-graph", }, + { + title: 'Cody Gateway', + href: '/cody/core-concepts/cody-gateway' + }, + { + title: 'Enterprise Architecture', + href: '/cody/core-concepts/enterprise-architecture' + } + ] + }, + // { + // title: "Use Cases", href: "/cody/use-cases/generate-unit-tests", + // subsections: [ + // { title: "Generate Unit Tests", href: "/cody/use-cases/generate-unit-tests", }, + // // { title: "Build UI", href: "/cody/use-cases/build-ui", }, + // ] + // }, + { + title: 'Usage and Pricing', + href: '/cody/usage-and-pricing' + }, + {title: 'Troubleshooting', href: '/cody/troubleshooting'}, + {title: 'FAQs', href: '/cody/faq'} + ] + }, + { + title: 'Code Search', + href: '/code-search', + sections: [ + {title: 'Features', href: '/code-search/features'}, + { + title: 'Search Query Syntax', + href: '/code-search/queries', + subsections: [ + { + title: 'Search Filters Panel', + href: '/code-search/working/search_filters' + }, + { + title: 'Search Examples', + href: '/code-search/queries/examples' + }, + { + title: 'Symbol Search', + href: '/code-search/types/symbol' + }, + { + title: 'Language Reference', + href: '/code-search/queries/language' + } + ] + }, + { + title: 'Code Navigation', + href: '/code-search/code-navigation', + subsections: [ + { + title: 'Features', + href: '/code-search/code-navigation/features' + }, + { + title: 'Search-based code navigation', + href: '/code-search/code-navigation/search_based_code_navigation' + }, + { + title: 'Precise code navigation', + href: '/code-search/code-navigation/precise_code_navigation' + }, + { + title: 'Indexers', + href: '/code-search/code-navigation/writing_an_indexer' + }, + { + title: 'Auto-indexing', + href: '/code-search/code-navigation/auto_indexing' + }, + { + title: 'Environment Variables', + href: '/code-search/code-navigation/envvars' + }, + { + title: 'Troubleshooting', + href: '/code-search/code-navigation/troubleshooting' + } + ] + }, + { + title: 'Advanced Features', + href: '/code-search/working/saved_searches', + subsections: [ + { + title: 'Fuzzy Finder', + href: '/code-search/types/fuzzy' + }, + { + title: 'Search Contexts', + href: '/code-search/working/search_contexts' + }, + { + title: 'Search Subexpressions', + href: '/code-search/working/search_subexpressions' + }, + { + title: 'Search Jobs', + href: '/code-search/types/search-jobs' + }, + { + title: 'Search Snippets', + href: '/code-search/working/snippets' + }, + { + title: 'Saved Searches', + href: '/code-search/working/saved_searches' + }, + { + title: 'Structural Search', + href: '/code-search/types/structural' + } + ] + }, + { + title: 'FAQs', + href: '/code-search/faq' + } + ] + } + ] + }, + { + separator: 'Code Management', + topics: [ + { + title: 'Batch Changes', + href: '/batch-changes', + sections: [ + {title: 'Quickstart', href: '/batch-changes/quickstart'}, + { + title: 'Create a Batch Change', + href: '/batch-changes/create-a-batch-change', + subsections: [ + { + title: 'Update a Batch Change', + href: '/batch-changes/update-a-batch-change' + }, + { + title: 'Close a Batch Change', + href: '/batch-changes/delete-a-batch-change' + }, + { + title: 'Run Server Side', + href: '/batch-changes/server-side' + } + ] + }, + { + title: 'Bulk Operations', + href: '/batch-changes/bulk-operations-on-changesets', + subsections: [ + { + title: 'Publishing Changesets', + href: '/batch-changes/publishing-changesets' + }, + { + title: 'Tracking Changesets', + href: '/batch-changes/tracking-existing-changesets' + } + ] + }, + { + title: 'Configuring Credentials', + href: '/batch-changes/configuring-credentials', + subsections: [ + { + title: 'Site Admin Config', + href: '/batch-changes/site-admin-configuration' + } + ] + }, + { + title: 'Access Permissions', + href: '/batch-changes/permissions-in-batch-changes' + }, + { + title: 'Changesets in Monorepo', + href: '/batch-changes/creating-changesets-per-project-in-monorepos', + subsections: [ + { + title: 'Changesets in Large Repos', + href: '/batch-changes/creating-multiple-changesets-in-large-repositories' + } + ] + }, + { + title: 'Error Handling', + href: '/batch-changes/handling-errored-changesets' + }, + { + title: 'Examples', + href: '/batch-changes/examples', + subsections: [ + { + title: 'Refactor Go Code via Comby', + href: '/batch-changes/refactor-go-comby' + }, + { + title: 'Update Go Import Statements via Comby', + href: '/batch-changes/updating-go-import-statements' + }, + { + title: 'Update Base Images in Dockerfiles', + href: '/batch-changes/update-base-images-in-dockerfiles' + }, + { + title: 'Search and Replace Specific Terms', + href: '/batch-changes/search-and-replace-specific-terms' + } + ] + }, + { + title: 'Batch Spec Reference', + href: '/batch-changes/batch-spec-yaml-reference', + subsections: [ + { + title: 'Batch Spec Templating', + href: '/batch-changes/batch-spec-templating' + }, + { + title: 'Batch Spec Cheatsheet', + href: '/batch-changes/batch-spec-cheat-sheet' + } + ] + }, + { + title: 'Working with CLI', + href: '/batch-changes/how-src-executes-a-batch-spec', + subsections: [ + { + title: 'Re-executing Batch Specs', + href: '/batch-changes/reexecuting-batch-specs-multiple-times' + } + ] + }, + { + title: 'Troubleshooting', + href: '/batch-changes/troubleshooting' + }, + {title: 'FAQs', href: '/batch-changes/faq'} + ] + }, + { + title: 'Code Monitoring', + href: '/code_monitoring', + sections: [ + {title: 'Quickstart', href: '/code_monitoring/quickstart'}, + { + title: 'Explanations', + href: '/code_monitoring/explanations' + }, + {title: 'How-to Guides', href: '/code_monitoring/how-tos'} + ] + }, + { + title: 'Code Ownership', + href: '/own', + sections: [ + { + title: 'CODEOWNERS Format', + href: '/own/codeowners_format' + }, + { + title: 'CODEOWNERS Ingestion', + href: '/own/codeowners_ingestion' + }, + { + title: 'Configuration Reference', + href: '/own/configuration_reference' + }, + { + title: 'Assigned Ownership', + href: '/own/assigned_ownership' + } + ] + }, + { + title: 'Code Insights', + href: '/code_insights', + sections: [ + {title: 'Quickstart', href: '/code_insights/quickstart'}, + { + title: 'Explanations', + href: '/code_insights/explanations' + }, + {title: 'How-to Guides', href: '/code_insights/how-tos'}, + {title: 'References', href: '/code_insights/references'} + ] + }, + { + title: 'Notebooks', + href: '/notebooks', + sections: [{title: 'Quickstart', href: '/notebooks/quickstart'}] + } + ] + }, + { + separator: 'Platform', + topics: [ + { + title: 'Sourcegraph Admin', + href: '/admin', + sections: [ + {title: 'Deploy', href: '/admin/deploy'}, + {title: 'Upgrade', href: '/admin/updates'}, + {title: 'Configuration', href: '/admin/config'}, + {title: 'Licensing', href: '/admin/licensing'}, + {title: 'Codehosts', href: '/admin/code_hosts'}, + {title: 'User Authentication', href: '/admin/auth'}, + {title: 'Access Control', href: '/admin/access_control'}, + { + title: 'Repository Permissions', + href: '/admin/permissions' + }, + {title: 'Observability', href: '/admin/observability'}, + {title: 'Analytics', href: '/admin/analytics'}, + {title: 'Executors', href: '/admin/executors'}, + {title: 'FAQs', href: '/admin/faq'}, + {title: 'Troubleshooting', href: '/admin/troubleshooting'}, + {title: 'How-to Guides', href: '/admin/how-to'}, + { + title: 'Enterprise Getting Started', + href: '/admin/enterprise_getting_started_guide' + }, + {title: 'Pricing', href: '/admin/pricing'} + ] + }, + { + title: 'Sourcegraph Cloud', + href: '/cloud' + }, + { + title: 'Integrations', + href: '/integration', + sections: [ + { + title: 'Browser Extension', + href: '/integration/browser_extension' + }, + {title: 'Editors', href: '/integration/editor'}, + { + title: 'Browser Search Engine', + href: '/integration/browser_extension/how-tos/browser_search_engine' + } + ] + } + ] + }, + { + separator: 'CLI & API', + topics: [ + { + title: 'Sourcegraph CLI', + href: '/cli', + sections: [ + {title: 'Quickstart', href: '/cli/quickstart'}, + {title: 'Explanations', href: '/cli/explanations'}, + {title: 'How-to Guides', href: '/cli/how-tos'}, + {title: 'References', href: '/cli/references'} + ] + }, + { + title: 'Sourcegraph GraphQL API', + href: '/api/graphql' + }, + { + title: 'Sourcegraph Stream API', + href: '/api/stream_api' + } + ] + }, + { + separator: 'Help & Support', + topics: [ + { + title: 'SLAs & Premium Support', + href: '/sla' + }, + { + title: 'Tutorials', + href: '/tutorials' + }, + { + title: 'Sourcegraph Accounts', + href: '/sourcegraph-accounts' + }, + { + title: 'Changelog', + href: 'https://sourcegraph.com/changelog' + }, + { + title: 'Technical changelog', + href: '/technical-changelog' + } + ] + } ]; export const versionNavigations: VersionNavigations = { - navigation, // latest version - '5.2': navigation_5_2, - '5.3': navigation_5_3, -} + navigation, // latest version + '5.2': navigation_5_2, + '5.3': navigation_5_3 +}; From cb0fa6a0711df7007281158368920bf441d74e39 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Wed, 30 Oct 2024 19:02:42 +0200 Subject: [PATCH 07/58] wip --- docs/cody/model-configuration/examples.mdx | 119 +++++++++++++++++++++ docs/cody/model-configuration/index.mdx | 1 + 2 files changed, 120 insertions(+) diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx index df635b4e6..fae7675c6 100644 --- a/docs/cody/model-configuration/examples.mdx +++ b/docs/cody/model-configuration/examples.mdx @@ -1 +1,120 @@ # Examples + +## Sourcegraph-supplied models only + +This section includes examples how to configure Cody to use Sourcegraph-supplied models. + +- [Minimal configuration](/cody/model-configuration#configure-sourcegraph-supplied-models) +- [Using model filters](/cody/model-configuration#model-filters) +- [Change default models](/cody/model-configuration#default-models) + +## Sourcegraph-supplied models and BYOK (Bring Your Own Key) + +Sourcegraph-supplied models come with preconfigured providers, identified by the following IDs (namespaces): + +- "anthropic" +- "google" +- "fireworks" +- "mistral" +- "openai" + +### Override provider config for all models in the namespace + +When Sourcegraph-supplied models are used and a provider override for a Sourcegraph-supported provider (same ID) is specified, the override applies to all Sourcegraph-supplied models within that provider. +For example, if you specify an override for a provider with ID `"anthropic"`, it will apply to all models from the `"anthropic"` provider. + +Example configuration: + +```json +{ +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": {}, + "providerOverrides": [ + { + "id": "anthropic", + "displayName": "Anthropic BYOK", + "serverSideConfig": { + "type": "anthropic", + "accessToken": "sk-ant-token", + "endpoint": "https://api.anthropic.com/v1/messages" + } + } + ], + "defaultModels": { + "chat": "anthropic::2024-10-22::claude-3.5-sonnet", + "fastChat": "anthropic::2023-06-01::claude-3-haiku", + "autocomplete": "fireworks::v1::deepseek-coder-v2-lite-base" + } +} +``` + +In the configuration above, we: + +- Enable Sourcegraph-supplied models and do not set any overrides (note that `"modelConfiguration.modelOverrides"` is not specified). +- Route requests for Anthropic models directly to the Anthropic API (via the provider override specified for "anthropic"). +- Route requests for other models (such as the Fireworks model for "autocomplete") through Cody Gateway. + +### Override provider configur for some models and use the Sourcegraph-configured provider config for the rest + +It's possible to route requests directly to the LLM provider (bypassing the Cody Gateway) for some models while using the Sourcegraph-configured provider config for the rest. + +Example configuration: + +```json +{ +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": {}, + "providerOverrides": [ + { + "id": "anthropic-byok", + "displayName": "Anthropic BYOK", + "serverSideConfig": { + "type": "anthropic", + "accessToken": "sk-ant-token", + "endpoint": "https://api.anthropic.com/v1/messages" + } + } + ], + "modelOverrides": [ + { + "modelRef": "anthropic-byok::2024-10-22::claude-3.5-sonnet", + "displayName": "Claude 3.5 Sonnet", + "modelName": "claude-3-5-sonnet-latest", + "capabilities": ["edit", "chat", "vision"], + "category": "accuracy", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + } + ], + "defaultModels": { + "chat": "anthropic-byok::2024-10-22::claude-3.5-sonnet", + "fastChat": "anthropic::2023-06-01::claude-3-haiku", + "autocomplete": "fireworks::v1::deepseek-coder-v2-lite-base" + } +} +``` + +In the configuration above, we: + +- Enable Sourcegraph-supplied models (the `sourcegraph` field is not empty). +- Define a new provider with the ID `"anthropic-byok"` and configure it to use the Anthropic API. +- Since this provider is unknown to Sourcegraph, no Sourcegraph-supplied models are available for it. Therefore, + we add a custom model in the `"modelOverrides"` section. +- Use the custom model configured in the previous step (`"anthropic-byok::2024-10-22::claude-3.5-sonnet"`) for `"chat"`. + Requests are sent directly to the Anthropic API as set in the provider override. +- For `"fastChat"` and `"autocomplete"`, we use Sourcegraph-supplied models via Cody Gateway. + +## Config examples for various LLM providers + +Below are configuration examples for setting up various LLM providers using BYOK. +These examples are applicable whether or not you are using Sourcegraph-supported models. + + + + diff --git a/docs/cody/model-configuration/index.mdx b/docs/cody/model-configuration/index.mdx index 1ef1994b0..a972c8842 100644 --- a/docs/cody/model-configuration/index.mdx +++ b/docs/cody/model-configuration/index.mdx @@ -223,6 +223,7 @@ This field is an array of items, each with the following fields: - `"speed"` - Ideal for low-parameter models that may not be suited for general-purpose chat but are beneficial for specialized tasks, such as query rewriting. - `"accuracy"` - Reserved for models, like OpenAI o1, that use advanced reasoning techniques to improve response accuracy, though with slower latency. - `"other"` - Used for older models without distinct advantages in reasoning or speed. Select this category if uncertain about which category to choose. + - `"deprecated"` - For models that are no longer supported by the provider and are filtered out on the client side (not available for use). - `contextWindow` - An object that defines the number of "tokens" (units of text) that can be sent to the LLM. This setting influences response time and request cost, and may vary according to the limits set by each LLM model or provider. From a9f7f4c53a9eade78c85615573d5e03838494efc Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Thu, 31 Oct 2024 13:58:30 +0200 Subject: [PATCH 08/58] wip --- docs/cody/model-configuration/examples.mdx | 166 ++++++++++++++++++--- 1 file changed, 144 insertions(+), 22 deletions(-) diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx index fae7675c6..f2f492de3 100644 --- a/docs/cody/model-configuration/examples.mdx +++ b/docs/cody/model-configuration/examples.mdx @@ -20,7 +20,8 @@ Sourcegraph-supplied models come with preconfigured providers, identified by the ### Override provider config for all models in the namespace -When Sourcegraph-supplied models are used and a provider override for a Sourcegraph-supported provider (same ID) is specified, the override applies to all Sourcegraph-supplied models within that provider. +When Sourcegraph-supplied models are used and a provider override for a Sourcegraph-supported provider (same ID) is specified, +the override applies to all Sourcegraph-supplied models within that provider. For example, if you specify an override for a provider with ID `"anthropic"`, it will apply to all models from the `"anthropic"` provider. Example configuration: @@ -55,34 +56,60 @@ In the configuration above, we: - Route requests for Anthropic models directly to the Anthropic API (via the provider override specified for "anthropic"). - Route requests for other models (such as the Fireworks model for "autocomplete") through Cody Gateway. -### Override provider configur for some models and use the Sourcegraph-configured provider config for the rest +### Override provider config for some models in the namespace and use the Sourcegraph-configured provider config for the rest -It's possible to route requests directly to the LLM provider (bypassing the Cody Gateway) for some models while using the Sourcegraph-configured provider config for the rest. +It's possible to route requests directly to the LLM provider (bypassing the Cody Gateway) for some models while using the +Sourcegraph-configured provider config for the rest. Example configuration: +In the configuration above, we: + +- Enable Sourcegraph-supplied models (the `sourcegraph` field is not empty or `null`). +- Define a new provider with the ID `"anthropic-byok"` and configure it to use the Anthropic API. +- Since this provider is unknown to Sourcegraph, no Sourcegraph-supplied models are available for it. + Therefore, we add a custom model in the `"modelOverrides"` section. +- Use the custom model configured in the previous step (`"anthropic-byok::2024-10-22::claude-3.5-sonnet"`) for `"chat"`. + Requests are sent directly to the Anthropic API as set in the provider override. +- For `"fastChat"` and `"autocomplete"`, we use Sourcegraph-supplied models via Cody Gateway. + +## Config examples for various LLM providers + +Below are configuration examples for setting up various LLM providers using BYOK. +These examples are applicable whether or not you are using Sourcegraph-supported models. + +**Note:** + +- In this section, all configuration examples have Sourcegraph-supplied models disabled. To use a combination of + Sourcegraph-supplied models and BYOK, please refer to the previous section. +- Ensure that at least one model is available for each Cody feature ("chat", "edit", "autocomplete"), regardless of + the provider and model overrides configured. To verify this, [view the configuration](/cody/model-configuration#view-configuration) + and confirm that appropriate models are listed in the `"defaultModels"` section. + + + ```json { "cody.enabled": true, "modelConfiguration": { - "sourcegraph": {}, + "sourcegraph": null, "providerOverrides": [ { - "id": "anthropic-byok", - "displayName": "Anthropic BYOK", + "id": "anthropic", + "displayName": "Anthropic", "serverSideConfig": { "type": "anthropic", "accessToken": "sk-ant-token", - "endpoint": "https://api.anthropic.com/v1/messages" + "endpoint": "https://api.anthropic.com/v1/messages" } } ], "modelOverrides": [ { - "modelRef": "anthropic-byok::2024-10-22::claude-3.5-sonnet", + "modelRef": "anthropic::2024-10-22::claude-3.5-sonnet", "displayName": "Claude 3.5 Sonnet", "modelName": "claude-3-5-sonnet-latest", - "capabilities": ["edit", "chat", "vision"], + "capabilities": ["edit", "chat"], "category": "accuracy", "status": "stable", "tier": "free", @@ -90,31 +117,126 @@ Example configuration: "maxInputTokens": 45000, "maxOutputTokens": 4000 } + }, + { + "modelRef": "anthropic::2023-06-01::claude-3-haiku", + "displayName": "Claude 3 Haiku", + "modelName": "claude-3-haiku-20240307", + "capabilities": ["edit", "chat"], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "anthropic::2023-01-01::claude-instant-1.2", + "displayName": "Claude Instant", + "modelName": "claude-instant-1.2", + "capabilities": ["autocomplete", "edit", "chat"], + "category": "other", + "status": "deprecated", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } } ], "defaultModels": { - "chat": "anthropic-byok::2024-10-22::claude-3.5-sonnet", + "chat": "anthropic::2024-10-22::claude-3.5-sonnet", "fastChat": "anthropic::2023-06-01::claude-3-haiku", - "autocomplete": "fireworks::v1::deepseek-coder-v2-lite-base" + "autocomplete": "anthropic::2023-01-01::claude-instant-1.2" } } ``` In the configuration above, we: -- Enable Sourcegraph-supplied models (the `sourcegraph` field is not empty). -- Define a new provider with the ID `"anthropic-byok"` and configure it to use the Anthropic API. -- Since this provider is unknown to Sourcegraph, no Sourcegraph-supplied models are available for it. Therefore, - we add a custom model in the `"modelOverrides"` section. -- Use the custom model configured in the previous step (`"anthropic-byok::2024-10-22::claude-3.5-sonnet"`) for `"chat"`. - Requests are sent directly to the Anthropic API as set in the provider override. -- For `"fastChat"` and `"autocomplete"`, we use Sourcegraph-supplied models via Cody Gateway. + - Set up a provider override for Anthropic, routing requests for this provider directly to the specified Anthropic endpoint (bypassing Cody Gateway). + - Add three Anthropic models: + - Two models with chat capabilities (`"anthropic::2024-10-22::claude-3.5-sonnet"` and `"anthropic::2023-06-01::claude-3-haiku"`), + providing options for chat users. + - One model with autocomplete capability (`"anthropic::2023-01-01::claude-instant-1.2"`). + - Set the configured models as default models for Cody features in the `"defaultModels"` field. -## Config examples for various LLM providers + -Below are configuration examples for setting up various LLM providers using BYOK. -These examples are applicable whether or not you are using Sourcegraph-supported models. + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "fireworks", + "displayName": "Fireworks", + "serverSideConfig": { + "type": "fireworks", + "accessToken": "token", + "endpoint": "https://api.fireworks.ai/inference/v1/completions" + } + } + ], + "modelOverrides": [ + { + "modelRef": "fireworks::v1::mixtral-8x22b-instruct", + "displayName": "Mixtral 8x22B", + "modelName": "accounts/fireworks/models/mixtral-8x22b-instruct", + "capabilities": ["edit", "chat"], + "category": "other", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "fireworks::v1::starcoder-16b", + "modelName": "accounts/fireworks/models/starcoder-16b", + "displayName": "(Fireworks) Starcoder 16B", + "contextWindow": { + "maxInputTokens": 8192, + "maxOutputTokens": 4096 + }, + "capabilities": ["autocomplete"], + "category": "balanced", + "status": "stable" + } + ], + "defaultModels": { + "chat": "fireworks::v1::mixtral-8x22b-instruct", + "fastChat": "fireworks::v1::mixtral-8x22b-instruct", + "autocomplete": "fireworks::v1::starcoder-16b" + } +} +``` - +In the configuration above, we: + +- Set up a provider override for Fireworks, routing requests for this provider directly to the specified Fireworks endpoint (bypassing Cody Gateway). +- Add two Fireworks models: - `"fireworks::v1::mixtral-8x22b-instruct"` with "edit" and "chat" capabiities - used for "chat" + and "fastChat" - `"fireworks::v1::starcoder-16b"` with "autocomplete" capability - used for "autocomplete". + + + + + + + + + + + +## Self-hosted models + +TODO + +``` + +``` From d935f69cb690dbe2f20b9ec11602b9e1b3143abc Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Thu, 31 Oct 2024 15:09:40 +0200 Subject: [PATCH 09/58] wip --- docs/cody/model-configuration/examples.mdx | 69 +++++++++++++++++++++- 1 file changed, 67 insertions(+), 2 deletions(-) diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx index f2f492de3..9cc631788 100644 --- a/docs/cody/model-configuration/examples.mdx +++ b/docs/cody/model-configuration/examples.mdx @@ -223,9 +223,74 @@ In the configuration above, we: - + + +```json +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "openai", + "displayName": "OpenAI", + "serverSideConfig": { + "type": "openai", + "accessToken": "token", + "endpoint": "https://api.openai.com" + } + } + ], + "modelOverrides": [ + { + "modelRef": "openai::2024-02-01::gpt-4o", + "displayName": "GPT-4o", + "modelName": "gpt-4o", + "capabilities": ["edit", "chat"], + "category": "accuracy", + "status": "stable", + "tier": "pro", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 400 + } + }, + { + "modelRef": "openai::unknown::gpt-3.5-turbo-instruct", + "displayName": "GPT-3.5 Turbo Instruct", + "modelName": "gpt-3.5-turbo-instruct", + "capabilities": ["autocomplete"], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 400 + } + } +], + "defaultModels": { + "chat": "openai::2024-02-01::gpt-4o", + "fastChat": "openai::2024-02-01::gpt-4o", + "autocomplete": "openai::unknown::gpt-3.5-turbo-instruct" + } +} +``` + +In the configuration above, we: + +- Set up a provider override for OpenAI, routing requests for this provider directly to the specified OpenAI endpoint (bypassing Cody Gateway). +- Add two OpenAI models: + - `"openai::2024-02-01::gpt-4o"` with "edit" and "chat" capabiities - used for "chat" and "fastChat" + - `"openai::unknown::gpt-3.5-turbo-instruct"` with "autocomplete" capability - used for "autocomplete". + + + - + + + +TODO + + From 90ac8d0f867aad7610131f6b1e0b2866c892e06d Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Thu, 31 Oct 2024 16:28:59 +0200 Subject: [PATCH 10/58] wip --- docs/cody/model-configuration/examples.mdx | 67 +++++++++++++++++++++- 1 file changed, 66 insertions(+), 1 deletion(-) diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx index 9cc631788..02771178b 100644 --- a/docs/cody/model-configuration/examples.mdx +++ b/docs/cody/model-configuration/examples.mdx @@ -284,7 +284,72 @@ In the configuration above, we: - + + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "azure-openai", + "displayName": "Azure OpenAI", + "serverSideConfig": { + "type": "azureOpenAI", + "accessToken": "token", + "endpoint": "https://acme-test.openai.azure.com/", + "user": "", + "useDeprecatedCompletionsAPI": true + } + } + ], + "modelOverrides": [ + { + "modelRef": "azure-openai::unknown::gpt-4o", + "displayName": "GPT-4o", + "modelName": "gpt-4o", + "capabilities": ["edit", "chat"], + "category": "accuracy", + "status": "stable", + "tier": "pro", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "azure-openai::unknown::gpt-35-turbo-instruct-test", + "displayName": "GPT-3.5 Turbo Instruct", + "modelName": "gpt-35-turbo-instruct-test", + "capabilities": ["autocomplete"], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } + } + ], + "defaultModels": { + "chat": "azure-openai::unknown::gpt-4o", + "fastChat": "azure-openai::unknown::gpt-4o", + "autocomplete": "azure-openai::unknown::gpt-35-turbo-instruct-test" + } +} +``` + +In the configuration above, we: + +- Set up a provider override for Azure OpenAI, routing requests for this provider directly to the specified Azure OpenAI endpoint (bypassing Cody Gateway). +- Add two OpenAI models: + - `"azure-openai::unknown::gpt-4o"` with "edit" and "chat" capabiities - used for "chat" and "fastChat" + - `"azure-openai::unknown::gpt-35-turbo-instruct-test"` with "autocomplete" capability - used for "autocomplete". +- Since `"azure-openai::unknown::gpt-35-turbo-instruct-test"` is not supported on the newer OpenAI `"v1/chat/completions"` endpoint, + we set `"useDeprecatedCompletionsAPI"` to `true` to route requests to the legacy `"v1/completions"` endpoint. This setting is + unnecessary if you are using a model supported on the `"v1/chat/completions"` endpoint. + + From 1467b0066dd99ee1c2a2ea49e334f970797b27dd Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Thu, 31 Oct 2024 17:25:19 +0200 Subject: [PATCH 11/58] wip --- docs/cody/model-configuration/examples.mdx | 72 ++++++++++++++++++++-- 1 file changed, 66 insertions(+), 6 deletions(-) diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx index 02771178b..ea9e8d311 100644 --- a/docs/cody/model-configuration/examples.mdx +++ b/docs/cody/model-configuration/examples.mdx @@ -357,16 +357,76 @@ TODO - + + +```json +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "google", + "displayName": "Google Anthropic", + "serverSideConfig": { + "type": "google", + "accessToken": "token", + "endpoint": "https://us-east5-aiplatform.googleapis.com/v1/projects/project-name/locations/us-east5/publishers/anthropic/models" + } + } + ], + "modelOverrides": [ + { + "modelRef": "google::unknown::claude-3-5-sonnet", + "displayName": "Claude 3.5 Sonnet (via Google/Vertex)", + "modelName": "claude-3-5-sonnet@20240620", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + }, + "capabilities": ["chat"], + "category": "accuracy", + "status": "stable" + }, + { + "modelRef": "google::unknown::claude-3-haiku", + "displayName": "Claude 3 Haiku", + "modelName": "claude-3-haiku@20240307", + "capabilities": ["autocomplete", "edit", "chat"], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } + }, + ], + "defaultModels": { + "chat": "google::unknown::claude-3-5-sonnet", + "fastChat": "google::unknown::claude-3-5-sonnet", + "autocomplete": "google::unknown::claude-3-haiku" + } +} +``` + +In the configuration above, we: + +- Set up a provider override for Google Anthropic, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway). +- Add two OpenAI models: + - `"google::unknown::claude-3-5-sonnet"` with "chat" capabiity - used for "chat" and "fastChat" + - `"google::unknown::claude-3-haiku"` with "autocomplete" capability - used for "autocomplete". + + + - + + + +TODO + + ## Self-hosted models TODO - -``` - -``` From 3551c9d19c2e7728fe118f97b9394210cc16634c Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Mon, 4 Nov 2024 14:23:34 +0200 Subject: [PATCH 12/58] wip --- docs/cody/model-configuration/examples.mdx | 68 +++++++++++++++++++++- docs/cody/model-configuration/index.mdx | 2 +- 2 files changed, 67 insertions(+), 3 deletions(-) diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx index ea9e8d311..8d6c7baa7 100644 --- a/docs/cody/model-configuration/examples.mdx +++ b/docs/cody/model-configuration/examples.mdx @@ -417,7 +417,11 @@ In the configuration above, we: - + + +TODO + + @@ -425,7 +429,67 @@ TODO - + + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "aws-bedrock", + "displayName": "AWS Bedrock", + "serverSideConfig": { + "type": "awsBedrock", + "accessToken": "token", + "endpoint": "us-west-2", + "region": "us-west-2" + } + } + ], + "modelOverrides": [ + { + "modelRef": "aws-bedrock::2023-06-01::claude-3-opus", + "displayName": "Claude 3 Opus (AWS Bedrock)", + "modelName": "anthropic.claude-3-opus-20240229-v1:0", + "capabilities": ["edit", "chat", "autocomplete"], + "category": "other", + "status": "stable", + "tier": "pro", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + } + ], + "defaultModels": { + "chat": "aws-bedrock::2023-06-01::claude-3-opus", + "fastChat": "aws-bedrock::2023-06-01::claude-3-opus", + "autocomplete": "aws-bedrock::2023-06-01::claude-3-opus", + } +} +``` + +In the configuration described above, we: + +- Set up a provider override for AWS Bedrock, routing requests for this provider directly to the specified endpoint, + bypassing Cody Gateway. +- Add the `"aws-bedrock::2023-06-01::claude-3-opus"` model, which is used for all Cody features. + We do not add other models for simplicity, as adding multiple models is already covered in the examples above. + +Provider override `serverSideConfig` fields: + +| Field | Description | +| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `type` | Must be `"awsBedrock"`. | +| `accessToken` | Leave empty to rely on instance role bindings or other AWS configurations in the frontend service. Use `:` for direct credential configuration, or `::` if a session token is also required. | +| `endpoint` | For Pay-as-you-go, set it to an AWS region code (e.g., `us-west-2`) when using a public Amazon Bedrock endpoint. For Provisioned Throughput, set it to the provisioned VPC endpoint for the bedrock-runtime API (e.g., `https://vpce-0a10b2345cd67e89f-abc0defg.bedrock-runtime.us-west-2.vpce.amazonaws.com`). | +| `region` | The region to use when configuring API clients. This is necessary because the 'frontend' binary's container cannot access environment variables from the host OS. | + +Provisioned throughput for AWS Bedrock models can be configured using the `"awsBedrockProvisionedThroughput"` server-side +configuration type. For more details, refer to the [Model Overrides](/cody/model-configuration#model-overrides) section. + + ## Self-hosted models diff --git a/docs/cody/model-configuration/index.mdx b/docs/cody/model-configuration/index.mdx index a972c8842..d6fb6ca4a 100644 --- a/docs/cody/model-configuration/index.mdx +++ b/docs/cody/model-configuration/index.mdx @@ -230,7 +230,7 @@ This field is an array of items, each with the following fields: It includes two fields: - `maxInputTokens` - Specifies the maximum number of tokens for the contextual data in the prompt (e.g., question, relevant snippets). - `maxOutputTokens` - Specifies the maximum number of tokens allowed in the response. -- `serverSideConfig` - Additional configuration for the model. The available fields include: +- `serverSideConfig` - Additional configuration for the model. Can be one of the following: - `awsBedrockProvisionedThroughput` - Specifies provisioned throughput settings for AWS Bedrock models, with the following fields: From be6f553d54659acec1efc26e74b1da2ce020047f Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Mon, 4 Nov 2024 14:25:14 +0200 Subject: [PATCH 13/58] wip --- docs/cody/model-configuration/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cody/model-configuration/index.mdx b/docs/cody/model-configuration/index.mdx index d6fb6ca4a..6ac76ea62 100644 --- a/docs/cody/model-configuration/index.mdx +++ b/docs/cody/model-configuration/index.mdx @@ -325,7 +325,7 @@ This field is an array of items, each with the following fields: | `apiVersion` | (Optional) The API version to use. Defaults to `"v1"` if not specified. | | `model` | (Optional) Specifies which default model configuration to use. Sourcegraph provides default configurations for select models. For arbitrary models, use the `modelOverrides` section. Available options include: `"starcoder2-7b@v1"`, `"starcoder2-15b@v1"`, `"mistral-7b-instruct@v1"`, `"mixtral-8x7b-instruct@v1"`, `"mixtral-8x22b-instruct@v1"` | -TODO: add examples here. +For examples, refer to the [examples page](/cody/model-configuration/examples#self-hosted-models). ## Default Models From f6a58b0e928090c885095f2397309829a20f259f Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Mon, 4 Nov 2024 17:02:33 +0200 Subject: [PATCH 14/58] wip --- docs/cody/model-configuration/examples.mdx | 269 +++++++++++++++++---- docs/cody/model-configuration/index.mdx | 10 +- 2 files changed, 234 insertions(+), 45 deletions(-) diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx index 8d6c7baa7..f853ef2c8 100644 --- a/docs/cody/model-configuration/examples.mdx +++ b/docs/cody/model-configuration/examples.mdx @@ -37,7 +37,7 @@ Example configuration: "displayName": "Anthropic BYOK", "serverSideConfig": { "type": "anthropic", - "accessToken": "sk-ant-token", + "accessToken": "token", "endpoint": "https://api.anthropic.com/v1/messages" } } @@ -99,7 +99,7 @@ These examples are applicable whether or not you are using Sourcegraph-supported "displayName": "Anthropic", "serverSideConfig": { "type": "anthropic", - "accessToken": "sk-ant-token", + "accessToken": "token", "endpoint": "https://api.anthropic.com/v1/messages" } } @@ -291,50 +291,50 @@ In the configuration above, we: "modelConfiguration": { "sourcegraph": null, "providerOverrides": [ - { - "id": "azure-openai", - "displayName": "Azure OpenAI", - "serverSideConfig": { - "type": "azureOpenAI", - "accessToken": "token", - "endpoint": "https://acme-test.openai.azure.com/", - "user": "", - "useDeprecatedCompletionsAPI": true - } - } + { + "id": "azure-openai", + "displayName": "Azure OpenAI", + "serverSideConfig": { + "type": "azureOpenAI", + "accessToken": "token", + "endpoint": "https://acme-test.openai.azure.com/", + "user": "", + "useDeprecatedCompletionsAPI": true + } + } ], "modelOverrides": [ { - "modelRef": "azure-openai::unknown::gpt-4o", - "displayName": "GPT-4o", - "modelName": "gpt-4o", - "capabilities": ["edit", "chat"], - "category": "accuracy", - "status": "stable", - "tier": "pro", - "contextWindow": { - "maxInputTokens": 45000, - "maxOutputTokens": 4000 - } + "modelRef": "azure-openai::unknown::gpt-4o", + "displayName": "GPT-4o", + "modelName": "gpt-4o", + "capabilities": ["edit", "chat"], + "category": "accuracy", + "status": "stable", + "tier": "pro", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } }, - { - "modelRef": "azure-openai::unknown::gpt-35-turbo-instruct-test", - "displayName": "GPT-3.5 Turbo Instruct", - "modelName": "gpt-35-turbo-instruct-test", - "capabilities": ["autocomplete"], - "category": "speed", - "status": "stable", - "tier": "free", - "contextWindow": { - "maxInputTokens": 7000, - "maxOutputTokens": 4000 - } + { + "modelRef": "azure-openai::unknown::gpt-35-turbo-instruct-test", + "displayName": "GPT-3.5 Turbo Instruct", + "modelName": "gpt-35-turbo-instruct-test", + "capabilities": ["autocomplete"], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } } ], "defaultModels": { - "chat": "azure-openai::unknown::gpt-4o", - "fastChat": "azure-openai::unknown::gpt-4o", - "autocomplete": "azure-openai::unknown::gpt-35-turbo-instruct-test" + "chat": "azure-openai::unknown::gpt-4o", + "fastChat": "azure-openai::unknown::gpt-4o", + "autocomplete": "azure-openai::unknown::gpt-35-turbo-instruct-test" } } ``` @@ -353,7 +353,92 @@ In the configuration above, we: -TODO +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "fireworks", + "displayName": "Fireworks", + "serverSideConfig": { + "type": "openaicompatible", + "endpoints": [ + { + "url": "https://api.fireworks.ai/inference/v1", + "accessToken": "token" + } + ] + } + }, + { + "id": "huggingface-codellama", + "displayName": "Hugging Face", + "serverSideConfig": { + "type": "openaicompatible", + "endpoints": [ + { + "url": "https://api-inference.huggingface.co/models/meta-llama/CodeLlama-7b-hf/v1/", + "accessToken": "token" + } + ] + } + }, + ], + "modelOverrides": [ + { + "modelRef": "fireworks::v1::llama-v3p1-70b-instruct", + "modelName": "llama-v3p1-70b-instruct", + "displayName": "(Fireworks) Llama 3.1 70B Instruct", + "contextWindow": { + "maxInputTokens": 64000, + "maxOutputTokens": 8192 + }, + "serverSideConfig": { + "type": "openaicompatible", + "apiModel": "accounts/fireworks/models/llama-v3p1-70b-instruct" + }, + "clientSideConfig": { + "openaicompatible": {} + }, + "capabilities": ["chat"], + "category": "balanced", + "status": "stable" + }, + { + "modelRef": "huggingface-codellama::v1::CodeLlama-7b-hf", + "modelName": "CodeLlama-7b-hf", + "displayName": "(HuggingFace) CodeLlama-7b-hf", + "contextWindow": { + "maxInputTokens": 8192, + "maxOutputTokens": 4096 + }, + "serverSideConfig": { + "type": "openaicompatible", + "apiModel": "meta-llama/CodeLlama-7b-hf" + }, + "clientSideConfig": { + "openaicompatible": {} + }, + "capabilities": ["autocomplete", "chat"], + "category": "balanced", + "status": "stable" + } + ], + "defaultModels": { + "chat": "fireworks::v1::llama-v3p1-70b-instruct", + "fastChat": "fireworks::v1::llama-v3p1-70b-instruct", + "autocomplete": "huggingface-codellama::v1::CodeLlama-7b-hf" + } +} +``` + +In the configuration above, we: + +- Configure two OpenAI-compatible providers: `"fireworks"` and `"huggingface-codellama"`. +- Add two OpenAI-compatible models: `"fireworks::v1::llama-v3p1-70b-instruct"` and `"huggingface-codellama::v1::CodeLlama-7b-hf"`. Additionally: + - Set `clientSideConfig.openaicompatible` to `{}` to indicate to Cody clients that these models are OpenAI-compatible, ensuring the appropriate code paths are utilized. + - Designate these models as the default choices for chat and autocomplete, respectively. @@ -411,7 +496,7 @@ TODO In the configuration above, we: - Set up a provider override for Google Anthropic, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway). -- Add two OpenAI models: +- Add two Anthropic models: - `"google::unknown::claude-3-5-sonnet"` with "chat" capabiity - used for "chat" and "fastChat" - `"google::unknown::claude-3-haiku"` with "autocomplete" capability - used for "autocomplete". @@ -419,7 +504,48 @@ In the configuration above, we: -TODO +```json +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "google", + "displayName": "Google Gemini", + "serverSideConfig": { + "type": "google", + "accessToken": "token", + "endpoint": "https://generativelanguage.googleapis.com/v1beta/models" + } + } + ], + "modelOverrides": [ + { + "modelRef": "google::v1::gemini-1.5-pro", + "displayName": "Gemini 1.5 Pro", + "modelName": "gemini-1.5-pro", + "capabilities": ["chat", "autocomplete"], + "category": "balanced", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 400 + } + } + ], + "defaultModels": { + "chat": "google::v1::gemini-1.5-pro", + "fastChat": "google::v1::gemini-1.5-pro", + "autocomplete": "google::v1::gemini-1.5-pro" + } +} +``` + +In the configuration above, we: + +- Set up a provider override for Google Gemini, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway). +- Add the `"google::v1::gemini-1.5-pro"` model, which is used for all Cody features. + We do not add other models for simplicity, as adding multiple models is already covered in the examples above. @@ -493,4 +619,61 @@ configuration type. For more details, refer to the [Model Overrides](/cody/model ## Self-hosted models -TODO + + Configuring self-hosted models can be nuanced and complex. Consider reaching + out to Sourcegraph support for assistance with the setup. + + +For detailed information about configuring self-hosted models, please refer to the [self-hosted models](/cody/model-configuration#self-hosted-models) documentation. + +Example configuration: + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "fireworks", + "displayName": "Fireworks", + "serverSideConfig": { + "type": "openaicompatible", + "endpoints": [ + { + "url": "https://api.fireworks.ai/inference/v1", + "accessToken": "token" + } + ] + } + } + ], + "selfHostedModels": [ + { + "provider": "fireworks", + "model": "mixtral-8x7b-instruct@v1", + "override": { + "displayName": "(Fireworks) Mixtral 8x7b Instruct", + "serverSideConfig": { + "type": "openaicompatible", + "apiModel": "accounts/fireworks/models/mixtral-8x7b-instruct" + } + } + }, + { + "provider": "fireworks", + "model": "mixtral-8x22b-instruct@v1", + "override": { + "displayName": "(Fireworks) Mixtral 8x22b Instruct", + "serverSideConfig": { + "type": "openaicompatible", + "apiModel": "accounts/fireworks/models/mixtral-8x22b-instruct" + } + } + }, + ] +} +``` + +In the configuration above we: + +- TODO diff --git a/docs/cody/model-configuration/index.mdx b/docs/cody/model-configuration/index.mdx index 6ac76ea62..261038fac 100644 --- a/docs/cody/model-configuration/index.mdx +++ b/docs/cody/model-configuration/index.mdx @@ -156,7 +156,7 @@ Example configuration: "displayName": "OpenAI (via BYOK)", "serverSideConfig": { "type": "openai", - "accessToken": "sk-token", + "accessToken": "token", "endpoint": "https://api.openai.com" } }, @@ -238,6 +238,7 @@ This field is an array of items, each with the following fields: - `arn` - The ARN (Amazon Resource Name) for provisioned throughput to use when sending requests to AWS Bedrock. - `openaicompatible` - Configuration specific to models provided by an OpenAI-compatible provider, with the following fields: + - `type` - Must be `"openaicompatible"`. - `apiModel` - The literal string value of the `model` field to be sent to the `/chat/completions` API. If set, Sourcegraph treats this as an opaque string and sends it directly to the API without inferring any additional information. @@ -261,7 +262,7 @@ Example configuration: "endpoints": [ { "url": "https://api-inference.huggingface.co/models/meta-llama/CodeLlama-7b-hf/v1/", - "accessToken": "hf_token" + "accessToken": "token" } ] } @@ -310,6 +311,11 @@ For additional examples, refer to the [examples page](/cody/model-configuration/ ## Self-hosted Models + + Configuring self-hosted models can be nuanced and complex. Consider reaching + out to Sourcegraph support for assistance with the setup. + + With a provider defined (either a Sourcegraph-supplied provider or a custom provider configured via the `providerOverrides` field), self-hosted models can be added to Cody’s recognized models list by configuring the `"modelConfiguration.selfHostedModels"` section. These models use Sourcegraph’s default configuration settings and are only available for select models. From fb2e71b6d28a62b07d942d2f6f1eb9b47fe89c3c Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Mon, 4 Nov 2024 17:07:01 +0200 Subject: [PATCH 15/58] rm "edit" capability mentions --- docs/cody/model-configuration/examples.mdx | 28 ++++++----- docs/cody/model-configuration/index.mdx | 56 +++++----------------- 2 files changed, 28 insertions(+), 56 deletions(-) diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx index f853ef2c8..59400a7d5 100644 --- a/docs/cody/model-configuration/examples.mdx +++ b/docs/cody/model-configuration/examples.mdx @@ -82,7 +82,7 @@ These examples are applicable whether or not you are using Sourcegraph-supported - In this section, all configuration examples have Sourcegraph-supplied models disabled. To use a combination of Sourcegraph-supplied models and BYOK, please refer to the previous section. -- Ensure that at least one model is available for each Cody feature ("chat", "edit", "autocomplete"), regardless of +- Ensure that at least one model is available for each Cody feature ("chat" and "autocomplete"), regardless of the provider and model overrides configured. To verify this, [view the configuration](/cody/model-configuration#view-configuration) and confirm that appropriate models are listed in the `"defaultModels"` section. @@ -109,7 +109,7 @@ These examples are applicable whether or not you are using Sourcegraph-supported "modelRef": "anthropic::2024-10-22::claude-3.5-sonnet", "displayName": "Claude 3.5 Sonnet", "modelName": "claude-3-5-sonnet-latest", - "capabilities": ["edit", "chat"], + "capabilities": ["chat"], "category": "accuracy", "status": "stable", "tier": "free", @@ -122,7 +122,7 @@ These examples are applicable whether or not you are using Sourcegraph-supported "modelRef": "anthropic::2023-06-01::claude-3-haiku", "displayName": "Claude 3 Haiku", "modelName": "claude-3-haiku-20240307", - "capabilities": ["edit", "chat"], + "capabilities": ["chat"], "category": "speed", "status": "stable", "tier": "free", @@ -135,7 +135,7 @@ These examples are applicable whether or not you are using Sourcegraph-supported "modelRef": "anthropic::2023-01-01::claude-instant-1.2", "displayName": "Claude Instant", "modelName": "claude-instant-1.2", - "capabilities": ["autocomplete", "edit", "chat"], + "capabilities": ["autocomplete", "chat"], "category": "other", "status": "deprecated", "tier": "free", @@ -185,7 +185,7 @@ In the configuration above, we: "modelRef": "fireworks::v1::mixtral-8x22b-instruct", "displayName": "Mixtral 8x22B", "modelName": "accounts/fireworks/models/mixtral-8x22b-instruct", - "capabilities": ["edit", "chat"], + "capabilities": ["chat"], "category": "other", "status": "stable", "tier": "free", @@ -218,8 +218,10 @@ In the configuration above, we: In the configuration above, we: - Set up a provider override for Fireworks, routing requests for this provider directly to the specified Fireworks endpoint (bypassing Cody Gateway). -- Add two Fireworks models: - `"fireworks::v1::mixtral-8x22b-instruct"` with "edit" and "chat" capabiities - used for "chat" - and "fastChat" - `"fireworks::v1::starcoder-16b"` with "autocomplete" capability - used for "autocomplete". +- Add two Fireworks models: + - `"fireworks::v1::mixtral-8x22b-instruct"` with "chat" capabiity - used for "chat" + and "fastChat" + - `"fireworks::v1::starcoder-16b"` with "autocomplete" capability - used for "autocomplete". @@ -244,7 +246,7 @@ In the configuration above, we: "modelRef": "openai::2024-02-01::gpt-4o", "displayName": "GPT-4o", "modelName": "gpt-4o", - "capabilities": ["edit", "chat"], + "capabilities": ["chat"], "category": "accuracy", "status": "stable", "tier": "pro", @@ -279,7 +281,7 @@ In the configuration above, we: - Set up a provider override for OpenAI, routing requests for this provider directly to the specified OpenAI endpoint (bypassing Cody Gateway). - Add two OpenAI models: - - `"openai::2024-02-01::gpt-4o"` with "edit" and "chat" capabiities - used for "chat" and "fastChat" + - `"openai::2024-02-01::gpt-4o"` with "chat" capabilities - used for "chat" and "fastChat" - `"openai::unknown::gpt-3.5-turbo-instruct"` with "autocomplete" capability - used for "autocomplete". @@ -308,7 +310,7 @@ In the configuration above, we: "modelRef": "azure-openai::unknown::gpt-4o", "displayName": "GPT-4o", "modelName": "gpt-4o", - "capabilities": ["edit", "chat"], + "capabilities": ["chat"], "category": "accuracy", "status": "stable", "tier": "pro", @@ -343,7 +345,7 @@ In the configuration above, we: - Set up a provider override for Azure OpenAI, routing requests for this provider directly to the specified Azure OpenAI endpoint (bypassing Cody Gateway). - Add two OpenAI models: - - `"azure-openai::unknown::gpt-4o"` with "edit" and "chat" capabiities - used for "chat" and "fastChat" + - `"azure-openai::unknown::gpt-4o"` with "chat" capability - used for "chat" and "fastChat" - `"azure-openai::unknown::gpt-35-turbo-instruct-test"` with "autocomplete" capability - used for "autocomplete". - Since `"azure-openai::unknown::gpt-35-turbo-instruct-test"` is not supported on the newer OpenAI `"v1/chat/completions"` endpoint, we set `"useDeprecatedCompletionsAPI"` to `true` to route requests to the legacy `"v1/completions"` endpoint. This setting is @@ -475,7 +477,7 @@ In the configuration above, we: "modelRef": "google::unknown::claude-3-haiku", "displayName": "Claude 3 Haiku", "modelName": "claude-3-haiku@20240307", - "capabilities": ["autocomplete", "edit", "chat"], + "capabilities": ["autocomplete", "chat"], "category": "speed", "status": "stable", "tier": "free", @@ -578,7 +580,7 @@ TODO "modelRef": "aws-bedrock::2023-06-01::claude-3-opus", "displayName": "Claude 3 Opus (AWS Bedrock)", "modelName": "anthropic.claude-3-opus-20240229-v1:0", - "capabilities": ["edit", "chat", "autocomplete"], + "capabilities": ["chat", "autocomplete"], "category": "other", "status": "stable", "tier": "pro", diff --git a/docs/cody/model-configuration/index.mdx b/docs/cody/model-configuration/index.mdx index 261038fac..6a648e3c2 100644 --- a/docs/cody/model-configuration/index.mdx +++ b/docs/cody/model-configuration/index.mdx @@ -26,7 +26,7 @@ For more detailed information, refer to the documentation for each field. | [`providerOverrides`](/cody/model-configuration#provider-overrides) | Configures model providers, allowing you to customize Cody's connection to model providers, such as using your own API keys or self-hosted models. | | [`modelOverrides`](/cody/model-configuration#model-overrides) | Extends or modifies the list of models Cody recognizes, along with their configurations. | | [`selfHostedModels`](/cody/model-configuration#self-hosted-models) | Adds models to Cody’s recognized models list with default configurations provided by Sourcegraph. Only available for certain models; general models can be configured in `modelOverrides`. | -| [`defaultModels`](/cody/model-configuration#default-models) | Specifies the models assigned to each Cody feature (chat, fast edit, autocomplete). | +| [`defaultModels`](/cody/model-configuration#default-models) | Specifies the models assigned to each Cody feature (chat, fast chat, autocomplete). | ## Quickstart @@ -216,7 +216,7 @@ This field is an array of items, each with the following fields: - `displayName` - An optional, user-friendly name for the model. If not set, clients should display the `ModelID` part of the `modelRef` instead (not the `modelName`). - `modelName` - A unique identifier used by the API provider to specify which model is being invoked. This is the identifier that the LLM provider recognizes to determine the model you are calling. -- `capabilities` - A list of capabilities that the model supports. Supported values: "autocomplete", "chat", "edit". +- `capabilities` - A list of capabilities that the model supports. Supported values: "autocomplete" and "chat". - `category` - Specifies the model's category, with the following options: - `"balanced"` - Typically the best default choice for most users. This category is suited for state-of-the-art models, like Sonnet 3.5 (as of October 2024). @@ -405,11 +405,7 @@ Example response: "modelRef": "anthropic::2024-10-22::claude-3-5-sonnet-latest", "displayName": "Claude 3.5 Sonnet (Latest)", "modelName": "claude-3-5-sonnet-latest", - "capabilities": [ - "edit", - "chat", - "vision" - ], + "capabilities": ["chat"], "category": "accuracy", "status": "stable", "tier": "free", @@ -422,10 +418,7 @@ Example response: "modelRef": "anthropic::2023-06-01::claude-3-opus", "displayName": "Claude 3 Opus", "modelName": "claude-3-opus-20240229", - "capabilities": [ - "edit", - "chat" - ], + "capabilities": ["chat"], "category": "other", "status": "stable", "tier": "pro", @@ -438,10 +431,7 @@ Example response: "modelRef": "anthropic::2023-06-01::claude-3-haiku", "displayName": "Claude 3 Haiku", "modelName": "claude-3-haiku-20240307", - "capabilities": [ - "edit", - "chat" - ], + "capabilities": ["chat"], "category": "speed", "status": "stable", "tier": "free", @@ -454,9 +444,7 @@ Example response: "modelRef": "fireworks::v1::starcoder", "displayName": "StarCoder", "modelName": "starcoder", - "capabilities": [ - "autocomplete" - ], + "capabilities": ["autocomplete"], "category": "speed", "status": "stable", "tier": "free", @@ -469,9 +457,7 @@ Example response: "modelRef": "fireworks::v1::deepseek-coder-v2-lite-base", "displayName": "DeepSeek V2 Lite Base", "modelName": "accounts/sourcegraph/models/deepseek-coder-v2-lite-base", - "capabilities": [ - "autocomplete" - ], + "capabilities": ["autocomplete"], "category": "speed", "status": "stable", "tier": "free", @@ -484,10 +470,7 @@ Example response: "modelRef": "google::v1::gemini-1.5-pro", "displayName": "Gemini 1.5 Pro", "modelName": "gemini-1.5-pro", - "capabilities": [ - "edit", - "chat" - ], + "capabilities": ["chat"], "category": "balanced", "status": "stable", "tier": "free", @@ -500,10 +483,7 @@ Example response: "modelRef": "google::v1::gemini-1.5-flash", "displayName": "Gemini 1.5 Flash", "modelName": "gemini-1.5-flash", - "capabilities": [ - "edit", - "chat" - ], + "capabilities": ["chat"], "category": "speed", "status": "stable", "tier": "free", @@ -516,9 +496,7 @@ Example response: "modelRef": "mistral::v1::mixtral-8x7b-instruct", "displayName": "Mixtral 8x7B", "modelName": "accounts/fireworks/models/mixtral-8x7b-instruct", - "capabilities": [ - "chat" - ], + "capabilities": ["chat"], "category": "speed", "status": "stable", "tier": "free", @@ -531,11 +509,7 @@ Example response: "modelRef": "openai::2024-02-01::gpt-4o", "displayName": "GPT-4o", "modelName": "gpt-4o", - "capabilities": [ - "edit", - "chat", - "vision" - ], + "capabilities": ["chat"], "category": "accuracy", "status": "stable", "tier": "pro", @@ -548,9 +522,7 @@ Example response: "modelRef": "openai::2024-02-01::cody-chat-preview-001", "displayName": "OpenAI o1-preview", "modelName": "cody-chat-preview-001", - "capabilities": [ - "chat" - ], + "capabilities": ["chat"], "category": "accuracy", "status": "waitlist", "tier": "pro", @@ -563,9 +535,7 @@ Example response: "modelRef": "openai::2024-02-01::cody-chat-preview-002", "displayName": "OpenAI o1-mini", "modelName": "cody-chat-preview-002", - "capabilities": [ - "chat" - ], + "capabilities": ["chat"], "category": "accuracy", "status": "waitlist", "tier": "pro", From e716b9cfd1410c192d5cb373ab2cbb3812cdd993 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Fri, 8 Nov 2024 16:58:28 +0200 Subject: [PATCH 16/58] Update docs/cody/model-configuration/examples.mdx Co-authored-by: Emi (stephen) Gutekanst --- docs/cody/model-configuration/examples.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx index 59400a7d5..4f725a0ca 100644 --- a/docs/cody/model-configuration/examples.mdx +++ b/docs/cody/model-configuration/examples.mdx @@ -10,7 +10,7 @@ This section includes examples how to configure Cody to use Sourcegraph-supplied ## Sourcegraph-supplied models and BYOK (Bring Your Own Key) -Sourcegraph-supplied models come with preconfigured providers, identified by the following IDs (namespaces): +Out of the box, Sourcegraph is already aware of many models from the following providers: - "anthropic" - "google" From d25fdfcec78047089f492c83d6c360711314b21f Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Fri, 8 Nov 2024 16:58:34 +0200 Subject: [PATCH 17/58] Update docs/cody/model-configuration/examples.mdx Co-authored-by: Emi (stephen) Gutekanst --- docs/cody/model-configuration/examples.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx index 4f725a0ca..af515bc11 100644 --- a/docs/cody/model-configuration/examples.mdx +++ b/docs/cody/model-configuration/examples.mdx @@ -18,7 +18,7 @@ Out of the box, Sourcegraph is already aware of many models from the following p - "mistral" - "openai" -### Override provider config for all models in the namespace +### Overriding configuration of a model provider When Sourcegraph-supplied models are used and a provider override for a Sourcegraph-supported provider (same ID) is specified, the override applies to all Sourcegraph-supplied models within that provider. From a881d5f7c1971c4fc260a5f3cca1c60e14eaf926 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Wed, 20 Nov 2024 21:55:55 +0200 Subject: [PATCH 18/58] upd copy --- docs/cody/model-configuration/examples.mdx | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx index 59400a7d5..e1ed75d93 100644 --- a/docs/cody/model-configuration/examples.mdx +++ b/docs/cody/model-configuration/examples.mdx @@ -20,11 +20,7 @@ Sourcegraph-supplied models come with preconfigured providers, identified by the ### Override provider config for all models in the namespace -When Sourcegraph-supplied models are used and a provider override for a Sourcegraph-supported provider (same ID) is specified, -the override applies to all Sourcegraph-supplied models within that provider. -For example, if you specify an override for a provider with ID `"anthropic"`, it will apply to all models from the `"anthropic"` provider. - -Example configuration: +Instead of Sourcegraph using its own servers to make LLM requests, it is possible to bring your own API keys for a given model provider. For example, if you wish for all Anthropic API requests to go directly to your own Anthropic account and use your own API keys instead of going via Sourcegraph's servers, you could override the `anthropic` provider's configuration: ```json { From b01326bb56cc3cc8c9ce4f0abbea7a09ca4c2179 Mon Sep 17 00:00:00 2001 From: Maedah Batool Date: Wed, 20 Nov 2024 16:47:20 -0800 Subject: [PATCH 19/58] Add nav items --- src/data/navigation.ts | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/data/navigation.ts b/src/data/navigation.ts index f1d8c9237..0406ea0d6 100644 --- a/src/data/navigation.ts +++ b/src/data/navigation.ts @@ -42,6 +42,12 @@ export const navigation: NavigationItem[] = [ { title: "Model Configuration", href: "/cody/clients/model-configuration", }, ] }, + { + title: "Model Configuration", href: "/cody/model-configuration", + subsections: [ + { title: "Examples", href: "/cody/model-configuration/examples", }, + ] + }, { title: "Capabilities", href: "/cody/capabilities", subsections: [ From 76213c563e7508a79fb14fcbb83f30a0ff58ce87 Mon Sep 17 00:00:00 2001 From: Maedah Batool Date: Wed, 20 Nov 2024 17:07:27 -0800 Subject: [PATCH 20/58] Resolve merge conflicts --- src/data/navigation.ts | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/data/navigation.ts b/src/data/navigation.ts index 8cd66542f..57b132b6d 100644 --- a/src/data/navigation.ts +++ b/src/data/navigation.ts @@ -42,6 +42,12 @@ export const navigation: NavigationItem[] = [ { title: "Model Configuration", href: "/cody/clients/model-configuration", }, ] }, + { + title: "Model Configuration", href: "/cody/model-configuration", + subsections: [ + { title: "Examples", href: "/cody/model-configuration/examples", }, + ] + }, { title: "Capabilities", href: "/cody/capabilities", subsections: [ From 9b33ea1cf0f92b936fa64d5049b36aad06b7ef63 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Thu, 21 Nov 2024 17:29:50 +0200 Subject: [PATCH 21/58] remove Google Vertex (public) example --- docs/cody/model-configuration/examples.mdx | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx index 36ced10bc..89069b23a 100644 --- a/docs/cody/model-configuration/examples.mdx +++ b/docs/cody/model-configuration/examples.mdx @@ -547,12 +547,6 @@ In the configuration above, we: - - -TODO - - - ```json From bf4ac995a4e3c347e053db32d46030ce157f62dd Mon Sep 17 00:00:00 2001 From: Maedah Batool Date: Thu, 21 Nov 2024 08:32:15 -0800 Subject: [PATCH 22/58] First draft still working --- docs/cody/clients/enable-cody-enterprise.mdx | 132 +++-- docs/cody/clients/model-configuration.mdx | 536 ------------------ .../enterprise/completions-configuration.mdx | 186 ++++++ docs/cody/enterprise/features.mdx | 93 +++ .../model-config-examples.mdx} | 0 .../model-configuration.mdx} | 40 +- src/data/navigation.ts | 9 +- 7 files changed, 374 insertions(+), 622 deletions(-) delete mode 100644 docs/cody/clients/model-configuration.mdx create mode 100644 docs/cody/enterprise/completions-configuration.mdx create mode 100644 docs/cody/enterprise/features.mdx rename docs/cody/{model-configuration/examples.mdx => enterprise/model-config-examples.mdx} (100%) rename docs/cody/{model-configuration/index.mdx => enterprise/model-configuration.mdx} (91%) diff --git a/docs/cody/clients/enable-cody-enterprise.mdx b/docs/cody/clients/enable-cody-enterprise.mdx index 905cd2f25..f82da0ed7 100644 --- a/docs/cody/clients/enable-cody-enterprise.mdx +++ b/docs/cody/clients/enable-cody-enterprise.mdx @@ -1,4 +1,4 @@ -# Cody on Sourcegraph Enterprise +# Cody for Enterprise

Cody enhances your coding experience by providing intelligent code suggestions, context-aware completions, and advanced code analysis. These docs will help you use Cody on your Sourcegraph Enterprise instance.

@@ -6,96 +6,114 @@ -## Cody Enterprise features +## Setting up Cody Enterprise -To cater to your Enterprise requirements, Cody offers the following features: +You can set up Cody for your Enterprise instance by two methods: -### IDE token expiry +1. [Sourcegraph Cloud](#cody-on-sourcegraph-cloud) +2. [Self-hosted Sourcegraph](#cody-on-self-hosted-sourcegraph-enterprise) -Site administrators can set the duration of access tokens for users connecting Cody from their IDEs (VS Code, JetBrains, etc.). This can be configured from the **Site admin** page of the Sourcegraph Enterprise instance. Available options include **7, 14, 30, 60, and 90 days**. +## Cody on Sourcegraph Cloud -![ide-token-expiry](https://storage.googleapis.com/sourcegraph-assets/Docs/token-expiry.png) +With [Sourcegraph Cloud](/cloud/), you get Cody as a managed service, and you **do not** need to enable Cody as is required for self-hosted setup. However, by contacting your account manager, Cody can still be enabled or disabled on-demand on your Sourcegraph instance. -### Guardrails +## Self-hosted Sourcegraph Enterprise -Guardrails for public code is currently in Beta and is supported with VS Code and JetBrains IDEs extensions. +### Prerequisites -Open source attribution guardrails for public code, commonly called copyright guardrails, reduce the exposure to copyrighted code. This involves implementing a verification mechanism within Cody to ensure that any code generated by the platform does not replicate open source code. +- You have Sourcegraph version `5.1.0` or more +- A Sourcegraph Enterprise subscription with [Cody Gateway](/cody/core-concepts/cody-gateway) or an account with a third-party LLM provider -Guardrails for public code are available to all Sourcegraph Enterprise instances and are **disabled** by default. You can enable them from the Site configuration section by setting `attribution.enabled` to `true`. +### Enable Cody on your Sourcegraph instance -Guardrails don't differentiate between license types. It matches any code snippet that is at least **ten lines** long from the **290,000** indexed open source repositories. +Site admins can only enable Cody on the Sourcegraph instance. To do so, -### Admin controls +- First, configure your desired LLM provider either by [using Sourcegraph Cody Gateway](/cody/core-concepts/cody-gateway) (recommended) or by directly using a third-party LLM provider +- Next, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: -Admin controls are supported with VS Code and JetBrains IDE extension. +```json + { + // [...] + "cody.enabled": true, + "completions": { + "provider": "sourcegraph" + } + } +``` -Site administrators have selective control over users' access to Cody Enterprise, which is managed via the Sourcegraph role-based access control system. This provides a more intuitive user interface for assigning permission to use Cody. +- Cody is enabled on your self-hosted Sourcegraph enterprise instance -### Analytics +## Disable Cody -Cody Analytics are supported with VS Code IDE extension and on the latest versions of JetBrains IDEs. +To turn Cody off: -Cody Enterprise users can view analytics for their instance. A separately managed cloud service for Cody analytics handles user auth, gets metrics data from Sourcegraph's BigQuery instance, and visualizes the metrics data. +- Go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: -The following metrics are available for Cody Enterprise users: +```json + { + // [...] + "cody.enabled": false + } +``` -| **Metric Type** | **What is measured?** | -| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Active users | - Total active users
- Average daily users
- Average no. of days each user used Cody (of last 30 days)
- Cody users by day (last 30 days)
- Cody users by month (last two months)
- Cody users by number of days used | -| Completions | - Total accepted completions
- Minutes saved per completion
- Hours saved by completions
- Cody completions by day
- Completions acceptance rate
- Weighted completions acceptance rate
- Average completion latency
- Acceptance rate by language
| -| Chat | - Total chat events
- Minutes saved per chat
- Hours saved by chats
- Cody chats by day | -| Commands | - Total command events
- Minutes saved per command
- Hours saved by commands
- Cody commands by day
- Most used commands | +- Next, remove `completions` configuration if they exist -To enable Cody Analytics: +## Enable Cody only for some users -- Create an account on [Sourcegraph Accounts](https://accounts.sourcegraph.com/) -- A user already having a Sourcegraph.com account gets automatically migrated to Sourcegraph Accounts. Users can sign in to Cody Analytics using their email and password -- Users without a Sourcegraph.com account should contact one of our team members. They can help with both the account setup and assigning instances to specific users -- Map your user account to a Sourcegraph instance, and this gives you access to Cody analytics +To enable Cody only for some users it depends on what version of Sourcegraph you are running. -### Multi-repository context +### Sourcegraph v5.3+ -Cody supports multi-repository context, allowing you to search up to 10 repositories simultaneously for relevant information. Open a new chat, type `@`, and select `Remote Repositories.` +In Sourcegraph v5.3+, access to Cody is managed via user roles. By default, all users have access. -Keep @-mentioning repos that you want to include in your context. This flexibility lets you get more comprehensive and accurate responses by leveraging information across multiple codebases. +First, ensure Cody is enabled in your site configuration. Go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: -### @-mention directory +```json + { + // [...] + "cody.enabled": true, + // Make sure cody.restrictUsersFeatureFlag is not in your configuration! If it is, remove it. + } +``` -To better support teams working with large monorepos, Enterprise users can `@-mention` directories when chatting with Cody. This helps you define more specific directories and sub-directories within that monorepo to give more precise context. + Ensure `cody.restrictUsersFeatureFlag` is **not** in your site configuration. If it is, remove it or else the old feature-flag approach from Sourcegraph 5.2 and earlier will be used. -To do this, type `@` in the chat, and then select **Directories** to search other repositories for context in your codebase. +Next, go to **Site admin > Users & Auth > Roles** (`/site-admin/roles`) on your instance. On that page, you can: -![@-mention-directory](https://storage.googleapis.com/sourcegraph-assets/Docs/cody-ent-at-mention-dir-1124.png) +- Control whether users **by default** have access to Cody (expand `User [System]` and toggle **Cody** > **Access** as desired) +- Control whether groups of users have access to Cody (`+Create role` and enable the **Cody** > **Access** toggle as desired) -Please note that you can only `@-mention` remote directories (i.e., directories in your Sourcegraph instance) but not local directories. This means any recent changes to your directories can't be utilized as context until your Sourcegraph instance re-indexes any changes. +### Sourcegraph v5.2 and earlier -If you want to include recent changes that haven't been indexed in your Sourcegraph instance, you can `@-mention` specific files, lines of code, or symbols. +In Sourcegraph v5.2 and earlier, you should use the feature flag `cody` to turn Cody on selectively for some users. To do so: -## Supported LLM models +- Go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: -Sourcegraph Enterprise supports different LLM providers and models, such as models from Anthropic and OpenAI. You can do this by adjusting your Sourcegraph instance configuration. +```json + { + // [...] + "cody.enabled": true, + "cody.restrictUsersFeatureFlag": true + } +``` - -For the supported LLM models listed above, refer to the following notes: +- Next, go to **Site admin > Feature flags** (`/site-admin/feature-flags`) +- Add a feature flag called `cody` +- Select the `boolean` type and set it to `false` +- Once added, click on the feature flag and use **add overrides** to pick users that will have access to Cody -1. Microsoft Azure is planning to deprecate the APIs used in Sourcegraph version `>5.3.3` on July 1, 2024 [Source](https://learn.microsoft.com/en-us/azure/ai-services/openai/api-version-deprecation) -2. Claude 2.1 is not recommended -3. Sourcegraph doesn’t recommend using the GPT-4 (non-Turbo), Claude 1, or Claude 2 models anymore -4. Only supported through legacy completions API -5. BYOK (Bring Your Own Key) with managed services are only supported for Self-hosted Sourcegraph instances -6. GPT-4 and GPT-4o for completions have a bug that is resulting in many failed completions +![add-overrides](https://user-images.githubusercontent.com/25070988/235454594-9f1a6b27-6882-44d9-be32-258d6c244880.png) -### Supported model configuration +## Configure Cody for LLM providers -Use the drop-down menu to make your desired selection and get a detailed breakdown of the supported model configuration for each provider on Cody Enterprise. This is an on-site configuration. Admins should pick a value from the table for `chatModel` to configure their chat model. +Cody supports several LLM providers and models. You can access these models via the Cody Gateway, directly using your own model provider account or infrastructure. - +There are two ways of configuring Cody for LLM providers: -For the supported LLM model configuration listed above, refer to the following notes: + -1. Microsoft Azure is planning to deprecate the APIs used in Sourcegraph version `>5.3.3` on July 1, 2024 [Source](https://learn.microsoft.com/en-us/azure/ai-services/openai/api-version-deprecation) -2. Claude 2.1 is not recommended -3. Sourcegraph doesn't recommend GPT-4 non-turbo, Claude 1 or 2 models -4. Only supported through legacy completions API -5. BYOK (Bring Your Own Key) with managed services are only supported for Self-hosted Sourcegraph instances + + + + + diff --git a/docs/cody/clients/model-configuration.mdx b/docs/cody/clients/model-configuration.mdx deleted file mode 100644 index 799f383b6..000000000 --- a/docs/cody/clients/model-configuration.mdx +++ /dev/null @@ -1,536 +0,0 @@ -# LLM Model Configuration - -

This guide will walk you through the steps to customize the LLM models available from your Sourcegraph Enterprise instance.

- -For Sourcegraph Cloud customers, configuring the available LLM models requires contacting your Sourcegraph account team representative - -Cody Enterprise can be configured using one of two methods: -1. "Completions" Configuration -2. Model Configuration (Early Access Program) - -The Model Configuration method is in Early Access Program and only avaiable on Sourcegraph v5.6.0 or later. In the future when Model Configuration exits EAP, configuring Cody via your Sourcegraph Enterprise instance via the "Completions" Configuration site configuration section will be deprecated. For now, both methods remain supported. We recommend you continue to use the Completions Configuration unless you have specific reason to do otherwise. - -## "Completions" Configuration - - -## Setting up Cody Enterprise - -You can set up Cody for your Enterprise instance in one of the following ways: - -- [Self-hosted Sourcegraph](#cody-on-self-hosted-sourcegraph-enterprise) -- [Sourcegraph Cloud](#cody-on-sourcegraph-cloud) - -## Cody on self-hosted Sourcegraph Enterprise - -### Prerequisites - -- You have Sourcegraph version 5.1.0 or above -- A Sourcegraph Enterprise subscription with [Cody Gateway access](/cody/core-concepts/cody-gateway) or [an account with a third-party LLM provider](#supported-models-and-model-providers) - -### Enable Cody on your Sourcegraph instance - -Cody uses one or more third-party LLM (Large Language Model) providers. Make sure you review Cody's usage and privacy notice. Code snippets are sent to a third-party language model provider when you use the Cody extension. - -This requires site-admin privileges. To do so, - -1. First, configure your desired LLM provider either by [Using Sourcegraph Cody Gateway](/cody/core-concepts/cody-gateway#using-cody-gateway-in-sourcegraph-enterprise) (recommended) or [Using a third-party LLM provider directly](#supported-models-and-model-providers) - - If you are a Sourcegraph Cloud customer, skip directly to step 3. - -2. Next, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: - -```json - { - // [...] - "cody.enabled": true, - "completions": { - "provider": "sourcegraph" - } - } -``` - -Cody is now fully enabled on your self-hosted Sourcegraph enterprise instance! - -## Cody on Sourcegraph Cloud - -- With [Sourcegraph Cloud](/cloud/), you get Cody as a managed service, and you **do not** need to [enable Cody as is required for self-hosted setup](#enable-cody-on-your-sourcegraph-instance) -- However, by contacting your account manager, Cody can still be enabled on-demand on your Sourcegraph instance. The Sourcegraph team will refer to the handbook -- Next, you can configure the [VS Code extension](#configure-the-vs-code-extension) by following the same steps as mentioned for the self-hosted environment -- After which, you are all set to use Cody with Sourcegraph Cloud - -[Learn more about running Cody on Sourcegraph Cloud](/cloud/#cody). - -## Disable Cody - -To turn Cody off: - -- Go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: - -```json - { - // [...] - "cody.enabled": false - } -``` - -- Next, remove `completions` configuration if they exist - -## Enable Cody only for some users - -To enable Cody only for some users, for example, when rolling out a Cody POC, follow all the steps mentioned in [Enabling Cody on your Sourcegraph instance](#enable-cody-on-your-sourcegraph-instance). Then, do the following: - -### Sourcegraph 5.3+ - -In Sourcegraph 5.3+, access to Cody is managed via user roles. By default, all users have access. - -First, ensure Cody is enabled in your site configuration. Go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: - -```json - { - // [...] - "cody.enabled": true, - // Make sure cody.restrictUsersFeatureFlag is not in your configuration! If it is, remove it. - } -``` - - Ensure `cody.restrictUsersFeatureFlag` is **not** in your site configuration. If it is, remove it or else the old feature-flag approach from Sourcegraph 5.2 and earlier will be used. - -Next, go to **Site admin > Users & Auth > Roles** (`/site-admin/roles`) on your instance. On that page, you can: - -- Control whether users _by default_ have access to Cody (expand `User [System]` and toggle **Cody** > **Access** as desired) -- Control whether groups of users have access to Cody (`+Create role` and enable the **Cody** > **Access** toggle as desired) - -### Sourcegraph 5.2 and earlier - -In Sourcegraph 5.2 and earlier, you should use the feature flag `cody` to turn Cody on selectively for some users. To do so: - -- Go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: - -```json - { - // [...] - "cody.enabled": true, - "cody.restrictUsersFeatureFlag": true - } -``` - -- Next, go to **Site admin > Feature flags** (`/site-admin/feature-flags`) -- Add a feature flag called `cody` -- Select the `boolean` type and set it to `false` -- Once added, click on the feature flag and use **add overrides** to pick users that will have access to Cody - -![add-overrides](https://user-images.githubusercontent.com/25070988/235454594-9f1a6b27-6882-44d9-be32-258d6c244880.png) - -## Supported models and model providers - -[Cody Enterprise](https://sourcegraph.com/enterprise) supports many models and model providers. You can configure Cody Enterprise to access models via Sourcegraph Cody Gateway or directly using your own model provider account or infrastructure. - -- Using [Sourcegraph Cody Gateway](/cody/core-concepts/cody-gateway): - - Recommended for most organizations. - - Supports [state-of-the-art models](/cody/capabilities/supported-models) from Anthropic, OpenAI, and more, without needing a separate account or incurring separate charges. -- Using your organization's account with a model provider: - - [Use your organization's Anthropic account](#use-your-organizations-anthropic-account) - - [Use your organization's OpenAI account](#use-your-organizations-openai-account) -- Using your organization's public cloud infrastructure: - - [Use Amazon Bedrock (AWS)](#use-amazon-bedrock-aws) - - [Use Azure OpenAI Service](#use-azure-openai-service) - - *Use Vertex AI on Google Cloud (coming soon)* - -### Use your organization's Anthropic account - -First, [create your own key with Anthropic](https://console.anthropic.com/account/keys). Once you have the key, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: - -```json -{ - // [...] - "cody.enabled": true, - "completions": { - "provider": "anthropic", - "chatModel": "claude-2.0", // Or any other model you would like to use - "fastChatModel": "claude-instant-1.2", // Or any other model you would like to use - "completionModel": "claude-instant-1.2", // Or any other model you would like to use - "accessToken": "" - } -} -``` - -### Use your organization's OpenAI account - -First, [create your own key with OpenAI](https://beta.openai.com/account/api-keys). Once you have the key, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: - -```json -{ - // [...] - "cody.enabled": true, - "completions": { - "provider": "openai", - "chatModel": "gpt-4", // Or any other model you would like to use - "fastChatModel": "gpt-3.5-turbo", // Or any other model you would like to use - "completionModel": "gpt-3.5-turbo-instruct", // Or any other model that supports the legacy completions endpoint - "accessToken": "" - } -} -``` - -[Learn more about OpenAI models.](https://platform.openai.com/docs/models) - -### Use Amazon Bedrock (AWS) - -You can use Anthropic Claude models on [Amazon Bedrock](https://aws.amazon.com/bedrock/). - -First, make sure you can access Amazon Bedrock. Then, request access to the Anthropic Claude models in Bedrock. -This may take some time to provision. - -Next, create an IAM user with programmatic access in your AWS account. Depending on your AWS setup, different ways may be required to provide access. All completion requests are made from the `frontend` service, so this service needs to be able to access AWS. You can use instance role bindings or directly configure the IAM user credentials in the configuration. Additionally, the `AWS_REGION` environment variable will need to be set in the `frontend` container for scoping the IAM credentials to the AWS region hosting the Bedrock endpoint. - -Once ready, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: - -```json -{ - // [...] - "cody.enabled": true, - "completions": { - "provider": "aws-bedrock", - "chatModel": "anthropic.claude-3-opus-20240229-v1:0", - "completionModel": "anthropic.claude-instant-v1", - "endpoint": "", - "accessToken": "" - } -} -``` - -For the `chatModel` and `completionModel` fields, see [Amazon's Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html) for an up-to-date list of supported model IDs, and cross reference against Sourcegraph's [supported LLM list](/cody/capabilities/supported-models) to verify compatibility with Cody. - -For `endpoint`, you can either: - -- For Pay-as-you-go, set it to an AWS region code (e.g., `us-west-2`) when using a public Amazon Bedrock endpoint -- For Provisioned Throughput, set it to the provisioned VPC endpoint for the `bedrock-runtime` API (e.g., `"https://vpce-0a10b2345cd67e89f-abc0defg.bedrock-runtime.us-west-2.vpce.amazonaws.com"`) - -For `accessToken`, you can either: - -- Leave it empty and rely on instance role bindings or other AWS configurations in the `frontend` service -- Set it to `:` if directly configuring the credentials -- Set it to `::` if a session token is also required - - -### Using GCP Vertex AI - -Right now, We only support Anthropic Claude models on [GCP Vertex](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-claude). - -1. Enable the [Vertex AI API](https://console.cloud.google.com/marketplace/product/google/aiplatform.googleapis.com) in the GCP console. Once Vertex has been enabled in your project, navigate to the [Vertex Model Garden](https://console.cloud.google.com/vertex-ai/model-garden) to select & enable the Anthropic Claude model(s) which you wish to use with Cody. See [Supported LLM Models](../capabilities/supported-models) for an up-to-date list of Anthropic Claude models supported by Cody. - -It may take some time to enable Vertex and provision access to the models you plan to use - -2. **Create a Service Account**: - - Create a [service account](https://cloud.google.com/iam/docs/service-account-overview). - - Assign the `Vertex AI User` role to the service account. - - Generate a JSON key for the service account and download it. - -3. **Convert JSON Key to Base64** by doing: -```python -cat | base64 -``` - -Once ready, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: - - -```json -{ - // [...] - "cody.enabled": true, - "completions": { - "chatModel": "claude-3-opus@20240229", - "completionModel": "claude-3-haiku@20240307", - "provider": "google", - "endpoint": "", - "accessToken": "" - } -} - -``` - -For the `Endpoint`, you can -1. Navigate to the Documentation Page: - Go to the Claude 3 Haiku Documentation on the GCP Console Model garden -2. Locate the Example: if you scroll enough through the page to find the example that shows how to use the cURL command with the Claude 3 Haiku model. The example will include a sample request JSON body and the necessary endpoint URL. Copy the URL in the site-admin config: - The endpoint URL will look something like this: - `https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/anthropic/models/` - -3. Example URL: -`https://us-east5-aiplatform.googleapis.com/v1/projects/sourcegraph-vertex-staging/locations/us-east5/publishers/anthropic/models` - - -### Use Azure OpenAI Service - -Create a project in the Azure OpenAI Service portal. Go to **Keys and Endpoint** from the project overview and get **one of the keys** on that page and the **endpoint**. - -Next, under **Model deployments**, click "manage deployments" and ensure you deploy the models you want, for example, `gpt-35-turbo`. Take note of the **deployment name**. - -Once done, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: - -```json -{ - // [...] - "cody.enabled": true, - "completions": { - "provider": "azure-openai", - "chatModel": "", - "fastChatModel": "", - "completionModel": "", // the model must support the legacy completions endpoint such as gpt-3.5-turbo-instruct - "endpoint": "", - "accessToken": "" - } -} -``` - -For the access token, you can either: - -- As of 5.2.4 the access token can be left empty and it will rely on Environmental, Workload Identity or Managed Identity credentials configured for the `frontend` and `worker` services -- Set it to `` if directly configuring the credentials using the API key specified in the Azure portal - -### Use StarCoder for Autocomplete - -When tested with other coder models for the autocomplete use case, [StarCoder](https://huggingface.co/blog/starcoder) offered significant improvements in quality and latency compared to our control groups for users on Sourcegraph.com. You can read more about the improvements in our [October 2023 release notes](https://sourcegraph.com/blog/feature-release-october-2023) and the [GA release notes](https://sourcegraph.com/blog/cody-is-generally-available). - -To ensure a fast and reliable experience, we are partnering with [Fireworks](https://fireworks.ai/) and have set up a dedicated hardware deployment for our Enterprise users. Sourcegraph supports StarCoder using the [Cody Gateway](/cody/core-concepts/cody-gateway). - -To enable StarCoder go to **Site admin > Site configuration** (`/site-admin/configuration`) and change the `completionModel`: - -```json -{ - // [...] - "cody.enabled": true, - "completions": { - "provider": "sourcegraph", - "completionModel": "fireworks/starcoder" - } -} -``` - -Users of the Cody Extensions will automatically pick up this change when connected to your Enterprise instance. - - -# Model Configuration - -Sourcegraph v5.6.0 or later supports the ability to choose between different LLM models, allowing developers to use the best model for Cody Chat as needed. This is accomplished exposing much more flexible configuration options for Cody when using Sourcegraph Enterprise. The newer style of configuration is described next. However, you can still use the [Older style "Completions" Configuration](#legacy-completions-configuration). - -## Quickstart - -Before you start, please note that the model configuration is an early access program (EAP) and we are working towards improving on its coverage of supported providers. If you are having any issues with this configuration, please reach out to your Sourcegraph Account Representative or roll back your configuration to the Legacy "Completions" configuration - -The simplest way to configure your Sourcegraph Enterprise would be to add the following configuration section to your instance's [site configuration](/admin/config/site_config): - -```json - ... - "cody.enabled": true, - "modelConfiguration": { - "sourcegraph": {} - }, - ... -``` - -The `"modelConfiguration"` section defines which LLM models are supported by the Sourcegraph instance, and how to invoke them. The `"sourcegraph"` section defines how Sourcegraph-supplied LLM models should be configured. (That is, LLM models made available by the [Cody Gateway](/cody/core-concepts/cody-gateway) service.) The default settings will expose all current Cody Gateway models from your Sourcegraph instance, and make them available to users. - -However, if you are seeking more control and wish to restrict which LLM models are available, or if you wish to use your own API access key, you can expand upon the `"modelConfiguration"` section as needed. - -## Concepts - -The LLM models available for use from a Sourcegraph Enterprise instance are the union of "Sourcegraph-supplied models" and any custom models providers that you explicitly add to your Sourcegraph instance's site configuration. For most administrators, just relying on Sourcegraph-supplied models will ensure that you are using quality models without needing to worry about the specifics. - -### Sourcegraph-supplied Models - -The Sourcegraph-supplied models are those that are available from [Cody Gateway](/cody/core-concepts/cody-gateway), and your site configuration controls which of those models can be used. - -If you wish to not use _any_ Sourcegraph-supplied models, and instead _only_ rely on those you have explicitly defined in your site configuration, you can set the `"sourcegraph"` field to `null`. - -There are three top-level settings for configuring Sourcegraph-supplied LLM models: - -| Field | Description | -| ----------- | ---------------------------------------------------------------------------------------- | -| `endpoint` (optional) | The URL for connecting to Cody Gateway, defaults to the production instance. | -| `accessToken` (optional) | The access token used to connect to Cody Gateway, defaulting to the current license key. | -| `modelFilters` (optional) | Filters for which models to include from Cody Gateway. | - -**Model Filters** - -The `"modelFilters"` section is how you restrict which Cody Gateway models are made available to your Sourcegraph Enterprise instance's users. - -The first field is the `"statusFilter"`. Each LLM model is given a label by Sourcegraph as per its release, such as "stable", beta", or "experimental". By default, all models available on -Cody Gateway are exposed. Using the category filter ensures that only models with a particular category are made available to your users. - -The `"allow"` and `"deny"` fields, are arrays of [model references](#model-configuration) for what models should or should not be included. These values accept wild cards. - - -The following examples illustrate how to use all these settings in conjunction: - -```json -"cody.enabled": true, -"modelConfiguration": { - "sourcegraph": { - "modelFilters": { - // Only allow "beta" and "stable" models. - // Not "experimental" or "deprecated". - "statusFilter": ["beta", "stable"], - - // Allow any models provided by Anthropic, OpenAI, Google and Fireworks. - "allow": [ - "anthropic::*", // Anthropic models - "openai::*", // OpenAI models - "google::*", // Google Gemini models - "fireworks::*", // Autocomplete models like StarCoder and DeepSeek-V2-Coder hosted on Fireworks - ], - - // Do not include any models with the Model ID containing "turbo", - // or any from AcmeCo. - "deny": [ - "*turbo*", - "acmeco::*" - ] - } - } -} -``` - -## Default Models - -The `"modelConfiguration"` setting also contains a `"defaultModels"` field that allows you to specify the LLM model used depending on the situation. If no default is specified, or refers to a model that isn't found, it will silently fallback to a suitable alternative. - -```json - ... - "cody.enabled": true, - "modelConfiguration": { - "defaultModels": { - "chat": "anthropic::2023-06-01::claude-3.5-sonnet", - "codeCompletion": "anthropic::2023-06-01::claude-3.5-sonnet", - "fastChat": "anthropic::2023-06-01::claude-3-haiku" - } - } - ... -``` - -The format of these strings is a "Model Reference", which is a format for uniquely identifying each LLM model exposed from your Sourcegraph instance. - -## Advanced Configuration - -For most administrators, relying on the LLM models made available by Cody Gateway is sufficient. However, if even more customization is required, you can configure your own LLM providers and models. - -Defining your own LLM providers and models is an advanced use-case and requires care to get the correct results. It also may bypass protections to ensure compatibility between your Sourcegraph instance and LLMs. If you need help contact your Sourcegraph account executive. - -### Overview - -The `"modelConfiguration"` section exposes two fields `"providerOverrides"` and `"modelOverrides"`. These may override any Sourcegraph-supplied data, or simply introduce new ones entirely. - -### Provider Configuration - -A "provider" is a way to organize LLM models. Typically a provider would be referring to the company that produced the model. Or the specific API/service being used to access the model. But conceptually, it's just a namespace. - -By defining a provider override in your Sourcegraph site configuration, you are introducing a new namespace to contain models. Or customize the existing provider namespace supplied by Sourcegraph. (e.g. all `"anthropic"` models.) - -The following configuration shippet defines a single provider override with the ID `"anthropic"`. - -```json -"cody.enabled": true, -"modelConfiguration": { - // Do not use any Sourcegraph-supplied models. - "sourcegraph": null, - - // Define a provider for "anthropic". - "providerOverrides": [ - { - "id": "anthropic", - "displayName": "Anthropic models, sent directly to anthropic.com", - - // The server-side config section defines how this provider operates. - "serverSideConfig": { - "type": "anthropic", - "accessToken": "sk-ant-api03-xxxxxxxxx", - "endpoint": "https://api.anthropic.com/v1/messages" - }, - - // The default model configuration provides defaults for all LLM - // models using this provider. - "defaultModelConfig": { - "capabilities": [ - "chat", - "autocomplete" - ], - "contextWindow": { - "maxInputTokens": 10000, - "maxOutputTokens": 4000 - }, - "category": "balanced", - "status": "stable" - } - } - ], - ... -} -``` - -**Server-side Configuration** - -The most important part of a provider's configuration is the `"serverSideConfig"` field. That defines how the LLM model's should be invoked, i.e. which external service or API will be called to serve LLM requests. - -In the example, the `"type"` field was `"anthropic"`. Meaning that any interactions using the `"anthropic"` provider would be sent directly to Anthropic, at the supplied `endpoint` URL using the given `accessToken`. - -However, Sourcegraph supports several different types of LLM API providers natively. The current set of supported LLM API providers is: - -| Provider type | Description | -| -------------------- | ------------ | -| `"sourcegraph"` | [Cody Gateway](/cody/core-concepts/cody-gateway), which supports many different models from various services | -| `"openaicompatible"` | Any OpenAI-compatible API implementation | -| `"awsBedrock"` | [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/) | -| `"azureOpenAI"` | [Microsoft Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service/) | -| `"anthropic"` | [Anthropic](https://www.anthropic.com) | -| `"fireworks"` | [Fireworks AI](https://fireworks.ai) | -| `"google"` | [Google Gemini](http://cloud.google.com/gemini) and [Vertex](https://cloud.google.com/vertex-ai/) | -| `"openai"` | [OpenAI](http://platform.openai.com) | -| `"huggingface-tgi"` | [Hugging Face Text Generation Interface](https://huggingface.co/docs/text-generation-inference/en/index) | - -### Model Configuration - -With a provider defined, we can now specify custom models using that provider by adding them to the `"modelOverrides"` section. - -**Model Reference** - -The following configuration snippet defines a custom model, using the `"anthropic"` provider from the previous example. - -```json -"cody.enabled": true, -"modelConfiguration": { - ... - "modelOverrides": [ - { - "modelRef": "anthropic::2024-06-20::claude-3-5-sonnet", - "displayName": "Claude 3.5 Sonnet", - "modelName": "claude-3-5-sonnet-20240620", - "contextWindow": { - "maxInputTokens": 45000, - "maxOutputTokens": 4000 - }, - "capabilities": ["chat", "autocomplete"], - "category": "balanced", - "status": "stable" - }, - ], - ... -} -``` - -Most of the configuration fields are self-explanatory, such as labeling the model's category ("stable", "beta") or category ("accuracy", "speed"). The more important fields are described below: - -**modelRef**. Each model is given a unique identifier, referred to as a model reference or "mref". This is a string of the form `${providerId}::${apiVersionId}::${modelId}`. - -In order to associate a model with your provider, the `${providerId}` must match. The `${modelId}` can be almost any URL-safe string. - -The `${apiVersionId}` is required in order to detect compatibility issues between newer models and Sourcegraph instances. In the following example, the string "2023-06-01" is used to clarify that this LLM model should formulate API requests using that version of the Anthropic API. If you are unsure, when defining your own models you can leave this as `"unknown"`. - -**contextWindow**. The context window is the number of "tokens" sent to an LLM. Either in the amount of contextual data sent in the LLM prompt (e.g. the question, relevant snippets, etc.) and the maximum size of the output allowed in the response. These values directly control factors such as the time it takes to respond to a prompt and the cost of the LLM request. And each LLM model or provider may have their own limits as well. - -**modelName**. The model _name_, is the value required by the LLM model's API provider. In this example, the `modelRef` defined the model's ID as `claude-3-sonnet` but the `modelName` was the more specific "claude-3-sonnet-20240229". - -**capabilities**. The capabilities of a model determine which situations the model can be used. For example, models only supported for "autocomplete" will not be available for Cody chats. - - - -It's recommended that every instance admin not using a third-party LLM provider makes this change and we are planning to make this the default in a future release. diff --git a/docs/cody/enterprise/completions-configuration.mdx b/docs/cody/enterprise/completions-configuration.mdx new file mode 100644 index 000000000..0be4bb03d --- /dev/null +++ b/docs/cody/enterprise/completions-configuration.mdx @@ -0,0 +1,186 @@ +# Completions Configuration + +

Learn how to configure Cody via `completions` on a Sourcegraph Enterprise instance.

+ +Configuring Cody via `completions` is legacy but it's still supported. We recommend using the new [`modelConfiguration`](/cody/enterprise/model-configuration) for flexible LLM model selection. + +[Cody Enterprise](https://sourcegraph.com/enterprise) supports many models and model providers. You can configure Cody Enterprise to access models via Sourcegraph Cody Gateway or directly using your own model provider account or infrastructure. Let's look at these options in more detail. + +## Using Sourcegraph Cody Gateway + +This is the recommended way to configure Cody Enterprise. It supports all the latest models from Anthropic, OpenAI, Mistral, and more without requiring a separate account or incurring separate charges. You can learn more about these in our [supported models](/cody/capabilities/supported-models) docs. + +## Using your organization's account with a model provider + +### Example: Your organization's Anthropic account + +First, [create your own key with Anthropic](https://console.anthropic.com/account/keys). Once you have the key, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: + +```json +{ + // [...] + "cody.enabled": true, + "completions": { + "provider": "anthropic", + "chatModel": "claude-2.0", // Or any other model you would like to use + "fastChatModel": "claude-instant-1.2", // Or any other model you would like to use + "completionModel": "claude-instant-1.2", // Or any other model you would like to use + "accessToken": "" + } +} +``` + +### Example: Your organization's OpenAI account + +First, [create your own key with OpenAI](https://beta.openai.com/account/api-keys). Once you have the key, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: + +```json +{ + // [...] + "cody.enabled": true, + "completions": { + "provider": "openai", + "chatModel": "gpt-4", // Or any other model you would like to use + "fastChatModel": "gpt-3.5-turbo", // Or any other model you would like to use + "completionModel": "gpt-3.5-turbo-instruct", // Or any other model that supports the legacy completions endpoint + "accessToken": "" + } +} +``` + +[Learn more about OpenAI models.](https://platform.openai.com/docs/models) + +## Using your organization's public cloud infrastructure + +### Example: Use Amazon Bedrock + +You can use Anthropic Claude models on [Amazon Bedrock](https://aws.amazon.com/bedrock/). + +First, make sure you can access Amazon Bedrock. Then, request access to the Anthropic Claude models in Bedrock. This may take some time to provision. + +Next, create an IAM user with programmatic access in your AWS account. Depending on your AWS setup, different ways may be required to provide access. All completion requests are made from the `frontend` service, so this service needs to be able to access AWS. + +You can use instance role bindings or directly configure the IAM user credentials in the configuration. The `AWS_REGION` environment variable must also be set in the `frontend` container to scope the IAM credentials for the AWS region hosting the Bedrock endpoint. + +Once ready, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: + +```json +{ + // [...] + "cody.enabled": true, + "completions": { + "provider": "aws-bedrock", + "chatModel": "anthropic.claude-3-opus-20240229-v1:0", + "completionModel": "anthropic.claude-instant-v1", + "endpoint": "", + "accessToken": "" + } +} +``` + +For the `chatModel` and `completionModel` fields, see [Amazon's Bedrock docs](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html) for an up-to-date list of supported model IDs, and cross reference against Sourcegraph's [supported LLM list](/cody/capabilities/supported-models) to verify compatibility with Cody. + +For `endpoint`, you can either: + +- For **pay-as-you-go**, set it to an AWS region code (e.g., `us-west-2`) when using a public Amazon Bedrock endpoint +- For **provisioned throughput**, set it to the provisioned VPC endpoint for the `bedrock-runtime` API (e.g., `"https://vpce-0a10b2345cd67e89f-abc0defg.bedrock-runtime.us-west-2.vpce.amazonaws.com"`) + +For `accessToken`, you can either: + +- Leave it empty and rely on instance role bindings or other AWS configurations in the `frontend` service +- Set it to `:` if directly configuring the credentials +- Set it to `::` if a session token is also required + +### Example: Using GCP Vertex AI + +We only support Anthropic Claude models on [GCP Vertex](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-claude). + +- Enable the [Vertex AI API](https://console.cloud.google.com/marketplace/product/google/aiplatform.googleapis.com) in the GCP console. Once Vertex has been enabled in your project, navigate to the [Vertex Model Garden](https://console.cloud.google.com/vertex-ai/model-garden) to select and enable the Anthropic Claude model(s) that you wish to use with Cody. See [Supported LLM Models](/capabilities/supported-models) for an up-to-date list of Anthropic Claude models supported by Cody. + +It may take some time to enable Vertex and provision access to the models you plan to use + +1. **Create a Service Account**: + - Create a [service account](https://cloud.google.com/iam/docs/service-account-overview) + - Assign the `Vertex AI User` role to the service account + - Generate a JSON key for the service account and download it + +2. **Convert JSON Key to Base64** by doing: + +```python +cat | base64 +``` + +Once ready, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: + +```json +{ + // [...] + "cody.enabled": true, + "completions": { + "chatModel": "claude-3-opus@20240229", + "completionModel": "claude-3-haiku@20240307", + "provider": "google", + "endpoint": "", + "accessToken": "" + } +} + +``` + +For the `endpoint`, you can: + +- Go to the Claude 3 Haiku docs on the GCP Console Model garden +- Scroll through the page to find the example that shows how to use the `cURL` command with the Claude 3 Haiku model. The example will include a sample request JSON body and the necessary endpoint URL. Copy the URL in the site-admin config +- The endpoint URL will look something like this: + `https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/publishers/anthropic/models/` +- Example URL: +`https://us-east5-aiplatform.googleapis.com/v1/projects/sourcegraph-vertex-staging/locations/us-east5/publishers/anthropic/models` + +### Example: Use Azure OpenAI service + +Create a project in the Azure OpenAI Service portal. From the project overview, go to **Keys and Endpoint and get one of the keys** on that page and the **endpoint**. + +Next, under **Model deployments**, click **manage deployments** and ensure you deploy the models you want, for example, `gpt-35-turbo`. Take note of the **deployment name**. + +Once done, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: + +```json +{ + // [...] + "cody.enabled": true, + "completions": { + "provider": "azure-openai", + "chatModel": "", + "fastChatModel": "", + "completionModel": "", // the model must support the legacy completions endpoint such as gpt-3.5-turbo-instruct + "endpoint": "", + "accessToken": "" + } +} +``` + +For the access token, + +- For Sourcegraph `v5.2.4` or more, the access token can be left empty, and it will rely on Environmental, Workload Identity, or Managed Identity credentials configured for the `frontend` and `worker` services +- Set it to `` if directly configuring the credentials using the API key specified in the Azure portal + +### Use StarCoder for Autocomplete + +When tested with other coder models for the autocomplete use case, [StarCoder](https://huggingface.co/blog/starcoder) offered significant improvements in quality and latency compared to our control groups for users on Sourcegraph.com. You can read more about the improvements in our [October 2023 release notes](https://sourcegraph.com/blog/feature-release-october-2023) and the [GA release notes](https://sourcegraph.com/blog/cody-is-generally-available). + +To ensure a fast and reliable experience, we are partnering with [Fireworks](https://fireworks.ai/) and have set up a dedicated hardware deployment for our Enterprise users. Sourcegraph supports StarCoder using the [Cody Gateway](/cody/core-concepts/cody-gateway). + +To enable StarCoder, go to **Site admin > Site configuration** (`/site-admin/configuration`) and change the `completionModel`: + +```json +{ + // [...] + "cody.enabled": true, + "completions": { + "provider": "sourcegraph", + "completionModel": "fireworks/starcoder" + } +} +``` + +Users of the Cody extensions will automatically pick up this change when connected to your Enterprise instance. diff --git a/docs/cody/enterprise/features.mdx b/docs/cody/enterprise/features.mdx new file mode 100644 index 000000000..2805b47e0 --- /dev/null +++ b/docs/cody/enterprise/features.mdx @@ -0,0 +1,93 @@ +# Cody Enterprise Features + +

Along with the core features, Cody Enterprise offers additional features to enhance your coding experience.

+ +## IDE token expiry + +Site administrators can set the duration of access tokens for users connecting Cody from their IDEs (VS Code, JetBrains, etc.). This can be configured from the **Site admin** page of the Sourcegraph Enterprise instance. Available options include **7, 14, 30, 60, and 90 days**. + +![ide-token-expiry](https://storage.googleapis.com/sourcegraph-assets/Docs/token-expiry.png) + +## Guardrails + +Guardrails for public code is currently in Beta and is supported with VS Code and JetBrains IDEs extensions. + +Open source attribution guardrails for public code, commonly called copyright guardrails, reduce the exposure to copyrighted code. This involves implementing a verification mechanism within Cody to ensure that any code generated by the platform does not replicate open source code. + +Guardrails for public code are available to all Sourcegraph Enterprise instances and are **disabled** by default. You can enable them from the Site configuration section by setting `attribution.enabled` to `true`. + +Guardrails don't differentiate between license types. It matches any code snippet that is at least **ten lines** long from the **290,000** indexed open source repositories. + +## Admin controls + +Admin controls are supported with VS Code and JetBrains IDE extension. + +Site administrators have selective control over users' access to Cody Enterprise, which is managed via the Sourcegraph role-based access control system. This provides a more intuitive user interface for assigning permission to use Cody. + +## Analytics + +Cody Analytics are supported with VS Code IDE extension and on the latest versions of JetBrains IDEs. + +Cody Enterprise users can view analytics for their instance. A separately managed cloud service for Cody analytics handles user auth, gets metrics data from Sourcegraph's BigQuery instance, and visualizes the metrics data. + +The following metrics are available for Cody Enterprise users: + +| **Metric Type** | **What is measured?** | +| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Active users | - Total active users
- Average daily users
- Average no. of days each user used Cody (of last 30 days)
- Cody users by day (last 30 days)
- Cody users by month (last two months)
- Cody users by number of days used | +| Completions | - Total accepted completions
- Minutes saved per completion
- Hours saved by completions
- Cody completions by day
- Completions acceptance rate
- Weighted completions acceptance rate
- Average completion latency
- Acceptance rate by language
| +| Chat | - Total chat events
- Minutes saved per chat
- Hours saved by chats
- Cody chats by day | +| Commands | - Total command events
- Minutes saved per command
- Hours saved by commands
- Cody commands by day
- Most used commands | + +To enable Cody Analytics: + +- Create an account on [Sourcegraph Accounts](https://accounts.sourcegraph.com/) +- A user already having a Sourcegraph.com account gets automatically migrated to Sourcegraph Accounts. Users can sign in to Cody Analytics using their email and password +- Users without a Sourcegraph.com account should contact one of our team members. They can help with both the account setup and assigning instances to specific users +- Map your user account to a Sourcegraph instance, and this gives you access to Cody analytics + +## Multi-repository context + +Cody supports multi-repository context, allowing you to search up to 10 repositories simultaneously for relevant information. Open a new chat, type `@`, and select `Remote Repositories.` + +Keep @-mentioning repos that you want to include in your context. This flexibility lets you get more comprehensive and accurate responses by leveraging information across multiple codebases. + +## @-mention directories + +To better support teams working with large monorepos, Enterprise users can `@-mention` directories when chatting with Cody. This helps you define more specific directories and sub-directories within that monorepo to give more precise context. + +To do this, type `@` in the chat, and then select **Directories** to search other repositories for context in your codebase. + +![@-mention-directory](https://storage.googleapis.com/sourcegraph-assets/Docs/cody-ent-at-mention-dir-1124.png) + +Please note that you can only `@-mention` remote directories (i.e., directories in your Sourcegraph instance) but not local directories. This means any recent changes to your directories can't be utilized as context until your Sourcegraph instance re-indexes any changes. + +If you want to include recent changes that haven't been indexed in your Sourcegraph instance, you can `@-mention` specific files, lines of code, or symbols. + +## Supported LLM models + +Sourcegraph Enterprise supports different LLM providers and models, such as models from Anthropic and OpenAI. You can do this by adjusting your Sourcegraph instance configuration. + + +For the supported LLM models listed above, refer to the following notes: + +1. Microsoft Azure is planning to deprecate the APIs used in Sourcegraph version `>5.3.3` on July 1, 2024 [Source](https://learn.microsoft.com/en-us/azure/ai-services/openai/api-version-deprecation) +2. Claude 2.1 is not recommended +3. Sourcegraph doesn’t recommend using the GPT-4 (non-Turbo), Claude 1, or Claude 2 models anymore +4. Only supported through legacy completions API +5. BYOK (Bring Your Own Key) with managed services are only supported for Self-hosted Sourcegraph instances +6. GPT-4 and GPT-4o for completions have a bug that is resulting in many failed completions + +## Supported model configuration + +Use the drop-down menu to make your desired selection and get a detailed breakdown of the supported model configuration for each provider on Cody Enterprise. This is an on-site configuration. Admins should pick a value from the table for `chatModel` to configure their chat model. + + + +For the supported LLM model configuration listed above, refer to the following notes: + +1. Microsoft Azure is planning to deprecate the APIs used in Sourcegraph version `>5.3.3` on July 1, 2024 [Source](https://learn.microsoft.com/en-us/azure/ai-services/openai/api-version-deprecation) +2. Claude 2.1 is not recommended +3. Sourcegraph doesn't recommend GPT-4 non-turbo, Claude 1 or 2 models +4. Only supported through legacy completions API +5. BYOK (Bring Your Own Key) with managed services are only supported for Self-hosted Sourcegraph instances diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/enterprise/model-config-examples.mdx similarity index 100% rename from docs/cody/model-configuration/examples.mdx rename to docs/cody/enterprise/model-config-examples.mdx diff --git a/docs/cody/model-configuration/index.mdx b/docs/cody/enterprise/model-configuration.mdx similarity index 91% rename from docs/cody/model-configuration/index.mdx rename to docs/cody/enterprise/model-configuration.mdx index 6a648e3c2..51ed7010f 100644 --- a/docs/cody/model-configuration/index.mdx +++ b/docs/cody/enterprise/model-configuration.mdx @@ -1,26 +1,16 @@ # Model Configuration -Sourcegraph v5.6.0 or later supports the ability to choose between different LLM models, allowing developers to use the -best model for Cody Chat as needed. This is accomplished exposing much more flexible configuration options for Cody when -using Sourcegraph Enterprise. The newer style of configuration is described next. However, you can still use the -[Older style "Completions" Configuration](#legacy-completions-configuration). - - - Before you start, please note that the model configuration is an early - access program (EAP) and we are working towards improving on its coverage of - supported providers. If you are having any issues with this configuration, - please reach out to your Sourcegraph Account Representative or roll back - your configuration to the Legacy "Completions" configuration{' '} - +

Learn how to configure Cody via `modelConfigurations` on a Sourcegraph Enterprise instance.

+ +For Sourcegraph `v5.6.0` and more, users on an Enterprise Sourcegraph instance can choose between different LLM models for Cody chat, enabling greater flexibility to select the best model for their needs. + +The LLM models available for use from a Sourcegraph Enterprise instance are the union of Sourcegraph-supplied models and any custom models providers that you explicitly add to your Sourcegraph instance's site configuration. -The LLM models available for use from a Sourcegraph Enterprise instance are the union of [Sourcegraph-supplied models](/cody/model-configuration#sourcegraph-supplied-models) -and any custom models providers that you explicitly add to your Sourcegraph instance's site configuration. +The model configuration for Cody is managed through the `"modelConfiguration"` field in the **Site config** section. -The model configuration for Cody is managed through the `"modelConfiguration"` field in the site config. -It includes the fields listed below, with a brief overview of each in the table. -For more detailed information, refer to the documentation for each field. +It includes the following fields: -| Field | Description | +| **Field** | **Description** | | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [`sourcegraph`](/cody/model-configuration#sourcegraph-supplied-models) | Configures access to Sourcegraph-supplied models available through Cody Gateway. | | [`providerOverrides`](/cody/model-configuration#provider-overrides) | Configures model providers, allowing you to customize Cody's connection to model providers, such as using your own API keys or self-hosted models. | @@ -28,9 +18,9 @@ For more detailed information, refer to the documentation for each field. | [`selfHostedModels`](/cody/model-configuration#self-hosted-models) | Adds models to Cody’s recognized models list with default configurations provided by Sourcegraph. Only available for certain models; general models can be configured in `modelOverrides`. | | [`defaultModels`](/cody/model-configuration#default-models) | Specifies the models assigned to each Cody feature (chat, fast chat, autocomplete). | -## Quickstart +## Getting started with `modelConfiguration` -The quickest and recommended way to set up model configuration is by using Sourcegraph-supplied models through the Cody Gateway. +The recommended way to set up model configuration is by using Sourcegraph-supplied models through the Cody Gateway. For a minimal configuration example, see [Configure Sourcegraph-supplied models](/cody/model-configuration#configure-sourcegraph-supplied-models). @@ -81,7 +71,7 @@ The above configuration sets up the following: There are three main settings for configuring Sourcegraph-supplied LLM models: -| Field | Description | +| Field | Description | | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | `endpoint` | (Optional) The URL for connecting to Cody Gateway, defaulting to the production instance. | | `accessToken` | (Optional) The access token for connecting to Cody Gateway, which defaults to the current license key. | @@ -91,7 +81,7 @@ There are three main settings for configuring Sourcegraph-supplied LLM models: The `"modelFilters"` section allows you to control which Cody Gateway models are available to users of your Sourcegraph Enterprise instance. The following table describes each field: -| Field | Description | +| Field | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `statusFilter` | Filters models by their release status, such as "stable", "beta", "deprecated" or "experimental." By default, all models available on Cody Gateway are accessible. | | `allow` | An array of `modelRef`s specifying which models to include. Supports wildcards. | @@ -138,7 +128,7 @@ models or customize the existing provider namespace supplied by Sourcegraph (e.g Provider overrides are configured via the `"modelConfiguration.providerOverrides"` field in the site configuration. This field is an array of items, each containing the following fields: -| Field | Description | +| Field | Description | | ------------------ | --------------------------------------------------------------------------------- | | `id` | The namespace for models accessed via the provider. | | `displayName` | A human-readable name for the provider. | @@ -184,7 +174,7 @@ should be invoked—that is, which external service or API will handle the LLM r Sourcegraph natively supports several types of LLM API providers. The current set of supported providers includes: -| Provider type | Description | +| Provider type | Description | | -------------------- | ----------------------------------------------------------------------------------------------------------- | | `"sourcegraph"` | [Cody Gateway](/cody/core-concepts/cody-gateway), offering access to various models from multiple services. | | `"openaicompatible"` | Any OpenAI-compatible API implementation. | @@ -325,7 +315,7 @@ Specifying the same model in both `"selfHostedModels"` and `"modelOverrides"` is This field is an array of items, each with the following fields: -| Field | Description | +| Field | Description | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `provider` | (Required) The provider ID of the model, configured in the `providerOverrides` field. | | `apiVersion` | (Optional) The API version to use. Defaults to `"v1"` if not specified. | diff --git a/src/data/navigation.ts b/src/data/navigation.ts index 57b132b6d..39414f628 100644 --- a/src/data/navigation.ts +++ b/src/data/navigation.ts @@ -38,14 +38,15 @@ export const navigation: NavigationItem[] = [ { title: "Cody for JetBrains", href: "/cody/clients/install-jetbrains", }, { title: "Cody for Visual Studio", href: "/cody/clients/install-visual-studio", }, { title: "Cody for Web", href: "/cody/clients/cody-with-sourcegraph", }, - { title: "Cody for Enterprise", href: "/cody/clients/enable-cody-enterprise", }, - { title: "Model Configuration", href: "/cody/clients/model-configuration", }, ] }, { - title: "Model Configuration", href: "/cody/model-configuration", + title: "Cody for Enterprise", href: "/cody/clients/enable-cody-enterprise", subsections: [ - { title: "Examples", href: "/cody/model-configuration/examples", }, + { title: "Features", href: "/cody/enterprise/features", }, + { title: "Completions Configuration", href: "/cody/enterprise/completions-configuration", }, + { title: "Model Configuration", href: "/cody/enterprise/model-configuration", }, + { title: "Examples modelConfiguration", href: "/cody/enterprise/model-config-examples", }, ] }, { From 8ace100a424fe8aa4db8b7dc7362cfba23b5b6af Mon Sep 17 00:00:00 2001 From: Maedah Batool Date: Thu, 21 Nov 2024 09:15:04 -0800 Subject: [PATCH 23/58] Rmv model config matrix --- docs/cody/enterprise/features.mdx | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/cody/enterprise/features.mdx b/docs/cody/enterprise/features.mdx index 2805b47e0..3685d5d1c 100644 --- a/docs/cody/enterprise/features.mdx +++ b/docs/cody/enterprise/features.mdx @@ -78,7 +78,9 @@ For the supported LLM models listed above, refer to the following notes: 5. BYOK (Bring Your Own Key) with managed services are only supported for Self-hosted Sourcegraph instances 6. GPT-4 and GPT-4o for completions have a bug that is resulting in many failed completions -## Supported model configuration +{/*Tempporarily removed*/} + + {/* ## Supported model configuration Use the drop-down menu to make your desired selection and get a detailed breakdown of the supported model configuration for each provider on Cody Enterprise. This is an on-site configuration. Admins should pick a value from the table for `chatModel` to configure their chat model. @@ -91,3 +93,5 @@ For the supported LLM model configuration listed above, refer to the following n 3. Sourcegraph doesn't recommend GPT-4 non-turbo, Claude 1 or 2 models 4. Only supported through legacy completions API 5. BYOK (Bring Your Own Key) with managed services are only supported for Self-hosted Sourcegraph instances + + */} From b680cf52f2e5596c7eecadb46482ac4f06eeecf7 Mon Sep 17 00:00:00 2001 From: Maedah Batool Date: Thu, 21 Nov 2024 16:04:21 -0800 Subject: [PATCH 24/58] Add improvements --- docs/cody/enterprise/model-configuration.mdx | 214 +++++++++---------- 1 file changed, 98 insertions(+), 116 deletions(-) diff --git a/docs/cody/enterprise/model-configuration.mdx b/docs/cody/enterprise/model-configuration.mdx index 51ed7010f..9324c2d14 100644 --- a/docs/cody/enterprise/model-configuration.mdx +++ b/docs/cody/enterprise/model-configuration.mdx @@ -1,45 +1,49 @@ # Model Configuration -

Learn how to configure Cody via `modelConfigurations` on a Sourcegraph Enterprise instance.

+

Learn how to configure Cody via `modelConfiguration` on a Sourcegraph Enterprise instance.

-For Sourcegraph `v5.6.0` and more, users on an Enterprise Sourcegraph instance can choose between different LLM models for Cody chat, enabling greater flexibility to select the best model for their needs. +`modelConfiguration` is the recommended way to configure chat and autocomplete models in Sourcegraph `v5.6.0` and later. -The LLM models available for use from a Sourcegraph Enterprise instance are the union of Sourcegraph-supplied models and any custom models providers that you explicitly add to your Sourcegraph instance's site configuration. +The `modelConfiguration` field in the **Site config** section allows you to configure Cody to use different LLM models for chat and autocomplete, enabling greater flexibility in selecting the best model for your needs. -The model configuration for Cody is managed through the `"modelConfiguration"` field in the **Site config** section. +Using this configuration, you get an LLM selector in the Cody chat with an Enterprise instance that allows you to select the model you want to use. -It includes the following fields: +The LLM models available for a Sourcegraph Enterprise instance offer a combination of Sourcegraph-provided models and any custom models providers that you explicitly add to your Sourcegraph instance site configuration. -| **Field** | **Description** | +## `modelConfiguration` + +The model configuration for Cody is managed through the `"modelConfiguration"` field in the **Site config** section. It includes the following fields: + +| **Field** | **Description** | | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| [`sourcegraph`](/cody/model-configuration#sourcegraph-supplied-models) | Configures access to Sourcegraph-supplied models available through Cody Gateway. | -| [`providerOverrides`](/cody/model-configuration#provider-overrides) | Configures model providers, allowing you to customize Cody's connection to model providers, such as using your own API keys or self-hosted models. | -| [`modelOverrides`](/cody/model-configuration#model-overrides) | Extends or modifies the list of models Cody recognizes, along with their configurations. | +| [`sourcegraph`](/cody/model-configuration#sourcegraph-supplied-models) | Configures access to Sourcegraph-provided models available through Cody Gateway. | +| [`providerOverrides`](/cody/model-configuration#provider-overrides) | Configures model providers, allowing you to customize Cody's connection to model providers using your own API keys or self-hosted models. | +| [`modelOverrides`](/cody/model-configuration#model-overrides) | Extends or modifies the list of models Cody recognizes and their configurations. | | [`selfHostedModels`](/cody/model-configuration#self-hosted-models) | Adds models to Cody’s recognized models list with default configurations provided by Sourcegraph. Only available for certain models; general models can be configured in `modelOverrides`. | | [`defaultModels`](/cody/model-configuration#default-models) | Specifies the models assigned to each Cody feature (chat, fast chat, autocomplete). | ## Getting started with `modelConfiguration` -The recommended way to set up model configuration is by using Sourcegraph-supplied models through the Cody Gateway. +The recommended way to set up model configuration is using Sourcegraph-provided models through the Cody Gateway. For a minimal configuration example, see [Configure Sourcegraph-supplied models](/cody/model-configuration#configure-sourcegraph-supplied-models). -## Sourcegraph-supplied models +## Sourcegraph-provided models -Sourcegraph-supplied models, accessible through the [Cody Gateway](/cody/core-concepts/cody-gateway), are managed via your site configuration. +Sourcegraph-provided models, accessible through the [Cody Gateway](/cody/core-concepts/cody-gateway), are managed via your site configuration. For most administrators, relying on these models alone ensures access to high-quality models without needing to manage specific configurations. -Usage of these models is controlled through the `"modelConfiguration.sourcegraph"` field in the site config. +The use of these models is controlled through the `"modelConfiguration.sourcegraph"` field in the site config. -### Disable Sourcegraph-supplied models +### Disable Sourcegraph-provided models -To disable all Sourcegraph-supplied models and use only the models explicitly defined in your site configuration, set the `"sourcegraph"` field to `null` (see example below). +To disable all Sourcegraph-provided models and use only the models explicitly defined in your site configuration, set the `"sourcegraph"` field to `null` as shown in the example below. ```json "cody.enabled": true, "modelConfiguration": { - "sourcegraph": null, // ignore Sourcegraph-supplied models + "sourcegraph": null, // ignore Sourcegraph-provided models "providerOverrides": { // define access to the LLM providers }, @@ -52,9 +56,9 @@ To disable all Sourcegraph-supplied models and use only the models explicitly de } ``` -### Configure Sourcegraph-supplied models +### Configure Sourcegraph-provided models -The minimal configuration for Sourcegraph-supplied models is: +The minimal configuration for Sourcegraph-provided models is: ```json "cody.enabled": true, @@ -65,27 +69,27 @@ The minimal configuration for Sourcegraph-supplied models is: The above configuration sets up the following: -- Sourcegraph-supplied models are enabled (`sourcegraph` is not set to `null`). -- Requests to LLM providers are routed through the Cody Gateway (no `providerOverrides` field specified). -- Sourcegraph-defined default models are used for Cody features (no `defaultModels` field specified). +- Sourcegraph-provided models are enabled (`sourcegraph` is not set to `null`) +- Requests to LLM providers are routed through the Cody Gateway (no `providerOverrides` field specified) +- Sourcegraph-defined default models are used for Cody features (no `defaultModels` field specified) -There are three main settings for configuring Sourcegraph-supplied LLM models: +There are three main settings for configuring Sourcegraph-provided LLM models: -| Field | Description | +| **Field** | **Description** | | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | -| `endpoint` | (Optional) The URL for connecting to Cody Gateway, defaulting to the production instance. | -| `accessToken` | (Optional) The access token for connecting to Cody Gateway, which defaults to the current license key. | +| `endpoint` | (Optional) The URL for connecting to Cody Gateway. The default is set to the production instance. | +| `accessToken` | (Optional) The access token for connecting to Cody Gateway defaults to the current license key. | | [`modelFilters`](/cody/model-configuration#model-filters) | (Optional) Filters specifying which models to include from Cody Gateway. | ### Model Filters The `"modelFilters"` section allows you to control which Cody Gateway models are available to users of your Sourcegraph Enterprise instance. The following table describes each field: -| Field | Description | -| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `statusFilter` | Filters models by their release status, such as "stable", "beta", "deprecated" or "experimental." By default, all models available on Cody Gateway are accessible. | -| `allow` | An array of `modelRef`s specifying which models to include. Supports wildcards. | -| `deny` | An array of `modelRef`s specifying which models to exclude. Supports wildcards. | +| **Field** | **Description** | +| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `statusFilter` | Filters models based on their release status, such as stable, beta, deprecated or experimental. By default, all models available on Cody Gateway are accessible. | +| `allow` | An array of `modelRef`s specifying which models to include. Supports wildcards. | +| `deny` | An array of `modelRef`s specifying which models to exclude. Supports wildcards. | The following examples demonstrate how to use each of these settings together: @@ -119,16 +123,14 @@ The following examples demonstrate how to use each of these settings together: ## Provider Overrides -A "provider" is an organizational concept for grouping LLM models. Typically, a provider refers to the company that -produced the model or the specific API/service used to access it, serving as a namespace. +A `provider` is an organizational concept for grouping LLM models. Typically, a provider refers to the company that produced the model or the specific API/service used to access it, serving as a namespace. -By defining a provider override in your Sourcegraph site configuration, you can introduce a new namespace to organize -models or customize the existing provider namespace supplied by Sourcegraph (e.g., for all `"anthropic"` models). +By defining a provider override in your Sourcegraph site configuration, you can introduce a new namespace to organize models or customize the existing provider namespace supplied by Sourcegraph (e.g., for all `"anthropic"` models). Provider overrides are configured via the `"modelConfiguration.providerOverrides"` field in the site configuration. This field is an array of items, each containing the following fields: -| Field | Description | +| **Field** | **Description** | | ------------------ | --------------------------------------------------------------------------------- | | `id` | The namespace for models accessed via the provider. | | `displayName` | A human-readable name for the provider. | @@ -161,20 +163,18 @@ Example configuration: In the example above: -- Sourcegraph-supplied models are enabled (`sourcegraph` is not set to `null`). -- The `"openai"` provider configuration is overridden to be accessed using your own key, with the provider API accessed - directly via the specified `endpoint` URL. In contrast, models from the `"anthropic"` provider are accessed through the Cody Gateway. +- Sourcegraph-provided models are enabled (`sourcegraph` is not set to `null`) +- The `"openai"` provider configuration is overridden to be accessed using your own key, with the provider API accessed directly via the specified `endpoint` URL. In contrast, models from the `"anthropic"` provider are accessed through the Cody Gateway -For additional examples, refer to the [examples page](/cody/model-configuration/examples). +Refer to the [examples page](/cody/model-configuration/examples) for additional examples. -### Server-side Config +### Server-side config -The most important part of a provider's configuration is the `"serverSideConfig"` field, which defines how the LLM models -should be invoked—that is, which external service or API will handle the LLM requests. +The most important part of a provider's configuration is the `"serverSideConfig"` field, which defines how the LLM models should be invoked, i.e., which external service or API will handle the LLM requests. Sourcegraph natively supports several types of LLM API providers. The current set of supported providers includes: -| Provider type | Description | +| **Provider type** | **Description** | | -------------------- | ----------------------------------------------------------------------------------------------------------- | | `"sourcegraph"` | [Cody Gateway](/cody/core-concepts/cody-gateway), offering access to various models from multiple services. | | `"openaicompatible"` | Any OpenAI-compatible API implementation. | @@ -188,56 +188,44 @@ Sourcegraph natively supports several types of LLM API providers. The current se The configuration for `serverSideConfig` varies by provider type. -For examples, refer to the [examples page](/cody/model-configuration/examples). +Refer to the [examples page](/cody/model-configuration/examples) for examples. ## Model Overrides -With a provider defined (either a Sourcegraph-supplied provider or a custom provider configured via the `providerOverrides` -field), custom models can be specified for that provider by adding them to the `"modelConfiguration.modelOverrides"` section. +With a provider defined (either a Sourcegraph-provided provider or a custom provider configured via the `providerOverrides` field), custom models can be specified for that provider by adding them to the `"modelConfiguration.modelOverrides"` section. This field is an array of items, each with the following fields: -- `modelRef` - Uniquely identifies the model within the provider namespace. - - A string in the format `${providerId}::${apiVersionId}::${modelId}`. - - To associate a model with your provider, `${providerId}` must match the provider’s ID. - - `${modelId}` can be any URL-safe string. - - `${apiVersionId}` specifies the API version, which helps detect compatibility issues between models and Sourcegraph instances. - For example, `"2023-06-01"` can indicate that the model uses that version of the Anthropic API. If unsure, you may set this to `"unknown"` when defining custom models. -- `displayName` - An optional, user-friendly name for the model. If not set, clients should display the `ModelID` part of the `modelRef` instead (not the `modelName`). -- `modelName` - A unique identifier used by the API provider to specify which model is being invoked. - This is the identifier that the LLM provider recognizes to determine the model you are calling. -- `capabilities` - A list of capabilities that the model supports. Supported values: "autocomplete" and "chat". -- `category` - Specifies the model's category, with the following options: - - - `"balanced"` - Typically the best default choice for most users. This category is suited for state-of-the-art models, like Sonnet 3.5 (as of October 2024). - - `"speed"` - Ideal for low-parameter models that may not be suited for general-purpose chat but are beneficial for specialized tasks, such as query rewriting. - - `"accuracy"` - Reserved for models, like OpenAI o1, that use advanced reasoning techniques to improve response accuracy, though with slower latency. - - `"other"` - Used for older models without distinct advantages in reasoning or speed. Select this category if uncertain about which category to choose. - - `"deprecated"` - For models that are no longer supported by the provider and are filtered out on the client side (not available for use). - -- `contextWindow` - An object that defines the number of "tokens" (units of text) that can be sent to the LLM. - This setting influences response time and request cost, and may vary according to the limits set by each LLM model or provider. - It includes two fields: - - `maxInputTokens` - Specifies the maximum number of tokens for the contextual data in the prompt (e.g., question, relevant snippets). - - `maxOutputTokens` - Specifies the maximum number of tokens allowed in the response. -- `serverSideConfig` - Additional configuration for the model. Can be one of the following: - - - `awsBedrockProvisionedThroughput` - Specifies provisioned throughput settings for AWS Bedrock models, with the following fields: - - - `type` - Must be `"awsBedrockProvisionedThroughput"`. - - `arn` - The ARN (Amazon Resource Name) for provisioned throughput to use when sending requests to AWS Bedrock. - - - `openaicompatible` - Configuration specific to models provided by an OpenAI-compatible provider, with the following fields: - - - `type` - Must be `"openaicompatible"`. - - `apiModel` - The literal string value of the `model` field to be sent to the `/chat/completions` API. - If set, Sourcegraph treats this as an opaque string and sends it directly to the API without inferring any additional information. - By default, the configured model name is sent. - -- `clientSideConfig` - Advanced configuration options that are only respected if the model is provided by the `"openaicompatible"` provider. - TODO: add more details. - -Example configuration: +- `modelRef`: Uniquely identifies the model within the provider namespace + - A string in the format `${providerId}::${apiVersionId}::${modelId}` + - To associate a model with your provider, `${providerId}` must match the provider’s ID + - `${modelId}` can be any URL-safe string + - `${apiVersionId}` specifies the API version, which helps detect compatibility issues between models and Sourcegraph instances. For example, `"2023-06-01"` can indicate that the model uses that version of the Anthropic API. If unsure, you may set this to `"unknown"` when defining custom models +- `displayName`: An optional, user-friendly name for the model. If not set, clients should display the `ModelID` part of the `modelRef` instead (not the `modelName`) +- `modelName`: A unique identifier the API provider uses to specify which model is being invoked. This is the identifier that the LLM provider recognizes to determine the model you are calling +- `capabilities`: A list of capabilities that the model supports. Supported values: **autocomplete** and **chat** +- `category`: Specifies the model's category with the following options: + - `"balanced"`: Typically the best default choice for most users. This category is suited for models like Sonnet 3.5 (as of October 2024) + - `"speed"`: Ideal for low-parameter models that may not suit general-purpose chat but are beneficial for specialized tasks, such as query rewriting + - `"accuracy"`: Reserved for models, like OpenAI o1, that use advanced reasoning techniques to improve response accuracy, though with slower latency + - `"other"`: Used for older models without distinct advantages in reasoning or speed. Select this category if you are uncertain about which category to choose + - `"deprecated"`: For models that are no longer supported by the provider and are filtered out on the client side (not available for use) +- `contextWindow`: An object that defines the **number of tokens** (units of text) that can be sent to the LLM. This setting influences response time and request cost and may vary according to the limits set by each LLM model or provider. It includes two fields: + - `maxInputTokens`: Specifies the maximum number of tokens for the contextual data in the prompt (e.g., question, relevant snippets) + - `maxOutputTokens`: Specifies the maximum number of tokens allowed in the response +- `serverSideConfig`: Additional configuration for the model. It can be one of the following: + - `awsBedrockProvisionedThroughput`: Specifies provisioned throughput settings for AWS Bedrock models with the following fields: + - `type`: Must be `"awsBedrockProvisionedThroughput"` + - `arn`: The ARN (Amazon Resource Name) for provisioned throughput to use when sending requests to AWS Bedrock +- `openaicompatible`: Configuration specific to models provided by an OpenAI-compatible provider, with the following fields: + - `type`: Must be `"openaicompatible"` + - `apiModel`: The literal string value of the `model` field to be sent to the `/chat/completions` API. If set, Sourcegraph treats this as an opaque string and sends it directly to the API without inferring any additional information. By default, the configured model name is sent + +- `clientSideConfig`: Advanced configuration options that are only respected if the model is provided by the `"openaicompatible"` provider + +{/*TODO: add more details.*/} + +#### Example configuration ```json "cody.enabled": true, @@ -289,54 +277,48 @@ Example configuration: In the example above: -- Sourcegraph-supplied models are enabled (`sourcegraph` is not set to `null`). -- An additional provider, `"huggingface-codellama"`, is configured to access Hugging Face’s OpenAI-compatible API directly. -- A custom model, `"CodeLlama-7b-hf"`, is added using the `"huggingface-codellama"` provider. -- Default models are set up as follows: - - Sourcegraph-supplied models are used for `"chat"` and `"fastChat"` (accessed via Cody Gateway). - - The newly configured model, `"huggingface-codellama::v1::CodeLlama-7b-hf"`, is used for `"autocomplete"` - (connecting directly to Hugging Face’s OpenAI-compatible API). +- Sourcegraph-provided models are enabled (`sourcegraph` is not set to `null`) +- An additional provider, `"huggingface-codellama"`, is configured to access Hugging Face’s OpenAI-compatible API directly +- A custom model, `"CodeLlama-7b-hf"`, is added using the `"huggingface-codellama"` provider +- Default models are set up as follows: + - Sourcegraph-provided models are used for `"chat"` and `"fastChat"` (accessed via Cody Gateway) + - The newly configured model, `"huggingface-codellama::v1::CodeLlama-7b-hf"`, is used for `"autocomplete"` (connecting directly to Hugging Face’s OpenAI-compatible API) -For additional examples, refer to the [examples page](/cody/model-configuration/examples). +Refer to the [examples page](/cody/model-configuration/examples) for additional examples. -## Self-hosted Models +## Self-hosted models - - Configuring self-hosted models can be nuanced and complex. Consider reaching - out to Sourcegraph support for assistance with the setup. - +Configuring self-hosted models can be nuanced and complex. Please reach + out to Sourcegraph support for assistance with the setup. -With a provider defined (either a Sourcegraph-supplied provider or a custom provider configured via the `providerOverrides` field), -self-hosted models can be added to Cody’s recognized models list by configuring the `"modelConfiguration.selfHostedModels"` section. +With a provider defined (either a Sourcegraph-provided provider or a custom provider configured via the `providerOverrides` field), self-hosted models can be added to Cody’s recognized models list by configuring the `"modelConfiguration.selfHostedModels"` section. These models use Sourcegraph’s default configuration settings and are only available for select models. -Generic models should instead be configured in the `"modelOverrides"` section. +Generic models should be configured in the `"modelOverrides"` section instead. Specifying the same model in both `"selfHostedModels"` and `"modelOverrides"` is not allowed. This field is an array of items, each with the following fields: -| Field | Description | +| **Field** | **Description** | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `provider` | (Required) The provider ID of the model, configured in the `providerOverrides` field. | | `apiVersion` | (Optional) The API version to use. Defaults to `"v1"` if not specified. | | `model` | (Optional) Specifies which default model configuration to use. Sourcegraph provides default configurations for select models. For arbitrary models, use the `modelOverrides` section. Available options include: `"starcoder2-7b@v1"`, `"starcoder2-15b@v1"`, `"mistral-7b-instruct@v1"`, `"mixtral-8x7b-instruct@v1"`, `"mixtral-8x22b-instruct@v1"` | -For examples, refer to the [examples page](/cody/model-configuration/examples#self-hosted-models). +Refer to the [examples page](/cody/model-configuration/examples#self-hosted-models) for examples. -## Default Models +## Default models -The `"modelConfiguration"` setting includes a `"defaultModels"` field, which allows you to specify the LLM model used for -each Cody feature (`"chat"`, `"fastChat"`, and `"autocomplete"`). The values for each feature should be `modelRef`s of -either Sourcegraph-supplied models or models configured in the `modelOverrides` section. +The `"modelConfiguration"` setting includes a `"defaultModels"` field, which allows you to specify the LLM model used for each Cody feature (`"chat"`, `"fastChat"`, and `"autocomplete"`). The values for each feature should be `modelRef`s of either Sourcegraph-provided models or models configured in the `modelOverrides` section. -If no default is specified or if the specified model is not found, the configuration will silently fall back to a suitable alternative. +If no default is specified or the specified model is not found, the configuration will silently fall back to a suitable alternative. -Example configuration: +#### Example configuration ```json "cody.enabled": true, -// allow Sourcegraph-supplied models +// allow Sourcegraph-provided models "sourcegraph": {}, "defaultModels": { @@ -346,7 +328,7 @@ Example configuration: } ``` -## View Configuration +## View configuration To view the current model configuration, run the following command: @@ -360,10 +342,10 @@ curl --location "${INSTANCE_URL}/.api/modelconfig/supported-models.json" \ The response includes: -- Configured providers and models—both Sourcegraph-supplied (if enabled, with any applied filters) and any overrides. -- Default models for Cody features. +- Configured providers and models—both Sourcegraph-provided (if enabled, with any applied filters) and any overrides +- Default models for Cody features -Example response: +#### Example response ```json "schemaVersion": "1.0", From e6c2576b637b9719c375181e39f80c49fb887fd4 Mon Sep 17 00:00:00 2001 From: Maedah Batool Date: Thu, 21 Nov 2024 16:47:02 -0800 Subject: [PATCH 25/58] Add examples --- .../cody/enterprise/model-config-examples.mdx | 256 ++++++++---------- 1 file changed, 116 insertions(+), 140 deletions(-) diff --git a/docs/cody/enterprise/model-config-examples.mdx b/docs/cody/enterprise/model-config-examples.mdx index 89069b23a..270dc34cf 100644 --- a/docs/cody/enterprise/model-config-examples.mdx +++ b/docs/cody/enterprise/model-config-examples.mdx @@ -1,24 +1,22 @@ -# Examples +# Model Configuration Examples -## Sourcegraph-supplied models only +

This section includes examples about how to configure Cody to use Sourcegraph-provided models with `modelConfiguration`. These examples will use the following:

-This section includes examples how to configure Cody to use Sourcegraph-supplied models. +- [Minimal configuration](/cody/model-configuration#configure-sourcegraph-supplied-models) +- [Using model filters](/cody/model-configuration#model-filters) +- [Change default models](/cody/model-configuration#default-models) -- [Minimal configuration](/cody/model-configuration#configure-sourcegraph-supplied-models) -- [Using model filters](/cody/model-configuration#model-filters) -- [Change default models](/cody/model-configuration#default-models) +## Sourcegraph-provided models and BYOK (Bring Your Own Key) -## Sourcegraph-supplied models and BYOK (Bring Your Own Key) +By default, Sourcegraph is fully aware of several models from the following providers: -Out of the box, Sourcegraph is already aware of many models from the following providers: +- "anthropic" +- "google" +- "fireworks" +- "mistral" +- "openai" -- "anthropic" -- "google" -- "fireworks" -- "mistral" -- "openai" - -### Overriding configuration of a model provider +### Override configuration of a model provider Instead of Sourcegraph using its own servers to make LLM requests, it is possible to bring your own API keys for a given model provider. For example, if you wish for all Anthropic API requests to go directly to your own Anthropic account and use your own API keys instead of going via Sourcegraph's servers, you could override the `anthropic` provider's configuration: @@ -46,41 +44,32 @@ Instead of Sourcegraph using its own servers to make LLM requests, it is possibl } ``` -In the configuration above, we: +In the configuration above: -- Enable Sourcegraph-supplied models and do not set any overrides (note that `"modelConfiguration.modelOverrides"` is not specified). -- Route requests for Anthropic models directly to the Anthropic API (via the provider override specified for "anthropic"). -- Route requests for other models (such as the Fireworks model for "autocomplete") through Cody Gateway. +- Enable Sourcegraph-provided models and do not set any overrides (note that `"modelConfiguration.modelOverrides"` is not specified) +- Route requests for Anthropic models directly to the Anthropic API (via the provider override specified for "anthropic") +- Route requests for other models (such as the Fireworks model for "autocomplete") through Cody Gateway -### Override provider config for some models in the namespace and use the Sourcegraph-configured provider config for the rest +### Partially override provider config in the namespace -It's possible to route requests directly to the LLM provider (bypassing the Cody Gateway) for some models while using the -Sourcegraph-configured provider config for the rest. +If you want to override the provider config for some models in the namespace and use the Sourcegraph-configured provider config for the rest, you can route requests directly to the LLM provider (bypassing the Cody Gateway) for some models while using the Sourcegraph-configured provider config for the rest. Example configuration: In the configuration above, we: -- Enable Sourcegraph-supplied models (the `sourcegraph` field is not empty or `null`). -- Define a new provider with the ID `"anthropic-byok"` and configure it to use the Anthropic API. -- Since this provider is unknown to Sourcegraph, no Sourcegraph-supplied models are available for it. - Therefore, we add a custom model in the `"modelOverrides"` section. -- Use the custom model configured in the previous step (`"anthropic-byok::2024-10-22::claude-3.5-sonnet"`) for `"chat"`. - Requests are sent directly to the Anthropic API as set in the provider override. -- For `"fastChat"` and `"autocomplete"`, we use Sourcegraph-supplied models via Cody Gateway. +- Enable Sourcegraph-supplied models (the `sourcegraph` field is not empty or `null`) +- Define a new provider with the ID `"anthropic-byok"` and configure it to use the Anthropic API +- Since this provider is unknown to Sourcegraph, no Sourcegraph-supplied models are available. Therefore, we add a custom model in the `"modelOverrides"` section +- Use the custom model configured in the previous step (`"anthropic-byok::2024-10-22::claude-3.5-sonnet"`) for `"chat"`. Requests are sent directly to the Anthropic API as set in the provider override +- For `"fastChat"` and `"autocomplete"`, we use Sourcegraph-provided models via Cody Gateway ## Config examples for various LLM providers -Below are configuration examples for setting up various LLM providers using BYOK. -These examples are applicable whether or not you are using Sourcegraph-supported models. - -**Note:** +Below are configuration examples for setting up various LLM providers using BYOK. These examples are applicable whether or not you are using Sourcegraph-supported models. -- In this section, all configuration examples have Sourcegraph-supplied models disabled. To use a combination of - Sourcegraph-supplied models and BYOK, please refer to the previous section. -- Ensure that at least one model is available for each Cody feature ("chat" and "autocomplete"), regardless of - the provider and model overrides configured. To verify this, [view the configuration](/cody/model-configuration#view-configuration) - and confirm that appropriate models are listed in the `"defaultModels"` section. +- In this section, all configuration examples have Sourcegraph-provided models disabled. Please refer to the previous section to use a combination of Sourcegraph-provided models and BYOK. +- Ensure that at least one model is available for each Cody feature ("chat" and "autocomplete"), regardless of the provider and model overrides configured. To verify this, [view the configuration](/cody/model-configuration#view-configuration) and confirm that appropriate models are listed in the `"defaultModels"` section. @@ -149,14 +138,13 @@ These examples are applicable whether or not you are using Sourcegraph-supported } ``` -In the configuration above, we: +In the configuration above, - - Set up a provider override for Anthropic, routing requests for this provider directly to the specified Anthropic endpoint (bypassing Cody Gateway). - - Add three Anthropic models: - - Two models with chat capabilities (`"anthropic::2024-10-22::claude-3.5-sonnet"` and `"anthropic::2023-06-01::claude-3-haiku"`), - providing options for chat users. - - One model with autocomplete capability (`"anthropic::2023-01-01::claude-instant-1.2"`). - - Set the configured models as default models for Cody features in the `"defaultModels"` field. +- Set up a provider override for Anthropic, routing requests for this provider directly to the specified Anthropic endpoint (bypassing Cody Gateway) +- Add three Anthropic models: + - Two models with chat capabilities (`"anthropic::2024-10-22::claude-3.5-sonnet"` and `"anthropic::2023-06-01::claude-3-haiku"`), providing options for chat users + - One model with autocomplete capability (`"anthropic::2023-01-01::claude-instant-1.2"`) +- Set the configured models as default models for Cody features in the `"defaultModels"` field @@ -164,60 +152,59 @@ In the configuration above, we: ```json "cody.enabled": true, "modelConfiguration": { - "sourcegraph": null, - "providerOverrides": [ - { - "id": "fireworks", - "displayName": "Fireworks", - "serverSideConfig": { - "type": "fireworks", - "accessToken": "token", - "endpoint": "https://api.fireworks.ai/inference/v1/completions" - } - } - ], - "modelOverrides": [ - { - "modelRef": "fireworks::v1::mixtral-8x22b-instruct", - "displayName": "Mixtral 8x22B", - "modelName": "accounts/fireworks/models/mixtral-8x22b-instruct", - "capabilities": ["chat"], - "category": "other", - "status": "stable", - "tier": "free", - "contextWindow": { - "maxInputTokens": 7000, - "maxOutputTokens": 4000 - } - }, - { - "modelRef": "fireworks::v1::starcoder-16b", - "modelName": "accounts/fireworks/models/starcoder-16b", - "displayName": "(Fireworks) Starcoder 16B", - "contextWindow": { - "maxInputTokens": 8192, - "maxOutputTokens": 4096 - }, - "capabilities": ["autocomplete"], - "category": "balanced", - "status": "stable" - } - ], - "defaultModels": { - "chat": "fireworks::v1::mixtral-8x22b-instruct", - "fastChat": "fireworks::v1::mixtral-8x22b-instruct", - "autocomplete": "fireworks::v1::starcoder-16b" - } + "sourcegraph": null, + "providerOverrides": [ + { + "id": "fireworks", + "displayName": "Fireworks", + "serverSideConfig": { + "type": "fireworks", + "accessToken": "token", + "endpoint": "https://api.fireworks.ai/inference/v1/completions" + } + } + ], + "modelOverrides": [ + { + "modelRef": "fireworks::v1::mixtral-8x22b-instruct", + "displayName": "Mixtral 8x22B", + "modelName": "accounts/fireworks/models/mixtral-8x22b-instruct", + "capabilities": ["chat"], + "category": "other", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "fireworks::v1::starcoder-16b", + "modelName": "accounts/fireworks/models/starcoder-16b", + "displayName": "(Fireworks) Starcoder 16B", + "contextWindow": { + "maxInputTokens": 8192, + "maxOutputTokens": 4096 + }, + "capabilities": ["autocomplete"], + "category": "balanced", + "status": "stable" + } + ], + "defaultModels": { + "chat": "fireworks::v1::mixtral-8x22b-instruct", + "fastChat": "fireworks::v1::mixtral-8x22b-instruct", + "autocomplete": "fireworks::v1::starcoder-16b" + } } ``` -In the configuration above, we: +In the configuration above, -- Set up a provider override for Fireworks, routing requests for this provider directly to the specified Fireworks endpoint (bypassing Cody Gateway). -- Add two Fireworks models: - - `"fireworks::v1::mixtral-8x22b-instruct"` with "chat" capabiity - used for "chat" - and "fastChat" - - `"fireworks::v1::starcoder-16b"` with "autocomplete" capability - used for "autocomplete". +- Set up a provider override for Fireworks, routing requests for this provider directly to the specified Fireworks endpoint (bypassing Cody Gateway) +- Add two Fireworks models: + - `"fireworks::v1::mixtral-8x22b-instruct"` with "chat" capabiity - used for "chat" and "fastChat" + - `"fireworks::v1::starcoder-16b"` with "autocomplete" capability - used for "autocomplete" @@ -273,12 +260,12 @@ In the configuration above, we: } ``` -In the configuration above, we: +In the configuration above, -- Set up a provider override for OpenAI, routing requests for this provider directly to the specified OpenAI endpoint (bypassing Cody Gateway). -- Add two OpenAI models: - - `"openai::2024-02-01::gpt-4o"` with "chat" capabilities - used for "chat" and "fastChat" - - `"openai::unknown::gpt-3.5-turbo-instruct"` with "autocomplete" capability - used for "autocomplete". +- Set up a provider override for OpenAI, routing requests for this provider directly to the specified OpenAI endpoint (bypassing Cody Gateway) +- Add two OpenAI models: + - `"openai::2024-02-01::gpt-4o"` with "chat" capabilities - used for "chat" and "fastChat" + - `"openai::unknown::gpt-3.5-turbo-instruct"` with "autocomplete" capability - used for "autocomplete" @@ -337,15 +324,13 @@ In the configuration above, we: } ``` -In the configuration above, we: +In the configuration above, -- Set up a provider override for Azure OpenAI, routing requests for this provider directly to the specified Azure OpenAI endpoint (bypassing Cody Gateway). -- Add two OpenAI models: - - `"azure-openai::unknown::gpt-4o"` with "chat" capability - used for "chat" and "fastChat" - - `"azure-openai::unknown::gpt-35-turbo-instruct-test"` with "autocomplete" capability - used for "autocomplete". -- Since `"azure-openai::unknown::gpt-35-turbo-instruct-test"` is not supported on the newer OpenAI `"v1/chat/completions"` endpoint, - we set `"useDeprecatedCompletionsAPI"` to `true` to route requests to the legacy `"v1/completions"` endpoint. This setting is - unnecessary if you are using a model supported on the `"v1/chat/completions"` endpoint. +- Set up a provider override for Azure OpenAI, routing requests for this provider directly to the specified Azure OpenAI endpoint (bypassing Cody Gateway) +- Add two OpenAI models: + - `"azure-openai::unknown::gpt-4o"` with "chat" capability - used for "chat" and "fastChat" + - `"azure-openai::unknown::gpt-35-turbo-instruct-test"` with "autocomplete" capability - used for "autocomplete" +- Since `"azure-openai::unknown::gpt-35-turbo-instruct-test"` is not supported on the newer OpenAI `"v1/chat/completions"` endpoint, we set `"useDeprecatedCompletionsAPI"` to `true` to route requests to the legacy `"v1/completions"` endpoint. This setting is unnecessary if you are using a model supported on the `"v1/chat/completions"` endpoint @@ -431,12 +416,12 @@ In the configuration above, we: } ``` -In the configuration above, we: +In the configuration above, -- Configure two OpenAI-compatible providers: `"fireworks"` and `"huggingface-codellama"`. -- Add two OpenAI-compatible models: `"fireworks::v1::llama-v3p1-70b-instruct"` and `"huggingface-codellama::v1::CodeLlama-7b-hf"`. Additionally: - - Set `clientSideConfig.openaicompatible` to `{}` to indicate to Cody clients that these models are OpenAI-compatible, ensuring the appropriate code paths are utilized. - - Designate these models as the default choices for chat and autocomplete, respectively. +- Configure two OpenAI-compatible providers: `"fireworks"` and `"huggingface-codellama"` +- Add two OpenAI-compatible models: `"fireworks::v1::llama-v3p1-70b-instruct"` and `"huggingface-codellama::v1::CodeLlama-7b-hf"`. Additionally: + - Set `clientSideConfig.openaicompatible` to `{}` to indicate to Cody clients that these models are OpenAI-compatible, ensuring the appropriate code paths are utilized + - Designate these models as the default choices for chat and autocomplete, respectively @@ -491,12 +476,12 @@ In the configuration above, we: } ``` -In the configuration above, we: +In the configuration above, -- Set up a provider override for Google Anthropic, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway). -- Add two Anthropic models: - - `"google::unknown::claude-3-5-sonnet"` with "chat" capabiity - used for "chat" and "fastChat" - - `"google::unknown::claude-3-haiku"` with "autocomplete" capability - used for "autocomplete". +- Set up a provider override for Google Anthropic, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway) +- Add two Anthropic models: + - `"google::unknown::claude-3-5-sonnet"` with "chat" capabiity - used for "chat" and "fastChat" + - `"google::unknown::claude-3-haiku"` with "autocomplete" capability - used for "autocomplete" @@ -539,11 +524,10 @@ In the configuration above, we: } ``` -In the configuration above, we: +In the configuration above, -- Set up a provider override for Google Gemini, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway). -- Add the `"google::v1::gemini-1.5-pro"` model, which is used for all Cody features. - We do not add other models for simplicity, as adding multiple models is already covered in the examples above. +- Set up a provider override for Google Gemini, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway) +- Add the `"google::v1::gemini-1.5-pro"` model, which is used for all Cody features. We do not add other models for simplicity, as adding multiple models is already covered in the examples above @@ -588,35 +572,29 @@ In the configuration above, we: } ``` -In the configuration described above, we: +In the configuration described above, -- Set up a provider override for AWS Bedrock, routing requests for this provider directly to the specified endpoint, - bypassing Cody Gateway. -- Add the `"aws-bedrock::2023-06-01::claude-3-opus"` model, which is used for all Cody features. - We do not add other models for simplicity, as adding multiple models is already covered in the examples above. +- Set up a provider override for AWS Bedrock, routing requests for this provider directly to the specified endpoint, bypassing Cody Gateway +- Add the `"aws-bedrock::2023-06-01::claude-3-opus"` model, which is used for all Cody features. We do not add other models for simplicity, as adding multiple models is already covered in the examples above Provider override `serverSideConfig` fields: -| Field | Description | +| **Field** | **Description** | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | Must be `"awsBedrock"`. | | `accessToken` | Leave empty to rely on instance role bindings or other AWS configurations in the frontend service. Use `:` for direct credential configuration, or `::` if a session token is also required. | -| `endpoint` | For Pay-as-you-go, set it to an AWS region code (e.g., `us-west-2`) when using a public Amazon Bedrock endpoint. For Provisioned Throughput, set it to the provisioned VPC endpoint for the bedrock-runtime API (e.g., `https://vpce-0a10b2345cd67e89f-abc0defg.bedrock-runtime.us-west-2.vpce.amazonaws.com`). | -| `region` | The region to use when configuring API clients. This is necessary because the 'frontend' binary's container cannot access environment variables from the host OS. | +| `endpoint` | For pay-as-you-go, set it to an AWS region code (e.g., `us-west-2`) when using a public Amazon Bedrock endpoint. For provisioned throughput, set it to the provisioned VPC endpoint for the bedrock-runtime API (e.g., `https://vpce-0a10b2345cd67e89f-abc0defg.bedrock-runtime.us-west-2.vpce.amazonaws.com`). | +| `region` | The region to use when configuring API clients. This is necessary because the 'frontend' binary container cannot access environment variables from the host OS. | -Provisioned throughput for AWS Bedrock models can be configured using the `"awsBedrockProvisionedThroughput"` server-side -configuration type. For more details, refer to the [Model Overrides](/cody/model-configuration#model-overrides) section. +Provisioned throughput for AWS Bedrock models can be configured using the `"awsBedrockProvisionedThroughput"` server-side configuration type. Refer to the [Model Overrides](/cody/model-configuration#model-overrides) section for more details. ## Self-hosted models - - Configuring self-hosted models can be nuanced and complex. Consider reaching - out to Sourcegraph support for assistance with the setup. - +Configuring self-hosted models can be nuanced and complex. Please reach out to Sourcegraph support for assistance with the setup. -For detailed information about configuring self-hosted models, please refer to the [self-hosted models](/cody/model-configuration#self-hosted-models) documentation. +Please refer to the [self-hosted models](/cody/model-configuration#self-hosted-models) documentation for detailed information about configuring self-hosted models. Example configuration: @@ -666,6 +644,4 @@ Example configuration: } ``` -In the configuration above we: - -- TODO +{/* In the configuration above: TODO */} From 237d732b9cf56d40c37902e1005f109afdaf3a20 Mon Sep 17 00:00:00 2001 From: Maedah Batool Date: Thu, 21 Nov 2024 17:00:15 -0800 Subject: [PATCH 26/58] Add links and hash links --- docs/cody/clients/enable-cody-enterprise.mdx | 4 +-- .../cody/enterprise/model-config-examples.mdx | 12 ++++----- docs/cody/enterprise/model-configuration.mdx | 26 +++++++++---------- 3 files changed, 21 insertions(+), 21 deletions(-) diff --git a/docs/cody/clients/enable-cody-enterprise.mdx b/docs/cody/clients/enable-cody-enterprise.mdx index f82da0ed7..146056fb9 100644 --- a/docs/cody/clients/enable-cody-enterprise.mdx +++ b/docs/cody/clients/enable-cody-enterprise.mdx @@ -10,8 +10,8 @@ You can set up Cody for your Enterprise instance by two methods: -1. [Sourcegraph Cloud](#cody-on-sourcegraph-cloud) -2. [Self-hosted Sourcegraph](#cody-on-self-hosted-sourcegraph-enterprise) +1. Sourcegraph Cloud +2. Self-hosted Sourcegraph ## Cody on Sourcegraph Cloud diff --git a/docs/cody/enterprise/model-config-examples.mdx b/docs/cody/enterprise/model-config-examples.mdx index 270dc34cf..e49e58c51 100644 --- a/docs/cody/enterprise/model-config-examples.mdx +++ b/docs/cody/enterprise/model-config-examples.mdx @@ -2,9 +2,9 @@

This section includes examples about how to configure Cody to use Sourcegraph-provided models with `modelConfiguration`. These examples will use the following:

-- [Minimal configuration](/cody/model-configuration#configure-sourcegraph-supplied-models) -- [Using model filters](/cody/model-configuration#model-filters) -- [Change default models](/cody/model-configuration#default-models) +- [Minimal configuration](/cody/enterprise/model-configuration#configure-sourcegraph-provided-models) +- [Using model filters](/cody/enterprise/model-configuration#model-filters) +- [Change default models](/cody/enterprise/model-configuration#default-models) ## Sourcegraph-provided models and BYOK (Bring Your Own Key) @@ -69,7 +69,7 @@ In the configuration above, we: Below are configuration examples for setting up various LLM providers using BYOK. These examples are applicable whether or not you are using Sourcegraph-supported models. - In this section, all configuration examples have Sourcegraph-provided models disabled. Please refer to the previous section to use a combination of Sourcegraph-provided models and BYOK. -- Ensure that at least one model is available for each Cody feature ("chat" and "autocomplete"), regardless of the provider and model overrides configured. To verify this, [view the configuration](/cody/model-configuration#view-configuration) and confirm that appropriate models are listed in the `"defaultModels"` section. +- Ensure that at least one model is available for each Cody feature ("chat" and "autocomplete"), regardless of the provider and model overrides configured. To verify this, [view the configuration](/cody/enterprise/model-configuration#view-configuration) and confirm that appropriate models are listed in the `"defaultModels"` section. @@ -586,7 +586,7 @@ Provider override `serverSideConfig` fields: | `endpoint` | For pay-as-you-go, set it to an AWS region code (e.g., `us-west-2`) when using a public Amazon Bedrock endpoint. For provisioned throughput, set it to the provisioned VPC endpoint for the bedrock-runtime API (e.g., `https://vpce-0a10b2345cd67e89f-abc0defg.bedrock-runtime.us-west-2.vpce.amazonaws.com`). | | `region` | The region to use when configuring API clients. This is necessary because the 'frontend' binary container cannot access environment variables from the host OS. | -Provisioned throughput for AWS Bedrock models can be configured using the `"awsBedrockProvisionedThroughput"` server-side configuration type. Refer to the [Model Overrides](/cody/model-configuration#model-overrides) section for more details. +Provisioned throughput for AWS Bedrock models can be configured using the `"awsBedrockProvisionedThroughput"` server-side configuration type. Refer to the [Model Overrides](/cody/enterprise/model-configuration#model-overrides) section for more details. @@ -594,7 +594,7 @@ Provisioned throughput for AWS Bedrock models can be configured using the `"awsB Configuring self-hosted models can be nuanced and complex. Please reach out to Sourcegraph support for assistance with the setup. -Please refer to the [self-hosted models](/cody/model-configuration#self-hosted-models) documentation for detailed information about configuring self-hosted models. +Please refer to the [self-hosted models](/cody/enterprise/model-configuration#self-hosted-models) documentation for detailed information about configuring self-hosted models. Example configuration: diff --git a/docs/cody/enterprise/model-configuration.mdx b/docs/cody/enterprise/model-configuration.mdx index 9324c2d14..9a431e81d 100644 --- a/docs/cody/enterprise/model-configuration.mdx +++ b/docs/cody/enterprise/model-configuration.mdx @@ -16,17 +16,17 @@ The model configuration for Cody is managed through the `"modelConfiguration"` f | **Field** | **Description** | | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| [`sourcegraph`](/cody/model-configuration#sourcegraph-supplied-models) | Configures access to Sourcegraph-provided models available through Cody Gateway. | -| [`providerOverrides`](/cody/model-configuration#provider-overrides) | Configures model providers, allowing you to customize Cody's connection to model providers using your own API keys or self-hosted models. | -| [`modelOverrides`](/cody/model-configuration#model-overrides) | Extends or modifies the list of models Cody recognizes and their configurations. | -| [`selfHostedModels`](/cody/model-configuration#self-hosted-models) | Adds models to Cody’s recognized models list with default configurations provided by Sourcegraph. Only available for certain models; general models can be configured in `modelOverrides`. | -| [`defaultModels`](/cody/model-configuration#default-models) | Specifies the models assigned to each Cody feature (chat, fast chat, autocomplete). | +| [`sourcegraph`](/cody/enterprise/model-configuration#sourcegraph-provided-models) | Configures access to Sourcegraph-provided models available through Cody Gateway. | +| [`providerOverrides`](/cody/enterprise/model-configuration#provider-overrides) | Configures model providers, allowing you to customize Cody's connection to model providers using your own API keys or self-hosted models. | +| [`modelOverrides`](/cody/enterprise/model-configuration#model-overrides) | Extends or modifies the list of models Cody recognizes and their configurations. | +| [`selfHostedModels`](/cody/enterprise/model-configuration#self-hosted-models) | Adds models to Cody’s recognized models list with default configurations provided by Sourcegraph. Only available for certain models; general models can be configured in `modelOverrides`. | +| [`defaultModels`](/cody/enterprise/model-configuration#default-models) | Specifies the models assigned to each Cody feature (chat, fast chat, autocomplete). | ## Getting started with `modelConfiguration` The recommended way to set up model configuration is using Sourcegraph-provided models through the Cody Gateway. -For a minimal configuration example, see [Configure Sourcegraph-supplied models](/cody/model-configuration#configure-sourcegraph-supplied-models). +For a minimal configuration example, see [Configure Sourcegraph-supplied models](/cody/enterprise/model-configuration#configure-sourcegraph-provided-models). ## Sourcegraph-provided models @@ -79,7 +79,7 @@ There are three main settings for configuring Sourcegraph-provided LLM models: | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | `endpoint` | (Optional) The URL for connecting to Cody Gateway. The default is set to the production instance. | | `accessToken` | (Optional) The access token for connecting to Cody Gateway defaults to the current license key. | -| [`modelFilters`](/cody/model-configuration#model-filters) | (Optional) Filters specifying which models to include from Cody Gateway. | +| [`modelFilters`](/cody/enterprise/model-configuration#model-filters) | (Optional) Filters specifying which models to include from Cody Gateway. | ### Model Filters @@ -166,7 +166,7 @@ In the example above: - Sourcegraph-provided models are enabled (`sourcegraph` is not set to `null`) - The `"openai"` provider configuration is overridden to be accessed using your own key, with the provider API accessed directly via the specified `endpoint` URL. In contrast, models from the `"anthropic"` provider are accessed through the Cody Gateway -Refer to the [examples page](/cody/model-configuration/examples) for additional examples. +Refer to the [examples page](/cody/enterprise/model-config-examples) for additional examples. ### Server-side config @@ -176,8 +176,8 @@ Sourcegraph natively supports several types of LLM API providers. The current se | **Provider type** | **Description** | | -------------------- | ----------------------------------------------------------------------------------------------------------- | -| `"sourcegraph"` | [Cody Gateway](/cody/core-concepts/cody-gateway), offering access to various models from multiple services. | -| `"openaicompatible"` | Any OpenAI-compatible API implementation. | +| `"sourcegraph"` | [Cody Gateway](/cody/core-concepts/cody-gateway), offering access to various models from multiple services | +| `"openaicompatible"` | Any OpenAI-compatible API implementation | | `"awsBedrock"` | [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/) | | `"azureOpenAI"` | [Microsoft Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service/) | | `"anthropic"` | [Anthropic](https://www.anthropic.com) | @@ -188,7 +188,7 @@ Sourcegraph natively supports several types of LLM API providers. The current se The configuration for `serverSideConfig` varies by provider type. -Refer to the [examples page](/cody/model-configuration/examples) for examples. +Refer to the [examples page](/cody/enterprise/model-config-examples) for examples. ## Model Overrides @@ -284,7 +284,7 @@ In the example above: - Sourcegraph-provided models are used for `"chat"` and `"fastChat"` (accessed via Cody Gateway) - The newly configured model, `"huggingface-codellama::v1::CodeLlama-7b-hf"`, is used for `"autocomplete"` (connecting directly to Hugging Face’s OpenAI-compatible API) -Refer to the [examples page](/cody/model-configuration/examples) for additional examples. +Refer to the [examples page](/cody/enterprise/model-config-examples) for additional examples. ## Self-hosted models @@ -305,7 +305,7 @@ This field is an array of items, each with the following fields: | `apiVersion` | (Optional) The API version to use. Defaults to `"v1"` if not specified. | | `model` | (Optional) Specifies which default model configuration to use. Sourcegraph provides default configurations for select models. For arbitrary models, use the `modelOverrides` section. Available options include: `"starcoder2-7b@v1"`, `"starcoder2-15b@v1"`, `"mistral-7b-instruct@v1"`, `"mixtral-8x7b-instruct@v1"`, `"mixtral-8x22b-instruct@v1"` | -Refer to the [examples page](/cody/model-configuration/examples#self-hosted-models) for examples. +Refer to the [examples page](/cody/enterprise/model-config-examples#self-hosted-models) for examples. ## Default models From b6669b3fe591525b9c7d61231180a7d8028d7bf1 Mon Sep 17 00:00:00 2001 From: Maedah Batool Date: Thu, 21 Nov 2024 17:03:41 -0800 Subject: [PATCH 27/58] Add note for missing code --- docs/cody/enterprise/model-config-examples.mdx | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/cody/enterprise/model-config-examples.mdx b/docs/cody/enterprise/model-config-examples.mdx index e49e58c51..3a9f5bef8 100644 --- a/docs/cody/enterprise/model-config-examples.mdx +++ b/docs/cody/enterprise/model-config-examples.mdx @@ -54,7 +54,9 @@ In the configuration above: If you want to override the provider config for some models in the namespace and use the Sourcegraph-configured provider config for the rest, you can route requests directly to the LLM provider (bypassing the Cody Gateway) for some models while using the Sourcegraph-configured provider config for the rest. -Example configuration: +Example configuration + +{/*TODO: Missing code?*/} In the configuration above, we: From af56933b1beb98ae5969ce8c715f40eb3fbc07a6 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Fri, 22 Nov 2024 15:58:38 +0200 Subject: [PATCH 28/58] add missing example --- .../cody/enterprise/model-config-examples.mdx | 39 ++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/docs/cody/enterprise/model-config-examples.mdx b/docs/cody/enterprise/model-config-examples.mdx index 3a9f5bef8..2134f998d 100644 --- a/docs/cody/enterprise/model-config-examples.mdx +++ b/docs/cody/enterprise/model-config-examples.mdx @@ -56,7 +56,44 @@ If you want to override the provider config for some models in the namespace and Example configuration -{/*TODO: Missing code?*/} +```json +{ +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": {}, + "providerOverrides": [ + { + "id": "anthropic-byok", + "displayName": "Anthropic BYOK", + "serverSideConfig": { + "type": "anthropic", + "accessToken": "token", + "endpoint": "https://api.anthropic.com/v1/messages" + } + } + ], + "modelOverrides": [ + { + "modelRef": "anthropic-byok::2023-06-01::claude-3.5-sonnet", + "displayName": "Claude 3.5 Sonnet", + "modelName": "claude-3-5-sonnet-latest", + "capabilities": ["edit", "chat"], + "category": "accuracy", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + }, + ], + "defaultModels": { + "chat": "anthropic-byok::2023-06-01::claude-3.5-sonnet", + "fastChat": "anthropic::2023-06-01::claude-3-haiku", + "autocomplete": "fireworks::v1::deepseek-coder-v2-lite-base" + } +} +``` In the configuration above, we: From 91da74b31e0469a38bd4d451d3b32c5a78493eef Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Fri, 22 Nov 2024 23:39:18 +0200 Subject: [PATCH 29/58] add note on azure openai models --- .../cody/enterprise/model-config-examples.mdx | 136 +++++++++--------- 1 file changed, 66 insertions(+), 70 deletions(-) diff --git a/docs/cody/enterprise/model-config-examples.mdx b/docs/cody/enterprise/model-config-examples.mdx index 2134f998d..6658f6be7 100644 --- a/docs/cody/enterprise/model-config-examples.mdx +++ b/docs/cody/enterprise/model-config-examples.mdx @@ -1,20 +1,24 @@ # Model Configuration Examples -

This section includes examples about how to configure Cody to use Sourcegraph-provided models with `modelConfiguration`. These examples will use the following:

+

+ This section includes examples about how to configure Cody to use + Sourcegraph-provided models with `modelConfiguration`. These examples will + use the following: +

-- [Minimal configuration](/cody/enterprise/model-configuration#configure-sourcegraph-provided-models) -- [Using model filters](/cody/enterprise/model-configuration#model-filters) -- [Change default models](/cody/enterprise/model-configuration#default-models) +- [Minimal configuration](/cody/enterprise/model-configuration#configure-sourcegraph-provided-models) +- [Using model filters](/cody/enterprise/model-configuration#model-filters) +- [Change default models](/cody/enterprise/model-configuration#default-models) ## Sourcegraph-provided models and BYOK (Bring Your Own Key) By default, Sourcegraph is fully aware of several models from the following providers: -- "anthropic" -- "google" -- "fireworks" -- "mistral" -- "openai" +- "anthropic" +- "google" +- "fireworks" +- "mistral" +- "openai" ### Override configuration of a model provider @@ -46,9 +50,9 @@ Instead of Sourcegraph using its own servers to make LLM requests, it is possibl In the configuration above: -- Enable Sourcegraph-provided models and do not set any overrides (note that `"modelConfiguration.modelOverrides"` is not specified) -- Route requests for Anthropic models directly to the Anthropic API (via the provider override specified for "anthropic") -- Route requests for other models (such as the Fireworks model for "autocomplete") through Cody Gateway +- Enable Sourcegraph-provided models and do not set any overrides (note that `"modelConfiguration.modelOverrides"` is not specified) +- Route requests for Anthropic models directly to the Anthropic API (via the provider override specified for "anthropic") +- Route requests for other models (such as the Fireworks model for "autocomplete") through Cody Gateway ### Partially override provider config in the namespace @@ -80,7 +84,6 @@ Example configuration "capabilities": ["edit", "chat"], "category": "accuracy", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 4000 @@ -97,18 +100,18 @@ Example configuration In the configuration above, we: -- Enable Sourcegraph-supplied models (the `sourcegraph` field is not empty or `null`) -- Define a new provider with the ID `"anthropic-byok"` and configure it to use the Anthropic API -- Since this provider is unknown to Sourcegraph, no Sourcegraph-supplied models are available. Therefore, we add a custom model in the `"modelOverrides"` section -- Use the custom model configured in the previous step (`"anthropic-byok::2024-10-22::claude-3.5-sonnet"`) for `"chat"`. Requests are sent directly to the Anthropic API as set in the provider override -- For `"fastChat"` and `"autocomplete"`, we use Sourcegraph-provided models via Cody Gateway +- Enable Sourcegraph-supplied models (the `sourcegraph` field is not empty or `null`) +- Define a new provider with the ID `"anthropic-byok"` and configure it to use the Anthropic API +- Since this provider is unknown to Sourcegraph, no Sourcegraph-supplied models are available. Therefore, we add a custom model in the `"modelOverrides"` section +- Use the custom model configured in the previous step (`"anthropic-byok::2024-10-22::claude-3.5-sonnet"`) for `"chat"`. Requests are sent directly to the Anthropic API as set in the provider override +- For `"fastChat"` and `"autocomplete"`, we use Sourcegraph-provided models via Cody Gateway ## Config examples for various LLM providers Below are configuration examples for setting up various LLM providers using BYOK. These examples are applicable whether or not you are using Sourcegraph-supported models. -- In this section, all configuration examples have Sourcegraph-provided models disabled. Please refer to the previous section to use a combination of Sourcegraph-provided models and BYOK. -- Ensure that at least one model is available for each Cody feature ("chat" and "autocomplete"), regardless of the provider and model overrides configured. To verify this, [view the configuration](/cody/enterprise/model-configuration#view-configuration) and confirm that appropriate models are listed in the `"defaultModels"` section. +- In this section, all configuration examples have Sourcegraph-provided models disabled. Please refer to the previous section to use a combination of Sourcegraph-provided models and BYOK. +- Ensure that at least one model is available for each Cody feature ("chat" and "autocomplete"), regardless of the provider and model overrides configured. To verify this, [view the configuration](/cody/enterprise/model-configuration#view-configuration) and confirm that appropriate models are listed in the `"defaultModels"` section. @@ -136,7 +139,6 @@ Below are configuration examples for setting up various LLM providers using BYOK "capabilities": ["chat"], "category": "accuracy", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 4000 @@ -149,20 +151,18 @@ Below are configuration examples for setting up various LLM providers using BYOK "capabilities": ["chat"], "category": "speed", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 7000, "maxOutputTokens": 4000 } }, { - "modelRef": "anthropic::2023-01-01::claude-instant-1.2", - "displayName": "Claude Instant", - "modelName": "claude-instant-1.2", - "capabilities": ["autocomplete", "chat"], - "category": "other", - "status": "deprecated", - "tier": "free", + "modelRef": "anthropic::2023-06-01::claude-3-haiku", + "displayName": "Claude 3 Haiku", + "modelName": "claude-3-haiku-20240307", + "capabilities": ["edit", "chat"], + "category": "speed", + "status": "stable", "contextWindow": { "maxInputTokens": 7000, "maxOutputTokens": 4000 @@ -172,18 +172,18 @@ Below are configuration examples for setting up various LLM providers using BYOK "defaultModels": { "chat": "anthropic::2024-10-22::claude-3.5-sonnet", "fastChat": "anthropic::2023-06-01::claude-3-haiku", - "autocomplete": "anthropic::2023-01-01::claude-instant-1.2" + "autocomplete": "fireworks::v1::deepseek-coder-v2-lite-base" } } ``` In the configuration above, -- Set up a provider override for Anthropic, routing requests for this provider directly to the specified Anthropic endpoint (bypassing Cody Gateway) -- Add three Anthropic models: - - Two models with chat capabilities (`"anthropic::2024-10-22::claude-3.5-sonnet"` and `"anthropic::2023-06-01::claude-3-haiku"`), providing options for chat users - - One model with autocomplete capability (`"anthropic::2023-01-01::claude-instant-1.2"`) -- Set the configured models as default models for Cody features in the `"defaultModels"` field +- Set up a provider override for Anthropic, routing requests for this provider directly to the specified Anthropic endpoint (bypassing Cody Gateway) +- Add three Anthropic models: + - Two models with chat capabilities (`"anthropic::2024-10-22::claude-3.5-sonnet"` and `"anthropic::2023-06-01::claude-3-haiku"`), providing options for chat users + - One model with autocomplete capability (`"fireworks::v1::deepseek-coder-v2-lite-base"`) +- Set the configured models as default models for Cody features in the `"defaultModels"` field @@ -211,7 +211,6 @@ In the configuration above, "capabilities": ["chat"], "category": "other", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 7000, "maxOutputTokens": 4000 @@ -240,10 +239,10 @@ In the configuration above, In the configuration above, -- Set up a provider override for Fireworks, routing requests for this provider directly to the specified Fireworks endpoint (bypassing Cody Gateway) -- Add two Fireworks models: - - `"fireworks::v1::mixtral-8x22b-instruct"` with "chat" capabiity - used for "chat" and "fastChat" - - `"fireworks::v1::starcoder-16b"` with "autocomplete" capability - used for "autocomplete" +- Set up a provider override for Fireworks, routing requests for this provider directly to the specified Fireworks endpoint (bypassing Cody Gateway) +- Add two Fireworks models: + - `"fireworks::v1::mixtral-8x22b-instruct"` with "chat" capabiity - used for "chat" and "fastChat" + - `"fireworks::v1::starcoder-16b"` with "autocomplete" capability - used for "autocomplete" @@ -271,7 +270,6 @@ In the configuration above, "capabilities": ["chat"], "category": "accuracy", "status": "stable", - "tier": "pro", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 400 @@ -284,7 +282,6 @@ In the configuration above, "capabilities": ["autocomplete"], "category": "speed", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 7000, "maxOutputTokens": 400 @@ -301,10 +298,10 @@ In the configuration above, In the configuration above, -- Set up a provider override for OpenAI, routing requests for this provider directly to the specified OpenAI endpoint (bypassing Cody Gateway) -- Add two OpenAI models: - - `"openai::2024-02-01::gpt-4o"` with "chat" capabilities - used for "chat" and "fastChat" - - `"openai::unknown::gpt-3.5-turbo-instruct"` with "autocomplete" capability - used for "autocomplete" +- Set up a provider override for OpenAI, routing requests for this provider directly to the specified OpenAI endpoint (bypassing Cody Gateway) +- Add two OpenAI models: + - `"openai::2024-02-01::gpt-4o"` with "chat" capabilities - used for "chat" and "fastChat" + - `"openai::unknown::gpt-3.5-turbo-instruct"` with "autocomplete" capability - used for "autocomplete" @@ -335,7 +332,6 @@ In the configuration above, "capabilities": ["chat"], "category": "accuracy", "status": "stable", - "tier": "pro", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 4000 @@ -348,7 +344,6 @@ In the configuration above, "capabilities": ["autocomplete"], "category": "speed", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 7000, "maxOutputTokens": 4000 @@ -363,13 +358,14 @@ In the configuration above, } ``` +In the configuration above, In the configuration above, -- Set up a provider override for Azure OpenAI, routing requests for this provider directly to the specified Azure OpenAI endpoint (bypassing Cody Gateway) +- Set up a provider override for Azure OpenAI, routing requests for this provider directly to the specified Azure OpenAI endpoint (bypassing Cody Gateway). **Note:** For Azure OpenAI, ensure that the `modelName` matches the name defined in your Azure portal configuration for the model. - Add two OpenAI models: - - `"azure-openai::unknown::gpt-4o"` with "chat" capability - used for "chat" and "fastChat" - - `"azure-openai::unknown::gpt-35-turbo-instruct-test"` with "autocomplete" capability - used for "autocomplete" -- Since `"azure-openai::unknown::gpt-35-turbo-instruct-test"` is not supported on the newer OpenAI `"v1/chat/completions"` endpoint, we set `"useDeprecatedCompletionsAPI"` to `true` to route requests to the legacy `"v1/completions"` endpoint. This setting is unnecessary if you are using a model supported on the `"v1/chat/completions"` endpoint + - `"azure-openai::unknown::gpt-4o"` with "chat" capability - used for "chat" and "fastChat" + - `"azure-openai::unknown::gpt-35-turbo-instruct-test"` with "autocomplete" capability - used for "autocomplete" +- Since `"azure-openai::unknown::gpt-35-turbo-instruct-test"` is not supported on the newer OpenAI `"v1/chat/completions"` endpoint, we set `"useDeprecatedCompletionsAPI"` to `true` to route requests to the legacy `"v1/completions"` endpoint. This setting is unnecessary if you are using a model supported on the `"v1/chat/completions"` endpoint. @@ -457,10 +453,10 @@ In the configuration above, In the configuration above, -- Configure two OpenAI-compatible providers: `"fireworks"` and `"huggingface-codellama"` -- Add two OpenAI-compatible models: `"fireworks::v1::llama-v3p1-70b-instruct"` and `"huggingface-codellama::v1::CodeLlama-7b-hf"`. Additionally: - - Set `clientSideConfig.openaicompatible` to `{}` to indicate to Cody clients that these models are OpenAI-compatible, ensuring the appropriate code paths are utilized - - Designate these models as the default choices for chat and autocomplete, respectively +- Configure two OpenAI-compatible providers: `"fireworks"` and `"huggingface-codellama"` +- Add two OpenAI-compatible models: `"fireworks::v1::llama-v3p1-70b-instruct"` and `"huggingface-codellama::v1::CodeLlama-7b-hf"`. Additionally: + - Set `clientSideConfig.openaicompatible` to `{}` to indicate to Cody clients that these models are OpenAI-compatible, ensuring the appropriate code paths are utilized + - Designate these models as the default choices for chat and autocomplete, respectively @@ -500,7 +496,6 @@ In the configuration above, "capabilities": ["autocomplete", "chat"], "category": "speed", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 7000, "maxOutputTokens": 4000 @@ -517,10 +512,10 @@ In the configuration above, In the configuration above, -- Set up a provider override for Google Anthropic, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway) -- Add two Anthropic models: - - `"google::unknown::claude-3-5-sonnet"` with "chat" capabiity - used for "chat" and "fastChat" - - `"google::unknown::claude-3-haiku"` with "autocomplete" capability - used for "autocomplete" +- Set up a provider override for Google Anthropic, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway) +- Add two Anthropic models: + - `"google::unknown::claude-3-5-sonnet"` with "chat" capabiity - used for "chat" and "fastChat" + - `"google::unknown::claude-3-haiku"` with "autocomplete" capability - used for "autocomplete" @@ -548,7 +543,6 @@ In the configuration above, "capabilities": ["chat", "autocomplete"], "category": "balanced", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 400 @@ -565,8 +559,8 @@ In the configuration above, In the configuration above, -- Set up a provider override for Google Gemini, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway) -- Add the `"google::v1::gemini-1.5-pro"` model, which is used for all Cody features. We do not add other models for simplicity, as adding multiple models is already covered in the examples above +- Set up a provider override for Google Gemini, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway) +- Add the `"google::v1::gemini-1.5-pro"` model, which is used for all Cody features. We do not add other models for simplicity, as adding multiple models is already covered in the examples above @@ -596,7 +590,6 @@ In the configuration above, "capabilities": ["chat", "autocomplete"], "category": "other", "status": "stable", - "tier": "pro", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 4000 @@ -613,17 +606,17 @@ In the configuration above, In the configuration described above, -- Set up a provider override for AWS Bedrock, routing requests for this provider directly to the specified endpoint, bypassing Cody Gateway -- Add the `"aws-bedrock::2023-06-01::claude-3-opus"` model, which is used for all Cody features. We do not add other models for simplicity, as adding multiple models is already covered in the examples above +- Set up a provider override for AWS Bedrock, routing requests for this provider directly to the specified endpoint, bypassing Cody Gateway +- Add the `"aws-bedrock::2023-06-01::claude-3-opus"` model, which is used for all Cody features. We do not add other models for simplicity, as adding multiple models is already covered in the examples above Provider override `serverSideConfig` fields: -| **Field** | **Description** | +| **Field** | **Description** | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | Must be `"awsBedrock"`. | | `accessToken` | Leave empty to rely on instance role bindings or other AWS configurations in the frontend service. Use `:` for direct credential configuration, or `::` if a session token is also required. | | `endpoint` | For pay-as-you-go, set it to an AWS region code (e.g., `us-west-2`) when using a public Amazon Bedrock endpoint. For provisioned throughput, set it to the provisioned VPC endpoint for the bedrock-runtime API (e.g., `https://vpce-0a10b2345cd67e89f-abc0defg.bedrock-runtime.us-west-2.vpce.amazonaws.com`). | -| `region` | The region to use when configuring API clients. This is necessary because the 'frontend' binary container cannot access environment variables from the host OS. | +| `region` | The region to use when configuring API clients. This is necessary because the 'frontend' binary container cannot access environment variables from the host OS. | Provisioned throughput for AWS Bedrock models can be configured using the `"awsBedrockProvisionedThroughput"` server-side configuration type. Refer to the [Model Overrides](/cody/enterprise/model-configuration#model-overrides) section for more details. @@ -631,7 +624,10 @@ Provisioned throughput for AWS Bedrock models can be configured using the `"awsB ## Self-hosted models -Configuring self-hosted models can be nuanced and complex. Please reach out to Sourcegraph support for assistance with the setup. + + Configuring self-hosted models can be nuanced and complex. Please reach out + to Sourcegraph support for assistance with the setup. + Please refer to the [self-hosted models](/cody/enterprise/model-configuration#self-hosted-models) documentation for detailed information about configuring self-hosted models. From e893b3c8588ee9531ba41c7bada5e1b4d0585df9 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Fri, 22 Nov 2024 23:39:49 +0200 Subject: [PATCH 30/58] fix formatting --- docs/cody/enterprise/model-config-examples.mdx | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/cody/enterprise/model-config-examples.mdx b/docs/cody/enterprise/model-config-examples.mdx index 6658f6be7..13c62a72d 100644 --- a/docs/cody/enterprise/model-config-examples.mdx +++ b/docs/cody/enterprise/model-config-examples.mdx @@ -361,11 +361,12 @@ In the configuration above, In the configuration above, In the configuration above, -- Set up a provider override for Azure OpenAI, routing requests for this provider directly to the specified Azure OpenAI endpoint (bypassing Cody Gateway). **Note:** For Azure OpenAI, ensure that the `modelName` matches the name defined in your Azure portal configuration for the model. -- Add two OpenAI models: - - `"azure-openai::unknown::gpt-4o"` with "chat" capability - used for "chat" and "fastChat" - - `"azure-openai::unknown::gpt-35-turbo-instruct-test"` with "autocomplete" capability - used for "autocomplete" -- Since `"azure-openai::unknown::gpt-35-turbo-instruct-test"` is not supported on the newer OpenAI `"v1/chat/completions"` endpoint, we set `"useDeprecatedCompletionsAPI"` to `true` to route requests to the legacy `"v1/completions"` endpoint. This setting is unnecessary if you are using a model supported on the `"v1/chat/completions"` endpoint. +- Set up a provider override for Azure OpenAI, routing requests for this provider directly to the specified Azure OpenAI endpoint (bypassing Cody Gateway). + **Note:** For Azure OpenAI, ensure that the `modelName` matches the name defined in your Azure portal configuration for the model. +- Add two OpenAI models: + - `"azure-openai::unknown::gpt-4o"` with "chat" capability - used for "chat" and "fastChat" + - `"azure-openai::unknown::gpt-35-turbo-instruct-test"` with "autocomplete" capability - used for "autocomplete" +- Since `"azure-openai::unknown::gpt-35-turbo-instruct-test"` is not supported on the newer OpenAI `"v1/chat/completions"` endpoint, we set `"useDeprecatedCompletionsAPI"` to `true` to route requests to the legacy `"v1/completions"` endpoint. This setting is unnecessary if you are using a model supported on the `"v1/chat/completions"` endpoint. From 097bfc23d8f9f8516b1f1158ed918e40f8d4c6de Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Fri, 22 Nov 2024 23:39:49 +0200 Subject: [PATCH 31/58] fix --- docs/cody/enterprise/model-config-examples.mdx | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/cody/enterprise/model-config-examples.mdx b/docs/cody/enterprise/model-config-examples.mdx index 6658f6be7..007546e3c 100644 --- a/docs/cody/enterprise/model-config-examples.mdx +++ b/docs/cody/enterprise/model-config-examples.mdx @@ -272,7 +272,7 @@ In the configuration above, "status": "stable", "contextWindow": { "maxInputTokens": 45000, - "maxOutputTokens": 400 + "maxOutputTokens": 4000 } }, { @@ -284,7 +284,7 @@ In the configuration above, "status": "stable", "contextWindow": { "maxInputTokens": 7000, - "maxOutputTokens": 400 + "maxOutputTokens": 4000 } } ], @@ -361,11 +361,12 @@ In the configuration above, In the configuration above, In the configuration above, -- Set up a provider override for Azure OpenAI, routing requests for this provider directly to the specified Azure OpenAI endpoint (bypassing Cody Gateway). **Note:** For Azure OpenAI, ensure that the `modelName` matches the name defined in your Azure portal configuration for the model. -- Add two OpenAI models: - - `"azure-openai::unknown::gpt-4o"` with "chat" capability - used for "chat" and "fastChat" - - `"azure-openai::unknown::gpt-35-turbo-instruct-test"` with "autocomplete" capability - used for "autocomplete" -- Since `"azure-openai::unknown::gpt-35-turbo-instruct-test"` is not supported on the newer OpenAI `"v1/chat/completions"` endpoint, we set `"useDeprecatedCompletionsAPI"` to `true` to route requests to the legacy `"v1/completions"` endpoint. This setting is unnecessary if you are using a model supported on the `"v1/chat/completions"` endpoint. +- Set up a provider override for Azure OpenAI, routing requests for this provider directly to the specified Azure OpenAI endpoint (bypassing Cody Gateway). + **Note:** For Azure OpenAI, ensure that the `modelName` matches the name defined in your Azure portal configuration for the model. +- Add two OpenAI models: + - `"azure-openai::unknown::gpt-4o"` with "chat" capability - used for "chat" and "fastChat" + - `"azure-openai::unknown::gpt-35-turbo-instruct-test"` with "autocomplete" capability - used for "autocomplete" +- Since `"azure-openai::unknown::gpt-35-turbo-instruct-test"` is not supported on the newer OpenAI `"v1/chat/completions"` endpoint, we set `"useDeprecatedCompletionsAPI"` to `true` to route requests to the legacy `"v1/completions"` endpoint. This setting is unnecessary if you are using a model supported on the `"v1/chat/completions"` endpoint. @@ -545,7 +546,7 @@ In the configuration above, "status": "stable", "contextWindow": { "maxInputTokens": 45000, - "maxOutputTokens": 400 + "maxOutputTokens": 4000 } } ], From 67cddad23e0f12b367ac8654c5c376e36904a973 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Fri, 22 Nov 2024 23:46:39 +0200 Subject: [PATCH 32/58] address feedback --- docs/cody/enterprise/model-config-examples.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cody/enterprise/model-config-examples.mdx b/docs/cody/enterprise/model-config-examples.mdx index 007546e3c..f3575210a 100644 --- a/docs/cody/enterprise/model-config-examples.mdx +++ b/docs/cody/enterprise/model-config-examples.mdx @@ -520,7 +520,7 @@ In the configuration above, - + ```json "modelConfiguration": { From 523ba07042690ca6f35ad3f2764064621766ba23 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Fri, 22 Nov 2024 23:48:19 +0200 Subject: [PATCH 33/58] fix --- docs/cody/enterprise/model-configuration.mdx | 151 ++++++++++--------- 1 file changed, 80 insertions(+), 71 deletions(-) diff --git a/docs/cody/enterprise/model-configuration.mdx b/docs/cody/enterprise/model-configuration.mdx index 9a431e81d..b6dfaeaed 100644 --- a/docs/cody/enterprise/model-configuration.mdx +++ b/docs/cody/enterprise/model-configuration.mdx @@ -1,8 +1,14 @@ # Model Configuration -

Learn how to configure Cody via `modelConfiguration` on a Sourcegraph Enterprise instance.

+

+ Learn how to configure Cody via `modelConfiguration` on a Sourcegraph + Enterprise instance. +

-`modelConfiguration` is the recommended way to configure chat and autocomplete models in Sourcegraph `v5.6.0` and later. + + `modelConfiguration` is the recommended way to configure chat and + autocomplete models in Sourcegraph `v5.6.0` and later. + The `modelConfiguration` field in the **Site config** section allows you to configure Cody to use different LLM models for chat and autocomplete, enabling greater flexibility in selecting the best model for your needs. @@ -14,11 +20,11 @@ The LLM models available for a Sourcegraph Enterprise instance offer a combinati The model configuration for Cody is managed through the `"modelConfiguration"` field in the **Site config** section. It includes the following fields: -| **Field** | **Description** | -| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **Field** | **Description** | +| --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [`sourcegraph`](/cody/enterprise/model-configuration#sourcegraph-provided-models) | Configures access to Sourcegraph-provided models available through Cody Gateway. | -| [`providerOverrides`](/cody/enterprise/model-configuration#provider-overrides) | Configures model providers, allowing you to customize Cody's connection to model providers using your own API keys or self-hosted models. | -| [`modelOverrides`](/cody/enterprise/model-configuration#model-overrides) | Extends or modifies the list of models Cody recognizes and their configurations. | +| [`providerOverrides`](/cody/enterprise/model-configuration#provider-overrides) | Configures model providers, allowing you to customize Cody's connection to model providers using your own API keys or self-hosted models. | +| [`modelOverrides`](/cody/enterprise/model-configuration#model-overrides) | Extends or modifies the list of models Cody recognizes and their configurations. | | [`selfHostedModels`](/cody/enterprise/model-configuration#self-hosted-models) | Adds models to Cody’s recognized models list with default configurations provided by Sourcegraph. Only available for certain models; general models can be configured in `modelOverrides`. | | [`defaultModels`](/cody/enterprise/model-configuration#default-models) | Specifies the models assigned to each Cody feature (chat, fast chat, autocomplete). | @@ -69,27 +75,27 @@ The minimal configuration for Sourcegraph-provided models is: The above configuration sets up the following: -- Sourcegraph-provided models are enabled (`sourcegraph` is not set to `null`) -- Requests to LLM providers are routed through the Cody Gateway (no `providerOverrides` field specified) -- Sourcegraph-defined default models are used for Cody features (no `defaultModels` field specified) +- Sourcegraph-provided models are enabled (`sourcegraph` is not set to `null`) +- Requests to LLM providers are routed through the Cody Gateway (no `providerOverrides` field specified) +- Sourcegraph-defined default models are used for Cody features (no `defaultModels` field specified) There are three main settings for configuring Sourcegraph-provided LLM models: -| **Field** | **Description** | -| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | -| `endpoint` | (Optional) The URL for connecting to Cody Gateway. The default is set to the production instance. | -| `accessToken` | (Optional) The access token for connecting to Cody Gateway defaults to the current license key. | -| [`modelFilters`](/cody/enterprise/model-configuration#model-filters) | (Optional) Filters specifying which models to include from Cody Gateway. | +| **Field** | **Description** | +| -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | +| `endpoint` | (Optional) The URL for connecting to Cody Gateway. The default is set to the production instance. | +| `accessToken` | (Optional) The access token for connecting to Cody Gateway defaults to the current license key. | +| [`modelFilters`](/cody/enterprise/model-configuration#model-filters) | (Optional) Filters specifying which models to include from Cody Gateway. | ### Model Filters The `"modelFilters"` section allows you to control which Cody Gateway models are available to users of your Sourcegraph Enterprise instance. The following table describes each field: -| **Field** | **Description** | -| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Field** | **Description** | +| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `statusFilter` | Filters models based on their release status, such as stable, beta, deprecated or experimental. By default, all models available on Cody Gateway are accessible. | -| `allow` | An array of `modelRef`s specifying which models to include. Supports wildcards. | -| `deny` | An array of `modelRef`s specifying which models to exclude. Supports wildcards. | +| `allow` | An array of `modelRef`s specifying which models to include. Supports wildcards. | +| `deny` | An array of `modelRef`s specifying which models to exclude. Supports wildcards. | The following examples demonstrate how to use each of these settings together: @@ -130,7 +136,7 @@ By defining a provider override in your Sourcegraph site configuration, you can Provider overrides are configured via the `"modelConfiguration.providerOverrides"` field in the site configuration. This field is an array of items, each containing the following fields: -| **Field** | **Description** | +| **Field** | **Description** | | ------------------ | --------------------------------------------------------------------------------- | | `id` | The namespace for models accessed via the provider. | | `displayName` | A human-readable name for the provider. | @@ -154,7 +160,7 @@ Example configuration: }, ], "defaultModels": { - "chat": "openai::v1::google::v1::gemini-1.5-pro", + "chat": "google::v1::gemini-1.5-pro", "fastChat": "anthropic::2023-06-01::claude-3-haiku", "autocomplete": "fireworks::v1::deepseek-coder-v2-lite-base" } @@ -163,8 +169,8 @@ Example configuration: In the example above: -- Sourcegraph-provided models are enabled (`sourcegraph` is not set to `null`) -- The `"openai"` provider configuration is overridden to be accessed using your own key, with the provider API accessed directly via the specified `endpoint` URL. In contrast, models from the `"anthropic"` provider are accessed through the Cody Gateway +- Sourcegraph-provided models are enabled (`sourcegraph` is not set to `null`) +- The `"openai"` provider configuration is overridden to be accessed using your own key, with the provider API accessed directly via the specified `endpoint` URL. In contrast, models from the `"anthropic"` provider are accessed through the Cody Gateway Refer to the [examples page](/cody/enterprise/model-config-examples) for additional examples. @@ -174,17 +180,17 @@ The most important part of a provider's configuration is the `"serverSideConfig" Sourcegraph natively supports several types of LLM API providers. The current set of supported providers includes: -| **Provider type** | **Description** | -| -------------------- | ----------------------------------------------------------------------------------------------------------- | +| **Provider type** | **Description** | +| -------------------- | ---------------------------------------------------------------------------------------------------------- | | `"sourcegraph"` | [Cody Gateway](/cody/core-concepts/cody-gateway), offering access to various models from multiple services | | `"openaicompatible"` | Any OpenAI-compatible API implementation | -| `"awsBedrock"` | [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/) | -| `"azureOpenAI"` | [Microsoft Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service/) | -| `"anthropic"` | [Anthropic](https://www.anthropic.com) | -| `"fireworks"` | [Fireworks AI](https://fireworks.ai) | -| `"google"` | [Google Gemini](http://cloud.google.com/gemini) and [Vertex](https://cloud.google.com/vertex-ai/) | -| `"openai"` | [OpenAI](http://platform.openai.com) | -| `"huggingface-tgi"` | [Hugging Face Text Generation Interface](https://huggingface.co/docs/text-generation-inference/en/index) | +| `"awsBedrock"` | [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/) | +| `"azureOpenAI"` | [Microsoft Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service/) | +| `"anthropic"` | [Anthropic](https://www.anthropic.com) | +| `"fireworks"` | [Fireworks AI](https://fireworks.ai) | +| `"google"` | [Google Gemini](http://cloud.google.com/gemini) and [Vertex](https://cloud.google.com/vertex-ai/) | +| `"openai"` | [OpenAI](http://platform.openai.com) | +| `"huggingface-tgi"` | [Hugging Face Text Generation Interface](https://huggingface.co/docs/text-generation-inference/en/index) | The configuration for `serverSideConfig` varies by provider type. @@ -196,34 +202,35 @@ With a provider defined (either a Sourcegraph-provided provider or a custom prov This field is an array of items, each with the following fields: -- `modelRef`: Uniquely identifies the model within the provider namespace - - A string in the format `${providerId}::${apiVersionId}::${modelId}` - - To associate a model with your provider, `${providerId}` must match the provider’s ID - - `${modelId}` can be any URL-safe string - - `${apiVersionId}` specifies the API version, which helps detect compatibility issues between models and Sourcegraph instances. For example, `"2023-06-01"` can indicate that the model uses that version of the Anthropic API. If unsure, you may set this to `"unknown"` when defining custom models -- `displayName`: An optional, user-friendly name for the model. If not set, clients should display the `ModelID` part of the `modelRef` instead (not the `modelName`) -- `modelName`: A unique identifier the API provider uses to specify which model is being invoked. This is the identifier that the LLM provider recognizes to determine the model you are calling -- `capabilities`: A list of capabilities that the model supports. Supported values: **autocomplete** and **chat** -- `category`: Specifies the model's category with the following options: - - `"balanced"`: Typically the best default choice for most users. This category is suited for models like Sonnet 3.5 (as of October 2024) - - `"speed"`: Ideal for low-parameter models that may not suit general-purpose chat but are beneficial for specialized tasks, such as query rewriting - - `"accuracy"`: Reserved for models, like OpenAI o1, that use advanced reasoning techniques to improve response accuracy, though with slower latency - - `"other"`: Used for older models without distinct advantages in reasoning or speed. Select this category if you are uncertain about which category to choose - - `"deprecated"`: For models that are no longer supported by the provider and are filtered out on the client side (not available for use) -- `contextWindow`: An object that defines the **number of tokens** (units of text) that can be sent to the LLM. This setting influences response time and request cost and may vary according to the limits set by each LLM model or provider. It includes two fields: - - `maxInputTokens`: Specifies the maximum number of tokens for the contextual data in the prompt (e.g., question, relevant snippets) - - `maxOutputTokens`: Specifies the maximum number of tokens allowed in the response -- `serverSideConfig`: Additional configuration for the model. It can be one of the following: - - `awsBedrockProvisionedThroughput`: Specifies provisioned throughput settings for AWS Bedrock models with the following fields: - - `type`: Must be `"awsBedrockProvisionedThroughput"` - - `arn`: The ARN (Amazon Resource Name) for provisioned throughput to use when sending requests to AWS Bedrock -- `openaicompatible`: Configuration specific to models provided by an OpenAI-compatible provider, with the following fields: - - `type`: Must be `"openaicompatible"` - - `apiModel`: The literal string value of the `model` field to be sent to the `/chat/completions` API. If set, Sourcegraph treats this as an opaque string and sends it directly to the API without inferring any additional information. By default, the configured model name is sent - -- `clientSideConfig`: Advanced configuration options that are only respected if the model is provided by the `"openaicompatible"` provider - -{/*TODO: add more details.*/} +- `modelRef`: Uniquely identifies the model within the provider namespace + - A string in the format `${providerId}::${apiVersionId}::${modelId}` + - To associate a model with your provider, `${providerId}` must match the provider’s ID + - `${modelId}` can be any URL-safe string + - `${apiVersionId}` specifies the API version, which helps detect compatibility issues between models and Sourcegraph instances. For example, `"2023-06-01"` can indicate that the model uses that version of the Anthropic API. If unsure, you may set this to `"unknown"` when defining custom models +- `displayName`: An optional, user-friendly name for the model. If not set, clients should display the `ModelID` part of the `modelRef` instead (not the `modelName`) +- `modelName`: A unique identifier the API provider uses to specify which model is being invoked. This is the identifier that the LLM provider recognizes to determine the model you are calling +- `capabilities`: A list of capabilities that the model supports. Supported values: **autocomplete** and **chat** +- `category`: Specifies the model's category with the following options: + - `"balanced"`: Typically the best default choice for most users. This category is suited for models like Sonnet 3.5 (as of October 2024) + - `"speed"`: Ideal for low-parameter models that may not suit general-purpose chat but are beneficial for specialized tasks, such as query rewriting + - `"accuracy"`: Reserved for models, like OpenAI o1, that use advanced reasoning techniques to improve response accuracy, though with slower latency + - `"other"`: Used for older models without distinct advantages in reasoning or speed. Select this category if you are uncertain about which category to choose + - `"deprecated"`: For models that are no longer supported by the provider and are filtered out on the client side (not available for use) +- `contextWindow`: An object that defines the **number of tokens** (units of text) that can be sent to the LLM. This setting influences response time and request cost and may vary according to the limits set by each LLM model or provider. It includes two fields: + - `maxInputTokens`: Specifies the maximum number of tokens for the contextual data in the prompt (e.g., question, relevant snippets) + - `maxOutputTokens`: Specifies the maximum number of tokens allowed in the response +- `serverSideConfig`: Additional configuration for the model. It can be one of the following: + - `awsBedrockProvisionedThroughput`: Specifies provisioned throughput settings for AWS Bedrock models with the following fields: + - `type`: Must be `"awsBedrockProvisionedThroughput"` + - `arn`: The ARN (Amazon Resource Name) for provisioned throughput to use when sending requests to AWS Bedrock +- `openaicompatible`: Configuration specific to models provided by an OpenAI-compatible provider, with the following fields: + + - `type`: Must be `"openaicompatible"` + - `apiModel`: The literal string value of the `model` field to be sent to the `/chat/completions` API. If set, Sourcegraph treats this as an opaque string and sends it directly to the API without inferring any additional information. By default, the configured model name is sent + +- `clientSideConfig`: Advanced configuration options that are only respected if the model is provided by the `"openaicompatible"` provider + +{/* TODO: add more details. */} #### Example configuration @@ -268,7 +275,7 @@ This field is an array of items, each with the following fields: } ], "defaultModels": { - "chat": "openai::v1::google::v1::gemini-1.5-pro", + "chat": "google::v1::gemini-1.5-pro", "fastChat": "anthropic::2023-06-01::claude-3-haiku", "autocomplete": "huggingface-codellama::v1::CodeLlama-7b-hf" } @@ -277,19 +284,21 @@ This field is an array of items, each with the following fields: In the example above: -- Sourcegraph-provided models are enabled (`sourcegraph` is not set to `null`) -- An additional provider, `"huggingface-codellama"`, is configured to access Hugging Face’s OpenAI-compatible API directly -- A custom model, `"CodeLlama-7b-hf"`, is added using the `"huggingface-codellama"` provider -- Default models are set up as follows: - - Sourcegraph-provided models are used for `"chat"` and `"fastChat"` (accessed via Cody Gateway) - - The newly configured model, `"huggingface-codellama::v1::CodeLlama-7b-hf"`, is used for `"autocomplete"` (connecting directly to Hugging Face’s OpenAI-compatible API) +- Sourcegraph-provided models are enabled (`sourcegraph` is not set to `null`) +- An additional provider, `"huggingface-codellama"`, is configured to access Hugging Face’s OpenAI-compatible API directly +- A custom model, `"CodeLlama-7b-hf"`, is added using the `"huggingface-codellama"` provider +- Default models are set up as follows: + - Sourcegraph-provided models are used for `"chat"` and `"fastChat"` (accessed via Cody Gateway) + - The newly configured model, `"huggingface-codellama::v1::CodeLlama-7b-hf"`, is used for `"autocomplete"` (connecting directly to Hugging Face’s OpenAI-compatible API) Refer to the [examples page](/cody/enterprise/model-config-examples) for additional examples. ## Self-hosted models -Configuring self-hosted models can be nuanced and complex. Please reach - out to Sourcegraph support for assistance with the setup. + + Configuring self-hosted models can be nuanced and complex. Please reach out + to Sourcegraph support for assistance with the setup. + With a provider defined (either a Sourcegraph-provided provider or a custom provider configured via the `providerOverrides` field), self-hosted models can be added to Cody’s recognized models list by configuring the `"modelConfiguration.selfHostedModels"` section. These models use Sourcegraph’s default configuration settings and are only available for select models. @@ -299,7 +308,7 @@ Specifying the same model in both `"selfHostedModels"` and `"modelOverrides"` is This field is an array of items, each with the following fields: -| **Field** | **Description** | +| **Field** | **Description** | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `provider` | (Required) The provider ID of the model, configured in the `providerOverrides` field. | | `apiVersion` | (Optional) The API version to use. Defaults to `"v1"` if not specified. | @@ -342,8 +351,8 @@ curl --location "${INSTANCE_URL}/.api/modelconfig/supported-models.json" \ The response includes: -- Configured providers and models—both Sourcegraph-provided (if enabled, with any applied filters) and any overrides -- Default models for Cody features +- Configured providers and models—both Sourcegraph-provided (if enabled, with any applied filters) and any overrides +- Default models for Cody features #### Example response From 3b22db8af0599ebb774f6f3efa35b8ffe3da7dc6 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Fri, 22 Nov 2024 23:49:55 +0200 Subject: [PATCH 34/58] upd --- docs/cody/enterprise/model-configuration.mdx | 40 ++++++++++---------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/docs/cody/enterprise/model-configuration.mdx b/docs/cody/enterprise/model-configuration.mdx index b6dfaeaed..66dadab2a 100644 --- a/docs/cody/enterprise/model-configuration.mdx +++ b/docs/cody/enterprise/model-configuration.mdx @@ -42,26 +42,6 @@ For most administrators, relying on these models alone ensures access to high-qu The use of these models is controlled through the `"modelConfiguration.sourcegraph"` field in the site config. -### Disable Sourcegraph-provided models - -To disable all Sourcegraph-provided models and use only the models explicitly defined in your site configuration, set the `"sourcegraph"` field to `null` as shown in the example below. - -```json -"cody.enabled": true, -"modelConfiguration": { - "sourcegraph": null, // ignore Sourcegraph-provided models - "providerOverrides": { - // define access to the LLM providers - }, - "modelOverrides": { - // define models available via providers defined in the providerOverrides - }, - "defaultModels": { - // set default models per Cody feature from the list of models defined in modelOverrides - } -} -``` - ### Configure Sourcegraph-provided models The minimal configuration for Sourcegraph-provided models is: @@ -87,6 +67,26 @@ There are three main settings for configuring Sourcegraph-provided LLM models: | `accessToken` | (Optional) The access token for connecting to Cody Gateway defaults to the current license key. | | [`modelFilters`](/cody/enterprise/model-configuration#model-filters) | (Optional) Filters specifying which models to include from Cody Gateway. | +### Disable Sourcegraph-provided models + +To disable all Sourcegraph-provided models and use only the models explicitly defined in your site configuration, set the `"sourcegraph"` field to `null` as shown in the example below. + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": null, // ignore Sourcegraph-provided models + "providerOverrides": { + // define access to the LLM providers + }, + "modelOverrides": { + // define models available via providers defined in the providerOverrides + }, + "defaultModels": { + // set default models per Cody feature from the list of models defined in modelOverrides + } +} +``` + ### Model Filters The `"modelFilters"` section allows you to control which Cody Gateway models are available to users of your Sourcegraph Enterprise instance. The following table describes each field: From 78aae4ca78b9379e5bd9ed585105add23d76e9e6 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Fri, 22 Nov 2024 23:53:32 +0200 Subject: [PATCH 35/58] change examples order --- docs/cody/model-configuration/examples.mdx | 671 +++++++++++++++++++++ 1 file changed, 671 insertions(+) create mode 100644 docs/cody/model-configuration/examples.mdx diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx new file mode 100644 index 000000000..e1ea4ea69 --- /dev/null +++ b/docs/cody/model-configuration/examples.mdx @@ -0,0 +1,671 @@ +# Examples + +## Sourcegraph-supplied models only + +This section includes examples how to configure Cody to use Sourcegraph-supplied models. + +- [Minimal configuration](/cody/model-configuration#configure-sourcegraph-supplied-models) +- [Using model filters](/cody/model-configuration#model-filters) +- [Change default models](/cody/model-configuration#default-models) + +## Sourcegraph-supplied models and BYOK (Bring Your Own Key) + +Out of the box, Sourcegraph is already aware of many models from the following providers: + +- "anthropic" +- "google" +- "fireworks" +- "mistral" +- "openai" + +### Overriding configuration of a model provider + +Instead of Sourcegraph using its own servers to make LLM requests, it is possible to bring your own API keys for a given model provider. For example, if you wish for all Anthropic API requests to go directly to your own Anthropic account and use your own API keys instead of going via Sourcegraph's servers, you could override the `anthropic` provider's configuration: + +```json +{ +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": {}, + "providerOverrides": [ + { + "id": "anthropic", + "displayName": "Anthropic BYOK", + "serverSideConfig": { + "type": "anthropic", + "accessToken": "token", + "endpoint": "https://api.anthropic.com/v1/messages" + } + } + ], + "defaultModels": { + "chat": "anthropic::2024-10-22::claude-3.5-sonnet", + "fastChat": "anthropic::2023-06-01::claude-3-haiku", + "autocomplete": "fireworks::v1::deepseek-coder-v2-lite-base" + } +} +``` + +In the configuration above, we: + +- Enable Sourcegraph-supplied models and do not set any overrides (note that `"modelConfiguration.modelOverrides"` is not specified). +- Route requests for Anthropic models directly to the Anthropic API (via the provider override specified for "anthropic"). +- Route requests for other models (such as the Fireworks model for "autocomplete") through Cody Gateway. + +### Override provider config for some models in the namespace and use the Sourcegraph-configured provider config for the rest + +It's possible to route requests directly to the LLM provider (bypassing the Cody Gateway) for some models while using the +Sourcegraph-configured provider config for the rest. + +Example configuration: + +In the configuration above, we: + +- Enable Sourcegraph-supplied models (the `sourcegraph` field is not empty or `null`). +- Define a new provider with the ID `"anthropic-byok"` and configure it to use the Anthropic API. +- Since this provider is unknown to Sourcegraph, no Sourcegraph-supplied models are available for it. + Therefore, we add a custom model in the `"modelOverrides"` section. +- Use the custom model configured in the previous step (`"anthropic-byok::2024-10-22::claude-3.5-sonnet"`) for `"chat"`. + Requests are sent directly to the Anthropic API as set in the provider override. +- For `"fastChat"` and `"autocomplete"`, we use Sourcegraph-supplied models via Cody Gateway. + +## Config examples for various LLM providers + +Below are configuration examples for setting up various LLM providers using BYOK. +These examples are applicable whether or not you are using Sourcegraph-supported models. + +**Note:** + +- In this section, all configuration examples have Sourcegraph-supplied models disabled. To use a combination of + Sourcegraph-supplied models and BYOK, please refer to the previous section. +- Ensure that at least one model is available for each Cody feature ("chat" and "autocomplete"), regardless of + the provider and model overrides configured. To verify this, [view the configuration](/cody/model-configuration#view-configuration) + and confirm that appropriate models are listed in the `"defaultModels"` section. + + + +```json +{ +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "anthropic", + "displayName": "Anthropic", + "serverSideConfig": { + "type": "anthropic", + "accessToken": "token", + "endpoint": "https://api.anthropic.com/v1/messages" + } + } + ], + "modelOverrides": [ + { + "modelRef": "anthropic::2024-10-22::claude-3.5-sonnet", + "displayName": "Claude 3.5 Sonnet", + "modelName": "claude-3-5-sonnet-latest", + "capabilities": ["chat"], + "category": "accuracy", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "anthropic::2023-06-01::claude-3-haiku", + "displayName": "Claude 3 Haiku", + "modelName": "claude-3-haiku-20240307", + "capabilities": ["chat"], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "anthropic::2023-01-01::claude-instant-1.2", + "displayName": "Claude Instant", + "modelName": "claude-instant-1.2", + "capabilities": ["autocomplete", "chat"], + "category": "other", + "status": "deprecated", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } + } + ], + "defaultModels": { + "chat": "anthropic::2024-10-22::claude-3.5-sonnet", + "fastChat": "anthropic::2023-06-01::claude-3-haiku", + "autocomplete": "anthropic::2023-01-01::claude-instant-1.2" + } +} +``` + +In the configuration above, we: + + - Set up a provider override for Anthropic, routing requests for this provider directly to the specified Anthropic endpoint (bypassing Cody Gateway). + - Add three Anthropic models: + - Two models with chat capabilities (`"anthropic::2024-10-22::claude-3.5-sonnet"` and `"anthropic::2023-06-01::claude-3-haiku"`), + providing options for chat users. + - One model with autocomplete capability (`"anthropic::2023-01-01::claude-instant-1.2"`). + - Set the configured models as default models for Cody features in the `"defaultModels"` field. + + + + + +```json +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "openai", + "displayName": "OpenAI", + "serverSideConfig": { + "type": "openai", + "accessToken": "token", + "endpoint": "https://api.openai.com" + } + } + ], + "modelOverrides": [ + { + "modelRef": "openai::2024-02-01::gpt-4o", + "displayName": "GPT-4o", + "modelName": "gpt-4o", + "capabilities": ["chat"], + "category": "accuracy", + "status": "stable", + "tier": "pro", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 400 + } + }, + { + "modelRef": "openai::unknown::gpt-3.5-turbo-instruct", + "displayName": "GPT-3.5 Turbo Instruct", + "modelName": "gpt-3.5-turbo-instruct", + "capabilities": ["autocomplete"], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 400 + } + } +], + "defaultModels": { + "chat": "openai::2024-02-01::gpt-4o", + "fastChat": "openai::2024-02-01::gpt-4o", + "autocomplete": "openai::unknown::gpt-3.5-turbo-instruct" + } +} +``` + +In the configuration above, we: + +- Set up a provider override for OpenAI, routing requests for this provider directly to the specified OpenAI endpoint (bypassing Cody Gateway). +- Add two OpenAI models: + - `"openai::2024-02-01::gpt-4o"` with "chat" capabilities - used for "chat" and "fastChat" + - `"openai::unknown::gpt-3.5-turbo-instruct"` with "autocomplete" capability - used for "autocomplete". + + + + + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "azure-openai", + "displayName": "Azure OpenAI", + "serverSideConfig": { + "type": "azureOpenAI", + "accessToken": "token", + "endpoint": "https://acme-test.openai.azure.com/", + "user": "", + "useDeprecatedCompletionsAPI": true + } + } + ], + "modelOverrides": [ + { + "modelRef": "azure-openai::unknown::gpt-4o", + "displayName": "GPT-4o", + "modelName": "gpt-4o", + "capabilities": ["chat"], + "category": "accuracy", + "status": "stable", + "tier": "pro", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "azure-openai::unknown::gpt-35-turbo-instruct-test", + "displayName": "GPT-3.5 Turbo Instruct", + "modelName": "gpt-35-turbo-instruct-test", + "capabilities": ["autocomplete"], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } + } + ], + "defaultModels": { + "chat": "azure-openai::unknown::gpt-4o", + "fastChat": "azure-openai::unknown::gpt-4o", + "autocomplete": "azure-openai::unknown::gpt-35-turbo-instruct-test" + } +} +``` + +In the configuration above, we: + +- Set up a provider override for Azure OpenAI, routing requests for this provider directly to the specified Azure OpenAI endpoint (bypassing Cody Gateway). +- Add two OpenAI models: + - `"azure-openai::unknown::gpt-4o"` with "chat" capability - used for "chat" and "fastChat" + - `"azure-openai::unknown::gpt-35-turbo-instruct-test"` with "autocomplete" capability - used for "autocomplete". +- Since `"azure-openai::unknown::gpt-35-turbo-instruct-test"` is not supported on the newer OpenAI `"v1/chat/completions"` endpoint, + we set `"useDeprecatedCompletionsAPI"` to `true` to route requests to the legacy `"v1/completions"` endpoint. This setting is + unnecessary if you are using a model supported on the `"v1/chat/completions"` endpoint. + + + + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "fireworks", + "displayName": "Fireworks", + "serverSideConfig": { + "type": "fireworks", + "accessToken": "token", + "endpoint": "https://api.fireworks.ai/inference/v1/completions" + } + } + ], + "modelOverrides": [ + { + "modelRef": "fireworks::v1::mixtral-8x22b-instruct", + "displayName": "Mixtral 8x22B", + "modelName": "accounts/fireworks/models/mixtral-8x22b-instruct", + "capabilities": ["chat"], + "category": "other", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "fireworks::v1::starcoder-16b", + "modelName": "accounts/fireworks/models/starcoder-16b", + "displayName": "(Fireworks) Starcoder 16B", + "contextWindow": { + "maxInputTokens": 8192, + "maxOutputTokens": 4096 + }, + "capabilities": ["autocomplete"], + "category": "balanced", + "status": "stable" + } + ], + "defaultModels": { + "chat": "fireworks::v1::mixtral-8x22b-instruct", + "fastChat": "fireworks::v1::mixtral-8x22b-instruct", + "autocomplete": "fireworks::v1::starcoder-16b" + } +} +``` + +In the configuration above, we: + +- Set up a provider override for Fireworks, routing requests for this provider directly to the specified Fireworks endpoint (bypassing Cody Gateway). +- Add two Fireworks models: + - `"fireworks::v1::mixtral-8x22b-instruct"` with "chat" capabiity - used for "chat" + and "fastChat" + - `"fireworks::v1::starcoder-16b"` with "autocomplete" capability - used for "autocomplete". + + + + + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "fireworks", + "displayName": "Fireworks", + "serverSideConfig": { + "type": "openaicompatible", + "endpoints": [ + { + "url": "https://api.fireworks.ai/inference/v1", + "accessToken": "token" + } + ] + } + }, + { + "id": "huggingface-codellama", + "displayName": "Hugging Face", + "serverSideConfig": { + "type": "openaicompatible", + "endpoints": [ + { + "url": "https://api-inference.huggingface.co/models/meta-llama/CodeLlama-7b-hf/v1/", + "accessToken": "token" + } + ] + } + }, + ], + "modelOverrides": [ + { + "modelRef": "fireworks::v1::llama-v3p1-70b-instruct", + "modelName": "llama-v3p1-70b-instruct", + "displayName": "(Fireworks) Llama 3.1 70B Instruct", + "contextWindow": { + "maxInputTokens": 64000, + "maxOutputTokens": 8192 + }, + "serverSideConfig": { + "type": "openaicompatible", + "apiModel": "accounts/fireworks/models/llama-v3p1-70b-instruct" + }, + "clientSideConfig": { + "openaicompatible": {} + }, + "capabilities": ["chat"], + "category": "balanced", + "status": "stable" + }, + { + "modelRef": "huggingface-codellama::v1::CodeLlama-7b-hf", + "modelName": "CodeLlama-7b-hf", + "displayName": "(HuggingFace) CodeLlama-7b-hf", + "contextWindow": { + "maxInputTokens": 8192, + "maxOutputTokens": 4096 + }, + "serverSideConfig": { + "type": "openaicompatible", + "apiModel": "meta-llama/CodeLlama-7b-hf" + }, + "clientSideConfig": { + "openaicompatible": {} + }, + "capabilities": ["autocomplete", "chat"], + "category": "balanced", + "status": "stable" + } + ], + "defaultModels": { + "chat": "fireworks::v1::llama-v3p1-70b-instruct", + "fastChat": "fireworks::v1::llama-v3p1-70b-instruct", + "autocomplete": "huggingface-codellama::v1::CodeLlama-7b-hf" + } +} +``` + +In the configuration above, we: + +- Configure two OpenAI-compatible providers: `"fireworks"` and `"huggingface-codellama"`. +- Add two OpenAI-compatible models: `"fireworks::v1::llama-v3p1-70b-instruct"` and `"huggingface-codellama::v1::CodeLlama-7b-hf"`. Additionally: + - Set `clientSideConfig.openaicompatible` to `{}` to indicate to Cody clients that these models are OpenAI-compatible, ensuring the appropriate code paths are utilized. + - Designate these models as the default choices for chat and autocomplete, respectively. + + + + + +```json +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "google", + "displayName": "Google Anthropic", + "serverSideConfig": { + "type": "google", + "accessToken": "token", + "endpoint": "https://us-east5-aiplatform.googleapis.com/v1/projects/project-name/locations/us-east5/publishers/anthropic/models" + } + } + ], + "modelOverrides": [ + { + "modelRef": "google::unknown::claude-3-5-sonnet", + "displayName": "Claude 3.5 Sonnet (via Google/Vertex)", + "modelName": "claude-3-5-sonnet@20240620", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + }, + "capabilities": ["chat"], + "category": "accuracy", + "status": "stable" + }, + { + "modelRef": "google::unknown::claude-3-haiku", + "displayName": "Claude 3 Haiku", + "modelName": "claude-3-haiku@20240307", + "capabilities": ["autocomplete", "chat"], + "category": "speed", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } + }, + ], + "defaultModels": { + "chat": "google::unknown::claude-3-5-sonnet", + "fastChat": "google::unknown::claude-3-5-sonnet", + "autocomplete": "google::unknown::claude-3-haiku" + } +} +``` + +In the configuration above, we: + +- Set up a provider override for Google Anthropic, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway). +- Add two Anthropic models: + - `"google::unknown::claude-3-5-sonnet"` with "chat" capabiity - used for "chat" and "fastChat" + - `"google::unknown::claude-3-haiku"` with "autocomplete" capability - used for "autocomplete". + + + + + +```json +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "google", + "displayName": "Google Gemini", + "serverSideConfig": { + "type": "google", + "accessToken": "token", + "endpoint": "https://generativelanguage.googleapis.com/v1beta/models" + } + } + ], + "modelOverrides": [ + { + "modelRef": "google::v1::gemini-1.5-pro", + "displayName": "Gemini 1.5 Pro", + "modelName": "gemini-1.5-pro", + "capabilities": ["chat", "autocomplete"], + "category": "balanced", + "status": "stable", + "tier": "free", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 400 + } + } + ], + "defaultModels": { + "chat": "google::v1::gemini-1.5-pro", + "fastChat": "google::v1::gemini-1.5-pro", + "autocomplete": "google::v1::gemini-1.5-pro" + } +} +``` + +In the configuration above, we: + +- Set up a provider override for Google Gemini, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway). +- Add the `"google::v1::gemini-1.5-pro"` model, which is used for all Cody features. + We do not add other models for simplicity, as adding multiple models is already covered in the examples above. + + + + + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "aws-bedrock", + "displayName": "AWS Bedrock", + "serverSideConfig": { + "type": "awsBedrock", + "accessToken": "token", + "endpoint": "us-west-2", + "region": "us-west-2" + } + } + ], + "modelOverrides": [ + { + "modelRef": "aws-bedrock::2023-06-01::claude-3-opus", + "displayName": "Claude 3 Opus (AWS Bedrock)", + "modelName": "anthropic.claude-3-opus-20240229-v1:0", + "capabilities": ["chat", "autocomplete"], + "category": "other", + "status": "stable", + "tier": "pro", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + } + ], + "defaultModels": { + "chat": "aws-bedrock::2023-06-01::claude-3-opus", + "fastChat": "aws-bedrock::2023-06-01::claude-3-opus", + "autocomplete": "aws-bedrock::2023-06-01::claude-3-opus", + } +} +``` + +In the configuration described above, we: + +- Set up a provider override for AWS Bedrock, routing requests for this provider directly to the specified endpoint, + bypassing Cody Gateway. +- Add the `"aws-bedrock::2023-06-01::claude-3-opus"` model, which is used for all Cody features. + We do not add other models for simplicity, as adding multiple models is already covered in the examples above. + +Provider override `serverSideConfig` fields: + +| Field | Description | +| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `type` | Must be `"awsBedrock"`. | +| `accessToken` | Leave empty to rely on instance role bindings or other AWS configurations in the frontend service. Use `:` for direct credential configuration, or `::` if a session token is also required. | +| `endpoint` | For Pay-as-you-go, set it to an AWS region code (e.g., `us-west-2`) when using a public Amazon Bedrock endpoint. For Provisioned Throughput, set it to the provisioned VPC endpoint for the bedrock-runtime API (e.g., `https://vpce-0a10b2345cd67e89f-abc0defg.bedrock-runtime.us-west-2.vpce.amazonaws.com`). | +| `region` | The region to use when configuring API clients. This is necessary because the 'frontend' binary's container cannot access environment variables from the host OS. | + +Provisioned throughput for AWS Bedrock models can be configured using the `"awsBedrockProvisionedThroughput"` server-side +configuration type. For more details, refer to the [Model Overrides](/cody/model-configuration#model-overrides) section. + + + +## Self-hosted models + + + Configuring self-hosted models can be nuanced and complex. Consider reaching + out to Sourcegraph support for assistance with the setup. + + +For detailed information about configuring self-hosted models, please refer to the [self-hosted models](/cody/model-configuration#self-hosted-models) documentation. + +Example configuration: + +```json +"cody.enabled": true, +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "fireworks", + "displayName": "Fireworks", + "serverSideConfig": { + "type": "openaicompatible", + "endpoints": [ + { + "url": "https://api.fireworks.ai/inference/v1", + "accessToken": "token" + } + ] + } + } + ], + "selfHostedModels": [ + { + "provider": "fireworks", + "model": "mixtral-8x7b-instruct@v1", + "override": { + "displayName": "(Fireworks) Mixtral 8x7b Instruct", + "serverSideConfig": { + "type": "openaicompatible", + "apiModel": "accounts/fireworks/models/mixtral-8x7b-instruct" + } + } + }, + { + "provider": "fireworks", + "model": "mixtral-8x22b-instruct@v1", + "override": { + "displayName": "(Fireworks) Mixtral 8x22b Instruct", + "serverSideConfig": { + "type": "openaicompatible", + "apiModel": "accounts/fireworks/models/mixtral-8x22b-instruct" + } + } + }, + ] +} +``` + +In the configuration above we: + +- TODO From db4f5626c8e779dbff72d53e260db672a4e6d68c Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Fri, 22 Nov 2024 23:54:56 +0200 Subject: [PATCH 36/58] Update docs/cody/enterprise/model-configuration.mdx Co-authored-by: Aravind Ramaraju <51247042+aramaraju@users.noreply.github.com> --- docs/cody/enterprise/model-configuration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cody/enterprise/model-configuration.mdx b/docs/cody/enterprise/model-configuration.mdx index 66dadab2a..0963cc137 100644 --- a/docs/cody/enterprise/model-configuration.mdx +++ b/docs/cody/enterprise/model-configuration.mdx @@ -27,7 +27,7 @@ The model configuration for Cody is managed through the `"modelConfiguration"` f | [`modelOverrides`](/cody/enterprise/model-configuration#model-overrides) | Extends or modifies the list of models Cody recognizes and their configurations. | | [`selfHostedModels`](/cody/enterprise/model-configuration#self-hosted-models) | Adds models to Cody’s recognized models list with default configurations provided by Sourcegraph. Only available for certain models; general models can be configured in `modelOverrides`. | | [`defaultModels`](/cody/enterprise/model-configuration#default-models) | Specifies the models assigned to each Cody feature (chat, fast chat, autocomplete). | - +| [`providerOverrides`](/cody/model-configuration#provider-overrides) | Configures access to models through your keys with most common LLM providers (BYOK) or hosted behind any openaicompatible endpoint| ## Getting started with `modelConfiguration` The recommended way to set up model configuration is using Sourcegraph-provided models through the Cody Gateway. From 50ea693841f288f422c1302f015ecdaa0803dcea Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Fri, 22 Nov 2024 23:56:38 +0200 Subject: [PATCH 37/58] change section order --- docs/cody/enterprise/model-configuration.mdx | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/cody/enterprise/model-configuration.mdx b/docs/cody/enterprise/model-configuration.mdx index 66dadab2a..89ea42aeb 100644 --- a/docs/cody/enterprise/model-configuration.mdx +++ b/docs/cody/enterprise/model-configuration.mdx @@ -87,6 +87,12 @@ To disable all Sourcegraph-provided models and use only the models explicitly de } ``` +## Default models + +The `"modelConfiguration"` setting includes a `"defaultModels"` field, which allows you to specify the LLM model used for each Cody feature (`"chat"`, `"fastChat"`, and `"autocomplete"`). The values for each feature should be `modelRef`s of either Sourcegraph-provided models or models configured in the `modelOverrides` section. + +If no default is specified or the specified model is not found, the configuration will silently fall back to a suitable alternative. + ### Model Filters The `"modelFilters"` section allows you to control which Cody Gateway models are available to users of your Sourcegraph Enterprise instance. The following table describes each field: @@ -316,12 +322,6 @@ This field is an array of items, each with the following fields: Refer to the [examples page](/cody/enterprise/model-config-examples#self-hosted-models) for examples. -## Default models - -The `"modelConfiguration"` setting includes a `"defaultModels"` field, which allows you to specify the LLM model used for each Cody feature (`"chat"`, `"fastChat"`, and `"autocomplete"`). The values for each feature should be `modelRef`s of either Sourcegraph-provided models or models configured in the `modelOverrides` section. - -If no default is specified or the specified model is not found, the configuration will silently fall back to a suitable alternative. - #### Example configuration ```json From 1bceb58cc48b1051f88594161d12da9c891156fc Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Fri, 22 Nov 2024 23:57:57 +0200 Subject: [PATCH 38/58] Update docs/cody/enterprise/model-configuration.mdx Co-authored-by: Aravind Ramaraju <51247042+aramaraju@users.noreply.github.com> --- docs/cody/enterprise/model-configuration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cody/enterprise/model-configuration.mdx b/docs/cody/enterprise/model-configuration.mdx index 496bdace8..279de4b1a 100644 --- a/docs/cody/enterprise/model-configuration.mdx +++ b/docs/cody/enterprise/model-configuration.mdx @@ -30,7 +30,7 @@ The model configuration for Cody is managed through the `"modelConfiguration"` f | [`providerOverrides`](/cody/model-configuration#provider-overrides) | Configures access to models through your keys with most common LLM providers (BYOK) or hosted behind any openaicompatible endpoint| ## Getting started with `modelConfiguration` -The recommended way to set up model configuration is using Sourcegraph-provided models through the Cody Gateway. +The recommended and easiest way to set up model configuration is using Sourcegraph-provided models through the Cody Gateway. For a minimal configuration example, see [Configure Sourcegraph-supplied models](/cody/enterprise/model-configuration#configure-sourcegraph-provided-models). From e9e7af61c92f06808811faa9afd863ddc203d755 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Fri, 22 Nov 2024 23:58:38 +0200 Subject: [PATCH 39/58] Update docs/cody/enterprise/model-configuration.mdx Co-authored-by: Aravind Ramaraju <51247042+aramaraju@users.noreply.github.com> --- docs/cody/enterprise/model-configuration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cody/enterprise/model-configuration.mdx b/docs/cody/enterprise/model-configuration.mdx index 279de4b1a..09964dbf7 100644 --- a/docs/cody/enterprise/model-configuration.mdx +++ b/docs/cody/enterprise/model-configuration.mdx @@ -119,7 +119,7 @@ The following examples demonstrate how to use each of these settings together: "anthropic::*", // Anthropic models "openai::*", // OpenAI models "google::*", // Google Gemini models - "fireworks::*", // Autocomplete models like StarCoder and DeepSeek-V2-Coder hosted on Fireworks + "fireworks::*", // Autocomplete models hosted on Fireworks such as DeepSeek-V2-Coder ], // Do not include any models with the Model ID containing "turbo", From 4477219f98a57ce13642a8c4c16579598623c962 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Sat, 23 Nov 2024 10:06:51 +0200 Subject: [PATCH 40/58] upd comment --- docs/cody/enterprise/model-configuration.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/cody/enterprise/model-configuration.mdx b/docs/cody/enterprise/model-configuration.mdx index 09964dbf7..47cbffb8e 100644 --- a/docs/cody/enterprise/model-configuration.mdx +++ b/docs/cody/enterprise/model-configuration.mdx @@ -122,8 +122,8 @@ The following examples demonstrate how to use each of these settings together: "fireworks::*", // Autocomplete models hosted on Fireworks such as DeepSeek-V2-Coder ], - // Do not include any models with the Model ID containing "turbo", - // or any from AcmeCo. + // Example: Do not include any models with the Model ID containing "turbo", + // or any models from a hypothetical provider "AcmeCo" "deny": [ "*turbo*", "acmeco::*" From a7d4290196cb097070f73e18b7f46145492a1ea8 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Sat, 23 Nov 2024 10:25:26 +0200 Subject: [PATCH 41/58] address feedback --- .../cody/enterprise/model-config-examples.mdx | 185 +++++++++--------- 1 file changed, 92 insertions(+), 93 deletions(-) diff --git a/docs/cody/enterprise/model-config-examples.mdx b/docs/cody/enterprise/model-config-examples.mdx index f3575210a..d2d981123 100644 --- a/docs/cody/enterprise/model-config-examples.mdx +++ b/docs/cody/enterprise/model-config-examples.mdx @@ -191,49 +191,49 @@ In the configuration above, ```json "cody.enabled": true, "modelConfiguration": { - "sourcegraph": null, - "providerOverrides": [ - { - "id": "fireworks", - "displayName": "Fireworks", - "serverSideConfig": { - "type": "fireworks", - "accessToken": "token", - "endpoint": "https://api.fireworks.ai/inference/v1/completions" - } - } - ], - "modelOverrides": [ - { - "modelRef": "fireworks::v1::mixtral-8x22b-instruct", - "displayName": "Mixtral 8x22B", - "modelName": "accounts/fireworks/models/mixtral-8x22b-instruct", - "capabilities": ["chat"], - "category": "other", - "status": "stable", - "contextWindow": { - "maxInputTokens": 7000, - "maxOutputTokens": 4000 - } - }, - { - "modelRef": "fireworks::v1::starcoder-16b", - "modelName": "accounts/fireworks/models/starcoder-16b", - "displayName": "(Fireworks) Starcoder 16B", - "contextWindow": { - "maxInputTokens": 8192, - "maxOutputTokens": 4096 - }, - "capabilities": ["autocomplete"], - "category": "balanced", - "status": "stable" - } - ], - "defaultModels": { - "chat": "fireworks::v1::mixtral-8x22b-instruct", - "fastChat": "fireworks::v1::mixtral-8x22b-instruct", - "autocomplete": "fireworks::v1::starcoder-16b" - } + "sourcegraph": null, + "providerOverrides": [ + { + "id": "fireworks", + "displayName": "Fireworks", + "serverSideConfig": { + "type": "fireworks", + "accessToken": "token", + "endpoint": "https://api.fireworks.ai/inference/v1/completions" + } + } + ], + "modelOverrides": [ + { + "modelRef": "fireworks::v1::mixtral-8x7b-instruct", + "displayName": "Mixtral 8x7B", + "modelName": "accounts/fireworks/models/mixtral-8x7b-instruct", + "capabilities": ["chat"], + "category": "other", + "status": "stable", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } + }, + { + "modelRef": "fireworks::v1::starcoder-16b", + "modelName": "accounts/fireworks/models/starcoder-16b", + "displayName": "(Fireworks) Starcoder 16B", + "contextWindow": { + "maxInputTokens": 8192, + "maxOutputTokens": 4096 + }, + "capabilities": ["autocomplete"], + "category": "balanced", + "status": "stable" + } + ], + "defaultModels": { + "chat": "fireworks::v1::mixtral-8x7b-instruct", + "fastChat": "fireworks::v1::mixtral-8x7b-instruct", + "autocomplete": "fireworks::v1::starcoder-16b" + } } ``` @@ -241,7 +241,7 @@ In the configuration above, - Set up a provider override for Fireworks, routing requests for this provider directly to the specified Fireworks endpoint (bypassing Cody Gateway) - Add two Fireworks models: - - `"fireworks::v1::mixtral-8x22b-instruct"` with "chat" capabiity - used for "chat" and "fastChat" + - `"fireworks::v1::mixtral-8x7b-instruct"` with "chat" capabiity - used for "chat" and "fastChat" - `"fireworks::v1::starcoder-16b"` with "autocomplete" capability - used for "autocomplete" @@ -358,7 +358,6 @@ In the configuration above, } ``` -In the configuration above, In the configuration above, - Set up a provider override for Azure OpenAI, routing requests for this provider directly to the specified Azure OpenAI endpoint (bypassing Cody Gateway). @@ -461,6 +460,51 @@ In the configuration above, + + +```json +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "google", + "displayName": "Google Gemini", + "serverSideConfig": { + "type": "google", + "accessToken": "token", + "endpoint": "https://generativelanguage.googleapis.com/v1beta/models" + } + } + ], + "modelOverrides": [ + { + "modelRef": "google::v1::gemini-1.5-pro", + "displayName": "Gemini 1.5 Pro", + "modelName": "gemini-1.5-pro", + "capabilities": ["chat", "autocomplete"], + "category": "balanced", + "status": "stable", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + } + } + ], + "defaultModels": { + "chat": "google::v1::gemini-1.5-pro", + "fastChat": "google::v1::gemini-1.5-pro", + "autocomplete": "google::v1::gemini-1.5-pro" + } +} +``` + +In the configuration above, + +- Set up a provider override for Google Gemini, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway) +- Add the `"google::v1::gemini-1.5-pro"` model, which is used for all Cody features. We do not add other models for simplicity, as adding multiple models is already covered in the examples above + + + ```json @@ -520,51 +564,6 @@ In the configuration above, - - -```json -"modelConfiguration": { - "sourcegraph": null, - "providerOverrides": [ - { - "id": "google", - "displayName": "Google Gemini", - "serverSideConfig": { - "type": "google", - "accessToken": "token", - "endpoint": "https://generativelanguage.googleapis.com/v1beta/models" - } - } - ], - "modelOverrides": [ - { - "modelRef": "google::v1::gemini-1.5-pro", - "displayName": "Gemini 1.5 Pro", - "modelName": "gemini-1.5-pro", - "capabilities": ["chat", "autocomplete"], - "category": "balanced", - "status": "stable", - "contextWindow": { - "maxInputTokens": 45000, - "maxOutputTokens": 4000 - } - } - ], - "defaultModels": { - "chat": "google::v1::gemini-1.5-pro", - "fastChat": "google::v1::gemini-1.5-pro", - "autocomplete": "google::v1::gemini-1.5-pro" - } -} -``` - -In the configuration above, - -- Set up a provider override for Google Gemini, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway) -- Add the `"google::v1::gemini-1.5-pro"` model, which is used for all Cody features. We do not add other models for simplicity, as adding multiple models is already covered in the examples above - - - ```json @@ -667,12 +666,12 @@ Example configuration: }, { "provider": "fireworks", - "model": "mixtral-8x22b-instruct@v1", + "model": "mixtral-8x7b-instruct@v1", "override": { - "displayName": "(Fireworks) Mixtral 8x22b Instruct", + "displayName": "(Fireworks) Mixtral 8x7b Instruct", "serverSideConfig": { "type": "openaicompatible", - "apiModel": "accounts/fireworks/models/mixtral-8x22b-instruct" + "apiModel": "accounts/fireworks/models/mixtral-8x7b-instruct" } } }, From 9e47078d8aeef26fafadbd08eb2d9fac4be790c1 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Sat, 23 Nov 2024 10:26:28 +0200 Subject: [PATCH 42/58] fix formatting --- .../cody/enterprise/model-config-examples.mdx | 46 +++++++++---------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/docs/cody/enterprise/model-config-examples.mdx b/docs/cody/enterprise/model-config-examples.mdx index d2d981123..fdebe0282 100644 --- a/docs/cody/enterprise/model-config-examples.mdx +++ b/docs/cody/enterprise/model-config-examples.mdx @@ -571,35 +571,35 @@ In the configuration above, "modelConfiguration": { "sourcegraph": null, "providerOverrides": [ - { - "id": "aws-bedrock", - "displayName": "AWS Bedrock", - "serverSideConfig": { - "type": "awsBedrock", - "accessToken": "token", - "endpoint": "us-west-2", - "region": "us-west-2" - } + { + "id": "aws-bedrock", + "displayName": "AWS Bedrock", + "serverSideConfig": { + "type": "awsBedrock", + "accessToken": "token", + "endpoint": "us-west-2", + "region": "us-west-2" } + } ], "modelOverrides": [ - { - "modelRef": "aws-bedrock::2023-06-01::claude-3-opus", - "displayName": "Claude 3 Opus (AWS Bedrock)", - "modelName": "anthropic.claude-3-opus-20240229-v1:0", - "capabilities": ["chat", "autocomplete"], - "category": "other", - "status": "stable", - "contextWindow": { - "maxInputTokens": 45000, - "maxOutputTokens": 4000 - } + { + "modelRef": "aws-bedrock::2023-06-01::claude-3-opus", + "displayName": "Claude 3 Opus (AWS Bedrock)", + "modelName": "anthropic.claude-3-opus-20240229-v1:0", + "capabilities": ["chat", "autocomplete"], + "category": "other", + "status": "stable", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 } + } ], "defaultModels": { - "chat": "aws-bedrock::2023-06-01::claude-3-opus", - "fastChat": "aws-bedrock::2023-06-01::claude-3-opus", - "autocomplete": "aws-bedrock::2023-06-01::claude-3-opus", + "chat": "aws-bedrock::2023-06-01::claude-3-opus", + "fastChat": "aws-bedrock::2023-06-01::claude-3-opus", + "autocomplete": "aws-bedrock::2023-06-01::claude-3-opus", } } ``` From ddc090cac03ed6330bc62b08af657b3a1165d332 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Sat, 23 Nov 2024 10:29:28 +0200 Subject: [PATCH 43/58] aws => amazon bedrock --- docs/cody/enterprise/model-config-examples.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/cody/enterprise/model-config-examples.mdx b/docs/cody/enterprise/model-config-examples.mdx index fdebe0282..7f1c310a9 100644 --- a/docs/cody/enterprise/model-config-examples.mdx +++ b/docs/cody/enterprise/model-config-examples.mdx @@ -564,7 +564,7 @@ In the configuration above, - + ```json "cody.enabled": true, @@ -573,7 +573,7 @@ In the configuration above, "providerOverrides": [ { "id": "aws-bedrock", - "displayName": "AWS Bedrock", + "displayName": "Amazon Bedrock", "serverSideConfig": { "type": "awsBedrock", "accessToken": "token", @@ -585,7 +585,7 @@ In the configuration above, "modelOverrides": [ { "modelRef": "aws-bedrock::2023-06-01::claude-3-opus", - "displayName": "Claude 3 Opus (AWS Bedrock)", + "displayName": "Claude 3 Opus (Amazon Bedrock)", "modelName": "anthropic.claude-3-opus-20240229-v1:0", "capabilities": ["chat", "autocomplete"], "category": "other", @@ -606,7 +606,7 @@ In the configuration above, In the configuration described above, -- Set up a provider override for AWS Bedrock, routing requests for this provider directly to the specified endpoint, bypassing Cody Gateway +- Set up a provider override for Amazon Bedrock, routing requests for this provider directly to the specified endpoint, bypassing Cody Gateway - Add the `"aws-bedrock::2023-06-01::claude-3-opus"` model, which is used for all Cody features. We do not add other models for simplicity, as adding multiple models is already covered in the examples above Provider override `serverSideConfig` fields: @@ -618,7 +618,7 @@ Provider override `serverSideConfig` fields: | `endpoint` | For pay-as-you-go, set it to an AWS region code (e.g., `us-west-2`) when using a public Amazon Bedrock endpoint. For provisioned throughput, set it to the provisioned VPC endpoint for the bedrock-runtime API (e.g., `https://vpce-0a10b2345cd67e89f-abc0defg.bedrock-runtime.us-west-2.vpce.amazonaws.com`). | | `region` | The region to use when configuring API clients. This is necessary because the 'frontend' binary container cannot access environment variables from the host OS. | -Provisioned throughput for AWS Bedrock models can be configured using the `"awsBedrockProvisionedThroughput"` server-side configuration type. Refer to the [Model Overrides](/cody/enterprise/model-configuration#model-overrides) section for more details. +Provisioned throughput for Amazon Bedrock models can be configured using the `"awsBedrockProvisionedThroughput"` server-side configuration type. Refer to the [Model Overrides](/cody/enterprise/model-configuration#model-overrides) section for more details. From 9c08f8967dde2aa6553712602471145afcca67a5 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Sat, 23 Nov 2024 10:33:13 +0200 Subject: [PATCH 44/58] remove duplication --- docs/cody/enterprise/model-configuration.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/cody/enterprise/model-configuration.mdx b/docs/cody/enterprise/model-configuration.mdx index 47cbffb8e..b051325b8 100644 --- a/docs/cody/enterprise/model-configuration.mdx +++ b/docs/cody/enterprise/model-configuration.mdx @@ -23,11 +23,11 @@ The model configuration for Cody is managed through the `"modelConfiguration"` f | **Field** | **Description** | | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [`sourcegraph`](/cody/enterprise/model-configuration#sourcegraph-provided-models) | Configures access to Sourcegraph-provided models available through Cody Gateway. | -| [`providerOverrides`](/cody/enterprise/model-configuration#provider-overrides) | Configures model providers, allowing you to customize Cody's connection to model providers using your own API keys or self-hosted models. | +| [`providerOverrides`](/cody/model-configuration#provider-overrides) | Configures access to models through your keys with most common LLM providers (BYOK) or hosted behind any openaicompatible endpoint | | [`modelOverrides`](/cody/enterprise/model-configuration#model-overrides) | Extends or modifies the list of models Cody recognizes and their configurations. | | [`selfHostedModels`](/cody/enterprise/model-configuration#self-hosted-models) | Adds models to Cody’s recognized models list with default configurations provided by Sourcegraph. Only available for certain models; general models can be configured in `modelOverrides`. | | [`defaultModels`](/cody/enterprise/model-configuration#default-models) | Specifies the models assigned to each Cody feature (chat, fast chat, autocomplete). | -| [`providerOverrides`](/cody/model-configuration#provider-overrides) | Configures access to models through your keys with most common LLM providers (BYOK) or hosted behind any openaicompatible endpoint| + ## Getting started with `modelConfiguration` The recommended and easiest way to set up model configuration is using Sourcegraph-provided models through the Cody Gateway. From bb76873f22152f262e2ea331136bef4bb0122e7f Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Sat, 23 Nov 2024 10:38:37 +0200 Subject: [PATCH 45/58] remove model tier mentions --- docs/cody/enterprise/model-configuration.mdx | 11 ----------- docs/cody/model-configuration/examples.mdx | 11 ----------- 2 files changed, 22 deletions(-) diff --git a/docs/cody/enterprise/model-configuration.mdx b/docs/cody/enterprise/model-configuration.mdx index b051325b8..3d0f95a29 100644 --- a/docs/cody/enterprise/model-configuration.mdx +++ b/docs/cody/enterprise/model-configuration.mdx @@ -389,7 +389,6 @@ The response includes: "capabilities": ["chat"], "category": "accuracy", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 4000 @@ -402,7 +401,6 @@ The response includes: "capabilities": ["chat"], "category": "other", "status": "stable", - "tier": "pro", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 4000 @@ -415,7 +413,6 @@ The response includes: "capabilities": ["chat"], "category": "speed", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 7000, "maxOutputTokens": 4000 @@ -428,7 +425,6 @@ The response includes: "capabilities": ["autocomplete"], "category": "speed", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 2048, "maxOutputTokens": 256 @@ -441,7 +437,6 @@ The response includes: "capabilities": ["autocomplete"], "category": "speed", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 2048, "maxOutputTokens": 256 @@ -454,7 +449,6 @@ The response includes: "capabilities": ["chat"], "category": "balanced", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 4000 @@ -467,7 +461,6 @@ The response includes: "capabilities": ["chat"], "category": "speed", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 4000 @@ -480,7 +473,6 @@ The response includes: "capabilities": ["chat"], "category": "speed", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 7000, "maxOutputTokens": 4000 @@ -493,7 +485,6 @@ The response includes: "capabilities": ["chat"], "category": "accuracy", "status": "stable", - "tier": "pro", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 4000 @@ -506,7 +497,6 @@ The response includes: "capabilities": ["chat"], "category": "accuracy", "status": "waitlist", - "tier": "pro", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 4000 @@ -519,7 +509,6 @@ The response includes: "capabilities": ["chat"], "category": "accuracy", "status": "waitlist", - "tier": "pro", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 4000 diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx index e1ea4ea69..d71c90e15 100644 --- a/docs/cody/model-configuration/examples.mdx +++ b/docs/cody/model-configuration/examples.mdx @@ -108,7 +108,6 @@ These examples are applicable whether or not you are using Sourcegraph-supported "capabilities": ["chat"], "category": "accuracy", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 4000 @@ -121,7 +120,6 @@ These examples are applicable whether or not you are using Sourcegraph-supported "capabilities": ["chat"], "category": "speed", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 7000, "maxOutputTokens": 4000 @@ -134,7 +132,6 @@ These examples are applicable whether or not you are using Sourcegraph-supported "capabilities": ["autocomplete", "chat"], "category": "other", "status": "deprecated", - "tier": "free", "contextWindow": { "maxInputTokens": 7000, "maxOutputTokens": 4000 @@ -184,7 +181,6 @@ In the configuration above, we: "capabilities": ["chat"], "category": "accuracy", "status": "stable", - "tier": "pro", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 400 @@ -197,7 +193,6 @@ In the configuration above, we: "capabilities": ["autocomplete"], "category": "speed", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 7000, "maxOutputTokens": 400 @@ -248,7 +243,6 @@ In the configuration above, we: "capabilities": ["chat"], "category": "accuracy", "status": "stable", - "tier": "pro", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 4000 @@ -261,7 +255,6 @@ In the configuration above, we: "capabilities": ["autocomplete"], "category": "speed", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 7000, "maxOutputTokens": 4000 @@ -312,7 +305,6 @@ In the configuration above, we: "capabilities": ["chat"], "category": "other", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 7000, "maxOutputTokens": 4000 @@ -476,7 +468,6 @@ In the configuration above, we: "capabilities": ["autocomplete", "chat"], "category": "speed", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 7000, "maxOutputTokens": 4000 @@ -524,7 +515,6 @@ In the configuration above, we: "capabilities": ["chat", "autocomplete"], "category": "balanced", "status": "stable", - "tier": "free", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 400 @@ -573,7 +563,6 @@ In the configuration above, we: "capabilities": ["chat", "autocomplete"], "category": "other", "status": "stable", - "tier": "pro", "contextWindow": { "maxInputTokens": 45000, "maxOutputTokens": 4000 From ec113707d2b58cadfe7c8765c0c397021b96288a Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Mon, 25 Nov 2024 13:18:06 +0200 Subject: [PATCH 46/58] upd comment --- docs/cody/enterprise/model-configuration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cody/enterprise/model-configuration.mdx b/docs/cody/enterprise/model-configuration.mdx index 3d0f95a29..0dd15612d 100644 --- a/docs/cody/enterprise/model-configuration.mdx +++ b/docs/cody/enterprise/model-configuration.mdx @@ -119,7 +119,7 @@ The following examples demonstrate how to use each of these settings together: "anthropic::*", // Anthropic models "openai::*", // OpenAI models "google::*", // Google Gemini models - "fireworks::*", // Autocomplete models hosted on Fireworks such as DeepSeek-V2-Coder + "fireworks::*", // Open-source models hosted by Sourcegraph on Fireworks.ai (typically used for autocomplete and Mistral Chat models) ], // Example: Do not include any models with the Model ID containing "turbo", From ed191effea49ad53711f9017aa64f9a7271f3050 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Mon, 25 Nov 2024 13:20:50 +0200 Subject: [PATCH 47/58] Update docs/cody/enterprise/features.mdx Co-authored-by: David Veszelovszki --- docs/cody/enterprise/features.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cody/enterprise/features.mdx b/docs/cody/enterprise/features.mdx index 3685d5d1c..0756576d2 100644 --- a/docs/cody/enterprise/features.mdx +++ b/docs/cody/enterprise/features.mdx @@ -78,7 +78,7 @@ For the supported LLM models listed above, refer to the following notes: 5. BYOK (Bring Your Own Key) with managed services are only supported for Self-hosted Sourcegraph instances 6. GPT-4 and GPT-4o for completions have a bug that is resulting in many failed completions -{/*Tempporarily removed*/} +{/*Temporarily removed*/} {/* ## Supported model configuration From 03c9d6d34f8a38767a4809e6f2b59a43880fa762 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Mon, 25 Nov 2024 13:21:05 +0200 Subject: [PATCH 48/58] Update docs/cody/enterprise/completions-configuration.mdx Co-authored-by: David Veszelovszki --- docs/cody/enterprise/completions-configuration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cody/enterprise/completions-configuration.mdx b/docs/cody/enterprise/completions-configuration.mdx index 0be4bb03d..aa7c46f85 100644 --- a/docs/cody/enterprise/completions-configuration.mdx +++ b/docs/cody/enterprise/completions-configuration.mdx @@ -164,7 +164,7 @@ For the access token, - For Sourcegraph `v5.2.4` or more, the access token can be left empty, and it will rely on Environmental, Workload Identity, or Managed Identity credentials configured for the `frontend` and `worker` services - Set it to `` if directly configuring the credentials using the API key specified in the Azure portal -### Use StarCoder for Autocomplete +### Use StarCoder for autocomplete When tested with other coder models for the autocomplete use case, [StarCoder](https://huggingface.co/blog/starcoder) offered significant improvements in quality and latency compared to our control groups for users on Sourcegraph.com. You can read more about the improvements in our [October 2023 release notes](https://sourcegraph.com/blog/feature-release-october-2023) and the [GA release notes](https://sourcegraph.com/blog/cody-is-generally-available). From 8a280f7b587ffbbba248be836cb1245e3aae1a38 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Mon, 25 Nov 2024 13:29:41 +0200 Subject: [PATCH 49/58] Update docs/cody/clients/enable-cody-enterprise.mdx Co-authored-by: David Veszelovszki --- docs/cody/clients/enable-cody-enterprise.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cody/clients/enable-cody-enterprise.mdx b/docs/cody/clients/enable-cody-enterprise.mdx index 146056fb9..60987fa95 100644 --- a/docs/cody/clients/enable-cody-enterprise.mdx +++ b/docs/cody/clients/enable-cody-enterprise.mdx @@ -60,7 +60,7 @@ To turn Cody off: ## Enable Cody only for some users -To enable Cody only for some users it depends on what version of Sourcegraph you are running. +How to enable Cody only for _some_ users depends on what version of Sourcegraph you are running. ### Sourcegraph v5.3+ From 324868b79042184dffe804aba98efa7d22e18f95 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Mon, 25 Nov 2024 13:30:40 +0200 Subject: [PATCH 50/58] Update docs/cody/enterprise/completions-configuration.mdx Co-authored-by: David Veszelovszki --- docs/cody/enterprise/completions-configuration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cody/enterprise/completions-configuration.mdx b/docs/cody/enterprise/completions-configuration.mdx index aa7c46f85..d543437d9 100644 --- a/docs/cody/enterprise/completions-configuration.mdx +++ b/docs/cody/enterprise/completions-configuration.mdx @@ -93,7 +93,7 @@ For `accessToken`, you can either: ### Example: Using GCP Vertex AI -We only support Anthropic Claude models on [GCP Vertex](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-claude). +On [GCP Vertex](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-claude), we only support Anthropic Claude models. - Enable the [Vertex AI API](https://console.cloud.google.com/marketplace/product/google/aiplatform.googleapis.com) in the GCP console. Once Vertex has been enabled in your project, navigate to the [Vertex Model Garden](https://console.cloud.google.com/vertex-ai/model-garden) to select and enable the Anthropic Claude model(s) that you wish to use with Cody. See [Supported LLM Models](/capabilities/supported-models) for an up-to-date list of Anthropic Claude models supported by Cody. From b1f1b9fa20a4813a185379620395a52d3a5ed784 Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Mon, 25 Nov 2024 17:30:53 +0200 Subject: [PATCH 51/58] upd amazon bedrock example and docs --- .../cody/enterprise/model-config-examples.mdx | 45 ++++++++++--------- docs/cody/enterprise/model-configuration.mdx | 10 ++--- 2 files changed, 30 insertions(+), 25 deletions(-) diff --git a/docs/cody/enterprise/model-config-examples.mdx b/docs/cody/enterprise/model-config-examples.mdx index 7f1c310a9..d200c934f 100644 --- a/docs/cody/enterprise/model-config-examples.mdx +++ b/docs/cody/enterprise/model-config-examples.mdx @@ -573,41 +573,46 @@ In the configuration above, "providerOverrides": [ { "id": "aws-bedrock", - "displayName": "Amazon Bedrock", + "displayName": "Anthropic models through AWS Bedrock", "serverSideConfig": { - "type": "awsBedrock", - "accessToken": "token", - "endpoint": "us-west-2", - "region": "us-west-2" + "type": "awsBedrock", + "accessToken": "", + "region": "us-west-2" } } ], "modelOverrides": [ { - "modelRef": "aws-bedrock::2023-06-01::claude-3-opus", - "displayName": "Claude 3 Opus (Amazon Bedrock)", - "modelName": "anthropic.claude-3-opus-20240229-v1:0", - "capabilities": ["chat", "autocomplete"], - "category": "other", - "status": "stable", + "modelRef": "aws-bedrock::2024-02-29::claude-3-sonnet", + "displayName": "Claude 3 Sonnet", + "modelName": "claude-3-sonnet", + "serverSideConfig": { + "type": "awsBedrockProvisionedThroughput", + "arn": "" // e.g., arn:aws:bedrock:us-west-2:548543007731:provisioned-model/47u2lgtk1rc1 + }, "contextWindow": { - "maxInputTokens": 45000, - "maxOutputTokens": 4000 - } - } + "maxInputTokens": 16000, + "maxOutputTokens": 4000 + }, + "capabilities": ["chat", "autocomplete"], + "category": "balanced", + "status": "stable" + }, ], "defaultModels": { - "chat": "aws-bedrock::2023-06-01::claude-3-opus", - "fastChat": "aws-bedrock::2023-06-01::claude-3-opus", - "autocomplete": "aws-bedrock::2023-06-01::claude-3-opus", - } + "chat": "aws-bedrock::2024-02-29::claude-3-sonnet", + "codeCompletion": "aws-bedrock::2024-02-29::claude-3-sonnet", + "fastChat": "aws-bedrock::2024-02-29::claude-3-sonnet" + }, } ``` In the configuration described above, - Set up a provider override for Amazon Bedrock, routing requests for this provider directly to the specified endpoint, bypassing Cody Gateway -- Add the `"aws-bedrock::2023-06-01::claude-3-opus"` model, which is used for all Cody features. We do not add other models for simplicity, as adding multiple models is already covered in the examples above +- Add the `"aws-bedrock::2024-02-29::claude-3-sonnet"` model, which is used for all Cody features. We do not add other models for simplicity, as adding multiple models is already covered in the examples above +- Note: Since the model in the example uses provisioned throughput, specify the ARN in the `serverSideConfig.arn` field of the model override. Provider override `serverSideConfig` fields: diff --git a/docs/cody/enterprise/model-configuration.mdx b/docs/cody/enterprise/model-configuration.mdx index 0dd15612d..a2333a929 100644 --- a/docs/cody/enterprise/model-configuration.mdx +++ b/docs/cody/enterprise/model-configuration.mdx @@ -226,13 +226,13 @@ This field is an array of items, each with the following fields: - `maxInputTokens`: Specifies the maximum number of tokens for the contextual data in the prompt (e.g., question, relevant snippets) - `maxOutputTokens`: Specifies the maximum number of tokens allowed in the response - `serverSideConfig`: Additional configuration for the model. It can be one of the following: + - `awsBedrockProvisionedThroughput`: Specifies provisioned throughput settings for AWS Bedrock models with the following fields: - `type`: Must be `"awsBedrockProvisionedThroughput"` - - `arn`: The ARN (Amazon Resource Name) for provisioned throughput to use when sending requests to AWS Bedrock -- `openaicompatible`: Configuration specific to models provided by an OpenAI-compatible provider, with the following fields: - - - `type`: Must be `"openaicompatible"` - - `apiModel`: The literal string value of the `model` field to be sent to the `/chat/completions` API. If set, Sourcegraph treats this as an opaque string and sends it directly to the API without inferring any additional information. By default, the configured model name is sent + - `arn`: The ARN (Amazon Resource Name) for provisioned throughput to use when sending requests to AWS Bedrock. The ARN format for provisioned models is: `arn:${Partition}:bedrock:${Region}:${Account}:provisioned-model/${ResourceId}`. + - `openaicompatible`: Configuration specific to models provided by an OpenAI-compatible provider, with the following fields: + - `type`: Must be `"openaicompatible"` + - `apiModel`: The literal string value of the `model` field to be sent to the `/chat/completions` API. If set, Sourcegraph treats this as an opaque string and sends it directly to the API without inferring any additional information. By default, the configured model name is sent - `clientSideConfig`: Advanced configuration options that are only respected if the model is provided by the `"openaicompatible"` provider From 1033347084a56f5cab1df96e54dded1b22cf431d Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Mon, 25 Nov 2024 18:39:28 +0200 Subject: [PATCH 52/58] fix default model field name --- docs/cody/enterprise/model-config-examples.mdx | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/cody/enterprise/model-config-examples.mdx b/docs/cody/enterprise/model-config-examples.mdx index d200c934f..3a21ec1a7 100644 --- a/docs/cody/enterprise/model-config-examples.mdx +++ b/docs/cody/enterprise/model-config-examples.mdx @@ -43,7 +43,7 @@ Instead of Sourcegraph using its own servers to make LLM requests, it is possibl "defaultModels": { "chat": "anthropic::2024-10-22::claude-3.5-sonnet", "fastChat": "anthropic::2023-06-01::claude-3-haiku", - "autocomplete": "fireworks::v1::deepseek-coder-v2-lite-base" + "codeCompletion": "fireworks::v1::deepseek-coder-v2-lite-base" } } ``` @@ -93,7 +93,7 @@ Example configuration "defaultModels": { "chat": "anthropic-byok::2023-06-01::claude-3.5-sonnet", "fastChat": "anthropic::2023-06-01::claude-3-haiku", - "autocomplete": "fireworks::v1::deepseek-coder-v2-lite-base" + "codeCompletion": "fireworks::v1::deepseek-coder-v2-lite-base" } } ``` @@ -172,7 +172,7 @@ Below are configuration examples for setting up various LLM providers using BYOK "defaultModels": { "chat": "anthropic::2024-10-22::claude-3.5-sonnet", "fastChat": "anthropic::2023-06-01::claude-3-haiku", - "autocomplete": "fireworks::v1::deepseek-coder-v2-lite-base" + "codeCompletion": "fireworks::v1::deepseek-coder-v2-lite-base" } } ``` @@ -232,7 +232,7 @@ In the configuration above, "defaultModels": { "chat": "fireworks::v1::mixtral-8x7b-instruct", "fastChat": "fireworks::v1::mixtral-8x7b-instruct", - "autocomplete": "fireworks::v1::starcoder-16b" + "codeCompletion": "fireworks::v1::starcoder-16b" } } ``` @@ -291,7 +291,7 @@ In the configuration above, "defaultModels": { "chat": "openai::2024-02-01::gpt-4o", "fastChat": "openai::2024-02-01::gpt-4o", - "autocomplete": "openai::unknown::gpt-3.5-turbo-instruct" + "codeCompletion": "openai::unknown::gpt-3.5-turbo-instruct" } } ``` @@ -353,7 +353,7 @@ In the configuration above, "defaultModels": { "chat": "azure-openai::unknown::gpt-4o", "fastChat": "azure-openai::unknown::gpt-4o", - "autocomplete": "azure-openai::unknown::gpt-35-turbo-instruct-test" + "codeCompletion": "azure-openai::unknown::gpt-35-turbo-instruct-test" } } ``` @@ -446,7 +446,7 @@ In the configuration above, "defaultModels": { "chat": "fireworks::v1::llama-v3p1-70b-instruct", "fastChat": "fireworks::v1::llama-v3p1-70b-instruct", - "autocomplete": "huggingface-codellama::v1::CodeLlama-7b-hf" + "codeCompletion": "huggingface-codellama::v1::CodeLlama-7b-hf" } } ``` @@ -493,7 +493,7 @@ In the configuration above, "defaultModels": { "chat": "google::v1::gemini-1.5-pro", "fastChat": "google::v1::gemini-1.5-pro", - "autocomplete": "google::v1::gemini-1.5-pro" + "codeCompletion": "google::v1::gemini-1.5-pro" } } ``` @@ -550,7 +550,7 @@ In the configuration above, "defaultModels": { "chat": "google::unknown::claude-3-5-sonnet", "fastChat": "google::unknown::claude-3-5-sonnet", - "autocomplete": "google::unknown::claude-3-haiku" + "codeCompletion": "google::unknown::claude-3-haiku" } } ``` From 6b11a5214c067959f3c9a7c48a77342060a33eef Mon Sep 17 00:00:00 2001 From: Taras Yemets Date: Mon, 25 Nov 2024 18:51:27 +0200 Subject: [PATCH 53/58] remove self-hosted models --- .../cody/enterprise/model-config-examples.mdx | 115 +++++++++--------- docs/cody/enterprise/model-configuration.mdx | 42 ------- docs/cody/model-configuration/examples.mdx | 61 ---------- 3 files changed, 56 insertions(+), 162 deletions(-) diff --git a/docs/cody/enterprise/model-config-examples.mdx b/docs/cody/enterprise/model-config-examples.mdx index 3a21ec1a7..ec2d3bfd0 100644 --- a/docs/cody/enterprise/model-config-examples.mdx +++ b/docs/cody/enterprise/model-config-examples.mdx @@ -564,6 +564,62 @@ In the configuration above, + + +```json +"modelConfiguration": { + "sourcegraph": null, + "providerOverrides": [ + { + "id": "google", + "displayName": "Google Gemini", + "serverSideConfig": { + "type": "google", + "accessToken": "token", + "endpoint": "https://us-east5-aiplatform.googleapis.com/v1/projects/project-name/locations/us-east5/publishers/anthropic/models" + } + } + ], + "modelOverrides": [ + { + "modelRef": "google::unknown::claude-3-5-sonnet", + "displayName": "Claude 3.5 Sonnet (via Google/Vertex)", + "modelName": "claude-3-5-sonnet@20240620", + "contextWindow": { + "maxInputTokens": 45000, + "maxOutputTokens": 4000 + }, + "capabilities": ["chat"], + "category": "accuracy", + "status": "stable" + }, + { + "modelRef": "google::unknown::claude-3-haiku", + "displayName": "Claude 3 Haiku", + "modelName": "claude-3-haiku@20240307", + "capabilities": ["autocomplete", "chat"], + "category": "speed", + "status": "stable", + "contextWindow": { + "maxInputTokens": 7000, + "maxOutputTokens": 4000 + } + }, + ], + "defaultModels": { + "chat": "google::unknown::claude-3-5-sonnet", + "fastChat": "google::unknown::claude-3-5-sonnet", + "codeCompletion": "google::unknown::claude-3-haiku" + } +} +``` + +In the configuration above, + +- Set up a provider override for Google Anthropic, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway) +- Add two Anthropic models: - `"google::unknown::claude-3-5-sonnet"` with "chat" capabiity - used for "chat" and "fastChat" - `"google::unknown::claude-3-haiku"` with "autocomplete" capability - used for "autocomplete" + + ```json @@ -626,62 +682,3 @@ Provider override `serverSideConfig` fields: Provisioned throughput for Amazon Bedrock models can be configured using the `"awsBedrockProvisionedThroughput"` server-side configuration type. Refer to the [Model Overrides](/cody/enterprise/model-configuration#model-overrides) section for more details. - -## Self-hosted models - - - Configuring self-hosted models can be nuanced and complex. Please reach out - to Sourcegraph support for assistance with the setup. - - -Please refer to the [self-hosted models](/cody/enterprise/model-configuration#self-hosted-models) documentation for detailed information about configuring self-hosted models. - -Example configuration: - -```json -"cody.enabled": true, -"modelConfiguration": { - "sourcegraph": null, - "providerOverrides": [ - { - "id": "fireworks", - "displayName": "Fireworks", - "serverSideConfig": { - "type": "openaicompatible", - "endpoints": [ - { - "url": "https://api.fireworks.ai/inference/v1", - "accessToken": "token" - } - ] - } - } - ], - "selfHostedModels": [ - { - "provider": "fireworks", - "model": "mixtral-8x7b-instruct@v1", - "override": { - "displayName": "(Fireworks) Mixtral 8x7b Instruct", - "serverSideConfig": { - "type": "openaicompatible", - "apiModel": "accounts/fireworks/models/mixtral-8x7b-instruct" - } - } - }, - { - "provider": "fireworks", - "model": "mixtral-8x7b-instruct@v1", - "override": { - "displayName": "(Fireworks) Mixtral 8x7b Instruct", - "serverSideConfig": { - "type": "openaicompatible", - "apiModel": "accounts/fireworks/models/mixtral-8x7b-instruct" - } - } - }, - ] -} -``` - -{/* In the configuration above: TODO */} diff --git a/docs/cody/enterprise/model-configuration.mdx b/docs/cody/enterprise/model-configuration.mdx index a2333a929..59f3c45b7 100644 --- a/docs/cody/enterprise/model-configuration.mdx +++ b/docs/cody/enterprise/model-configuration.mdx @@ -234,10 +234,6 @@ This field is an array of items, each with the following fields: - `type`: Must be `"openaicompatible"` - `apiModel`: The literal string value of the `model` field to be sent to the `/chat/completions` API. If set, Sourcegraph treats this as an opaque string and sends it directly to the API without inferring any additional information. By default, the configured model name is sent -- `clientSideConfig`: Advanced configuration options that are only respected if the model is provided by the `"openaicompatible"` provider - -{/* TODO: add more details. */} - #### Example configuration ```json @@ -299,44 +295,6 @@ In the example above: Refer to the [examples page](/cody/enterprise/model-config-examples) for additional examples. -## Self-hosted models - - - Configuring self-hosted models can be nuanced and complex. Please reach out - to Sourcegraph support for assistance with the setup. - - -With a provider defined (either a Sourcegraph-provided provider or a custom provider configured via the `providerOverrides` field), self-hosted models can be added to Cody’s recognized models list by configuring the `"modelConfiguration.selfHostedModels"` section. -These models use Sourcegraph’s default configuration settings and are only available for select models. -Generic models should be configured in the `"modelOverrides"` section instead. - -Specifying the same model in both `"selfHostedModels"` and `"modelOverrides"` is not allowed. - -This field is an array of items, each with the following fields: - -| **Field** | **Description** | -| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `provider` | (Required) The provider ID of the model, configured in the `providerOverrides` field. | -| `apiVersion` | (Optional) The API version to use. Defaults to `"v1"` if not specified. | -| `model` | (Optional) Specifies which default model configuration to use. Sourcegraph provides default configurations for select models. For arbitrary models, use the `modelOverrides` section. Available options include: `"starcoder2-7b@v1"`, `"starcoder2-15b@v1"`, `"mistral-7b-instruct@v1"`, `"mixtral-8x7b-instruct@v1"`, `"mixtral-8x22b-instruct@v1"` | - -Refer to the [examples page](/cody/enterprise/model-config-examples#self-hosted-models) for examples. - -#### Example configuration - -```json -"cody.enabled": true, - -// allow Sourcegraph-provided models -"sourcegraph": {}, - -"defaultModels": { - "chat": "openai::2024-02-01::gpt-4o", - "fastChat": "google::v1::gemini-1.5-flash", - "codeCompletion": "fireworks::v1::deepseek-coder-v2-lite-base" -} -``` - ## View configuration To view the current model configuration, run the following command: diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx index d71c90e15..01bde7670 100644 --- a/docs/cody/model-configuration/examples.mdx +++ b/docs/cody/model-configuration/examples.mdx @@ -597,64 +597,3 @@ Provisioned throughput for AWS Bedrock models can be configured using the `"awsB configuration type. For more details, refer to the [Model Overrides](/cody/model-configuration#model-overrides) section. - -## Self-hosted models - - - Configuring self-hosted models can be nuanced and complex. Consider reaching - out to Sourcegraph support for assistance with the setup. - - -For detailed information about configuring self-hosted models, please refer to the [self-hosted models](/cody/model-configuration#self-hosted-models) documentation. - -Example configuration: - -```json -"cody.enabled": true, -"modelConfiguration": { - "sourcegraph": null, - "providerOverrides": [ - { - "id": "fireworks", - "displayName": "Fireworks", - "serverSideConfig": { - "type": "openaicompatible", - "endpoints": [ - { - "url": "https://api.fireworks.ai/inference/v1", - "accessToken": "token" - } - ] - } - } - ], - "selfHostedModels": [ - { - "provider": "fireworks", - "model": "mixtral-8x7b-instruct@v1", - "override": { - "displayName": "(Fireworks) Mixtral 8x7b Instruct", - "serverSideConfig": { - "type": "openaicompatible", - "apiModel": "accounts/fireworks/models/mixtral-8x7b-instruct" - } - } - }, - { - "provider": "fireworks", - "model": "mixtral-8x22b-instruct@v1", - "override": { - "displayName": "(Fireworks) Mixtral 8x22b Instruct", - "serverSideConfig": { - "type": "openaicompatible", - "apiModel": "accounts/fireworks/models/mixtral-8x22b-instruct" - } - } - }, - ] -} -``` - -In the configuration above we: - -- TODO From bb313f0ea5d92b976b0438d82bf699fb721cd4fc Mon Sep 17 00:00:00 2001 From: Maedah Batool Date: Mon, 25 Nov 2024 13:49:22 -0800 Subject: [PATCH 54/58] Update src/data/navigation.ts Co-authored-by: David Veszelovszki --- src/data/navigation.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/data/navigation.ts b/src/data/navigation.ts index 39414f628..56ca46beb 100644 --- a/src/data/navigation.ts +++ b/src/data/navigation.ts @@ -46,7 +46,7 @@ export const navigation: NavigationItem[] = [ { title: "Features", href: "/cody/enterprise/features", }, { title: "Completions Configuration", href: "/cody/enterprise/completions-configuration", }, { title: "Model Configuration", href: "/cody/enterprise/model-configuration", }, - { title: "Examples modelConfiguration", href: "/cody/enterprise/model-config-examples", }, + { title: "modelConfiguration examples", href: "/cody/enterprise/model-config-examples", }, ] }, { From bbe8279bbdde2a4c4c7fcd29fe89f1d34189de66 Mon Sep 17 00:00:00 2001 From: Maedah Batool Date: Mon, 25 Nov 2024 13:49:56 -0800 Subject: [PATCH 55/58] Update docs/cody/enterprise/features.mdx Co-authored-by: David Veszelovszki --- docs/cody/enterprise/features.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cody/enterprise/features.mdx b/docs/cody/enterprise/features.mdx index 0756576d2..80840adc1 100644 --- a/docs/cody/enterprise/features.mdx +++ b/docs/cody/enterprise/features.mdx @@ -1,4 +1,4 @@ -# Cody Enterprise Features +# Cody Enterprise features

Along with the core features, Cody Enterprise offers additional features to enhance your coding experience.

From 3471de4bcd014e9085b386d24ae63b305fd29405 Mon Sep 17 00:00:00 2001 From: Maedah Batool Date: Mon, 25 Nov 2024 14:41:27 -0800 Subject: [PATCH 56/58] fix build errors --- docs/cody/enterprise/model-config-examples.mdx | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/cody/enterprise/model-config-examples.mdx b/docs/cody/enterprise/model-config-examples.mdx index ec2d3bfd0..6c936f4cf 100644 --- a/docs/cody/enterprise/model-config-examples.mdx +++ b/docs/cody/enterprise/model-config-examples.mdx @@ -618,7 +618,8 @@ In the configuration above, - Set up a provider override for Google Anthropic, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway) - Add two Anthropic models: - `"google::unknown::claude-3-5-sonnet"` with "chat" capabiity - used for "chat" and "fastChat" - `"google::unknown::claude-3-haiku"` with "autocomplete" capability - used for "autocomplete" - + + From aa709dd7c1d292ba4da9399c1cfc61cc5eee7829 Mon Sep 17 00:00:00 2001 From: Maedah Batool Date: Mon, 25 Nov 2024 15:04:31 -0800 Subject: [PATCH 57/58] prompt pre-instructions docs --- docs/cody/enterprise/features.mdx | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/cody/enterprise/features.mdx b/docs/cody/enterprise/features.mdx index 80840adc1..76b842b6c 100644 --- a/docs/cody/enterprise/features.mdx +++ b/docs/cody/enterprise/features.mdx @@ -64,6 +64,22 @@ Please note that you can only `@-mention` remote directories (i.e., directories If you want to include recent changes that haven't been indexed in your Sourcegraph instance, you can `@-mention` specific files, lines of code, or symbols. +## Prompt pre-instructions + +Prompt pre-instructions is supported for Sourcegraph `v5.10` and more. + +**Site admins** can add and configure prompt pre-instructions for Cody. These are text instructions that Cody uses with every chat query, allowing organizations to configure how Cody responds to their users. + +For example, if you don’t want Cody to answer questions relating to sensitive non-code matters, you can pre-instruct Cody about it. In this case, if a user asks an unrelated question, Cody responds with a pre-instructed context. + +To configure pre-instructions, add the following to your site admin configuration file: + +```json +{ ... "modelConfiguration": { "systemPreInstruction": "If the question is not directly related to software development, respond with \"I can only answer programming-related questions\"" } } +``` + +We cannot guarantee that these pre-instructions or their intended use case will be fully accurate. If your pre-instructions are not working as expected, please get in touch with us. + ## Supported LLM models Sourcegraph Enterprise supports different LLM providers and models, such as models from Anthropic and OpenAI. You can do this by adjusting your Sourcegraph instance configuration. From eadab3a1ae2f96fd90d4a1fbe0273bb3129a98b6 Mon Sep 17 00:00:00 2001 From: Maedah Batool Date: Mon, 25 Nov 2024 15:41:58 -0800 Subject: [PATCH 58/58] Rmv duplicate folder --- docs/cody/clients/enable-cody-enterprise.mdx | 4 +- docs/cody/model-configuration/examples.mdx | 599 ------------------- 2 files changed, 1 insertion(+), 602 deletions(-) delete mode 100644 docs/cody/model-configuration/examples.mdx diff --git a/docs/cody/clients/enable-cody-enterprise.mdx b/docs/cody/clients/enable-cody-enterprise.mdx index 60987fa95..245bdfb62 100644 --- a/docs/cody/clients/enable-cody-enterprise.mdx +++ b/docs/cody/clients/enable-cody-enterprise.mdx @@ -28,7 +28,7 @@ With [Sourcegraph Cloud](/cloud/), you get Cody as a managed service, and you ** Site admins can only enable Cody on the Sourcegraph instance. To do so, -- First, configure your desired LLM provider either by [using Sourcegraph Cody Gateway](/cody/core-concepts/cody-gateway) (recommended) or by directly using a third-party LLM provider +- First, configure your desired LLM provider either by [using Sourcegraph Cody Gateway](/cody/core-concepts/cody-gateway) or by directly using a third-party LLM provider - Next, go to **Site admin > Site configuration** (`/site-admin/configuration`) on your instance and set: ```json @@ -56,8 +56,6 @@ To turn Cody off: } ``` -- Next, remove `completions` configuration if they exist - ## Enable Cody only for some users How to enable Cody only for _some_ users depends on what version of Sourcegraph you are running. diff --git a/docs/cody/model-configuration/examples.mdx b/docs/cody/model-configuration/examples.mdx deleted file mode 100644 index 01bde7670..000000000 --- a/docs/cody/model-configuration/examples.mdx +++ /dev/null @@ -1,599 +0,0 @@ -# Examples - -## Sourcegraph-supplied models only - -This section includes examples how to configure Cody to use Sourcegraph-supplied models. - -- [Minimal configuration](/cody/model-configuration#configure-sourcegraph-supplied-models) -- [Using model filters](/cody/model-configuration#model-filters) -- [Change default models](/cody/model-configuration#default-models) - -## Sourcegraph-supplied models and BYOK (Bring Your Own Key) - -Out of the box, Sourcegraph is already aware of many models from the following providers: - -- "anthropic" -- "google" -- "fireworks" -- "mistral" -- "openai" - -### Overriding configuration of a model provider - -Instead of Sourcegraph using its own servers to make LLM requests, it is possible to bring your own API keys for a given model provider. For example, if you wish for all Anthropic API requests to go directly to your own Anthropic account and use your own API keys instead of going via Sourcegraph's servers, you could override the `anthropic` provider's configuration: - -```json -{ -"cody.enabled": true, -"modelConfiguration": { - "sourcegraph": {}, - "providerOverrides": [ - { - "id": "anthropic", - "displayName": "Anthropic BYOK", - "serverSideConfig": { - "type": "anthropic", - "accessToken": "token", - "endpoint": "https://api.anthropic.com/v1/messages" - } - } - ], - "defaultModels": { - "chat": "anthropic::2024-10-22::claude-3.5-sonnet", - "fastChat": "anthropic::2023-06-01::claude-3-haiku", - "autocomplete": "fireworks::v1::deepseek-coder-v2-lite-base" - } -} -``` - -In the configuration above, we: - -- Enable Sourcegraph-supplied models and do not set any overrides (note that `"modelConfiguration.modelOverrides"` is not specified). -- Route requests for Anthropic models directly to the Anthropic API (via the provider override specified for "anthropic"). -- Route requests for other models (such as the Fireworks model for "autocomplete") through Cody Gateway. - -### Override provider config for some models in the namespace and use the Sourcegraph-configured provider config for the rest - -It's possible to route requests directly to the LLM provider (bypassing the Cody Gateway) for some models while using the -Sourcegraph-configured provider config for the rest. - -Example configuration: - -In the configuration above, we: - -- Enable Sourcegraph-supplied models (the `sourcegraph` field is not empty or `null`). -- Define a new provider with the ID `"anthropic-byok"` and configure it to use the Anthropic API. -- Since this provider is unknown to Sourcegraph, no Sourcegraph-supplied models are available for it. - Therefore, we add a custom model in the `"modelOverrides"` section. -- Use the custom model configured in the previous step (`"anthropic-byok::2024-10-22::claude-3.5-sonnet"`) for `"chat"`. - Requests are sent directly to the Anthropic API as set in the provider override. -- For `"fastChat"` and `"autocomplete"`, we use Sourcegraph-supplied models via Cody Gateway. - -## Config examples for various LLM providers - -Below are configuration examples for setting up various LLM providers using BYOK. -These examples are applicable whether or not you are using Sourcegraph-supported models. - -**Note:** - -- In this section, all configuration examples have Sourcegraph-supplied models disabled. To use a combination of - Sourcegraph-supplied models and BYOK, please refer to the previous section. -- Ensure that at least one model is available for each Cody feature ("chat" and "autocomplete"), regardless of - the provider and model overrides configured. To verify this, [view the configuration](/cody/model-configuration#view-configuration) - and confirm that appropriate models are listed in the `"defaultModels"` section. - - - -```json -{ -"cody.enabled": true, -"modelConfiguration": { - "sourcegraph": null, - "providerOverrides": [ - { - "id": "anthropic", - "displayName": "Anthropic", - "serverSideConfig": { - "type": "anthropic", - "accessToken": "token", - "endpoint": "https://api.anthropic.com/v1/messages" - } - } - ], - "modelOverrides": [ - { - "modelRef": "anthropic::2024-10-22::claude-3.5-sonnet", - "displayName": "Claude 3.5 Sonnet", - "modelName": "claude-3-5-sonnet-latest", - "capabilities": ["chat"], - "category": "accuracy", - "status": "stable", - "contextWindow": { - "maxInputTokens": 45000, - "maxOutputTokens": 4000 - } - }, - { - "modelRef": "anthropic::2023-06-01::claude-3-haiku", - "displayName": "Claude 3 Haiku", - "modelName": "claude-3-haiku-20240307", - "capabilities": ["chat"], - "category": "speed", - "status": "stable", - "contextWindow": { - "maxInputTokens": 7000, - "maxOutputTokens": 4000 - } - }, - { - "modelRef": "anthropic::2023-01-01::claude-instant-1.2", - "displayName": "Claude Instant", - "modelName": "claude-instant-1.2", - "capabilities": ["autocomplete", "chat"], - "category": "other", - "status": "deprecated", - "contextWindow": { - "maxInputTokens": 7000, - "maxOutputTokens": 4000 - } - } - ], - "defaultModels": { - "chat": "anthropic::2024-10-22::claude-3.5-sonnet", - "fastChat": "anthropic::2023-06-01::claude-3-haiku", - "autocomplete": "anthropic::2023-01-01::claude-instant-1.2" - } -} -``` - -In the configuration above, we: - - - Set up a provider override for Anthropic, routing requests for this provider directly to the specified Anthropic endpoint (bypassing Cody Gateway). - - Add three Anthropic models: - - Two models with chat capabilities (`"anthropic::2024-10-22::claude-3.5-sonnet"` and `"anthropic::2023-06-01::claude-3-haiku"`), - providing options for chat users. - - One model with autocomplete capability (`"anthropic::2023-01-01::claude-instant-1.2"`). - - Set the configured models as default models for Cody features in the `"defaultModels"` field. - - - - - -```json -"modelConfiguration": { - "sourcegraph": null, - "providerOverrides": [ - { - "id": "openai", - "displayName": "OpenAI", - "serverSideConfig": { - "type": "openai", - "accessToken": "token", - "endpoint": "https://api.openai.com" - } - } - ], - "modelOverrides": [ - { - "modelRef": "openai::2024-02-01::gpt-4o", - "displayName": "GPT-4o", - "modelName": "gpt-4o", - "capabilities": ["chat"], - "category": "accuracy", - "status": "stable", - "contextWindow": { - "maxInputTokens": 45000, - "maxOutputTokens": 400 - } - }, - { - "modelRef": "openai::unknown::gpt-3.5-turbo-instruct", - "displayName": "GPT-3.5 Turbo Instruct", - "modelName": "gpt-3.5-turbo-instruct", - "capabilities": ["autocomplete"], - "category": "speed", - "status": "stable", - "contextWindow": { - "maxInputTokens": 7000, - "maxOutputTokens": 400 - } - } -], - "defaultModels": { - "chat": "openai::2024-02-01::gpt-4o", - "fastChat": "openai::2024-02-01::gpt-4o", - "autocomplete": "openai::unknown::gpt-3.5-turbo-instruct" - } -} -``` - -In the configuration above, we: - -- Set up a provider override for OpenAI, routing requests for this provider directly to the specified OpenAI endpoint (bypassing Cody Gateway). -- Add two OpenAI models: - - `"openai::2024-02-01::gpt-4o"` with "chat" capabilities - used for "chat" and "fastChat" - - `"openai::unknown::gpt-3.5-turbo-instruct"` with "autocomplete" capability - used for "autocomplete". - - - - - -```json -"cody.enabled": true, -"modelConfiguration": { - "sourcegraph": null, - "providerOverrides": [ - { - "id": "azure-openai", - "displayName": "Azure OpenAI", - "serverSideConfig": { - "type": "azureOpenAI", - "accessToken": "token", - "endpoint": "https://acme-test.openai.azure.com/", - "user": "", - "useDeprecatedCompletionsAPI": true - } - } - ], - "modelOverrides": [ - { - "modelRef": "azure-openai::unknown::gpt-4o", - "displayName": "GPT-4o", - "modelName": "gpt-4o", - "capabilities": ["chat"], - "category": "accuracy", - "status": "stable", - "contextWindow": { - "maxInputTokens": 45000, - "maxOutputTokens": 4000 - } - }, - { - "modelRef": "azure-openai::unknown::gpt-35-turbo-instruct-test", - "displayName": "GPT-3.5 Turbo Instruct", - "modelName": "gpt-35-turbo-instruct-test", - "capabilities": ["autocomplete"], - "category": "speed", - "status": "stable", - "contextWindow": { - "maxInputTokens": 7000, - "maxOutputTokens": 4000 - } - } - ], - "defaultModels": { - "chat": "azure-openai::unknown::gpt-4o", - "fastChat": "azure-openai::unknown::gpt-4o", - "autocomplete": "azure-openai::unknown::gpt-35-turbo-instruct-test" - } -} -``` - -In the configuration above, we: - -- Set up a provider override for Azure OpenAI, routing requests for this provider directly to the specified Azure OpenAI endpoint (bypassing Cody Gateway). -- Add two OpenAI models: - - `"azure-openai::unknown::gpt-4o"` with "chat" capability - used for "chat" and "fastChat" - - `"azure-openai::unknown::gpt-35-turbo-instruct-test"` with "autocomplete" capability - used for "autocomplete". -- Since `"azure-openai::unknown::gpt-35-turbo-instruct-test"` is not supported on the newer OpenAI `"v1/chat/completions"` endpoint, - we set `"useDeprecatedCompletionsAPI"` to `true` to route requests to the legacy `"v1/completions"` endpoint. This setting is - unnecessary if you are using a model supported on the `"v1/chat/completions"` endpoint. - - - - -```json -"cody.enabled": true, -"modelConfiguration": { - "sourcegraph": null, - "providerOverrides": [ - { - "id": "fireworks", - "displayName": "Fireworks", - "serverSideConfig": { - "type": "fireworks", - "accessToken": "token", - "endpoint": "https://api.fireworks.ai/inference/v1/completions" - } - } - ], - "modelOverrides": [ - { - "modelRef": "fireworks::v1::mixtral-8x22b-instruct", - "displayName": "Mixtral 8x22B", - "modelName": "accounts/fireworks/models/mixtral-8x22b-instruct", - "capabilities": ["chat"], - "category": "other", - "status": "stable", - "contextWindow": { - "maxInputTokens": 7000, - "maxOutputTokens": 4000 - } - }, - { - "modelRef": "fireworks::v1::starcoder-16b", - "modelName": "accounts/fireworks/models/starcoder-16b", - "displayName": "(Fireworks) Starcoder 16B", - "contextWindow": { - "maxInputTokens": 8192, - "maxOutputTokens": 4096 - }, - "capabilities": ["autocomplete"], - "category": "balanced", - "status": "stable" - } - ], - "defaultModels": { - "chat": "fireworks::v1::mixtral-8x22b-instruct", - "fastChat": "fireworks::v1::mixtral-8x22b-instruct", - "autocomplete": "fireworks::v1::starcoder-16b" - } -} -``` - -In the configuration above, we: - -- Set up a provider override for Fireworks, routing requests for this provider directly to the specified Fireworks endpoint (bypassing Cody Gateway). -- Add two Fireworks models: - - `"fireworks::v1::mixtral-8x22b-instruct"` with "chat" capabiity - used for "chat" - and "fastChat" - - `"fireworks::v1::starcoder-16b"` with "autocomplete" capability - used for "autocomplete". - - - - - -```json -"cody.enabled": true, -"modelConfiguration": { - "sourcegraph": null, - "providerOverrides": [ - { - "id": "fireworks", - "displayName": "Fireworks", - "serverSideConfig": { - "type": "openaicompatible", - "endpoints": [ - { - "url": "https://api.fireworks.ai/inference/v1", - "accessToken": "token" - } - ] - } - }, - { - "id": "huggingface-codellama", - "displayName": "Hugging Face", - "serverSideConfig": { - "type": "openaicompatible", - "endpoints": [ - { - "url": "https://api-inference.huggingface.co/models/meta-llama/CodeLlama-7b-hf/v1/", - "accessToken": "token" - } - ] - } - }, - ], - "modelOverrides": [ - { - "modelRef": "fireworks::v1::llama-v3p1-70b-instruct", - "modelName": "llama-v3p1-70b-instruct", - "displayName": "(Fireworks) Llama 3.1 70B Instruct", - "contextWindow": { - "maxInputTokens": 64000, - "maxOutputTokens": 8192 - }, - "serverSideConfig": { - "type": "openaicompatible", - "apiModel": "accounts/fireworks/models/llama-v3p1-70b-instruct" - }, - "clientSideConfig": { - "openaicompatible": {} - }, - "capabilities": ["chat"], - "category": "balanced", - "status": "stable" - }, - { - "modelRef": "huggingface-codellama::v1::CodeLlama-7b-hf", - "modelName": "CodeLlama-7b-hf", - "displayName": "(HuggingFace) CodeLlama-7b-hf", - "contextWindow": { - "maxInputTokens": 8192, - "maxOutputTokens": 4096 - }, - "serverSideConfig": { - "type": "openaicompatible", - "apiModel": "meta-llama/CodeLlama-7b-hf" - }, - "clientSideConfig": { - "openaicompatible": {} - }, - "capabilities": ["autocomplete", "chat"], - "category": "balanced", - "status": "stable" - } - ], - "defaultModels": { - "chat": "fireworks::v1::llama-v3p1-70b-instruct", - "fastChat": "fireworks::v1::llama-v3p1-70b-instruct", - "autocomplete": "huggingface-codellama::v1::CodeLlama-7b-hf" - } -} -``` - -In the configuration above, we: - -- Configure two OpenAI-compatible providers: `"fireworks"` and `"huggingface-codellama"`. -- Add two OpenAI-compatible models: `"fireworks::v1::llama-v3p1-70b-instruct"` and `"huggingface-codellama::v1::CodeLlama-7b-hf"`. Additionally: - - Set `clientSideConfig.openaicompatible` to `{}` to indicate to Cody clients that these models are OpenAI-compatible, ensuring the appropriate code paths are utilized. - - Designate these models as the default choices for chat and autocomplete, respectively. - - - - - -```json -"modelConfiguration": { - "sourcegraph": null, - "providerOverrides": [ - { - "id": "google", - "displayName": "Google Anthropic", - "serverSideConfig": { - "type": "google", - "accessToken": "token", - "endpoint": "https://us-east5-aiplatform.googleapis.com/v1/projects/project-name/locations/us-east5/publishers/anthropic/models" - } - } - ], - "modelOverrides": [ - { - "modelRef": "google::unknown::claude-3-5-sonnet", - "displayName": "Claude 3.5 Sonnet (via Google/Vertex)", - "modelName": "claude-3-5-sonnet@20240620", - "contextWindow": { - "maxInputTokens": 45000, - "maxOutputTokens": 4000 - }, - "capabilities": ["chat"], - "category": "accuracy", - "status": "stable" - }, - { - "modelRef": "google::unknown::claude-3-haiku", - "displayName": "Claude 3 Haiku", - "modelName": "claude-3-haiku@20240307", - "capabilities": ["autocomplete", "chat"], - "category": "speed", - "status": "stable", - "contextWindow": { - "maxInputTokens": 7000, - "maxOutputTokens": 4000 - } - }, - ], - "defaultModels": { - "chat": "google::unknown::claude-3-5-sonnet", - "fastChat": "google::unknown::claude-3-5-sonnet", - "autocomplete": "google::unknown::claude-3-haiku" - } -} -``` - -In the configuration above, we: - -- Set up a provider override for Google Anthropic, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway). -- Add two Anthropic models: - - `"google::unknown::claude-3-5-sonnet"` with "chat" capabiity - used for "chat" and "fastChat" - - `"google::unknown::claude-3-haiku"` with "autocomplete" capability - used for "autocomplete". - - - - - -```json -"modelConfiguration": { - "sourcegraph": null, - "providerOverrides": [ - { - "id": "google", - "displayName": "Google Gemini", - "serverSideConfig": { - "type": "google", - "accessToken": "token", - "endpoint": "https://generativelanguage.googleapis.com/v1beta/models" - } - } - ], - "modelOverrides": [ - { - "modelRef": "google::v1::gemini-1.5-pro", - "displayName": "Gemini 1.5 Pro", - "modelName": "gemini-1.5-pro", - "capabilities": ["chat", "autocomplete"], - "category": "balanced", - "status": "stable", - "contextWindow": { - "maxInputTokens": 45000, - "maxOutputTokens": 400 - } - } - ], - "defaultModels": { - "chat": "google::v1::gemini-1.5-pro", - "fastChat": "google::v1::gemini-1.5-pro", - "autocomplete": "google::v1::gemini-1.5-pro" - } -} -``` - -In the configuration above, we: - -- Set up a provider override for Google Gemini, routing requests for this provider directly to the specified endpoint (bypassing Cody Gateway). -- Add the `"google::v1::gemini-1.5-pro"` model, which is used for all Cody features. - We do not add other models for simplicity, as adding multiple models is already covered in the examples above. - - - - - -```json -"cody.enabled": true, -"modelConfiguration": { - "sourcegraph": null, - "providerOverrides": [ - { - "id": "aws-bedrock", - "displayName": "AWS Bedrock", - "serverSideConfig": { - "type": "awsBedrock", - "accessToken": "token", - "endpoint": "us-west-2", - "region": "us-west-2" - } - } - ], - "modelOverrides": [ - { - "modelRef": "aws-bedrock::2023-06-01::claude-3-opus", - "displayName": "Claude 3 Opus (AWS Bedrock)", - "modelName": "anthropic.claude-3-opus-20240229-v1:0", - "capabilities": ["chat", "autocomplete"], - "category": "other", - "status": "stable", - "contextWindow": { - "maxInputTokens": 45000, - "maxOutputTokens": 4000 - } - } - ], - "defaultModels": { - "chat": "aws-bedrock::2023-06-01::claude-3-opus", - "fastChat": "aws-bedrock::2023-06-01::claude-3-opus", - "autocomplete": "aws-bedrock::2023-06-01::claude-3-opus", - } -} -``` - -In the configuration described above, we: - -- Set up a provider override for AWS Bedrock, routing requests for this provider directly to the specified endpoint, - bypassing Cody Gateway. -- Add the `"aws-bedrock::2023-06-01::claude-3-opus"` model, which is used for all Cody features. - We do not add other models for simplicity, as adding multiple models is already covered in the examples above. - -Provider override `serverSideConfig` fields: - -| Field | Description | -| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `type` | Must be `"awsBedrock"`. | -| `accessToken` | Leave empty to rely on instance role bindings or other AWS configurations in the frontend service. Use `:` for direct credential configuration, or `::` if a session token is also required. | -| `endpoint` | For Pay-as-you-go, set it to an AWS region code (e.g., `us-west-2`) when using a public Amazon Bedrock endpoint. For Provisioned Throughput, set it to the provisioned VPC endpoint for the bedrock-runtime API (e.g., `https://vpce-0a10b2345cd67e89f-abc0defg.bedrock-runtime.us-west-2.vpce.amazonaws.com`). | -| `region` | The region to use when configuring API clients. This is necessary because the 'frontend' binary's container cannot access environment variables from the host OS. | - -Provisioned throughput for AWS Bedrock models can be configured using the `"awsBedrockProvisionedThroughput"` server-side -configuration type. For more details, refer to the [Model Overrides](/cody/model-configuration#model-overrides) section. - -