From c02a2b6c6a6b94b8d6507029df57f4d308862d2e Mon Sep 17 00:00:00 2001 From: Lionello Lunesu Date: Tue, 13 May 2025 17:49:01 -0700 Subject: [PATCH] Add provider model docs --- .../managed-llms/managed-language-models.md | 4 +-- .../managed-llms/openai-access-gateway.md | 32 ++++++++++++++++--- 2 files changed, 30 insertions(+), 6 deletions(-) diff --git a/docs/concepts/managed-llms/managed-language-models.md b/docs/concepts/managed-llms/managed-language-models.md index 3acc01efa..570adb028 100644 --- a/docs/concepts/managed-llms/managed-language-models.md +++ b/docs/concepts/managed-llms/managed-language-models.md @@ -1,5 +1,5 @@ --- -title: Leveraging Managed Language Models with Defang +title: Using Managed LLMs description: Defang makes it easy to leverage cloud-native managed language models. sidebar_position: 3000 --- @@ -32,7 +32,7 @@ If you already have an OpenAI-compatible application, Defang makes it easy to de | Provider | Managed Language Models | | --- | --- | -| [Playground](/docs/providers/playground#managed-large-language-models) | ❌ | +| [Playground](/docs/providers/playground#managed-large-language-models) | ✅ | | [AWS Bedrock](/docs/providers/aws#managed-large-language-models) | ✅ | | [DigitalOcean GenAI](/docs/providers/digitalocean#future-improvements) | ❌ | | [GCP Vertex AI](/docs/providers/gcp#managed-large-language-models) | ✅ | diff --git a/docs/concepts/managed-llms/openai-access-gateway.md b/docs/concepts/managed-llms/openai-access-gateway.md index e6c940e5c..2b0b6b079 100644 --- a/docs/concepts/managed-llms/openai-access-gateway.md +++ b/docs/concepts/managed-llms/openai-access-gateway.md @@ -1,20 +1,44 @@ --- -title: Deploying OpenAI-compatible apps with Defang +title: OpenAI Access Gateway description: Defang makes it easy to leverage cloud-native managed language models for your OpenAI-compatible application. sidebar_position: 3000 --- -# Deploying OpenAI-compatible applications to cloud-native managed language models with Defang +# OpenAI Access Gateway -Defang makes it easy to deploy on your favourite cloud's managed LLM service with our [OpenAI Access Gateway](https://github.com/DefangLabs/openai-access-gateway). This service sits between your application and the cloud service and acts as a compatibility layer. It handles incoming OpenAI requests, translates those requests to the appropriate cloud-native API, handles the native response, and re-constructs an OpenAI-compatible response. +Defang makes it easy to deploy on your favourite cloud's managed LLM service with our [OpenAI Access Gateway](https://github.com/DefangLabs/openai-access-gateway). This service sits between your application and the cloud service and acts as a compatibility layer. +It handles incoming OpenAI requests, translates those requests to the appropriate cloud-native API, handles the native response, and re-constructs an OpenAI-compatible response. See [our tutorial](/docs/tutorials/deploying-openai-apps-aws-bedrock-gcp-vertex/) which describes how to configure the OpenAI Access Gateway for your application +## Docker Provider Services + +As of Docker Compose v2.35 and Docker Desktop v4.41, Compose introduces a new service type called `provider` that allows you to declare platform capabilities required by your application. +For AI models, you use the `model` type to declare model dependencies. This will expose an OpenAI compatible API for your service. Check the [Docker Model Runner documentation](https://docs.docker.com/compose/how-tos/model-runner/) for more details. + +```yaml +services: + chat: + build: . + depends_on: + - ai_runner + + ai_runner: + provider: + type: model + options: + model: ai/mistral + x-defang-llm: true +``` + +Under the hood, when you use the `model` provider, Defang will deploy the **OpenAI Access Gateway** in a private network. This allows you to use the same code for both local development and cloud deployment. +The `x-defang-llm` extension is used to configure the appropriate roles and permissions for your service. See the [Managed Language Models](/docs/concepts/managed-llms/managed-language-models/) page for more details. + ## Current Support | Provider | Managed Language Models | | --- | --- | -| [Playground](/docs/providers/playground#managed-services) | ❌ | +| [Playground](/docs/providers/playground#managed-services) | ✅ | | [AWS Bedrock](/docs/providers/aws#managed-llms) | ✅ | | [DigitalOcean GenAI](/docs/providers/digitalocean#future-improvements) | ❌ | | [GCP Vertex AI](/docs/providers/gcp#managed-llms) | ✅ |