From 2172f9efbea196f35ef0152d4ed29c3a04537dcd Mon Sep 17 00:00:00 2001 From: aevesdocker Date: Tue, 1 Jul 2025 11:23:43 +0100 Subject: [PATCH 1/4] models in Compose apps --- content/manuals/ai/compose/_index.md | 6 + .../manuals/ai/compose/models-and-compose.md | 183 ++++++++++++++++++ 2 files changed, 189 insertions(+) create mode 100644 content/manuals/ai/compose/_index.md create mode 100644 content/manuals/ai/compose/models-and-compose.md diff --git a/content/manuals/ai/compose/_index.md b/content/manuals/ai/compose/_index.md new file mode 100644 index 000000000000..dfc04cd2f2d7 --- /dev/null +++ b/content/manuals/ai/compose/_index.md @@ -0,0 +1,6 @@ +--- +build: + render: never +title: AI and Docker Compose +weight: 40 +--- \ No newline at end of file diff --git a/content/manuals/ai/compose/models-and-compose.md b/content/manuals/ai/compose/models-and-compose.md new file mode 100644 index 000000000000..fe08fdeccbef --- /dev/null +++ b/content/manuals/ai/compose/models-and-compose.md @@ -0,0 +1,183 @@ +--- +title: Define models in Compose applications +description: Learn how to define and use AI models in Docker Compose applications using the models top-level element +keywords: compose, docker compose, models, ai, machine learning, cloud providers, specification +weight: 10 +params: + sidebar: + badge: + color: green + text: New +--- + +{{< summary-bar feature_name="Compose models" >}} + +Compose includes support for defining AI models as first-class resources in your applications. This lets you declare model dependencies alongside your services, making your applications portable across different platforms that support the Compose Specification. + +## Prerequisites + +- Docker Compose v2.38 or later +- A platform that supports Compose models such as Docker Model Runner or compatible cloud providers + +## What are Compose models? + +Compose `models` are a standardized way to define AI model dependencies in your application. By using the `models` top-level element in your Compose file, you can: + +- Declare which AI models your application needs +- Specify model configurations and requirements +- Make your application portable across different platforms +- Let the platform handle model provisioning and lifecycle management + +## Basic model definition + +To define models in your Compose application, use the `models` top-level element: + +```yaml +services: + chat-app: + image: my-chat-app + models: + - llm + +models: + llm: + image: ai/smollm2 +``` + +This example defines: +- A service called `chat-app` that uses a model named `llm` +- A model definition for `llm` that references the `ai/smollm2` model image + +## Model configuration options + +Models support various configuration options: + +```yaml +models: + llm: + image: ai/smollm2 + context_size: 1024 + runtime_flags: + - "--a-flag" + - "--another-flag=42" +``` + +Common configuration options include: +- `model` (required): The OCI artifact identifier for the model. This is what Compose pulls and runs via the model runner. +- `context_size`: Defines the maximum token context size for the model. +- `runtime_flags`: A list of raw command-line flags passed to the inference engine when the model is started. +- Platform-specific options may also be available via extensions attributes `x-*` + +## Service model binding + +Services can reference models in two ways: short syntax and long syntax. + +### Short syntax + +The short syntax is the simplest way to bind a model to a service: + +```yaml +services: + app: + image: my-app + models: + - llm + - embedding-model + +models: + llm: + image: ai/smollm2 + embedding-model: + image: ai/all-minilm +``` + +With short syntax, the platform automatically generates environment variables based on the model name: +- `LLM_URL` - URL to access the llm model +- `LLM_MODEL` - Model identifier for the llm model +- `EMBEDDING_MODEL_URL` - URL to access the embedding-model +- `EMBEDDING_MODEL_MODEL` - Model identifier for the embedding-model + +### Long syntax + +The long syntax allows you to customize environment variable names: + +```yaml +services: + app: + image: my-app + models: + llm: + endpoint_var: AI_MODEL_URL + model_var: AI_MODEL_NAME + embedding-model: + endpoint_var: EMBEDDING_URL + model_var: EMBEDDING_NAME + +models: + llm: + image: ai/smollm2 + embedding-model: + image: ai/all-minilm +``` + +With this configuration, your service receives: +- `AI_MODEL_URL` and `AI_MODEL_NAME` for the LLM model +- `EMBEDDING_URL` and `EMBEDDING_NAME` for the embedding model + +## Platform portability + +One of the key benefits of using Compose models is portability across different platforms that support the Compose specification. + +### Docker Model Runner + +When Docker Model Runner is enabled: + +```yaml +services: + chat-app: + image: my-chat-app + models: + - llm + +models: + llm: + image: ai/smollm2 +``` + +Docker Model Runner will: +- Pull and run the specified model locally +- Provide endpoint URLs for accessing the model +- Inject environment variables into the service + +### Cloud providers + +The same Compose file can run on cloud providers that support Compose models: + +```yaml +services: + chat-app: + image: my-chat-app + models: + - llm + +models: + llm: + image: ai/smollm2 + # Cloud-specific configurations + labels: + - "cloud.instance-type=gpu-small" + - "cloud.region=us-west-2" +``` + +Cloud providers might: +- Use managed AI services instead of running models locally +- Apply cloud-specific optimizations and scaling +- Provide additional monitoring and logging capabilities +- Handle model versioning and updates automatically + +## Reference + +- [`models` top-level element](/reference/compose-file/models.md) +- [`models` attribute](/reference/compose-file/services.md#models) +- [Docker Model Runner documentation](/manuals/ai/model-runner.md) +- [Compose Model Runner documentation](/manuals/compose/how-tos/model-runner.md)] \ No newline at end of file From feb04579f99df2df9e95a348f22531bbde44bd08 Mon Sep 17 00:00:00 2001 From: aevesdocker Date: Tue, 1 Jul 2025 11:32:11 +0100 Subject: [PATCH 2/4] edits --- content/manuals/ai/compose/_index.md | 3 +++ content/manuals/ai/compose/models-and-compose.md | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/content/manuals/ai/compose/_index.md b/content/manuals/ai/compose/_index.md index dfc04cd2f2d7..a861d426a60e 100644 --- a/content/manuals/ai/compose/_index.md +++ b/content/manuals/ai/compose/_index.md @@ -3,4 +3,7 @@ build: render: never title: AI and Docker Compose weight: 40 +params: + sidebar: + group: AI --- \ No newline at end of file diff --git a/content/manuals/ai/compose/models-and-compose.md b/content/manuals/ai/compose/models-and-compose.md index fe08fdeccbef..63251ef98b69 100644 --- a/content/manuals/ai/compose/models-and-compose.md +++ b/content/manuals/ai/compose/models-and-compose.md @@ -1,5 +1,5 @@ --- -title: Define models in Compose applications +title: Define AI Models in Docker Compose applications description: Learn how to define and use AI models in Docker Compose applications using the models top-level element keywords: compose, docker compose, models, ai, machine learning, cloud providers, specification weight: 10 @@ -12,7 +12,7 @@ params: {{< summary-bar feature_name="Compose models" >}} -Compose includes support for defining AI models as first-class resources in your applications. This lets you declare model dependencies alongside your services, making your applications portable across different platforms that support the Compose Specification. +Compose lets you define AI models as core components of your application, so you can declare model dependencies alongside services and run the application on any platform that supports the Compose Specification. ## Prerequisites From 4bccb7f6ad6e410756f4a8b58f92c61c2f851e15 Mon Sep 17 00:00:00 2001 From: aevesdocker Date: Tue, 1 Jul 2025 11:35:08 +0100 Subject: [PATCH 3/4] edits --- content/manuals/ai/compose/models-and-compose.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/manuals/ai/compose/models-and-compose.md b/content/manuals/ai/compose/models-and-compose.md index 63251ef98b69..106e597fa6cc 100644 --- a/content/manuals/ai/compose/models-and-compose.md +++ b/content/manuals/ai/compose/models-and-compose.md @@ -1,5 +1,6 @@ --- title: Define AI Models in Docker Compose applications +linkTitle: Use AI models in Compose description: Learn how to define and use AI models in Docker Compose applications using the models top-level element keywords: compose, docker compose, models, ai, machine learning, cloud providers, specification weight: 10 From 34b6fe2fc68462d05762406ed0cceecfb7402dd8 Mon Sep 17 00:00:00 2001 From: aevesdocker Date: Tue, 1 Jul 2025 11:42:59 +0100 Subject: [PATCH 4/4] add link --- content/manuals/ai/compose/models-and-compose.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/manuals/ai/compose/models-and-compose.md b/content/manuals/ai/compose/models-and-compose.md index 106e597fa6cc..bdfb0de5eb65 100644 --- a/content/manuals/ai/compose/models-and-compose.md +++ b/content/manuals/ai/compose/models-and-compose.md @@ -22,7 +22,7 @@ Compose lets you define AI models as core components of your application, so you ## What are Compose models? -Compose `models` are a standardized way to define AI model dependencies in your application. By using the `models` top-level element in your Compose file, you can: +Compose `models` are a standardized way to define AI model dependencies in your application. By using the []`models` top-level element](/reference/compose-file/models.md) in your Compose file, you can: - Declare which AI models your application needs - Specify model configurations and requirements