From f120e8300d4d9e33af8399850f7dcb00ee8f5f74 Mon Sep 17 00:00:00 2001 From: Meygha Machado Date: Tue, 14 Jul 2026 21:36:36 -0700 Subject: [PATCH 1/5] shared-compute implementation to support multiple GPU SKUs --- CHANGELOG.md | 12 + api/hastefuncapi/requirements.txt | 2 +- api/hastefuncqueues/requirements.txt | 2 +- docker/imageryprep/requirements.txt | 2 +- docs/configuration.md | 50 +++- docs/deployment.md | 11 +- hastelib/pyproject.toml | 5 +- hastelib/src/hastegeo/__about__.py | 2 +- hastelib/src/hastegeo/core/config.py | 34 +++ .../src/hastegeo/core/processors/artifacts.py | 3 + .../src/hastegeo/core/processors/embedding.py | 3 + .../src/hastegeo/core/processors/imagery.py | 3 + .../src/hastegeo/core/processors/inference.py | 3 + .../src/hastegeo/core/processors/train.py | 3 + .../src/hastegeo/core/runners/azure_batch.py | 185 ++++++++++++--- hastelib/src/hastegeo/core/runners/local.py | 6 +- hastelib/tests/core/runners/__init__.py | 2 + .../core/runners/test_azure_batch_routing.py | 129 +++++++++++ infra/main.bicep | 2 +- infra/main.bicepparam | 2 +- infra/modules/batchPool.bicep | 69 ++++-- infra/modules/functionApp.bicep | 16 ++ infra/shared-pools.bicep | 79 +++++++ infra/shared-pools.bicepparam | 22 ++ setup/README.md | 4 +- .../batch-compute-expansion/README.md | 114 +++++++++ .../batch-compute-expansion/design.md | 206 +++++++++++++++++ .../impact-analysis.md | 93 ++++++++ spec/features/batch-compute-expansion/plan.md | 108 +++++++++ .../batch-compute-expansion/rollout.md | 118 ++++++++++ .../batch-compute-expansion/test-plan.md | 104 +++++++++ .../batch-compute-expansion/user-stories.md | 216 ++++++++++++++++++ 32 files changed, 1552 insertions(+), 58 deletions(-) create mode 100644 hastelib/tests/core/runners/__init__.py create mode 100644 hastelib/tests/core/runners/test_azure_batch_routing.py create mode 100644 infra/shared-pools.bicep create mode 100644 infra/shared-pools.bicepparam create mode 100644 spec/features/batch-compute-expansion/README.md create mode 100644 spec/features/batch-compute-expansion/design.md create mode 100644 spec/features/batch-compute-expansion/impact-analysis.md create mode 100644 spec/features/batch-compute-expansion/plan.md create mode 100644 spec/features/batch-compute-expansion/rollout.md create mode 100644 spec/features/batch-compute-expansion/test-plan.md create mode 100644 spec/features/batch-compute-expansion/user-stories.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 298d29c..510dcad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,18 @@ Versioning follows the Docker image tags defined in the CI workflows (see [.gith --- +## [Unreleased] + +### Added +- **Shared multi-tenant GPU Batch pools** — For deployments running many environments against scarce GPU quota, HASTE can now share a small set of multi-tenant Batch pools (H100 for training, T4 for inference/imageryprep + spillover) instead of one pool per environment. Data isolation is enforced at the credential boundary: each job mints a short-lived **user-delegation SAS** scoped to its own storage container (the pool identity is used only for ACR pull and holds no storage access), so a tenant's task can never read another tenant's data. Pools autoscale on low-priority nodes and scale to zero when idle. New `hastelib` routing picks a pool from an ordered candidate list **at submit time** (first with an idle node, else the preferred pool). Provisioned by the standalone [`infra/shared-pools.bicep`](infra/shared-pools.bicep) + [`shared-pools.bicepparam`](infra/shared-pools.bicepparam); opted into per environment via `AZURE_BATCH_*_POOL_IDS`, `AZURE_BATCH_USE_SAS`, and `AZURE_BATCH_MANAGE_POOLS` (all default to the legacy single-pool behavior). Full design in [`spec/features/batch-compute-expansion/`](spec/features/batch-compute-expansion). See [docs/configuration.md](docs/configuration.md#shared-multi-tenant-gpu-pools). + +### Changed +- **`infra/modules/batchPool.bicep` parameterized** — one module now serves fixed-dedicated (dev/prod) and autoscale-low-priority (shared) pools via `scaleMode` / `nodeType` / `minNodes` params, with optional VNet injection. Backward-compatible defaults. +- **Generic-default IaC for reuse by other partners** — `HASTE_RESOURCE_PREFIX` now defaults to the neutral `haste` (overridable per deployment); the shared-pools template keeps its account/ACR as bring-your-own params. The `api`/`queues` Function App identity is granted **Storage Blob Delegator** (in `functionApp.bicep`) so it can mint user-delegation SAS. +- **Pinned `azure-batch==14.2.0`** — the 15.x track-2 rewrite restructures the batch models this code uses; migration is tracked separately. + +--- + ## [v2.0.0] — Building labeling workflow & one-step `azd` setup ### Added diff --git a/api/hastefuncapi/requirements.txt b/api/hastefuncapi/requirements.txt index b78539a..3d399cb 100644 --- a/api/hastefuncapi/requirements.txt +++ b/api/hastefuncapi/requirements.txt @@ -29,5 +29,5 @@ tenacity==9.1.2 typing-extensions==4.14.1 docker==7.1.0 # Use wheel for remote, comment out for local Docker (source is copied directly) -hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.26-py3-none-any.whl +hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.27-py3-none-any.whl # -e ../../hastelib # for local development only diff --git a/api/hastefuncqueues/requirements.txt b/api/hastefuncqueues/requirements.txt index b78539a..3d399cb 100644 --- a/api/hastefuncqueues/requirements.txt +++ b/api/hastefuncqueues/requirements.txt @@ -29,5 +29,5 @@ tenacity==9.1.2 typing-extensions==4.14.1 docker==7.1.0 # Use wheel for remote, comment out for local Docker (source is copied directly) -hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.26-py3-none-any.whl +hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.27-py3-none-any.whl # -e ../../hastelib # for local development only diff --git a/docker/imageryprep/requirements.txt b/docker/imageryprep/requirements.txt index ff9b2a6..f170887 100644 --- a/docker/imageryprep/requirements.txt +++ b/docker/imageryprep/requirements.txt @@ -31,4 +31,4 @@ pyarrow==23.0.1 fiona==1.10.1 fsspec==2024.10.0 adlfs==2024.12.0 -hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.26-py3-none-any.whl +hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.27-py3-none-any.whl diff --git a/docs/configuration.md b/docs/configuration.md index b457397..02e1822 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -14,6 +14,7 @@ This guide documents each configuration mode. For the end-to-end workflow, see - [Core settings](#core-settings) - [Batch (create vs. bring-your-own)](#batch-create-vs-bring-your-own) - [Batch image tags and pool immutability](#batch-image-tags-and-pool-immutability) +- [Shared multi-tenant GPU pools](#shared-multi-tenant-gpu-pools) - [Email sender domain](#email-sender-domain) - [Front Door](#front-door) - [Development mode](#development-mode) @@ -24,7 +25,7 @@ This guide documents each configuration mode. For the end-to-end workflow, see | Variable | Default | Purpose | |---|---|---| -| `HASTE_RESOURCE_PREFIX` | `ai4gl` | Prefix for all resource names. | +| `HASTE_RESOURCE_PREFIX` | `haste` | Prefix for all resource names. Generic default; override per deployment. | | `HASTE_RANDOM_SUFFIX` | `dev1` | Per-environment suffix; keeps names unique. | | `AZURE_LOCATION` | `westus2` | Azure region. | | `HASTE_APIM_PUBLISHER_EMAIL` | — | APIM publisher email (required). | @@ -98,6 +99,53 @@ reads the existing pool's `containerImageNames` and sets `HASTE_TRAINING_IMAGE` only in that mode and never clobbers a tag you set explicitly — set either variable yourself to override the auto-resolved value. +## Shared multi-tenant GPU pools + +For deployments that run many environments against **scarce GPU quota**, HASTE +supports a small set of **shared, multi-tenant** Batch pools (H100 for training, +T4 for inference/imageryprep + spillover) instead of one pool per environment, so +GPU quota is pooled and rationed centrally rather than fragmented. See +[`spec/features/batch-compute-expansion/`](https://github.com/microsoft/haste/tree/main/spec/features/batch-compute-expansion) +for the full design. + +**Creating the pools.** The shared pools are provisioned by +[`infra/shared-pools.bicep`](https://github.com/microsoft/haste/blob/main/infra/shared-pools.bicep) +— a standalone deployment into the shared Batch account's resource group, +separate from `azd up` — configured by +[`infra/shared-pools.bicepparam`](https://github.com/microsoft/haste/blob/main/infra/shared-pools.bicepparam): + +```bash +az deployment group create -g \ + --parameters infra/shared-pools.bicepparam +``` + +Pools are named `-shared---pool` (e.g. the dev-group H100 +pool); `HASTE_SHARED_GROUP` selects the group (`dev`, +`demo`, …). They autoscale on low-priority nodes and scale to zero when idle. The +pool identity is used **only** for ACR pull (`haste-shared-acr-umi`) — it holds no +storage access. + +**Data isolation.** Tenants share compute but not data: each job mints a +short-lived **user-delegation SAS** scoped to its own storage container, so a +tenant's task can never read another tenant's data. This requires the submitting +Function App identity to hold **Storage Blob Delegator** on its storage account +(granted in `functionApp.bicep`). + +**Per-environment app settings** (set on the `api`/`queues` Function Apps to opt +an environment into the shared pools — all default to the legacy single-pool, +pool-identity behavior, so existing environments are unaffected until opted in): + +| Variable | Default | Purpose | +|---|---|---| +| `AZURE_BATCH_TRAINING_POOL_IDS` | `AZURE_BATCH_TRAINING_POOL_ID` | Ordered candidate pools for training (preference-first, spillover-second), comma-separated. | +| `AZURE_BATCH_INFERENCE_POOL_IDS` | training pool | Ordered candidate pools for inference/embedding (e.g. T4-first). | +| `AZURE_BATCH_IMAGERYPREP_POOL_IDS` | `AZURE_BATCH_IMAGERYPREP_POOL_ID` | Ordered candidate pools for imageryprep/artifacts. | +| `AZURE_BATCH_USE_SAS` | `false` | Use per-job user-delegation SAS for blob I/O instead of the pool's managed identity. Required for shared pools. | +| `AZURE_BATCH_MANAGE_POOLS` | `true` | Whether the runner auto-creates/resizes its pool. Set `false` for pre-created autoscale pools (resize fails on an autoscale pool). | + +The runner picks a pool from the candidate list **at submit time** — the first +with an idle node, otherwise the preferred (first) pool, which scales up / queues. + ## Email sender domain The email backend (Azure Communication Services) is provisioned in-IaC, so its diff --git a/docs/deployment.md b/docs/deployment.md index 30a9244..b448872 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -27,7 +27,7 @@ az login # Create an environment and set the required configuration. azd env new dev3 -azd env set HASTE_RESOURCE_PREFIX ai4gl +azd env set HASTE_RESOURCE_PREFIX myorg azd env set HASTE_RANDOM_SUFFIX dev3 azd env set AZURE_LOCATION westus2 azd env set HASTE_APIM_PUBLISHER_EMAIL you@example.com @@ -66,6 +66,15 @@ brings up the full stack with Docker Compose. ## Production Considerations +### GPU compute at scale + +Running many environments against limited GPU quota? HASTE supports **shared, +multi-tenant Batch pools** (provisioned separately from `azd up` via +[`infra/shared-pools.bicep`](https://github.com/microsoft/haste/blob/main/infra/shared-pools.bicep)), +with per-job user-delegation SAS for tenant data isolation and capacity-aware +routing. See [Shared multi-tenant GPU pools](configuration.md#shared-multi-tenant-gpu-pools) +in the configuration guide. + ### Security For production deployments, follow the [Secure Configuration Guidance](security-configuration.md) — it covers identity and authentication setup, secrets management with managed identity and Key Vault, CORS and HTTP security headers, container hardening, logging and monitoring, and known limitations with operational mitigations. The guide also includes a pre-production checklist. diff --git a/hastelib/pyproject.toml b/hastelib/pyproject.toml index d661783..aa6adff 100644 --- a/hastelib/pyproject.toml +++ b/hastelib/pyproject.toml @@ -34,7 +34,10 @@ dependencies = [ "azure-storage-file-datalake", "azure-storage-queue", "azure-mgmt-containerregistry", - "azure-batch", + # Pinned: azure-batch 15.x is a track-2 rewrite that restructures the models + # this code uses (e.g. ComputeNodeIdentityReference). Migration tracked + # separately; keep on the last track-1 release until then. + "azure-batch==14.2.0", "azure-core", "azure-identity", "psycopg2-binary", diff --git a/hastelib/src/hastegeo/__about__.py b/hastelib/src/hastegeo/__about__.py index 4d1e3df..bbf14bb 100644 --- a/hastelib/src/hastegeo/__about__.py +++ b/hastelib/src/hastegeo/__about__.py @@ -5,4 +5,4 @@ # storage (see ../../../haste_build.py) so that source/Docker installs report # the same version that was published. Committed value tracks the latest # published release. -__version__ = "1.0.26" +__version__ = "1.0.27" diff --git a/hastelib/src/hastegeo/core/config.py b/hastelib/src/hastegeo/core/config.py index f82de69..32ffa1c 100644 --- a/hastelib/src/hastegeo/core/config.py +++ b/hastelib/src/hastegeo/core/config.py @@ -423,6 +423,16 @@ def get_azure_batch_config(): imageryprep_pool_id = os.getenv( "AZURE_BATCH_IMAGERYPREP_POOL_ID", "imageryprep-pool" ) + + def _split_ids(raw, fallback): + # Ordered candidate pool ids for capacity-aware routing (v2.1.0). + # Comma-separated env override; fall back to the single legacy id. + if raw: + ids = [p.strip() for p in raw.split(",") if p.strip()] + if ids: + return ids + return [fallback] + return { "account_name": os.getenv( "AZURE_BATCH_ACCOUNT_NAME", "" @@ -438,6 +448,30 @@ def get_azure_batch_config(): "imageprep_pool_id": os.getenv( "AZURE_BATCH_IMAGERYPREP_POOL_ID", "imageryprep-pool" ), + # Ordered candidate pools per workload (v2.1.0 capacity-aware + # routing): preference-first, spillover-second + # (e.g. AZURE_BATCH_TRAINING_POOL_IDS="h100-pool,t4-pool"). + "training_pool_ids": _split_ids( + os.getenv("AZURE_BATCH_TRAINING_POOL_IDS"), training_pool_id + ), + "inference_pool_ids": _split_ids( + os.getenv("AZURE_BATCH_INFERENCE_POOL_IDS"), training_pool_id + ), + "imageryprep_pool_ids": _split_ids( + os.getenv("AZURE_BATCH_IMAGERYPREP_POOL_IDS"), + imageryprep_pool_id, + ), + # Per-job user-delegation SAS instead of pool-identity for blob I/O + # (required for multi-tenant shared pools). Default off = legacy + # identity_reference path. + "use_sas": os.getenv("AZURE_BATCH_USE_SAS", "false").lower() + == "true", + # Whether the runner auto-creates/resizes its pool. Off for + # pre-created IaC/autoscale pools (resize fails on autoscale). + "manage_pools": os.getenv( + "AZURE_BATCH_MANAGE_POOLS", "true" + ).lower() + == "true", "registry_server": os.getenv( "AZURE_BATCH_REGISTRY_SERVER", ".azurecr.io", diff --git a/hastelib/src/hastegeo/core/processors/artifacts.py b/hastelib/src/hastegeo/core/processors/artifacts.py index 5c34026..489b306 100644 --- a/hastelib/src/hastegeo/core/processors/artifacts.py +++ b/hastelib/src/hastegeo/core/processors/artifacts.py @@ -63,6 +63,9 @@ def __init__( runner_type=self.config.runner_type, config=self.config, pool_id=self.config.get_azure_batch_config()["imageprep_pool_id"], + candidate_pool_ids=self.config.get_azure_batch_config()[ + "imageryprep_pool_ids" + ], ) self.model_artifacts = model_artifacts if self.model_data is not None: diff --git a/hastelib/src/hastegeo/core/processors/embedding.py b/hastelib/src/hastegeo/core/processors/embedding.py index e075f19..cfd18c6 100644 --- a/hastelib/src/hastegeo/core/processors/embedding.py +++ b/hastelib/src/hastegeo/core/processors/embedding.py @@ -75,6 +75,9 @@ def __init__( runner_type=config.runner_type, config=self.config, pool_id=self.config.get_azure_batch_config()["training_pool_id"], + candidate_pool_ids=self.config.get_azure_batch_config()[ + "training_pool_ids" + ], ) self.queue_client = AzureQueueHandler( config.queue_config["queue_connection_string"], diff --git a/hastelib/src/hastegeo/core/processors/imagery.py b/hastelib/src/hastegeo/core/processors/imagery.py index 0196974..24ad74f 100644 --- a/hastelib/src/hastegeo/core/processors/imagery.py +++ b/hastelib/src/hastegeo/core/processors/imagery.py @@ -181,6 +181,9 @@ def __init__(self, image_data: ImageLayer, config: Config = None): runner_type=config.runner_type, config=self.config, pool_id=self.config.get_azure_batch_config()["imageprep_pool_id"], + candidate_pool_ids=self.config.get_azure_batch_config()[ + "imageryprep_pool_ids" + ], ) self.queue = AzureQueueHandler( config.queue_config["queue_connection_string"], diff --git a/hastelib/src/hastegeo/core/processors/inference.py b/hastelib/src/hastegeo/core/processors/inference.py index 011158e..5924517 100644 --- a/hastelib/src/hastegeo/core/processors/inference.py +++ b/hastelib/src/hastegeo/core/processors/inference.py @@ -45,6 +45,9 @@ def __init__( runner_type=config.runner_type, config=self.config, pool_id=self.config.get_azure_batch_config()["training_pool_id"], + candidate_pool_ids=self.config.get_azure_batch_config()[ + "inference_pool_ids" + ], ) diff --git a/hastelib/src/hastegeo/core/processors/train.py b/hastelib/src/hastegeo/core/processors/train.py index 0eb2bec..fe18a0d 100644 --- a/hastelib/src/hastegeo/core/processors/train.py +++ b/hastelib/src/hastegeo/core/processors/train.py @@ -51,6 +51,9 @@ def __init__( runner_type=config.runner_type, config=self.config, pool_id=self.config.get_azure_batch_config()["training_pool_id"], + candidate_pool_ids=self.config.get_azure_batch_config()[ + "training_pool_ids" + ], ) diff --git a/hastelib/src/hastegeo/core/runners/azure_batch.py b/hastelib/src/hastegeo/core/runners/azure_batch.py index bfbd812..0cb7686 100644 --- a/hastelib/src/hastegeo/core/runners/azure_batch.py +++ b/hastelib/src/hastegeo/core/runners/azure_batch.py @@ -3,6 +3,8 @@ import io import time +from datetime import datetime, timedelta, timezone +from urllib.parse import urlparse from azure.batch import BatchServiceClient from azure.batch.batch_auth import SharedKeyCredentials @@ -39,6 +41,11 @@ VirtualMachineConfiguration, ) from azure.identity import DefaultAzureCredential +from azure.storage.blob import ( + BlobServiceClient, + ContainerSasPermissions, + generate_container_sas, +) from hastegeo.core.config import Config from hastegeo.core.utils.logs import Logger from tenacity import ( @@ -52,11 +59,18 @@ class AzureBatchRunner(BaseRunner): - def __init__(self, config: Config = None, pool_id=None): + def __init__( + self, config: Config = None, pool_id=None, candidate_pool_ids=None + ): super().__init__(config) config = config or Config() self.batch_config = config.get_azure_batch_config() + # Ordered candidate pools for capacity-aware routing (v2.1.0): the runner + # binds the actual pool at submit time (see add_task). Falls back to the + # single pool_id (or the training pool) for backward compatibility. self.pool_id = pool_id or self.batch_config["training_pool_id"] + self.candidate_pool_ids = candidate_pool_ids or [self.pool_id] + self.manage_pools = self.batch_config.get("manage_pools", True) self.batch_cluster = AzureBatchJob( account_name=self.batch_config["account_name"], account_key=self.batch_config["account_key"], @@ -65,6 +79,8 @@ def __init__(self, config: Config = None, pool_id=None): user_assigned_identity_resource_id=self.batch_config[ "user_assigned_identity_resource_id" ], + use_sas=self.batch_config.get("use_sas", False), + manage_pools=self.manage_pools, ) self.logger = Logger.get_logger(__name__) @@ -125,21 +141,36 @@ def add_task( # NOTE: test workdir option to eliminate the cd into /app here command = command + # Capacity-aware routing (v2.1.0): pick the pool at submit time from the + # ordered candidates (preference-first, spillover-second), then bind the + # job to it. + selected_pool = self.batch_cluster.select_pool(self.candidate_pool_ids) + self.batch_cluster.pool_id = selected_pool self.logger.info( - "Creating pool for job_id: %s and task_id: %s", job_id, task_id - ) - self.batch_cluster.create_pool_if_not_exists( - self.batch_config["vm_size"], - self.batch_config["vm_publisher"], - self.batch_config["vm_offer"], - self.batch_config["vm_sku"], - self.batch_config["vm_version"], - self.batch_config["target_dedicated_nodes"], - self.batch_config["target_low_priority_nodes"], - self.batch_config["registry_server"], - [self.batch_config["registry_image"]], - self.batch_config["node_agent_sku_id"], + "Selected pool %s for job_id: %s task_id: %s", + selected_pool, + job_id, + task_id, ) + + # Pre-created IaC/autoscale pools manage their own lifecycle; only + # auto-create/resize for legacy single-pool envs (manage_pools=True). + if self.manage_pools: + self.logger.info( + "Creating pool for job_id: %s and task_id: %s", job_id, task_id + ) + self.batch_cluster.create_pool_if_not_exists( + self.batch_config["vm_size"], + self.batch_config["vm_publisher"], + self.batch_config["vm_offer"], + self.batch_config["vm_sku"], + self.batch_config["vm_version"], + self.batch_config["target_dedicated_nodes"], + self.batch_config["target_low_priority_nodes"], + self.batch_config["registry_server"], + [self.batch_config["registry_image"]], + self.batch_config["node_agent_sku_id"], + ) self.logger.info( "Creating job for job_id: %s and task_id: %s", job_id, task_id ) @@ -201,6 +232,8 @@ def __init__( batch_url: str, pool_id: str, user_assigned_identity_resource_id: str, + use_sas: bool = False, + manage_pools: bool = True, ): self.logger = Logger.get_logger(__name__) if account_name and account_key: @@ -212,6 +245,95 @@ def __init__( ) self.pool_id = pool_id self.user_assigned_identity = user_assigned_identity_resource_id + # v2.1.0: per-job user-delegation SAS for blob I/O on multi-tenant shared + # pools (instead of the pool's managed identity); and whether the runner + # manages (creates/resizes) its own pool. + self.use_sas = use_sas + self.manage_pools = manage_pools + self._sas_credential = None + # Cache user-delegation keys per storage account (valid for hours). + self._udk_cache = {} + + def select_pool(self, candidate_pool_ids): + # Capacity-aware routing (v2.1.0): return the first candidate with an + # idle node (spillover to a free tier). If none has an idle node, return + # the preferred (first) candidate and let it scale up / queue. A single + # candidate is returned as-is (no API calls). + if not candidate_pool_ids: + return self.pool_id + if len(candidate_pool_ids) == 1: + return candidate_pool_ids[0] + for pid in candidate_pool_ids: + try: + if self._pool_has_idle_node(pid): + return pid + except BatchErrorException as e: + self.logger.warning( + "Capacity check failed for pool %s (%s); trying next.", + pid, + getattr(e.error, "code", e), + ) + return candidate_pool_ids[0] + + def _pool_has_idle_node(self, pool_id): + pool = self.batch_client.pool.get(pool_id) + if (pool.current_dedicated_nodes or 0) + ( + pool.current_low_priority_nodes or 0 + ) == 0: + return False + nodes = self.batch_client.compute_node.list(pool_id) + return any(n.state == ComputeNodeState.idle for n in nodes) + + def _sas_url(self, url, permissions): + # Append a user-delegation SAS scoped to the URL's container so a Batch + # ResourceFile/OutputFile can read/write WITHOUT the pool holding any + # standing data access — the isolation boundary for multi-tenant shared + # pools. The submitting identity needs `Storage Blob Delegator` on the + # account. User-delegation keys are cached per account. + parsed = urlparse(url) + account_url = f"{parsed.scheme}://{parsed.netloc}" + account_name = parsed.netloc.split(".")[0] + container = parsed.path.lstrip("/").split("/", 1)[0] + if self._sas_credential is None: + self._sas_credential = DefaultAzureCredential() + now = datetime.now(timezone.utc) + start = now - timedelta(minutes=5) + entry = self._udk_cache.get(account_url) + if entry is None or entry[1] <= now + timedelta(hours=1): + expiry = now + timedelta(hours=24) + bsc = BlobServiceClient( + account_url, credential=self._sas_credential + ) + udk = bsc.get_user_delegation_key(start, expiry) + entry = (udk, expiry) + self._udk_cache[account_url] = entry + udk, expiry = entry + sas = generate_container_sas( + account_name=account_name, + container_name=container, + user_delegation_key=udk, + permission=ContainerSasPermissions.from_string(permissions), + expiry=expiry, + start=start, + ) + sep = "&" if "?" in url else "?" + return f"{url}{sep}{sas}" + + def _maybe_sas(self, url, permissions): + # SAS-augment the URL when per-job SAS is enabled; else return as-is + # (paired with identity_reference from _blob_identity()). + if url and self.use_sas: + return self._sas_url(url, permissions) + return url + + def _blob_identity(self): + # No pool identity on the blob transfer when using SAS (the SAS IS the + # credential); otherwise the pool's user-assigned identity. + if self.use_sas: + return None + return ComputeNodeIdentityReference( + resource_id=self.user_assigned_identity + ) def create_pool_if_not_exists( self, @@ -309,7 +431,11 @@ def create_pool_if_not_exists( raise def create_job(self, job_id): - self.wait_for_pool_to_be_ready() + # Fixed pools we manage need a ready node before the job is created; + # autoscale / pre-created pools scale up in response to queued tasks, so + # waiting here would deadlock (0 nodes until tasks exist). + if self.manage_pools: + self.wait_for_pool_to_be_ready() try: job = self.batch_client.job.get(job_id) if job.state != JobState.active: @@ -355,15 +481,15 @@ def add_task( if resource_files_for_upload is not None: resource_files = [ ResourceFile( - http_url=resource_file.get("http_url"), - storage_container_url=resource_file.get( - "storage_container_url" + http_url=self._maybe_sas( + resource_file.get("http_url"), "rl" + ), + storage_container_url=self._maybe_sas( + resource_file.get("storage_container_url"), "rl" ), blob_prefix=resource_file.get("blob_prefix"), file_path=resource_file.get("file_path"), - identity_reference=ComputeNodeIdentityReference( - resource_id=self.user_assigned_identity - ), + identity_reference=self._blob_identity(), ) for resource_file in resource_files_for_upload.values() ] @@ -380,6 +506,11 @@ def add_task( retention_time=retention_time, ) + # Blob transfer credential: per-job SAS (multi-tenant shared pools) or + # the pool's managed identity (legacy). Output needs write/create/list. + output_sas_url = self._maybe_sas(output_container_url, "racwl") + output_identity = self._blob_identity() + task = TaskAddParameter( id=task_id, constraints=task_constraints, @@ -399,11 +530,9 @@ def add_task( file_pattern=file_pattern, destination=OutputFileDestination( container=OutputFileBlobContainerDestination( - container_url=output_container_url, + container_url=output_sas_url, path=output_prefix, - identity_reference=ComputeNodeIdentityReference( - resource_id=self.user_assigned_identity - ), + identity_reference=output_identity, ) ), upload_options=OutputFileUploadOptions( @@ -415,11 +544,9 @@ def add_task( file_pattern="../*.txt", destination=OutputFileDestination( container=OutputFileBlobContainerDestination( - container_url=output_container_url, + container_url=output_sas_url, path=output_prefix, - identity_reference=ComputeNodeIdentityReference( - resource_id=self.user_assigned_identity - ), + identity_reference=output_identity, ) ), upload_options=OutputFileUploadOptions( diff --git a/hastelib/src/hastegeo/core/runners/local.py b/hastelib/src/hastegeo/core/runners/local.py index 36a191f..bc39b37 100644 --- a/hastelib/src/hastegeo/core/runners/local.py +++ b/hastelib/src/hastegeo/core/runners/local.py @@ -37,9 +37,13 @@ def _normalize_azurite_url(url: Optional[str]) -> Optional[str]: class LocalRunner(BaseRunner): """Local runner that executes containers using Docker instead of Azure Batch.""" - def __init__(self, config: Config = None, pool_id=None): + def __init__( + self, config: Config = None, pool_id=None, candidate_pool_ids=None + ): super().__init__(config) self.config = config or Config() + # candidate_pool_ids is accepted for interface parity with the Azure + # Batch runner (capacity-aware routing); local runs use a single pool. self.pool_id = pool_id or "local-pool" self.logger = Logger.get_logger(__name__) self.verbose = os.getenv("HASTE_DEBUG_VERBOSE", "0") == "1" diff --git a/hastelib/tests/core/runners/__init__.py b/hastelib/tests/core/runners/__init__.py new file mode 100644 index 0000000..5b7f7a9 --- /dev/null +++ b/hastelib/tests/core/runners/__init__.py @@ -0,0 +1,2 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. diff --git a/hastelib/tests/core/runners/test_azure_batch_routing.py b/hastelib/tests/core/runners/test_azure_batch_routing.py new file mode 100644 index 0000000..f0fe505 --- /dev/null +++ b/hastelib/tests/core/runners/test_azure_batch_routing.py @@ -0,0 +1,129 @@ +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. + +"""Unit tests for v2.1.0 capacity-aware pool routing + per-job SAS toggle. + +Covers the pure decision logic (no live Batch/Storage calls): +- config candidate pool-id lists + SAS/manage flags, +- AzureBatchJob.select_pool (single, spillover-to-idle, preferred fallback), +- the SAS-vs-identity blob-credential toggle. +""" + +from unittest.mock import MagicMock + +from azure.batch.models import ComputeNodeIdentityReference, ComputeNodeState +from hastegeo.core.config import Config +from hastegeo.core.runners.azure_batch import AzureBatchJob + + +def _job(use_sas=False, manage_pools=True): + job = AzureBatchJob( + account_name="acct", # pragma: allowlist secret + account_key="key", # pragma: allowlist secret + batch_url="https://acct.westus2.batch.azure.com", # pragma: allowlist secret + pool_id="default-pool", + user_assigned_identity_resource_id="/subscriptions/x/umi", + use_sas=use_sas, + manage_pools=manage_pools, + ) + job.batch_client = MagicMock() + return job + + +def _pool(dedicated=0, lowpri=0): + m = MagicMock() + m.current_dedicated_nodes = dedicated + m.current_low_priority_nodes = lowpri + return m + + +def _node(state): + m = MagicMock() + m.state = state + return m + + +def test_select_pool_single_candidate_makes_no_api_calls(): + job = _job() + assert job.select_pool(["only-pool"]) == "only-pool" + job.batch_client.pool.get.assert_not_called() + + +def test_select_pool_empty_falls_back_to_bound_pool(): + job = _job() + assert job.select_pool([]) == "default-pool" + + +def test_select_pool_spills_over_to_pool_with_idle_node(): + job = _job() + # h100 has a node but it's running (busy); t4 has an idle node. + job.batch_client.pool.get.side_effect = lambda pid: ( + _pool(dedicated=1) if pid == "h100" else _pool(lowpri=1) + ) + job.batch_client.compute_node.list.side_effect = lambda pid: ( + [_node(ComputeNodeState.running)] + if pid == "h100" + else [_node(ComputeNodeState.idle)] + ) + assert job.select_pool(["h100", "t4"]) == "t4" + + +def test_select_pool_prefers_first_candidate_with_idle_node(): + job = _job() + job.batch_client.pool.get.side_effect = lambda pid: _pool(dedicated=1) + job.batch_client.compute_node.list.side_effect = lambda pid: [ + _node(ComputeNodeState.idle) + ] + assert job.select_pool(["h100", "t4"]) == "h100" + + +def test_select_pool_falls_back_to_preferred_when_none_idle(): + job = _job() + # Both pools empty (0 nodes): nothing idle -> preferred first candidate, + # which will scale up / queue. + job.batch_client.pool.get.side_effect = lambda pid: _pool() + assert job.select_pool(["h100", "t4"]) == "h100" + + +def test_blob_identity_uses_pool_identity_in_legacy_mode(): + job = _job(use_sas=False) + ident = job._blob_identity() + assert isinstance(ident, ComputeNodeIdentityReference) + # legacy mode leaves URLs untouched (no SAS) + assert job._maybe_sas("https://a.blob/c", "rl") == "https://a.blob/c" + + +def test_blob_identity_none_and_sas_applied_in_sas_mode(): + job = _job(use_sas=True) + assert job._blob_identity() is None + job._sas_url = MagicMock(return_value="https://a.blob/c?sig=xyz") + assert ( + job._maybe_sas("https://a.blob/c", "rl") == "https://a.blob/c?sig=xyz" + ) + job._sas_url.assert_called_once_with("https://a.blob/c", "rl") + + +def test_maybe_sas_noop_on_empty_url(): + job = _job(use_sas=True) + assert job._maybe_sas(None, "rl") is None + + +def test_config_candidate_pool_lists_and_flags(monkeypatch): + monkeypatch.setenv("AZURE_BATCH_TRAINING_POOL_ID", "single-train") + monkeypatch.setenv("AZURE_BATCH_IMAGERYPREP_POOL_ID", "single-prep") + monkeypatch.setenv("AZURE_BATCH_TRAINING_POOL_IDS", "h100-pool, t4-pool") + monkeypatch.delenv("AZURE_BATCH_INFERENCE_POOL_IDS", raising=False) + monkeypatch.delenv("AZURE_BATCH_IMAGERYPREP_POOL_IDS", raising=False) + monkeypatch.setenv("AZURE_BATCH_USE_SAS", "true") + monkeypatch.delenv("AZURE_BATCH_MANAGE_POOLS", raising=False) + + cfg = Config().get_azure_batch_config() + + # explicit list is split + trimmed + assert cfg["training_pool_ids"] == ["h100-pool", "t4-pool"] + # unset lists fall back to the single legacy id + assert cfg["inference_pool_ids"] == ["single-train"] + assert cfg["imageryprep_pool_ids"] == ["single-prep"] + # flags: SAS on (explicit), manage_pools default true + assert cfg["use_sas"] is True + assert cfg["manage_pools"] is True diff --git a/infra/main.bicep b/infra/main.bicep index 75234c1..73df584 100644 --- a/infra/main.bicep +++ b/infra/main.bicep @@ -8,7 +8,7 @@ targetScope = 'subscription' // Core parameters (replace the positional CLI args of setup_infra.sh) // --------------------------------------------------------------------------- -@description('Short prefix used to build every resource name (e.g. "ai4gl").') +@description('Short prefix used to build every resource name (e.g. "haste").') param resourcePrefix string @description('Azure region for the environment resource group and resources.') diff --git a/infra/main.bicepparam b/infra/main.bicepparam index 78e2c44..4e9342e 100644 --- a/infra/main.bicepparam +++ b/infra/main.bicepparam @@ -4,7 +4,7 @@ using './main.bicep' // variables from `azd env set ...`). Defaults keep `az bicep build` / local // what-if runs working without azd. -param resourcePrefix = readEnvironmentVariable('HASTE_RESOURCE_PREFIX', 'ai4gl') +param resourcePrefix = readEnvironmentVariable('HASTE_RESOURCE_PREFIX', 'haste') param location = readEnvironmentVariable('AZURE_LOCATION', 'westus2') param randomSuffix = readEnvironmentVariable('HASTE_RANDOM_SUFFIX', 'dev1') diff --git a/infra/modules/batchPool.bicep b/infra/modules/batchPool.bicep index 33635c3..1f43bcc 100644 --- a/infra/modules/batchPool.bicep +++ b/infra/modules/batchPool.bicep @@ -13,14 +13,34 @@ param poolName string @description('Pool VM size.') param vmSize string -@description('Max dedicated nodes (autoscale cap).') +@description('Max nodes (autoscale cap).') param maxNodes int +@description('Scale mode: Fixed (dev/prod reserved) or Autoscale (shared-demo burst).') +@allowed([ + 'Fixed' + 'Autoscale' +]) +param scaleMode string = 'Autoscale' + +@description('Node cost tier: Dedicated (guaranteed) or LowPriority (spot, preemptible).') +@allowed([ + 'Dedicated' + 'LowPriority' +]) +param nodeType string = 'Dedicated' + +@description('Node count when scaleMode == Fixed.') +param fixedNodeCount int = 1 + +@description('Autoscale floor. 0 = scale-to-zero when idle (shared-demo burst); 1 keeps the legacy always-on behavior.') +param minNodes int = 1 + @description('User-assigned managed identity resource id (for ACR pull).') param umiResourceId string -@description('Resource id of the env VNet subnet for the pool.') -param subnetId string +@description('VNet subnet resource id for VNet-injected pools. Empty => no VNet injection (BatchManaged public IPs). Shared-demo pools finalize their subnet + per-demo-storage firewall allowlisting before running workloads.') +param subnetId string = '' @description('Shared ACR name (without .azurecr.io).') param acrName string @@ -33,7 +53,25 @@ param imageryprepImage string var registryServer = '${acrName}.azurecr.io' -var autoscaleFormula = '$samples = $ActiveTasks.GetSamplePercent(TimeInterval_Minute * 15);$tasks = $samples < 70 ? max(0, $ActiveTasks.GetSample(1)) : max($ActiveTasks.GetSample(1), avg($ActiveTasks.GetSample(TimeInterval_Minute * 15)));$targetVMs = $tasks > 0 ? $tasks : max(0, $TargetDedicatedNodes / 2);$cappedPoolSize = ${maxNodes};$TargetDedicatedNodes = max(1, min($targetVMs, $cappedPoolSize));$NodeDeallocationOption = taskcompletion;' +// Autoscale targets the node bucket matching nodeType; minNodes=0 => scale-to-zero +// when idle (shared-demo burst preserves scarce GPU quota). +var scaleTargetVar = nodeType == 'LowPriority' ? '$TargetLowPriorityNodes' : '$TargetDedicatedNodes' +var autoscaleFormula = '$samples = $ActiveTasks.GetSamplePercent(TimeInterval_Minute * 15);$tasks = $samples < 70 ? max(0, $ActiveTasks.GetSample(1)) : max($ActiveTasks.GetSample(1), avg($ActiveTasks.GetSample(TimeInterval_Minute * 15)));$targetVMs = $tasks > 0 ? $tasks : ${minNodes};${scaleTargetVar} = max(${minNodes}, min($targetVMs, ${maxNodes}));$NodeDeallocationOption = taskcompletion;' + +// Fixed (reserved dev/prod) vs Autoscale (shared-demo burst), targeting the +// Dedicated or LowPriority bucket per nodeType. +var scaleSettings = scaleMode == 'Fixed' ? { + fixedScale: { + targetDedicatedNodes: nodeType == 'Dedicated' ? fixedNodeCount : 0 + targetLowPriorityNodes: nodeType == 'LowPriority' ? fixedNodeCount : 0 + resizeTimeout: 'PT15M' + } +} : { + autoScale: { + formula: autoscaleFormula + evaluationInterval: 'PT5M' + } +} resource batchAccount 'Microsoft.Batch/batchAccounts@2024-07-01' existing = { name: batchAccountName @@ -91,20 +129,17 @@ resource pool 'Microsoft.Batch/batchAccounts/pools@2024-07-01' = { } } } - networkConfiguration: { - subnetId: subnetId - publicIPAddressConfiguration: { - provision: 'BatchManaged' - } - dynamicVnetAssignmentScope: 'none' - enableAcceleratedNetworking: false - } - scaleSettings: { - autoScale: { - formula: autoscaleFormula - evaluationInterval: 'PT5M' + ...(empty(subnetId) ? {} : { + networkConfiguration: { + subnetId: subnetId + publicIPAddressConfiguration: { + provision: 'BatchManaged' + } + dynamicVnetAssignmentScope: 'none' + enableAcceleratedNetworking: false } - } + }) + scaleSettings: scaleSettings } } diff --git a/infra/modules/functionApp.bicep b/infra/modules/functionApp.bicep index 1ac6023..5fb37f0 100644 --- a/infra/modules/functionApp.bicep +++ b/infra/modules/functionApp.bicep @@ -48,6 +48,12 @@ var queueDataContributorRoleId = subscriptionResourceId( 'Microsoft.Authorization/roleDefinitions', '974c5e8b-45b9-4653-ba55-5f855dd0fb88' ) +// Storage Blob Delegator: lets the app mint user-delegation SAS (v2.1.0 per-job +// SAS for multi-tenant shared Batch pools). Not included in Blob Data Owner. +var blobDelegatorRoleId = subscriptionResourceId( + 'Microsoft.Authorization/roleDefinitions', + 'db58b8e5-c6ad-4a2a-8342-4190687cbf4a' +) var deploymentContainerName = 'app-package-${name}' @@ -192,6 +198,16 @@ resource siteQueueContributor 'Microsoft.Authorization/roleAssignments@2022-04-0 } } +resource siteBlobDelegator 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: guid(storageAccount.id, site.id, 'blobDelegator') + scope: storageAccount + properties: { + roleDefinitionId: blobDelegatorRoleId + principalId: site.identity.principalId + principalType: 'ServicePrincipal' + } +} + output name string = site.name output defaultHostName string = site.properties.defaultHostName output systemPrincipalId string = site.identity.principalId diff --git a/infra/shared-pools.bicep b/infra/shared-pools.bicep new file mode 100644 index 0000000..8af2616 --- /dev/null +++ b/infra/shared-pools.bicep @@ -0,0 +1,79 @@ +// Shared multi-tenant Batch pools (H100 + T4) for a group of envs (e.g. dev, demo), +// created in the shared Batch account's resource group. +// Multi-tenant: data isolation is per-job user-delegation SAS (see +// spec/features/batch-compute-expansion/). ACR pull uses ONLY the shared UMI (no +// per-tenant identity, no storage grants). Autoscale low-priority, scale-to-zero +// when idle to preserve scarce dedicated GPU quota. +// +// Names: ${sharedPrefix}-shared-${sharedGroup}-{h100,t4}-pool +// e.g. group=dev -> -shared-dev-{h100,t4}-pool +// +// Deploy (values come from env vars via shared-pools.bicepparam): +// az deployment group create -g \ +// --parameters infra/shared-pools.bicepparam + +@description('Resource-name prefix. Generic default; override per deployment.') +param sharedPrefix string = 'haste' + +@description('Shared group these pools serve (e.g. dev, demo).') +param sharedGroup string + +@description('Existing shared Batch account name (BYO — no generic default).') +param batchAccountName string + +@description('Shared ACR-pull UMI resource id.') +param umiResourceId string + +@description('Existing shared ACR name, without .azurecr.io (BYO — no generic default).') +param acrName string + +@description('Training image (H100 tier + spillover).') +param trainingImage string = 'hastetraining:2.0.0' + +@description('Imageryprep image (T4 tier + spillover).') +param imageryprepImage string = 'hasteimageryprep:2.0.0' + +@description('H100 autoscale max nodes (formula cap).') +param h100MaxNodes int = 2 + +@description('T4 autoscale max nodes (formula cap).') +param t4MaxNodes int = 2 + +module h100Pool 'modules/batchPool.bicep' = { + name: 'shared-${sharedGroup}-h100' + params: { + batchAccountName: batchAccountName + poolName: '${sharedPrefix}-shared-${sharedGroup}-h100-pool' + vmSize: 'Standard_NC40ads_H100_v5' + scaleMode: 'Autoscale' + nodeType: 'LowPriority' + minNodes: 0 + maxNodes: h100MaxNodes + umiResourceId: umiResourceId + acrName: acrName + trainingImage: trainingImage + imageryprepImage: imageryprepImage + // subnetId omitted => no VNet injection yet; finalize subnet + per-tenant + // storage firewall allowlisting before running real workloads. + } +} + +module t4Pool 'modules/batchPool.bicep' = { + name: 'shared-${sharedGroup}-t4' + params: { + batchAccountName: batchAccountName + poolName: '${sharedPrefix}-shared-${sharedGroup}-t4-pool' + vmSize: 'Standard_NC16as_T4_v3' + scaleMode: 'Autoscale' + nodeType: 'LowPriority' + minNodes: 0 + maxNodes: t4MaxNodes + umiResourceId: umiResourceId + acrName: acrName + trainingImage: trainingImage + imageryprepImage: imageryprepImage + } +} + +output h100PoolName string = h100Pool.outputs.poolName +output t4PoolName string = t4Pool.outputs.poolName diff --git a/infra/shared-pools.bicepparam b/infra/shared-pools.bicepparam new file mode 100644 index 0000000..9b0e6c8 --- /dev/null +++ b/infra/shared-pools.bicepparam @@ -0,0 +1,22 @@ +// Deployment config for shared-pools.bicep. The template stays generic (prefix +// defaults to the neutral 'haste'); values are supplied per deployment from +// environment variables — mirroring main.bicepparam — so NO partner-specific +// subscription/account details are committed. Set these before deploying: +// HASTE_RESOURCE_PREFIX resource-name prefix (e.g. your org's prefix) +// HASTE_SHARED_GROUP dev | demo | ... (default: dev) +// HASTE_EXISTING_BATCH_ACCOUNT shared Batch account name +// HASTE_SHARED_ACR_NAME shared ACR name (without .azurecr.io) +// HASTE_SHARED_UMI_ID resource id of the ACR-pull-only identity +// +// Pool names: ${prefix}-shared-${group}-{h100,t4}-pool +// +// Deploy: az deployment group create -g \ +// --parameters infra/shared-pools.bicepparam + +using './shared-pools.bicep' + +param sharedPrefix = readEnvironmentVariable('HASTE_RESOURCE_PREFIX', 'haste') +param sharedGroup = readEnvironmentVariable('HASTE_SHARED_GROUP', 'dev') +param batchAccountName = readEnvironmentVariable('HASTE_EXISTING_BATCH_ACCOUNT', '') +param acrName = readEnvironmentVariable('HASTE_SHARED_ACR_NAME', '') +param umiResourceId = readEnvironmentVariable('HASTE_SHARED_UMI_ID', '') diff --git a/setup/README.md b/setup/README.md index e0ed8f1..c0ff37d 100644 --- a/setup/README.md +++ b/setup/README.md @@ -37,7 +37,7 @@ az login azd env new dev3 # Required configuration. -azd env set HASTE_RESOURCE_PREFIX ai4gl +azd env set HASTE_RESOURCE_PREFIX myorg azd env set HASTE_RANDOM_SUFFIX dev3 azd env set AZURE_LOCATION westus2 azd env set HASTE_APIM_PUBLISHER_EMAIL you@example.com @@ -83,7 +83,7 @@ Common knobs: | Variable | Default | Purpose | |---|---|---| -| `HASTE_RESOURCE_PREFIX` | `ai4gl` | Resource name prefix. | +| `HASTE_RESOURCE_PREFIX` | `haste` | Resource name prefix. | | `HASTE_RANDOM_SUFFIX` | `dev1` | Per-environment suffix. | | `AZURE_LOCATION` | `westus2` | Azure region. | | `HASTE_APIM_PUBLISHER_EMAIL` | — | APIM publisher email. | diff --git a/spec/features/batch-compute-expansion/README.md b/spec/features/batch-compute-expansion/README.md new file mode 100644 index 0000000..0fc3997 --- /dev/null +++ b/spec/features/batch-compute-expansion/README.md @@ -0,0 +1,114 @@ +# Feature: Batch compute expansion + multi-tenant shared GPU pools + +**Status:** in-progress +**Author:** HASTE engineering team +**Date:** 2026-07-14 +**Target Release:** TBD +**Priority:** P1 +**Work Item:** TBD + +## Summary + +Expand HASTE Batch compute to two GPU tiers (H100 for training, T4 for +inference/imageryprep/overflow) with capacity-aware routing, and re-architect +demo compute so that **many demo environments (targeting 20+) share a small set +of GPU pools** while keeping each environment's data/metadata fully isolated. +The scarce resource — GPU quota — is pooled and rationed centrally instead of +fragmented across per-env pools. + +## Motivation + +- **GPU quota is scarce and hard to obtain.** Per-env pools strand quota: an idle + node in env A cannot serve env B. A shared pool keeps every scarce node busy + whenever *any* tenant has work. +- **Per-env pools do not scale.** The Batch account allows ~20 pools total; 20 + demo envs × 2 tiers = 40 pools — physically impossible. Sharing keeps the pool + count at a handful regardless of tenant count. +- **Uncoordinated autoscale blows quota.** 20 independently-autoscaling pools + would exceed the account core quota and cause allocation failures. One shared, + centrally-capped burst pool rations the scarce nodes. +- **Single fixed pool per workload today.** `config.py` exposes one + `training_pool_id` and one `imageprep_pool_id`; the runner binds one pool at + init (`azure_batch.py`). No tiering, no routing, no overflow. + +## Non-negotiable constraint: data isolation on shared compute + +Tasks from different tenants may run on the same shared node. Isolation must be +enforced at the **credential boundary**, not by application correctness. The +pool holds **no standing data access**; each job receives a short-TTL +**user-delegation SAS** scoped to exactly its own container, minted by the +submitting Function App (which already knows the tenant). Adding tenant #21 +requires **zero pool/compute changes** — only its storage + a SAS-minting grant. + +Rejected alternative: a shared pool identity with blob grants on every tenant's +storage — that is shared compute *and* shared data access, defeating isolation. +Also rejected: attaching every tenant's UMI to the pool — bounded by identity +limits and requires mutating a shared, semi-immutable pool per tenant. + +## Success Criteria + +- [ ] Two GPU tiers per compute group: H100 (`Standard_NC40ads_H100_v5`) and + T4 (`Standard_NC16as_T4_v3`), created and managed in IaC (not the SDK + `create_pool_if_not_exists` path). +- [ ] Capacity-aware routing: training → H100 first, spill to T4; inference / + imageryprep → T4 first, spill to H100; "free" = idle dedicated (or + under-max autoscale) node. +- [ ] N demo envs (validated at ≥ 4, designed for 20+) submit to **shared** + demo pools; each env's Batch tasks can read/write **only** its own storage, + proven by a cross-tenant access-denied test. +- [ ] Shared-demo pools autoscale within a **single central core-quota ceiling**; + dev/prod get reserved (fixed) capacity. +- [ ] Adding a new demo env requires no pool or shared-compute change. +- [ ] Per-tenant fairness: no single demo can starve the others. + +## HASTE Components Affected + +| Component | Impact | +|---|---| +| `hastelib` `config.py` | Pool ids → ordered candidate lists per workload | +| `hastelib` `azure_batch.py` | `select_pool` routing; bind pool at submit; SAS URLs replace `identity_reference`; drop SDK pool creation for shared pools | +| `hastelib` processors (train, inference, embedding, imagery, artifacts) | Pass candidate lists + workload preference; mint per-job SAS | +| `hastelib` data layer / artifact storage | User-delegation SAS minting helper | +| `infra/modules/batchPool.bicep` | Parameterize SKU, scale-mode (fixed vs autoscale), node type (dedicated vs low-priority), optional VNet, ACR-pull-only identity | +| `infra/shared-pools.bicep` + `.bicepparam` | Shared multi-tenant pool set (create-once per group, referenced by envs) | +| `infra/modules/functionApp.bicep` | `Storage Blob Delegator` grant so the app can mint user-delegation SAS | +| `infra/main.bicepparam` + env `.env` | Generic `haste` prefix default; candidate pool-id + SAS/manage-pool env vars | + +## Related Specs + +| Spec | Relationship | +|---|---| +| [infra-iac-migration](../infra-iac-migration/README.md) | Owns the Bicep/azd that now creates the pools; this feature extends `batchPool.bicep`. Pool creation moved out of the SDK per that migration. | + +## Document Index + +| Document | Purpose | Status | +|---|---|---| +| [plan.md](plan.md) | Execution plan: phases, milestones, agent summary | approved | +| [impact-analysis.md](impact-analysis.md) | Risk, dependencies, blast radius, security | approved | +| [user-stories.md](user-stories.md) | User stories & acceptance criteria | approved | +| [design.md](design.md) | Technical design: topology, isolation, routing, fairness, IaC + app | approved | +| [test-plan.md](test-plan.md) | Test strategy & coverage matrix | approved | +| [rollout.md](rollout.md) | Rollout phases, opt-in flags, rollback, monitoring | approved | +| data-model.md | Cosmos/Blob/Data Lake schema changes | n/a — no schema changes | + +## Decision Log + +| Date | Decision | Rationale | +|---|---|---| +| 2026-07-14 | Shared multi-tenant pools (not per-env) for dev + demo groups; prod stays dedicated | Scarce GPU quota + ~20-pool account limit; per-env pools fragment quota and don't scale to 20+ tenants. | +| 2026-07-14 | Data isolation via per-job **user-delegation SAS**, not a shared pool identity | Enforces isolation at the credential boundary, not app correctness; adding a tenant needs zero pool changes. POC-validated. | +| 2026-07-14 | **Uniform** per-job SAS on every pool (one code path) | Simpler than SAS-for-shared + identity-for-dedicated; single maintained path. | +| 2026-07-14 | Demo burst on **low-priority / spot** nodes | Separate quota bucket; preserves scarce *dedicated* GPU for dev/prod. | +| 2026-07-14 | Fairness = per-tenant task caps + Batch job priority (Phase 1) | Ship simple; add an admission broker only if starvation is observed. | +| 2026-07-14 | Shared pools deploy into the existing shared framework RG | Where the Batch account + ACR already live; clean logical ownership, no new RG. | +| 2026-07-14 | Generic-default IaC (prefix defaults to `haste`, BYO account/ACR) | Reusable by other partners; this deployment overrides via `shared-pools.bicepparam`. | +| 2026-07-14 | Pin `azure-batch==14.2.0` | 15.x track-2 rewrite restructures the model API the code uses; migration tracked separately. | + +## Remaining gates (before/during build) + +- Confirm preemption-safety of demo workloads (spot). +- Request low-priority GPU quota for demo burst. +- ✅ ~~embedding models `/mnt` mount~~ — resolved: no submit path uses the mount. +- **Destructive confirm:** delete the 6 existing pools after draining jobs bound + to old pool ids (create-new → repoint → drain → delete-old). diff --git a/spec/features/batch-compute-expansion/design.md b/spec/features/batch-compute-expansion/design.md new file mode 100644 index 0000000..110c790 --- /dev/null +++ b/spec/features/batch-compute-expansion/design.md @@ -0,0 +1,206 @@ +# Design: Batch compute expansion + multi-tenant shared GPU pools + +## Overview + +HASTE runs GPU workloads (training, inference, embedding, imageryprep) on Azure +Batch. This design replaces the single fixed pool per workload with two GPU tiers +(H100 for training; T4 for inference/imageryprep + spillover) and a small set of +**shared, multi-tenant** pools, so many environments share scarce GPU quota +instead of fragmenting it across per-env pools. Data isolation on shared compute +is enforced by a **per-job user-delegation SAS** (the pool holds no standing +storage access); jobs are routed to a pool from an ordered candidate list at +submit time. See [user-stories.md](user-stories.md) for goals, +[impact-analysis.md](impact-analysis.md) for risk, [test-plan.md](test-plan.md) +for verification, and [rollout.md](rollout.md) for the phased rollout. + +## Architecture + +### Pool topology + +One shared Batch account in a shared resource group (westus2). Example account +quota: ~446 shared dedicated cores, pool quota 20, low-priority 6 (request an +increase for demo burst — see [rollout.md](rollout.md)). + +Pool names follow the generic `${prefix}-shared-${group}-${tier}-pool` (prefix +defaults to `haste`; a partner overrides it — shown as `` below). + +| Group | H100 pool (`NC40ads_H100_v5`, 40c) | T4 pool (`NC16as_T4_v3`, 16c) | Scale | Node type | Tenancy | Envs | +|---|---|---|---|---|---|---| +| shared-dev | `-shared-dev-h100-pool` | `-shared-dev-t4-pool` | autoscale, min 0 | low-priority / spot | **multi** | dev1, dev2 | +| shared-demo | `-shared-demo-h100-pool` | `-shared-demo-t4-pool` | autoscale, min 0 | low-priority / spot | **multi** | demo1–N | +| prod | `-haste-prod-pool` (existing) | *(optional, TBD)* | fixed | dedicated | single | prod | + +**shared-dev** and **shared-demo** pools are multi-tenant (data isolation = per-job +SAS, below); the same `shared-pools.bicep` template produces each group via +`sharedGroup`. **prod** stays single-tenant on its dedicated pool (guaranteed +capacity + isolation-by-dedication). Autoscale ceilings (`maxNodes` per pool) are +set so combined low-priority usage stays within the (to-be-increased) low-priority +GPU quota. + +## Data isolation (the core design) + +**Decision: uniform per-job SAS on *every* pool** — one code path, not two. Data +access is never standing on any pool; each pool keeps only an ACR-pull identity. +The multi-tenant shared-dev / shared-demo pools rely on SAS as their sole data +boundary; prod additionally has isolation-by-dedication. + +### How task storage auth works today + +`azure_batch.py` never puts storage credentials inside the container. Input +(`ResourceFile`), output (`OutputFileBlobContainerDestination`), and the optional +models mount all carry +`identity_reference=ComputeNodeIdentityReference(resource_id=)`. +Batch performs the transfer under that identity. **A referenced identity must be +attached to the pool** — so this model is inherently one-tenant. + +### Target: per-job user-delegation SAS + +The submitting Function App knows the tenant. For **every** job (all pools) it +mints a **user-delegation SAS** (signed by the env identity via +`get_user_delegation_key`, no account key) scoped to that tenant's container and +prefix, with a short TTL, read for inputs / write for outputs. It embeds the SAS +in the blob URL and passes it to Batch: + +- `ResourceFile(storage_container_url="https://.blob.core.windows.net/?", …)` — **no** `identity_reference`. +- `OutputFileBlobContainerDestination(container_url="https://…?", …)` — **no** `identity_reference`. + +Each pool then needs **only** an ACR-pull identity (dev/prod: their env UMI; +shared-demo: `haste-shared-acr-umi` with `AcrPull` on the shared ACR). No pool +holds storage grants. A compromised node holds, at most, the SAS of its +currently-running tasks, scoped to one tenant's container. Every submitting +Function App identity needs `Storage Blob Delegator` on its own storage account to +mint the SAS (granted in `functionApp.bicep`). + +### Why this scales + +Adding tenant #21 = create its storage + grant `Storage Blob Delegator` + point +its pool-id env vars at the shared pools. **No pool mutation, no new identity on +the pool, no redeploy of the shared compute.** + +### Models mount caveat + +Blob **mounts are pool-level**, fixed at pool creation — they cannot be per-tenant +on a shared pool. **Verified non-issue:** no submit path uses the mount — +`add_task` calls `create_pool_if_not_exists` without `storage_account_name`, and +`embedding.py` (the model-heavy flow) uses the standard `ResourceFile`/`OutputFile` +path. The mount code is dead for current flows. A future flow needing models on +shared pools would deliver them per-task via SAS `ResourceFile`s. + +### Validation (PoC, 2026-07-14) — PASSED + +Proven on a throwaway low-priority CPU pool in the shared account with a real +**user-delegation SAS** (minted via `Storage Blob Delegator`, `skoid`/`sktid` +present — no account key): + +- **SAS Batch I/O works.** A task with a container-scoped SAS on `ResourceFile` + (download) *and* `OutputFileBlobContainerDestination` (upload) — no + `identity_reference`, no UMI on the pool — completed exitCode 0 and wrote its + output blob. The mechanism swap is a drop-in. +- **Tenant isolation holds.** A task using tenant-A's container-scoped SAS to read + tenant-B's blob failed with `BlobAccessDenied` (UserError). The credential + boundary is enforced by Azure, not app code. +- **ACR-pull-only shared identity works.** `haste-shared-acr-umi` (AcrPull on the + shared ACR) **authenticated and downloaded** the real `hastetraining:2.0.0` on a + pool referencing only that UMI. The cheap CPU test VM hit `DiskFull` only because + Batch stores images on the node temp disk (16 GB on D2s_v3), which the GPU pools' + large NVMe disks dwarf — auth path proven; disk is a test-VM artifact. + +This retired the core architectural risks (SAS I/O, isolation, shared-identity ACR +pull). + +## Behavior & Logic + +### Capacity-aware routing + +- Config exposes ordered candidate lists instead of single ids: + `AZURE_BATCH_TRAINING_POOL_IDS="h100,t4"`, `AZURE_BATCH_INFERENCE_POOL_IDS="t4,h100"`. +- `select_pool(candidates)` in `azure_batch.py`: return the first candidate with an + **idle node** (spillover to a free tier); if none is idle, return the preferred + (first) candidate and let it scale up / queue. A single candidate is returned + with no API calls. +- Pool binding moves **out of `AzureBatchRunner.__init__`** to submit time, so + `add_task` targets the pool chosen per task. `manage_pools=false` skips + SDK create/resize + the node-wait for pre-created autoscale pools (both fail / + deadlock on a scale-to-zero pool). + +### Fairness (shared pools, 20+ tenants) + +Isolation is solved above; fairness is the remaining hard problem. **Decision: +start simple, escalate only if needed.** + +- **Phase 1 (ship):** per-tenant concurrent-task cap enforced app-side + Batch + **job priority** (interactive/inference > long training). No new service. +- **Phase 2 (only if starvation is observed):** a lightweight admission broker that + admits queued jobs round-robin / weighted across tenants. + +## Configuration + +### IaC (`infra/`) + +- **`modules/batchPool.bicep`** — parameterized `scaleMode` (`Fixed` | `Autoscale`), + `nodeType` (`Dedicated` | `LowPriority`), `fixedNodeCount`, `minNodes` (autoscale + floor; 0 = scale-to-zero), optional VNet injection. One pool per invocation. + Backward-compatible defaults. +- **`shared-pools.bicep`** (+ `shared-pools.bicepparam`) — instantiates the H100+T4 + pair for a group, deployed **standalone** into the shared account's RG (separate + from `azd up`). `sharedGroup` selects `dev` / `demo` / …. Uses the ACR-pull-only + `haste-shared-acr-umi`. +- **`modules/functionApp.bicep`** — grants the app identity `Storage Blob Delegator` + (mint SAS). +- **`main.bicepparam`** — `HASTE_RESOURCE_PREFIX` defaults to the generic `haste`; + partners override the prefix + BYO account/ACR via `shared-pools.bicepparam`. + +### Environment settings (per-env opt-in) + +Set on the `api`/`queues` Function Apps; all default to the legacy single-pool, +pool-identity path (see [docs/configuration.md](../../../docs/configuration.md#shared-multi-tenant-gpu-pools)): + +| Variable | Default | Purpose | +|---|---|---| +| `AZURE_BATCH_{TRAINING,INFERENCE,IMAGERYPREP}_POOL_IDS` | single pool id | Ordered candidate pools (comma-separated) | +| `AZURE_BATCH_USE_SAS` | `false` | Per-job SAS blob I/O vs. pool identity | +| `AZURE_BATCH_MANAGE_POOLS` | `true` | Runner auto-creates/resizes its pool (`false` for pre-created autoscale pools) | + +## Migration & rollout + +Full phased plan in [rollout.md](rollout.md). Summary (non-destructive order): +request quota → create shared pools alongside existing → ship v2.1.0 (flags off) → +enable on dev1+dev2 and validate E2E → onboard demos → (optional, confirmed) +drain + delete legacy per-env pools. + +## Observability + +Signals to watch during/after rollout (alerts detailed in +[rollout.md](rollout.md#monitoring--alerting)): + +- **Batch task failure rate** and any `BlobAccessDenied` in normal runs (SAS + scope/grant problems — expected only in the isolation *test*). +- **Node allocation failures** on the shared pools (low-priority GPU quota signal). +- **`getUserDelegationKey` errors** in Function App logs (missing `Storage Blob + Delegator`). +- **Shared pool node-hours** vs. the low-priority budget ceiling. + +## Build status (2026-07-14) + +- ✅ `batchPool.bicep` parameterized; `shared-pools.bicep` deployed → + `-shared-dev-{h100,t4}-pool` **live**, autoscale low-priority, 0 nodes + (scale-to-zero), ACR-pull via `haste-shared-acr-umi`. Waiting on the quota bump + to scale up. +- ✅ **v2.1.0 app code (hastelib)** — candidate-list routing + `select_pool`, + per-job SAS, `manage_pools` gate, all 5 processors, `Storage Blob Delegator` in + `functionApp.bicep`. 9 unit tests pass. `azure-batch` pinned `==14.2.0`. +- ⬜ Finalize pool **networking** (subnet + per-tenant-storage firewall allowlisting) + before real workloads. +- ⬜ dev/prod fixed H100+T4 pairs (they currently run their existing single pool). + +## Open Questions + +- **Low-priority GPU quota** increase — required before the shared pools can scale + up (H100=40c, T4=16c per node vs. the current 6 low-priority cores). +- **Preemption-safety** of demo workloads — spot requires retry-safe / idempotent + tasks (Batch auto-reschedules preempted low-priority tasks). +- **Shared-pool networking** — VNet subnet + per-tenant storage-firewall + allowlisting model for VNet-injected shared pools. +- **Destructive cleanup** — deleting legacy per-env pools after draining jobs bound + to old pool ids (create-new → repoint → drain → delete-old); needs explicit + confirmation. diff --git a/spec/features/batch-compute-expansion/impact-analysis.md b/spec/features/batch-compute-expansion/impact-analysis.md new file mode 100644 index 0000000..9a314ba --- /dev/null +++ b/spec/features/batch-compute-expansion/impact-analysis.md @@ -0,0 +1,93 @@ +# Impact Analysis: Batch compute expansion + multi-tenant shared GPU pools + +## Scope of Change + +### HASTE Components Affected + +| Component | Path | Type of Change | Severity | +|---|---|---|---| +| Core library — config | `hastelib/src/hastegeo/core/config.py` | modified (additive candidate lists + flags) | low | +| Core library — batch runner | `hastelib/src/hastegeo/core/runners/azure_batch.py` | modified (select_pool, submit-time binding, SAS, manage_pools gate) | medium | +| Core library — local runner | `hastelib/src/hastegeo/core/runners/local.py` | modified (interface parity) | low | +| Core library — processors | `hastelib/src/hastegeo/core/processors/{train,inference,embedding,imagery,artifacts}.py` | modified (pass candidate lists) | low | +| IaC — batch pool module | `infra/modules/batchPool.bicep` | modified (parameterized scale/node/VNet) | medium | +| IaC — shared pools | `infra/shared-pools.bicep` + `.bicepparam` | new | medium | +| IaC — function app roles | `infra/modules/functionApp.bicep` | modified (Storage Blob Delegator grant) | medium | +| IaC — params | `infra/main.bicepparam` | modified (generic prefix default) | low | +| Build | `hastelib/pyproject.toml` | modified (`azure-batch==14.2.0` pin) | low | + +## Azure Service Impact + +| Service | Change | New Cost Impact | +|---|---|---| +| Azure Batch | New shared H100+T4 pools (autoscale low-priority, scale-to-zero); `batchPool` module gains scale/node params | None while idle (0 nodes). Low-priority nodes are cheaper than dedicated; demo GPU moves to the low-priority quota bucket. | +| Blob Storage | Access pattern change: Batch task I/O uses per-job user-delegation SAS instead of the pool managed identity | None (same reads/writes; different credential) | +| Azure Functions | `api`/`queues` apps gain `Storage Blob Delegator` and mint SAS at submit time | Negligible (a cached `getUserDelegationKey` call) | +| Managed Identity | New `haste-shared-acr-umi` (ACR-pull only) for shared pools | None | + +> Cosmos DB, Data Lake, Queue Storage, Static Web Apps: **no change**. + +## Dependency Analysis + +### Upstream Dependencies (things this feature needs) + +| Dependency | Type | Status | Risk if Unavailable | +|---|---|---|---| +| Shared Batch account + ACR (shared framework RG) | infra | available | Pools can't be created | +| Low-priority GPU quota | quota | **pending increase** | Shared pools exist but can't scale up until granted | +| `Storage Blob Delegator` on the Function App identity | RBAC | new grant (in `functionApp.bicep`) | SAS minting fails → jobs can't do blob I/O in SAS mode | +| `azure-batch` 14.x model API | library | pinned `==14.2.0` | 15.x breaks the imports | + +### Downstream Impact (things affected by this feature) + +| Consumer | How Affected | Breaking? | Migration Needed? | +|---|---|---|---| +| Existing single-pool envs (dev1, prod, …) | New env vars all default to legacy behavior | no | no — opt-in only | +| `AzureBatchRunner` / `LocalRunner` callers | `candidate_pool_ids` added (optional) | no | no | +| Processors | Pass candidate lists (fallback to single id) | no | no | +| Batch pool definitions | Autoscale pools reject SDK resize; runner skips it when `manage_pools=false` | no (gated) | set the flag for pre-created pools | + +## Risk Assessment + +| Risk | Likelihood | Impact | Mitigation | Owner | +|---|---|---|---|---| +| Cross-tenant data leakage on shared nodes | Low | Critical | Per-job container-scoped user-delegation SAS; pool holds no storage access; POC-proven `BlobAccessDenied` | backend-dev / security | +| SAS minting fails (missing `Storage Blob Delegator`) | Medium | High | Grant added in `functionApp.bicep`; verify on each env before enabling `USE_SAS` | Platform Operator | +| Autoscale pool resize/wait deadlock | Medium | High | `manage_pools=false` skips create/resize and the node-wait for pre-created pools | backend-dev | +| Spot preemption thrashes long training | Medium | Medium | Demos (short/interactive) on spot; dev/prod stay dedicated; tasks must be retry-safe | Platform Operator | +| Destructive pool rebuild orphans in-flight jobs | Low | High | Sequenced create-new → repoint → drain → delete-old (see rollout.md) | Platform Operator | +| `azure-batch` auto-upgrade breaks build | Medium | Medium | Pinned `==14.2.0` | backend-dev | +| One tenant starves the shared pool | Medium | Medium | Per-tenant task caps + job priority (Phase 1) | backend-dev | + +## Performance Impact + +- **Submit latency:** +1 pool-capacity check per candidate at submit time (skipped for single-candidate lists) + a cached `getUserDelegationKey` per account. Negligible. +- **Batch compute:** Higher *utilization* of scarce GPU (shared nodes never idle if any tenant has work); cold-start latency when a scale-to-zero pool spins up its first node. +- **Storage I/O:** Unchanged volume; credential path differs (SAS vs identity). +- **API / tile serving:** No change. + +## Security Impact + +- [x] New data classification handled? — Multi-tenant compute; isolation is the core requirement. Enforced by per-job container-scoped SAS (credential boundary), **security-reviewed** (US-003). +- [x] New secrets or connection strings required? — No standing secrets. Short-TTL user-delegation SAS minted per job (no account keys). +- [x] New RBAC? — `Storage Blob Delegator` on the Function App identity (mint SAS); ACR-pull-only UMI for shared pools (no storage access). +- [ ] New API endpoints exposed? — No. +- [ ] MSAL/Entra ID auth changes? — No. +- [ ] CORS changes? — No. +- [ ] New federated credentials? — No. + +## Compliance & Data Impact + +- [x] Partner data sharing agreements — Shared *compute*, isolated *data*: a tenant's task cannot read another tenant's storage. This is the explicit design guarantee. +- [ ] Geospatial data sovereignty — No region change (all westus2). +- [ ] New data retention — No. +- [x] Component Governance — One dependency change: `azure-batch` pinned (no new package). New Bicep only. + +## Rollback Assessment + +- **Reversibility:** fully reversible. +- **App code:** All new behavior is behind flags defaulting to legacy (`USE_SAS=false`, `MANAGE_POOLS=true`, single pool id). Reverting the flags (or the branch) restores exact prior behavior. +- **Shared pools:** Additive — deleting them affects only envs that opted in; other envs are untouched. +- **RBAC:** The `Storage Blob Delegator` grant is additive and harmless if unused. +- **Blob/Cosmos data:** No schema or data migration; nothing to roll back. +- **Estimated rollback time:** < 15 min (flip env vars / revert deploy). diff --git a/spec/features/batch-compute-expansion/plan.md b/spec/features/batch-compute-expansion/plan.md new file mode 100644 index 0000000..dfaad9e --- /dev/null +++ b/spec/features/batch-compute-expansion/plan.md @@ -0,0 +1,108 @@ +# Execution Plan: Batch compute expansion + multi-tenant shared GPU pools + +Phases are adapted for a compute/infra + library feature — there is no UI or API +route change. See [user-stories.md](user-stories.md#agent-assignment-map) for the +agent→story mapping and [rollout.md](rollout.md) for the deployment sequence. + +## Phases + +### Phase 1: Design & isolation PoC — done + +**Goal:** De-risk the architecture before provisioning scarce GPU pools. + +| Task | Agent | Dependencies | Story Ref | Status | +|---|---|---|---|---| +| Write spec (README, design, impact-analysis, user-stories, test-plan, rollout) | `backend-dev` | — | — | done | +| PoC: per-job SAS Batch I/O on a throwaway pool | `backend-dev` | — | US-003 | done | +| PoC: cross-tenant SAS access denied | `backend-dev`, `security` | — | US-003 | done | +| PoC: ACR-pull via a shared ACR-pull-only identity | `backend-dev` | — | US-001 | done | + +**Exit Criteria:** +- [x] SAS Batch I/O works as a drop-in for `identity_reference` +- [x] Cross-tenant access is denied at the credential boundary +- [x] Shared-identity ACR pull authenticates + +### Phase 2: Shared-pool IaC — in-progress + +**Goal:** Parameterize the pool module and provision the shared pools. + +| Task | Agent | Dependencies | Story Ref | Status | +|---|---|---|---|---| +| Parameterize `infra/modules/batchPool.bicep` (scale/node/VNet) | `backend-dev` | — | US-001, US-005 | done | +| Add `infra/shared-pools.bicep` + `.bicepparam` (generic prefix, BYO account/ACR) | `backend-dev` | batchPool | US-001 | done | +| Create the ACR-pull-only identity + `AcrPull` grant | `backend-dev` | — | US-001 | done | +| Deploy the shared-dev pool pair (H100 + T4, scale-to-zero) | `backend-dev` | above | US-001, US-005 | done | +| Deploy the shared-demo pool pair | `backend-dev` | quota | US-001 | not-started | +| Finalize pool networking (subnet + per-tenant storage firewall) | `backend-dev` | — | US-003 | not-started | + +**Exit Criteria:** +- [x] Pools created, autoscale low-priority, 0 nodes idle +- [ ] Networking model finalized for real workloads + +### Phase 3: hastelib routing + per-job SAS (v2.1.0) — done + +**Goal:** Capacity-aware routing + per-job SAS in the core library. + +| Task | Agent | Dependencies | Story Ref | Status | +|---|---|---|---|---| +| Candidate pool-id lists + flags in `config.py` | `backend-dev` | — | US-004 | done | +| `select_pool` + submit-time binding + `manage_pools` gate in `azure_batch.py` | `backend-dev` | config | US-004 | done | +| Per-job user-delegation SAS on `ResourceFile`/`OutputFile` | `backend-dev` | — | US-003 | done | +| `Storage Blob Delegator` grant in `functionApp.bicep` | `backend-dev` | — | US-003 | done | +| Update all 5 processors to pass candidate lists | `backend-dev` | above | US-004 | done | +| Unit tests in `hastelib/tests/` (`test_azure_batch_routing.py`) | `backend-dev` | above | US-003, US-004 | done | +| Pin `azure-batch==14.2.0` | `backend-dev` | — | — | done | + +**Exit Criteria:** +- [x] 9 unit tests pass +- [x] All flags default to legacy behavior (no regression for existing envs) + +### Phase 4: Integration & rollout — pending (gated on GPU quota) + +**Goal:** Validate end-to-end on real envs, then roll out. + +| Task | Agent | Dependencies | Story Ref | Status | +|---|---|---|---|---| +| Obtain low-priority GPU quota increase | Platform Operator | — | US-005 | not-started | +| E2E on dev1 + dev2 (two live tenants on shared pools) | `backend-dev` | Phases 2,3 + quota | US-001..004 | not-started | +| Onboard demo environments to shared pools | Platform Operator | E2E | US-002 | not-started | +| (Confirmed) drain + delete legacy per-env pools | Platform Operator | full migration | — | not-started | + +**Exit Criteria:** +- [ ] Both tenants coexist on the shared pool; cross-env access denied (E2E) +- [ ] Routing + spillover behave as designed; pools scale to zero +- [ ] CI (`hatch run test:pytest`) green + +## Milestones + +| Milestone | Date | Deliverable | +|---|---|---| +| Spec approved | 2026-07-14 | Signed-off spec + isolation PoC | +| Shared-dev pools live | 2026-07-14 | Pools provisioned (0 nodes, waiting on quota) | +| v2.1.0 library done | 2026-07-14 | Routing + SAS merged; unit tests green | +| E2E validated | TBD (gated on quota) | dev1 + dev2 on shared pools | +| Release | TBD | Demos onboarded to shared compute | + +## Agent Summary + +| Agent | Tasks Owned | Phases | +|---|---|---| +| `backend-dev` | Library + IaC + PoC + tests | 1, 2, 3, 4 | +| `security` | Isolation-model review | 1 | +| Platform Operator (human) | Quota, onboarding, destructive cleanup | 2, 4 | + +> No `ui` or `gis` tasks — this feature has no UI or imagery-provider change. + +## Resource Requirements + +- **Agents:** `backend-dev` (implements), `backend-validation` (validates), `security` (isolation review). +- **Azure services:** Azure Batch (shared H100 + T4 pools), one ACR-pull-only UMI, `Storage Blob Delegator` grants. No Cosmos/Queue/SWA change. +- **GPU compute:** H100 (`Standard_NC40ads_H100_v5`, 40c) + T4 (`Standard_NC16as_T4_v3`, 16c) low-priority nodes. **Requires a low-priority GPU quota increase** before pools can scale up. +- **External data:** none. + +## Open Questions + +- [ ] Low-priority GPU quota increase — blocks Phase 4 scale-up. +- [ ] Preemption-safety of demo workloads on spot (retry/idempotency). +- [ ] Shared-pool networking (subnet + per-tenant storage-firewall allowlisting). +- [ ] Timing of the destructive legacy-pool cleanup (needs explicit confirmation). diff --git a/spec/features/batch-compute-expansion/rollout.md b/spec/features/batch-compute-expansion/rollout.md new file mode 100644 index 0000000..f2fe0a7 --- /dev/null +++ b/spec/features/batch-compute-expansion/rollout.md @@ -0,0 +1,118 @@ +# Rollout Plan: Batch compute expansion + multi-tenant shared GPU pools + +## Rollout Strategy + +**Type:** feature-flag + phased +**Target date:** TBD (gated on low-priority GPU quota) + +Every runtime behavior change is behind per-environment app settings that default +to the **legacy** single-pool, pool-identity path. Shipping the code is therefore a +no-op until an environment opts in — so code rollout and behavior rollout are +decoupled. + +## Deployment Targets + +| Component | Deployment Method | Target | +|---|---|---| +| Shared pools (`shared-pools.bicep`) | `az deployment group create` (standalone) | Shared Batch account RG | +| `haste-shared-acr-umi` + AcrPull | `az` / Bicep | Shared framework RG | +| `hastelib` (v2.1.0) | Docker rebuild + func app deploy (`azd deploy`) | `api` / `queues` Function Apps | +| `Storage Blob Delegator` grant | `functionApp.bicep` via `azd up`/`provision` | Per env storage | +| Per-env opt-in settings | `azd env set` → app settings | `api` / `queues` app settings | + +## Feature Flags + +| Flag | Location | Default | Description | Kill Switch? | +|---|---|---|---|---| +| `AZURE_BATCH_USE_SAS` | `api`/`queues` app setting | `false` | Per-job SAS blob I/O vs. pool identity | yes (set `false`) | +| `AZURE_BATCH_MANAGE_POOLS` | app setting | `true` | Runner auto-creates/resizes its pool | yes (set `true`) | +| `AZURE_BATCH_TRAINING_POOL_IDS` / `_INFERENCE_POOL_IDS` / `_IMAGERYPREP_POOL_IDS` | app setting | single pool id | Ordered candidate pools | yes (unset → single pool) | + +Reverting all three restores exact prior behavior with no redeploy of code. + +## Rollout Phases + +### Phase 0: Prerequisites + +- [ ] Low-priority GPU quota increase granted (H100=40c, T4=16c per node). +- [x] Shared pools deployed (`-shared-dev-*`, 0 nodes) + `haste-shared-acr-umi`. +- [x] v2.1.0 code merged with flags defaulting to legacy. +- [ ] Confirm demo workloads are preemption-safe (spot). + +### Phase 1: Ship code (no behavior change) + +- **Target:** all envs (via normal deploy). +- **Deployment:** merge v2.1.0; `azd up`/`deploy` applies the `Storage Blob Delegator` grant. Flags stay at defaults. +- **Success criteria:** + - [ ] Existing envs behave identically (single pool, pool identity) — no regression. + - [ ] `Storage Blob Delegator` present on each env's Function App identity. +- **Rollback trigger:** any regression on an existing env → revert branch. + +### Phase 2: Enable on dev1 + dev2 (internal) + +- **Target:** dev1 + dev2 pointed at `-shared-dev-*`. +- **Deployment:** set `USE_SAS=true`, `MANAGE_POOLS=false`, `*_POOL_IDS`=shared on both. +- **Success criteria (E2E, see test-plan.md):** + - [ ] Training + inference from both envs run on the shared pools. + - [ ] Cross-env access denied (dev1's SAS cannot read dev2's storage). + - [ ] Routing + spillover behave as designed; pools scale to zero when idle. +- **Rollback trigger:** isolation failure, SAS/auth errors, or job failures → unset the flags (immediate). + +### Phase 3: Onboard demo envs + +- **Target:** demo environments → `-shared-demo-*` (deploy `shared-pools.bicep` with `HASTE_SHARED_GROUP=demo`). +- **Per env:** own storage + `Storage Blob Delegator` + point `*_POOL_IDS` at the shared demo pools. +- **Success criteria:** + - [ ] Each demo runs jobs on shared compute with isolated data. + - [ ] Adding an env requires **no** shared-pool change. +- **Feature flag cleanup:** none — the flags are the long-term opt-in until every env migrates (then the SAS path can become default). + +### (Optional) Destructive cleanup of legacy per-env pools + +Only after all traffic is on shared pools, and with explicit confirmation: +create-new → repoint → **drain** jobs bound to old pool ids → delete old pools. + +## Rollback Plan + +| Step | Action | Owner | ETA | +|---|---|---|---| +| 1 | Unset `USE_SAS` / `MANAGE_POOLS` / `*_POOL_IDS` on the affected env | Platform Operator | immediate | +| 2 | (If code-level) revert the v2.1.0 branch / redeploy previous | backend-dev | < 15 min | +| 3 | Verify jobs submit to the env's original pool with pool identity | Platform Operator | | + +**Cosmos data rollback required?** no — no schema/data change. +**Blob artifacts cleanup needed?** no — same containers/paths; only the transfer credential changed. + +## Monitoring & Alerting + +### Key Metrics to Watch + +| Metric | Source | Baseline | Alert Threshold | +|---|---|---|---| +| Batch task failure rate | Azure Batch metrics | pre-rollout rate | sustained increase | +| `BlobAccessDenied` on task I/O | Batch task failure info | 0 (expected only for the isolation *test*) | any in normal runs | +| Pool node allocation failures | Batch metrics | 0 | any (quota/capacity signal) | +| Shared pool node-hours | Batch metrics | 0 idle | budget ceiling | +| SAS mint errors (`getUserDelegationKey`) | Function App logs / App Insights | 0 | any | + +### Alerts to Configure + +| Alert | Condition | Severity | Notify | +|---|---|---|---| +| Task auth failures | `BlobAccessDenied` / SAS mint errors in normal runs | P1 | eng team | +| Allocation failures | pool can't scale up (quota) | P2 | Platform Operator | + +## Communication Plan + +| Audience | Channel | When | Message | +|---|---|---|---| +| Engineering team | GitHub PR / Teams | Pre-deploy | Rollout plan + flag semantics | +| Demo owners / partners | — | Before onboarding | Their env moves to shared compute; data stays isolated | + +## Post-Rollout Checklist + +- [ ] Flags documented in `docs/configuration.md` (done) +- [ ] `CHANGELOG.md` updated (done — `[Unreleased]`) +- [ ] GitHub Pages docs rebuilt (`docs-deploy.yml`) +- [ ] Legacy per-env pools cleaned up (only after full migration + confirm) +- [ ] Update `design.md` Build status + this spec's `README.md` status → `implemented` diff --git a/spec/features/batch-compute-expansion/test-plan.md b/spec/features/batch-compute-expansion/test-plan.md new file mode 100644 index 0000000..2aac595 --- /dev/null +++ b/spec/features/batch-compute-expansion/test-plan.md @@ -0,0 +1,104 @@ +# Test Plan: Batch compute expansion + multi-tenant shared GPU pools + +## Test Strategy + +| Level | Scope | Tool/Framework | Coverage Target | +|---|---|---|---| +| Unit | `hastelib` routing + SAS toggle + config parsing | pytest (`hastelib/tests/`) | All new decision logic | +| Infra validation | Bicep compiles + what-if is clean | `az bicep build`, `az deployment ... what-if` | batchPool + shared-pools | +| Isolation PoC | SAS Batch I/O + cross-tenant denial on a real pool | `az batch` + user-delegation SAS | Both bets (done) | +| E2E | Two real envs (dev1 + dev2) on the shared pools | Live workflow runs | The full submit→SAS→Batch path | + +> No UI, API-endpoint, or queue-message-format changes — those test levels are N/A. + +## Test Scenarios + +### Unit Tests (`hastelib/tests/`) + +Implemented in `tests/core/runners/test_azure_batch_routing.py` (9 tests, passing): + +| ID | Module | Scenario | Expected Output | Story Ref | +|---|---|---|---|---| +| UT-001 | `runners/azure_batch.py` | `select_pool` with a single candidate | returns it, no API calls | US-004 | +| UT-002 | `runners/azure_batch.py` | `select_pool` with empty list | falls back to bound pool | US-004 | +| UT-003 | `runners/azure_batch.py` | preferred busy, second has idle node | spills over to the idle pool | US-004 | +| UT-004 | `runners/azure_batch.py` | first candidate has an idle node | returns the first (preferred) | US-004 | +| UT-005 | `runners/azure_batch.py` | no candidate has an idle node | returns preferred (scales/queues) | US-004 | +| UT-006 | `runners/azure_batch.py` | legacy mode (`use_sas=false`) | `_blob_identity()` = pool identity; URL untouched | US-003 | +| UT-007 | `runners/azure_batch.py` | SAS mode (`use_sas=true`) | `_blob_identity()` = None; URL gets SAS | US-003 | +| UT-008 | `runners/azure_batch.py` | `_maybe_sas` on empty URL | no-op | US-003 | +| UT-009 | `core/config.py` | candidate lists split + trimmed; unset → single-id fallback; flags | correct lists + `use_sas`/`manage_pools` | US-004 | + +### Infra Validation + +| ID | Scenario | Expected | +|---|---|---| +| INF-001 | `az bicep build infra/modules/batchPool.bicep` | compiles clean | +| INF-002 | `az bicep build infra/shared-pools.bicep` (via deploy) | compiles clean | +| INF-003 | `what-if` of `shared-pools.bicepparam` against live pools | no real change (Batch pool "Modify" is a known what-if false positive; live formula byte-matches) | + +### Isolation PoC (done — see design.md, "Data isolation → Validation") + +| ID | Scenario | Input | Expected Behavior | +|---|---|---|---| +| POC-001 | SAS-based Batch I/O | task with SAS `ResourceFile` + `OutputFile`, no `identity_reference` | task exit 0; output blob written via SAS | +| POC-002 | Cross-tenant isolation | tenant-A SAS reading tenant-B container | `BlobAccessDenied` (UserError) | +| POC-003 | Shared-identity ACR pull | pool with ACR-pull-only UMI prefetches the real image | authenticates + downloads (disk-size artifact on the cheap test VM only) | + +### End-to-End Tests (dev1 + dev2, gated on GPU quota) + +| ID | User Flow | Steps | Expected Outcome | Story Ref | +|---|---|---|---|---| +| E2E-001 | Two live tenants on shared pools | 1. Point dev1 + dev2 at `-shared-dev-*` with `USE_SAS=true`, `MANAGE_POOLS=false` 2. Run training + inference from each | Both tenants' jobs coexist; each reads/writes only its own storage | US-001, US-002, US-003 | +| E2E-002 | Capacity-aware routing | Submit training (H100-first) + inference (T4-first); saturate one tier | Preferred tier used; spillover when preferred has no idle node | US-004 | +| E2E-003 | Scale-to-zero | Leave pools idle | Pools return to 0 nodes | US-005 | + +### Edge Case & Negative Tests + +| ID | Scenario | Input | Expected Behavior | +|---|---|---|---| +| NEG-001 | SAS mode without `Storage Blob Delegator` | env with `USE_SAS=true`, no grant | `getUserDelegationKey` fails fast with a clear auth error | +| NEG-002 | Candidate pool does not exist | bad pool id in the list | capacity check catches `BatchErrorException`, tries next candidate | +| EDGE-001 | `manage_pools=true` against an autoscale pool | legacy resize path | (documented) resize fails on autoscale — hence `manage_pools=false` for shared pools | +| EDGE-002 | Spot node preempted mid-task | low-priority preemption | Batch reschedules the task (workload must be retry-safe) | + +### Performance Tests + +| ID | Scenario | Target Metric | Threshold | +|---|---|---|---| +| PERF-001 | Submit-time pool selection overhead | added latency per submit | < ~1s (1 `pool.get` per candidate; cached SAS key) | + +## Test Data Requirements + +| Dataset | Description | Source | Sensitive? | +|---|---|---|---| +| Two tenant storage containers | Distinct blobs per tenant for isolation test | Synthetic | no | +| Small input blob | Read by the Batch task | Synthetic | no | + +## Coverage Matrix + +| User Story | Unit | Infra | PoC | E2E | Perf | +|---|---|---|---|---|---| +| US-001 | — | INF-002/003 | — | E2E-001 | — | +| US-002 | UT-009 | — | — | E2E-001 | — | +| US-003 | UT-006/007/008 | — | POC-001/002 | E2E-001 | — | +| US-004 | UT-001..005 | — | — | E2E-002 | PERF-001 | +| US-005 | — | INF-001 | POC-003 | E2E-003 | — | +| US-006 | — | — | — | (manual) | — | + +## Environment Requirements + +| Environment | Purpose | Config | +|---|---|---| +| Local (minimal venv) | Unit tests (no geo/ML deps) | `azure-batch==14.2.0` + pytest; `PYTHONPATH=src` | +| CI (conda `test` env) | Full `hatch run test:pytest` | `dev_env.yml` | +| dev1 + dev2 | E2E on shared pools | `USE_SAS=true`, `MANAGE_POOLS=false`, `*_POOL_IDS`=shared | + +## Sign-off Criteria + +- [x] All new unit tests pass (9/9) +- [x] Bicep compiles clean; shared-pools what-if shows no real drift +- [x] Isolation PoC passes (SAS I/O + cross-tenant denial + ACR pull) +- [ ] E2E on dev1 + dev2 passes (gated on low-priority GPU quota) +- [ ] Full `hatch run test:pytest` green in CI +- [ ] Component Governance scan clean (`azure-batch` pin only) diff --git a/spec/features/batch-compute-expansion/user-stories.md b/spec/features/batch-compute-expansion/user-stories.md new file mode 100644 index 0000000..5e509ad --- /dev/null +++ b/spec/features/batch-compute-expansion/user-stories.md @@ -0,0 +1,216 @@ +# User Stories: Batch compute expansion + multi-tenant shared GPU pools + +## Personas + +| Persona | Description | Key Goals | +|---|---|---| +| ML Engineer | Runs training / inference / embedding jobs from the HASTE app | Jobs land on GPU compute quickly; no per-env pool wrangling | +| Platform Operator | Deploys and operates HASTE environments + shared compute | Serve many envs within scarce GPU quota; add an env cheaply | +| External Partner | Runs a demo environment with their own data | Their imagery/results are never visible to another tenant | + +> Disaster Analyst / Project Manager personas are unaffected by this (compute-layer) feature. + +--- + +## Stories + +### US-001: Provision shared GPU pools once for many environments + +**As a** Platform Operator, +**I want to** provision a small set of shared GPU pools that many environments submit to, +**So that** scarce GPU quota is pooled centrally instead of fragmented across per-env pools. + +**Priority:** P1 +**Estimate:** M +**Component(s):** `infra/shared-pools.bicep`, `infra/modules/batchPool.bicep` + +**Acceptance Criteria:** + +```gherkin +Given the shared Batch account and an ACR-pull-only identity exist +When I deploy shared-pools.bicep with sharedGroup=dev +Then two pools -shared-dev-{h100,t4}-pool are created, autoscale enabled, + low-priority, at 0 nodes (scale-to-zero), consuming no GPU while idle +``` + +```gherkin +Given the shared pools already exist +When I redeploy shared-pools.bicep with the same parameters +Then the deployment is idempotent (no destructive change to the pools) +``` + +**Notes:** Pool count stays at a handful regardless of tenant count (account pool-quota is ~20). + +--- + +### US-002: Add a new environment with zero shared-compute changes + +**As a** Platform Operator, +**I want to** onboard a new demo environment onto the shared pools without modifying them, +**So that** onboarding scales to 20+ tenants without touching a shared, semi-immutable resource. + +**Priority:** P1 +**Estimate:** S +**Component(s):** env `.env` / Function App settings, `infra/modules/functionApp.bicep` + +**Acceptance Criteria:** + +```gherkin +Given the shared pools exist +When I deploy a new env that points AZURE_BATCH_*_POOL_IDS at the shared pool ids, + sets AZURE_BATCH_USE_SAS=true, and whose Function App identity holds Storage Blob Delegator +Then the env can submit jobs to the shared pools with no change to the pools or their identity +``` + +**Notes:** No per-tenant UMI is attached to the pool; the pool is never mutated per tenant. + +--- + +### US-003: A tenant cannot access another tenant's data + +**As an** External Partner, +**I want** my environment's Batch tasks to read/write only my own storage, +**So that** running on shared compute never exposes my imagery/results to another tenant. + +**Priority:** P0 +**Estimate:** M +**Component(s):** `hastelib/runners/azure_batch.py`, `functionApp.bicep` + +**Acceptance Criteria:** + +```gherkin +Given two tenants submit jobs to the same shared pool +When tenant A's task uses tenant A's per-job SAS to access tenant B's container +Then the access is denied by Azure (BlobAccessDenied) — isolation is enforced at + the credential boundary, not by application correctness +``` + +**Notes:** PoC-validated 2026-07-14 (see design.md, "Data isolation → Validation"). The pool holds no standing storage access. + +--- + +### US-004: Training routes to H100, spilling over to T4 + +**As an** ML Engineer, +**I want** training jobs to prefer H100 nodes and spill over to T4 when H100 is busy, +**So that** my job starts on the best available GPU instead of queuing behind a saturated tier. + +**Priority:** P1 +**Estimate:** M +**Component(s):** `hastelib/config.py`, `hastelib/runners/azure_batch.py`, `processors/train.py` + +**Acceptance Criteria:** + +```gherkin +Given AZURE_BATCH_TRAINING_POOL_IDS="h100-pool,t4-pool" +When a training task is submitted and the H100 pool has an idle node +Then the task is bound to the H100 pool at submit time +``` + +```gherkin +Given the H100 pool has no idle node but the T4 pool does +When a training task is submitted +Then the task spills over to the T4 pool +``` + +```gherkin +Given neither candidate pool has an idle node +When a training task is submitted +Then it is bound to the preferred (first) pool, which scales up / queues +``` + +**Notes:** Inference/embedding use `AZURE_BATCH_INFERENCE_POOL_IDS` (T4-first); imageryprep/artifacts use `AZURE_BATCH_IMAGERYPREP_POOL_IDS`. + +--- + +### US-005: Demo burst preserves scarce dedicated GPU quota + +**As a** Platform Operator, +**I want** demo bursts to run on low-priority/spot nodes that scale to zero when idle, +**So that** scarce *dedicated* GPU quota stays reserved for dev/prod and idle demos cost nothing. + +**Priority:** P2 +**Estimate:** S +**Component(s):** `infra/shared-pools.bicep`, `infra/modules/batchPool.bicep` + +**Acceptance Criteria:** + +```gherkin +Given a shared demo pool with no queued tasks +When it is idle +Then it holds 0 nodes (scale-to-zero) and consumes no GPU/cost +``` + +```gherkin +Given demo pools are low-priority +When they scale up +Then they draw from the low-priority quota bucket, not the dedicated GPU quota +``` + +**Notes:** Requires demo workloads to tolerate preemption (Batch reschedules preempted low-priority tasks). + +--- + +### US-006: No single tenant starves the shared pool + +**As a** Platform Operator, +**I want** per-tenant limits so one environment can't monopolize the shared pool, +**So that** 20+ demo tenants get fair access to shared compute. + +**Priority:** P2 +**Estimate:** M +**Component(s):** `hastelib` (submit path), Batch job priority + +**Acceptance Criteria:** + +```gherkin +Given a tenant already has its cap of active tasks on the shared pool +When it submits another task +Then the task is held/queued rather than admitted ahead of other tenants +``` + +**Notes:** Phase 1 = per-tenant concurrent-task caps + Batch job priority. Escalate to an admission broker only if starvation is observed. + +--- + +## Agent Assignment Map + +### Available Agents + +| Agent | Scope | Touches Code? | +|---|---|---| +| `backend-dev` | Python backend + infra (hastelib runners/config, Bicep) | Yes | +| `backend-validation` | Validates backend/infra against specs, conventions, tests | No (validates only) | +| `security` | Reviews the multi-tenant isolation model + new grants/deps | No (reports only) | +| `orchestrator` | Tracks spec status | No (observes only) | + +> UI / GIS agents are not involved (no UI or imagery-provider change). + +### Story → Agent Mapping + +| Story | Implementing Agent(s) | Validating Agent(s) | Notes | +|---|---|---|---| +| US-001 | `backend-dev` | `backend-validation` | IaC (shared-pools/batchPool) | +| US-002 | `backend-dev` | `backend-validation` | env wiring + Blob Delegator grant | +| US-003 | `backend-dev` | `security`, `backend-validation` | isolation boundary — security-reviewed | +| US-004 | `backend-dev` | `backend-validation` | `select_pool` routing | +| US-005 | `backend-dev` | `backend-validation` | autoscale/spot config | +| US-006 | `backend-dev` | `backend-validation` | fairness caps | + +### Story Map + +| Priority | Story | Phase | Implementing Agent | Component | +|---|---|---|---|---| +| P0 | US-003 | Isolation | `backend-dev` | `hastelib` + `functionApp.bicep` | +| P1 | US-001 | Pools (IaC) | `backend-dev` | `infra/` | +| P1 | US-002 | Onboarding | `backend-dev` | env `.env` | +| P1 | US-004 | Routing | `backend-dev` | `hastelib` | +| P2 | US-005 | Cost/quota | `backend-dev` | `infra/` | +| P2 | US-006 | Fairness | `backend-dev` | `hastelib` | + +## Out of Scope + +- [ ] Migrating dev/prod off their existing single dedicated pool onto H100+T4 pairs — future work. +- [ ] An admission-broker fair-share service — only if per-tenant caps prove insufficient. +- [ ] Migrating to the `azure-batch` 15.x (track-2) SDK — tracked separately. +- [ ] Cross-region GPU bursting. From 5d2420b3c4a0a028cf5ed67d70db5dce7932ff9f Mon Sep 17 00:00:00 2001 From: Meygha Machado Date: Wed, 15 Jul 2026 00:27:03 -0700 Subject: [PATCH 2/5] use dedicated nodes --- infra/main.bicep | 20 ++++++++++++++++++++ infra/main.bicepparam | 7 +++++++ infra/modules/functions.bicep | 24 ++++++++++++++++++++++++ infra/shared-pools.bicep | 11 +++++++++-- infra/shared-pools.bicepparam | 1 + 5 files changed, 61 insertions(+), 2 deletions(-) diff --git a/infra/main.bicep b/infra/main.bicep index 73df584..2a7eaa8 100644 --- a/infra/main.bicep +++ b/infra/main.bicep @@ -102,6 +102,21 @@ param enableFrontDoor bool = false @description('Dev-only: api/queues auto-provision any authenticated user as admin and use anonymous auth. Keep false for production.') param developmentMode bool = false +@description('Ordered candidate training pool ids (comma-separated). Empty => single training pool.') +param trainingPoolIds string = '' + +@description('Ordered candidate inference/embedding pool ids (comma-separated).') +param inferencePoolIds string = '' + +@description('Ordered candidate imageryprep/artifacts pool ids (comma-separated).') +param imageryprepPoolIds string = '' + +@description('Use per-job user-delegation SAS for Batch blob I/O (multi-tenant shared pools).') +param useSas bool = false + +@description('Runner auto-creates/resizes its pool. False for pre-created autoscale pools.') +param managePools bool = true + // --------------------------------------------------------------------------- // Computed names — mirror the bash naming scheme exactly. // --------------------------------------------------------------------------- @@ -269,6 +284,11 @@ module functions 'modules/functions.bicep' = { emailConnectionString: communication.outputs.connectionString batchAccountKey: batchAccountRef.listKeys().primary developmentMode: developmentMode + trainingPoolIds: trainingPoolIds + inferencePoolIds: inferencePoolIds + imageryprepPoolIds: imageryprepPoolIds + useSas: useSas + managePools: managePools tags: tags } dependsOn: [ diff --git a/infra/main.bicepparam b/infra/main.bicepparam index 4e9342e..e2f7728 100644 --- a/infra/main.bicepparam +++ b/infra/main.bicepparam @@ -37,3 +37,10 @@ param enableFrontDoor = bool(readEnvironmentVariable('HASTE_ENABLE_FRONT_DOOR', // Dev-only auto-provisioning + anonymous auth (never true for production). param developmentMode = bool(readEnvironmentVariable('HASTE_DEVELOPMENT_MODE', 'false')) + +// v2.1.0 capacity-aware routing + per-job SAS (default = legacy single-pool). +param trainingPoolIds = readEnvironmentVariable('HASTE_BATCH_TRAINING_POOL_IDS', '') +param inferencePoolIds = readEnvironmentVariable('HASTE_BATCH_INFERENCE_POOL_IDS', '') +param imageryprepPoolIds = readEnvironmentVariable('HASTE_BATCH_IMAGERYPREP_POOL_IDS', '') +param useSas = bool(readEnvironmentVariable('HASTE_BATCH_USE_SAS', 'false')) +param managePools = bool(readEnvironmentVariable('HASTE_BATCH_MANAGE_POOLS', 'true')) diff --git a/infra/modules/functions.bicep b/infra/modules/functions.bicep index 50f16ba..63ea211 100644 --- a/infra/modules/functions.bicep +++ b/infra/modules/functions.bicep @@ -72,6 +72,23 @@ param batchAccountKey string @description('Dev-only: auto-provision any authenticated user as admin and drop function-key auth (anonymous). Must be false for production.') param developmentMode bool = false +// --- v2.1.0 capacity-aware routing + per-job SAS ----------------------------- + +@description('Ordered candidate training pool ids (comma-separated). Empty => single training pool.') +param trainingPoolIds string = '' + +@description('Ordered candidate inference/embedding pool ids (comma-separated).') +param inferencePoolIds string = '' + +@description('Ordered candidate imageryprep/artifacts pool ids (comma-separated).') +param imageryprepPoolIds string = '' + +@description('Use per-job user-delegation SAS for Batch blob I/O (multi-tenant shared pools).') +param useSas bool = false + +@description('Runner auto-creates/resizes its pool. False for pre-created autoscale pools.') +param managePools bool = true + @description('Resource tags.') param tags object = {} @@ -125,6 +142,13 @@ var appConfigSettings = [ { name: 'STATIC_APP_DOMAIN', value: staticAppDomain } { name: 'EMAIL_CONNECTION_STRING', value: emailConnectionString } { name: 'EMAIL_SENDER', value: 'DoNotReply@${emailSenderDomain}' } + // v2.1.0: capacity-aware routing candidate lists + per-job SAS toggle. Empty + // lists / false flags = legacy single-pool, pool-identity behavior. + { name: 'AZURE_BATCH_TRAINING_POOL_IDS', value: trainingPoolIds } + { name: 'AZURE_BATCH_INFERENCE_POOL_IDS', value: inferencePoolIds } + { name: 'AZURE_BATCH_IMAGERYPREP_POOL_IDS', value: imageryprepPoolIds } + { name: 'AZURE_BATCH_USE_SAS', value: useSas ? 'true' : 'false' } + { name: 'AZURE_BATCH_MANAGE_POOLS', value: managePools ? 'true' : 'false' } ] module apiApp 'functionApp.bicep' = { diff --git a/infra/shared-pools.bicep b/infra/shared-pools.bicep index 8af2616..d179288 100644 --- a/infra/shared-pools.bicep +++ b/infra/shared-pools.bicep @@ -39,6 +39,13 @@ param h100MaxNodes int = 2 @description('T4 autoscale max nodes (formula cap).') param t4MaxNodes int = 2 +@description('Node cost tier. Dedicated (draws from the account dedicated-core quota). Set LowPriority only where low-priority GPU quota is available (cheaper, preemptible).') +@allowed([ + 'Dedicated' + 'LowPriority' +]) +param sharedNodeType string = 'Dedicated' + module h100Pool 'modules/batchPool.bicep' = { name: 'shared-${sharedGroup}-h100' params: { @@ -46,7 +53,7 @@ module h100Pool 'modules/batchPool.bicep' = { poolName: '${sharedPrefix}-shared-${sharedGroup}-h100-pool' vmSize: 'Standard_NC40ads_H100_v5' scaleMode: 'Autoscale' - nodeType: 'LowPriority' + nodeType: sharedNodeType minNodes: 0 maxNodes: h100MaxNodes umiResourceId: umiResourceId @@ -65,7 +72,7 @@ module t4Pool 'modules/batchPool.bicep' = { poolName: '${sharedPrefix}-shared-${sharedGroup}-t4-pool' vmSize: 'Standard_NC16as_T4_v3' scaleMode: 'Autoscale' - nodeType: 'LowPriority' + nodeType: sharedNodeType minNodes: 0 maxNodes: t4MaxNodes umiResourceId: umiResourceId diff --git a/infra/shared-pools.bicepparam b/infra/shared-pools.bicepparam index 9b0e6c8..4617a32 100644 --- a/infra/shared-pools.bicepparam +++ b/infra/shared-pools.bicepparam @@ -17,6 +17,7 @@ using './shared-pools.bicep' param sharedPrefix = readEnvironmentVariable('HASTE_RESOURCE_PREFIX', 'haste') param sharedGroup = readEnvironmentVariable('HASTE_SHARED_GROUP', 'dev') +param sharedNodeType = readEnvironmentVariable('HASTE_SHARED_NODE_TYPE', 'Dedicated') param batchAccountName = readEnvironmentVariable('HASTE_EXISTING_BATCH_ACCOUNT', '') param acrName = readEnvironmentVariable('HASTE_SHARED_ACR_NAME', '') param umiResourceId = readEnvironmentVariable('HASTE_SHARED_UMI_ID', '') From e10bfe2b390de39b5069c763104ed4dbc1083b3e Mon Sep 17 00:00:00 2001 From: Meygha Machado Date: Wed, 15 Jul 2026 11:38:11 -0700 Subject: [PATCH 3/5] tweak shared pools --- infra/shared-pools.bicep | 18 +- infra/shared-pools.bicepparam | 3 + infra/shared-pools.json | 475 ++++++++++++++++++++++++++++++++++ 3 files changed, 494 insertions(+), 2 deletions(-) create mode 100644 infra/shared-pools.json diff --git a/infra/shared-pools.bicep b/infra/shared-pools.bicep index d179288..d4b3f39 100644 --- a/infra/shared-pools.bicep +++ b/infra/shared-pools.bicep @@ -39,6 +39,9 @@ param h100MaxNodes int = 2 @description('T4 autoscale max nodes (formula cap).') param t4MaxNodes int = 2 +@description('T4 autoscale floor. 0 = scale-to-zero when idle; >0 keeps a warm baseline of always-on nodes.') +param t4MinNodes int = 0 + @description('Node cost tier. Dedicated (draws from the account dedicated-core quota). Set LowPriority only where low-priority GPU quota is available (cheaper, preemptible).') @allowed([ 'Dedicated' @@ -46,13 +49,24 @@ param t4MaxNodes int = 2 ]) param sharedNodeType string = 'Dedicated' +@description('H100 pool scale mode. Fixed reserves a baseline for the scarce GPU tier (recommended for contended regions where scale-to-zero can never reacquire a node); Autoscale scales-to-zero when idle.') +@allowed([ + 'Fixed' + 'Autoscale' +]) +param h100ScaleMode string = 'Autoscale' + +@description('H100 reserved node count when h100ScaleMode == Fixed.') +param h100FixedNodeCount int = 1 + module h100Pool 'modules/batchPool.bicep' = { name: 'shared-${sharedGroup}-h100' params: { batchAccountName: batchAccountName poolName: '${sharedPrefix}-shared-${sharedGroup}-h100-pool' vmSize: 'Standard_NC40ads_H100_v5' - scaleMode: 'Autoscale' + scaleMode: h100ScaleMode + fixedNodeCount: h100FixedNodeCount nodeType: sharedNodeType minNodes: 0 maxNodes: h100MaxNodes @@ -73,7 +87,7 @@ module t4Pool 'modules/batchPool.bicep' = { vmSize: 'Standard_NC16as_T4_v3' scaleMode: 'Autoscale' nodeType: sharedNodeType - minNodes: 0 + minNodes: t4MinNodes maxNodes: t4MaxNodes umiResourceId: umiResourceId acrName: acrName diff --git a/infra/shared-pools.bicepparam b/infra/shared-pools.bicepparam index 4617a32..61bb37f 100644 --- a/infra/shared-pools.bicepparam +++ b/infra/shared-pools.bicepparam @@ -18,6 +18,9 @@ using './shared-pools.bicep' param sharedPrefix = readEnvironmentVariable('HASTE_RESOURCE_PREFIX', 'haste') param sharedGroup = readEnvironmentVariable('HASTE_SHARED_GROUP', 'dev') param sharedNodeType = readEnvironmentVariable('HASTE_SHARED_NODE_TYPE', 'Dedicated') +param h100ScaleMode = readEnvironmentVariable('HASTE_SHARED_H100_SCALE_MODE', 'Autoscale') +param t4MinNodes = int(readEnvironmentVariable('HASTE_SHARED_T4_MIN_NODES', '0')) +param t4MaxNodes = int(readEnvironmentVariable('HASTE_SHARED_T4_MAX_NODES', '2')) param batchAccountName = readEnvironmentVariable('HASTE_EXISTING_BATCH_ACCOUNT', '') param acrName = readEnvironmentVariable('HASTE_SHARED_ACR_NAME', '') param umiResourceId = readEnvironmentVariable('HASTE_SHARED_UMI_ID', '') diff --git a/infra/shared-pools.json b/infra/shared-pools.json new file mode 100644 index 0000000..291f0ca --- /dev/null +++ b/infra/shared-pools.json @@ -0,0 +1,475 @@ +{ + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "1769188842529029875" + } + }, + "parameters": { + "sharedPrefix": { + "type": "string", + "defaultValue": "haste", + "metadata": { + "description": "Resource-name prefix. Generic default; override per deployment." + } + }, + "sharedGroup": { + "type": "string", + "metadata": { + "description": "Shared group these pools serve (e.g. dev, demo)." + } + }, + "batchAccountName": { + "type": "string", + "metadata": { + "description": "Existing shared Batch account name (BYO — no generic default)." + } + }, + "umiResourceId": { + "type": "string", + "metadata": { + "description": "Shared ACR-pull UMI resource id." + } + }, + "acrName": { + "type": "string", + "metadata": { + "description": "Existing shared ACR name, without .azurecr.io (BYO — no generic default)." + } + }, + "trainingImage": { + "type": "string", + "defaultValue": "hastetraining:2.0.0", + "metadata": { + "description": "Training image (H100 tier + spillover)." + } + }, + "imageryprepImage": { + "type": "string", + "defaultValue": "hasteimageryprep:2.0.0", + "metadata": { + "description": "Imageryprep image (T4 tier + spillover)." + } + }, + "h100MaxNodes": { + "type": "int", + "defaultValue": 2, + "metadata": { + "description": "H100 autoscale max nodes (formula cap)." + } + }, + "t4MaxNodes": { + "type": "int", + "defaultValue": 2, + "metadata": { + "description": "T4 autoscale max nodes (formula cap)." + } + }, + "t4MinNodes": { + "type": "int", + "defaultValue": 0, + "metadata": { + "description": "T4 autoscale floor. 0 = scale-to-zero when idle; >0 keeps a warm baseline of always-on nodes." + } + }, + "sharedNodeType": { + "type": "string", + "defaultValue": "Dedicated", + "allowedValues": [ + "Dedicated", + "LowPriority" + ], + "metadata": { + "description": "Node cost tier. Dedicated (draws from the account dedicated-core quota). Set LowPriority only where low-priority GPU quota is available (cheaper, preemptible)." + } + }, + "h100ScaleMode": { + "type": "string", + "defaultValue": "Autoscale", + "allowedValues": [ + "Fixed", + "Autoscale" + ], + "metadata": { + "description": "H100 pool scale mode. Fixed reserves a baseline for the scarce GPU tier (recommended for contended regions where scale-to-zero can never reacquire a node); Autoscale scales-to-zero when idle." + } + }, + "h100FixedNodeCount": { + "type": "int", + "defaultValue": 1, + "metadata": { + "description": "H100 reserved node count when h100ScaleMode == Fixed." + } + } + }, + "resources": [ + { + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "[format('shared-{0}-h100', parameters('sharedGroup'))]", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "batchAccountName": { + "value": "[parameters('batchAccountName')]" + }, + "poolName": { + "value": "[format('{0}-shared-{1}-h100-pool', parameters('sharedPrefix'), parameters('sharedGroup'))]" + }, + "vmSize": { + "value": "Standard_NC40ads_H100_v5" + }, + "scaleMode": { + "value": "[parameters('h100ScaleMode')]" + }, + "fixedNodeCount": { + "value": "[parameters('h100FixedNodeCount')]" + }, + "nodeType": { + "value": "[parameters('sharedNodeType')]" + }, + "minNodes": { + "value": 0 + }, + "maxNodes": { + "value": "[parameters('h100MaxNodes')]" + }, + "umiResourceId": { + "value": "[parameters('umiResourceId')]" + }, + "acrName": { + "value": "[parameters('acrName')]" + }, + "trainingImage": { + "value": "[parameters('trainingImage')]" + }, + "imageryprepImage": { + "value": "[parameters('imageryprepImage')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "14366334891975298585" + } + }, + "parameters": { + "batchAccountName": { + "type": "string", + "metadata": { + "description": "Resolved Batch account name (existing or just-created)." + } + }, + "poolName": { + "type": "string", + "metadata": { + "description": "Pool name." + } + }, + "vmSize": { + "type": "string", + "metadata": { + "description": "Pool VM size." + } + }, + "maxNodes": { + "type": "int", + "metadata": { + "description": "Max nodes (autoscale cap)." + } + }, + "scaleMode": { + "type": "string", + "defaultValue": "Autoscale", + "allowedValues": [ + "Fixed", + "Autoscale" + ], + "metadata": { + "description": "Scale mode: Fixed (dev/prod reserved) or Autoscale (shared-demo burst)." + } + }, + "nodeType": { + "type": "string", + "defaultValue": "Dedicated", + "allowedValues": [ + "Dedicated", + "LowPriority" + ], + "metadata": { + "description": "Node cost tier: Dedicated (guaranteed) or LowPriority (spot, preemptible)." + } + }, + "fixedNodeCount": { + "type": "int", + "defaultValue": 1, + "metadata": { + "description": "Node count when scaleMode == Fixed." + } + }, + "minNodes": { + "type": "int", + "defaultValue": 1, + "metadata": { + "description": "Autoscale floor. 0 = scale-to-zero when idle (shared-demo burst); 1 keeps the legacy always-on behavior." + } + }, + "umiResourceId": { + "type": "string", + "metadata": { + "description": "User-assigned managed identity resource id (for ACR pull)." + } + }, + "subnetId": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "VNet subnet resource id for VNet-injected pools. Empty => no VNet injection (BatchManaged public IPs). Shared-demo pools finalize their subnet + per-demo-storage firewall allowlisting before running workloads." + } + }, + "acrName": { + "type": "string", + "metadata": { + "description": "Shared ACR name (without .azurecr.io)." + } + }, + "trainingImage": { + "type": "string", + "metadata": { + "description": "Training container image (tag included)." + } + }, + "imageryprepImage": { + "type": "string", + "metadata": { + "description": "Imageryprep container image (tag included)." + } + } + }, + "variables": { + "registryServer": "[format('{0}.azurecr.io', parameters('acrName'))]", + "scaleTargetVar": "[if(equals(parameters('nodeType'), 'LowPriority'), '$TargetLowPriorityNodes', '$TargetDedicatedNodes')]", + "autoscaleFormula": "[format('$samples = $ActiveTasks.GetSamplePercent(TimeInterval_Minute * 15);$tasks = $samples < 70 ? max(0, $ActiveTasks.GetSample(1)) : max($ActiveTasks.GetSample(1), avg($ActiveTasks.GetSample(TimeInterval_Minute * 15)));$targetVMs = $tasks > 0 ? $tasks : {0};{1} = max({2}, min($targetVMs, {3}));$NodeDeallocationOption = taskcompletion;', parameters('minNodes'), variables('scaleTargetVar'), parameters('minNodes'), parameters('maxNodes'))]", + "scaleSettings": "[if(equals(parameters('scaleMode'), 'Fixed'), createObject('fixedScale', createObject('targetDedicatedNodes', if(equals(parameters('nodeType'), 'Dedicated'), parameters('fixedNodeCount'), 0), 'targetLowPriorityNodes', if(equals(parameters('nodeType'), 'LowPriority'), parameters('fixedNodeCount'), 0), 'resizeTimeout', 'PT15M')), createObject('autoScale', createObject('formula', variables('autoscaleFormula'), 'evaluationInterval', 'PT5M')))]" + }, + "resources": [ + { + "type": "Microsoft.Batch/batchAccounts/pools", + "apiVersion": "2024-07-01", + "name": "[format('{0}/{1}', parameters('batchAccountName'), parameters('poolName'))]", + "identity": { + "type": "UserAssigned", + "userAssignedIdentities": { + "[format('{0}', parameters('umiResourceId'))]": {} + } + }, + "properties": "[shallowMerge(createArray(createObject('vmSize', parameters('vmSize'), 'interNodeCommunication', 'Disabled', 'taskSlotsPerNode', 1, 'taskSchedulingPolicy', createObject('nodeFillType', 'Pack'), 'deploymentConfiguration', createObject('virtualMachineConfiguration', createObject('imageReference', createObject('publisher', 'microsoft-dsvm', 'offer', 'ubuntu-hpc', 'sku', '2204', 'version', 'latest'), 'nodeAgentSkuId', 'batch.node.ubuntu 22.04', 'osDisk', createObject('caching', 'None', 'diskSizeGB', 1023, 'managedDisk', createObject('storageAccountType', 'Premium_LRS')), 'containerConfiguration', createObject('type', 'DockerCompatible', 'containerImageNames', createArray(format('{0}/{1}', variables('registryServer'), parameters('trainingImage')), format('{0}/{1}', variables('registryServer'), parameters('imageryprepImage'))), 'containerRegistries', createArray(createObject('registryServer', format('https://{0}', variables('registryServer')), 'identityReference', createObject('resourceId', parameters('umiResourceId'))))), 'nodePlacementConfiguration', createObject('policy', 'Regional')))), if(empty(parameters('subnetId')), createObject(), createObject('networkConfiguration', createObject('subnetId', parameters('subnetId'), 'publicIPAddressConfiguration', createObject('provision', 'BatchManaged'), 'dynamicVnetAssignmentScope', 'none', 'enableAcceleratedNetworking', false()))), createObject('scaleSettings', variables('scaleSettings'))))]" + } + ], + "outputs": { + "poolName": { + "type": "string", + "value": "[parameters('poolName')]" + } + } + } + } + }, + { + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "[format('shared-{0}-t4', parameters('sharedGroup'))]", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "batchAccountName": { + "value": "[parameters('batchAccountName')]" + }, + "poolName": { + "value": "[format('{0}-shared-{1}-t4-pool', parameters('sharedPrefix'), parameters('sharedGroup'))]" + }, + "vmSize": { + "value": "Standard_NC16as_T4_v3" + }, + "scaleMode": { + "value": "Autoscale" + }, + "nodeType": { + "value": "[parameters('sharedNodeType')]" + }, + "minNodes": { + "value": "[parameters('t4MinNodes')]" + }, + "maxNodes": { + "value": "[parameters('t4MaxNodes')]" + }, + "umiResourceId": { + "value": "[parameters('umiResourceId')]" + }, + "acrName": { + "value": "[parameters('acrName')]" + }, + "trainingImage": { + "value": "[parameters('trainingImage')]" + }, + "imageryprepImage": { + "value": "[parameters('imageryprepImage')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "14366334891975298585" + } + }, + "parameters": { + "batchAccountName": { + "type": "string", + "metadata": { + "description": "Resolved Batch account name (existing or just-created)." + } + }, + "poolName": { + "type": "string", + "metadata": { + "description": "Pool name." + } + }, + "vmSize": { + "type": "string", + "metadata": { + "description": "Pool VM size." + } + }, + "maxNodes": { + "type": "int", + "metadata": { + "description": "Max nodes (autoscale cap)." + } + }, + "scaleMode": { + "type": "string", + "defaultValue": "Autoscale", + "allowedValues": [ + "Fixed", + "Autoscale" + ], + "metadata": { + "description": "Scale mode: Fixed (dev/prod reserved) or Autoscale (shared-demo burst)." + } + }, + "nodeType": { + "type": "string", + "defaultValue": "Dedicated", + "allowedValues": [ + "Dedicated", + "LowPriority" + ], + "metadata": { + "description": "Node cost tier: Dedicated (guaranteed) or LowPriority (spot, preemptible)." + } + }, + "fixedNodeCount": { + "type": "int", + "defaultValue": 1, + "metadata": { + "description": "Node count when scaleMode == Fixed." + } + }, + "minNodes": { + "type": "int", + "defaultValue": 1, + "metadata": { + "description": "Autoscale floor. 0 = scale-to-zero when idle (shared-demo burst); 1 keeps the legacy always-on behavior." + } + }, + "umiResourceId": { + "type": "string", + "metadata": { + "description": "User-assigned managed identity resource id (for ACR pull)." + } + }, + "subnetId": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "VNet subnet resource id for VNet-injected pools. Empty => no VNet injection (BatchManaged public IPs). Shared-demo pools finalize their subnet + per-demo-storage firewall allowlisting before running workloads." + } + }, + "acrName": { + "type": "string", + "metadata": { + "description": "Shared ACR name (without .azurecr.io)." + } + }, + "trainingImage": { + "type": "string", + "metadata": { + "description": "Training container image (tag included)." + } + }, + "imageryprepImage": { + "type": "string", + "metadata": { + "description": "Imageryprep container image (tag included)." + } + } + }, + "variables": { + "registryServer": "[format('{0}.azurecr.io', parameters('acrName'))]", + "scaleTargetVar": "[if(equals(parameters('nodeType'), 'LowPriority'), '$TargetLowPriorityNodes', '$TargetDedicatedNodes')]", + "autoscaleFormula": "[format('$samples = $ActiveTasks.GetSamplePercent(TimeInterval_Minute * 15);$tasks = $samples < 70 ? max(0, $ActiveTasks.GetSample(1)) : max($ActiveTasks.GetSample(1), avg($ActiveTasks.GetSample(TimeInterval_Minute * 15)));$targetVMs = $tasks > 0 ? $tasks : {0};{1} = max({2}, min($targetVMs, {3}));$NodeDeallocationOption = taskcompletion;', parameters('minNodes'), variables('scaleTargetVar'), parameters('minNodes'), parameters('maxNodes'))]", + "scaleSettings": "[if(equals(parameters('scaleMode'), 'Fixed'), createObject('fixedScale', createObject('targetDedicatedNodes', if(equals(parameters('nodeType'), 'Dedicated'), parameters('fixedNodeCount'), 0), 'targetLowPriorityNodes', if(equals(parameters('nodeType'), 'LowPriority'), parameters('fixedNodeCount'), 0), 'resizeTimeout', 'PT15M')), createObject('autoScale', createObject('formula', variables('autoscaleFormula'), 'evaluationInterval', 'PT5M')))]" + }, + "resources": [ + { + "type": "Microsoft.Batch/batchAccounts/pools", + "apiVersion": "2024-07-01", + "name": "[format('{0}/{1}', parameters('batchAccountName'), parameters('poolName'))]", + "identity": { + "type": "UserAssigned", + "userAssignedIdentities": { + "[format('{0}', parameters('umiResourceId'))]": {} + } + }, + "properties": "[shallowMerge(createArray(createObject('vmSize', parameters('vmSize'), 'interNodeCommunication', 'Disabled', 'taskSlotsPerNode', 1, 'taskSchedulingPolicy', createObject('nodeFillType', 'Pack'), 'deploymentConfiguration', createObject('virtualMachineConfiguration', createObject('imageReference', createObject('publisher', 'microsoft-dsvm', 'offer', 'ubuntu-hpc', 'sku', '2204', 'version', 'latest'), 'nodeAgentSkuId', 'batch.node.ubuntu 22.04', 'osDisk', createObject('caching', 'None', 'diskSizeGB', 1023, 'managedDisk', createObject('storageAccountType', 'Premium_LRS')), 'containerConfiguration', createObject('type', 'DockerCompatible', 'containerImageNames', createArray(format('{0}/{1}', variables('registryServer'), parameters('trainingImage')), format('{0}/{1}', variables('registryServer'), parameters('imageryprepImage'))), 'containerRegistries', createArray(createObject('registryServer', format('https://{0}', variables('registryServer')), 'identityReference', createObject('resourceId', parameters('umiResourceId'))))), 'nodePlacementConfiguration', createObject('policy', 'Regional')))), if(empty(parameters('subnetId')), createObject(), createObject('networkConfiguration', createObject('subnetId', parameters('subnetId'), 'publicIPAddressConfiguration', createObject('provision', 'BatchManaged'), 'dynamicVnetAssignmentScope', 'none', 'enableAcceleratedNetworking', false()))), createObject('scaleSettings', variables('scaleSettings'))))]" + } + ], + "outputs": { + "poolName": { + "type": "string", + "value": "[parameters('poolName')]" + } + } + } + } + } + ], + "outputs": { + "h100PoolName": { + "type": "string", + "value": "[reference(resourceId('Microsoft.Resources/deployments', format('shared-{0}-h100', parameters('sharedGroup'))), '2025-04-01').outputs.poolName.value]" + }, + "t4PoolName": { + "type": "string", + "value": "[reference(resourceId('Microsoft.Resources/deployments', format('shared-{0}-t4', parameters('sharedGroup'))), '2025-04-01').outputs.poolName.value]" + } + } +} \ No newline at end of file From 95b51985b88f82b825362132df11eefe7483bf0a Mon Sep 17 00:00:00 2001 From: Meygha Machado Date: Wed, 15 Jul 2026 11:39:45 -0700 Subject: [PATCH 4/5] fresh build to make sure priority ordering works as expected --- api/hastefuncapi/requirements.txt | 2 +- api/hastefuncqueues/requirements.txt | 2 +- docker/imageryprep/requirements.txt | 2 +- hastelib/src/hastegeo/__about__.py | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/api/hastefuncapi/requirements.txt b/api/hastefuncapi/requirements.txt index 3d399cb..c5f5490 100644 --- a/api/hastefuncapi/requirements.txt +++ b/api/hastefuncapi/requirements.txt @@ -29,5 +29,5 @@ tenacity==9.1.2 typing-extensions==4.14.1 docker==7.1.0 # Use wheel for remote, comment out for local Docker (source is copied directly) -hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.27-py3-none-any.whl +hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.28-py3-none-any.whl # -e ../../hastelib # for local development only diff --git a/api/hastefuncqueues/requirements.txt b/api/hastefuncqueues/requirements.txt index 3d399cb..c5f5490 100644 --- a/api/hastefuncqueues/requirements.txt +++ b/api/hastefuncqueues/requirements.txt @@ -29,5 +29,5 @@ tenacity==9.1.2 typing-extensions==4.14.1 docker==7.1.0 # Use wheel for remote, comment out for local Docker (source is copied directly) -hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.27-py3-none-any.whl +hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.28-py3-none-any.whl # -e ../../hastelib # for local development only diff --git a/docker/imageryprep/requirements.txt b/docker/imageryprep/requirements.txt index f170887..d8e8489 100644 --- a/docker/imageryprep/requirements.txt +++ b/docker/imageryprep/requirements.txt @@ -31,4 +31,4 @@ pyarrow==23.0.1 fiona==1.10.1 fsspec==2024.10.0 adlfs==2024.12.0 -hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.27-py3-none-any.whl +hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.28-py3-none-any.whl diff --git a/hastelib/src/hastegeo/__about__.py b/hastelib/src/hastegeo/__about__.py index bbf14bb..97fa4ca 100644 --- a/hastelib/src/hastegeo/__about__.py +++ b/hastelib/src/hastegeo/__about__.py @@ -5,4 +5,4 @@ # storage (see ../../../haste_build.py) so that source/Docker installs report # the same version that was published. Committed value tracks the latest # published release. -__version__ = "1.0.27" +__version__ = "1.0.28" From 2e591d6f4c528b1cbc5f056c7c055aa169e8436d Mon Sep 17 00:00:00 2001 From: Meygha Machado Date: Wed, 15 Jul 2026 19:06:14 -0700 Subject: [PATCH 5/5] vnet fixes for network reachability between batch and blob --- docs/configuration.md | 36 +- infra/main.bicep | 4 + infra/main.bicepparam | 4 + infra/main.json | 3925 +++++++++++++++++ infra/modules/storage.bicep | 16 +- infra/shared-pools.bicep | 12 +- infra/shared-pools.bicepparam | 2 + infra/shared-pools.json | 24 +- .../batch-compute-expansion/README.md | 2 + .../batch-compute-expansion/networking.md | 95 + 10 files changed, 4109 insertions(+), 11 deletions(-) create mode 100644 infra/main.json create mode 100644 spec/features/batch-compute-expansion/networking.md diff --git a/docs/configuration.md b/docs/configuration.md index 02e1822..5f37af9 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -120,10 +120,23 @@ az deployment group create -g \ ``` Pools are named `-shared---pool` (e.g. the dev-group H100 -pool); `HASTE_SHARED_GROUP` selects the group (`dev`, -`demo`, …). They autoscale on low-priority nodes and scale to zero when idle. The -pool identity is used **only** for ACR pull (`haste-shared-acr-umi`) — it holds no -storage access. +pool); `HASTE_SHARED_GROUP` selects the group (`dev`, `demo`, …). They autoscale +on **dedicated** nodes within a central per-pool core-quota ceiling. A scarce tier +(H100) can keep a warm floor instead of scaling to zero — under regional GPU +contention an autoscale floor (or `Fixed` target) **auto-retries allocation each +evaluation** until capacity frees, whereas a one-shot fixed resize gets stuck in +its error state. The pool identity is used **only** for ACR pull +(`haste-shared-acr-umi`) — it holds no storage access. + +Shared-pool deploy knobs (env vars read by `shared-pools.bicep`): + +| Variable | Default | Purpose | +|---|---|---| +| `HASTE_SHARED_GROUP` | `dev` | Pool group (`dev`, `demo`, …) — drives the pool names. | +| `HASTE_SHARED_H100_SCALE_MODE` | `Autoscale` | `Autoscale`, or `Fixed` reserved baseline for the scarce H100 tier. | +| `HASTE_SHARED_H100_MIN_NODES` | `0` | H100 autoscale floor (>0 keeps chasing/holding a warm node). | +| `HASTE_SHARED_T4_MIN_NODES` / `HASTE_SHARED_T4_MAX_NODES` | `0` / `2` | T4 autoscale floor / cap. | +| `HASTE_SHARED_BATCH_SUBNET_ID` | — | Shared hub batch-subnet to VNet-inject both pools into (see Blob↔Batch networking below). | **Data isolation.** Tenants share compute but not data: each job mints a short-lived **user-delegation SAS** scoped to its own storage container, so a @@ -131,6 +144,21 @@ tenant's task can never read another tenant's data. This requires the submitting Function App identity to hold **Storage Blob Delegator** on its storage account (granted in `functionApp.bicep`). +**Blob↔Batch networking.** SAS is an *auth* boundary, not network reach — each +tenant storage is `Deny`, so the pools must additionally be *network-allowed* to +it or blob I/O fails with `403 AuthorizationFailure`. The shared pools are +VNet-injected into a **shared hub batch-subnet** (`haste-hub-vnet/batch-subnet`, +carrying the `Microsoft.Storage` service endpoint), and **each tenant's storage +allowlists that subnet** with a VNet rule. Onboarding a new env is *just that one +storage rule* — wired in `storage.bicep` via `HASTE_SHARED_BATCH_SUBNET_ID` so it +lands on `azd provision`; the shared pools are never touched. Two one-time +prerequisites per hub (not per env): create the subnet, and grant the *Microsoft +Azure Batch* service principal `Network Contributor` on it so Batch can +VNet-inject. `HASTE_SHARED_BATCH_SUBNET_ID` is read by **both** the shared-pools +deploy (which subnet to inject the pools into) and each env deploy (which subnet +its storage allowlists). Full design: +[`networking.md`](https://github.com/microsoft/haste/blob/main/spec/features/batch-compute-expansion/networking.md). + **Per-environment app settings** (set on the `api`/`queues` Function Apps to opt an environment into the shared pools — all default to the legacy single-pool, pool-identity behavior, so existing environments are unaffected until opted in): diff --git a/infra/main.bicep b/infra/main.bicep index 2a7eaa8..68b3dbf 100644 --- a/infra/main.bicep +++ b/infra/main.bicep @@ -72,6 +72,9 @@ param batchPoolMaxNodes int = 3 @description('Subnet name (in the env VNet) used by the Batch pool.') param batchPoolSubnetName string = 'batch-subnet' +@description('Shared hub batch-subnet resource id where the SHARED multi-tenant pools are VNet-injected. Set for shared-pool envs (dev/demo) so this env storage allowlists that subnet; empty for single-tenant prod. See spec/features/batch-compute-expansion/networking.md.') +param sharedBatchSubnetId string = '' + @description('Training container image (tag included).') param trainingImage string = 'hastetraining:1.4.1' @@ -211,6 +214,7 @@ module storage 'modules/storage.bicep' = { defaultSubnetName: 'default' functionsSubnetName: functionsSubnetName batchSubnetName: batchPoolSubnetName + sharedBatchSubnetId: sharedBatchSubnetId tags: tags } dependsOn: [ diff --git a/infra/main.bicepparam b/infra/main.bicepparam index e2f7728..8241bc5 100644 --- a/infra/main.bicepparam +++ b/infra/main.bicepparam @@ -44,3 +44,7 @@ param inferencePoolIds = readEnvironmentVariable('HASTE_BATCH_INFERENCE_POOL_IDS param imageryprepPoolIds = readEnvironmentVariable('HASTE_BATCH_IMAGERYPREP_POOL_IDS', '') param useSas = bool(readEnvironmentVariable('HASTE_BATCH_USE_SAS', 'false')) param managePools = bool(readEnvironmentVariable('HASTE_BATCH_MANAGE_POOLS', 'true')) + +// Shared hub batch-subnet the multi-tenant pools live in; this env's storage +// allowlists it so those pools can reach its blobs. Empty for single-tenant prod. +param sharedBatchSubnetId = readEnvironmentVariable('HASTE_SHARED_BATCH_SUBNET_ID', '') diff --git a/infra/main.json b/infra/main.json new file mode 100644 index 0000000..052568f --- /dev/null +++ b/infra/main.json @@ -0,0 +1,3925 @@ +{ + "$schema": "https://schema.management.azure.com/schemas/2018-05-01/subscriptionDeploymentTemplate.json#", + "languageVersion": "2.0", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "12934134840547041152" + } + }, + "parameters": { + "resourcePrefix": { + "type": "string", + "metadata": { + "description": "Short prefix used to build every resource name (e.g. \"haste\")." + } + }, + "location": { + "type": "string", + "metadata": { + "description": "Azure region for the environment resource group and resources." + } + }, + "randomSuffix": { + "type": "string", + "minLength": 1, + "metadata": { + "description": "4-digit suffix that disambiguates parallel environments." + } + }, + "tags": { + "type": "object", + "defaultValue": { + "project": "haste" + }, + "metadata": { + "description": "Tags applied to the resource group and resources." + } + }, + "sharedResourceGroup": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "Resource group holding BYO shared resources (Batch, ACR). Empty => env RG." + } + }, + "sharedAcrName": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "Name of an existing ACR to pull training/imageryprep images from. Empty => no ACR wiring." + } + }, + "apimPublisherEmail": { + "type": "string", + "metadata": { + "description": "APIM publisher email." + } + }, + "apimPublisherName": { + "type": "string", + "defaultValue": "AI For Good Lab", + "metadata": { + "description": "APIM publisher organisation name." + } + }, + "batchAccountMode": { + "type": "string", + "defaultValue": "Create", + "allowedValues": [ + "Create", + "Existing" + ], + "metadata": { + "description": "Create a new Batch account in the env RG, or reference an existing shared one." + } + }, + "existingBatchAccountName": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "Existing Batch account name (required when batchAccountMode == Existing)." + } + }, + "batchPoolMode": { + "type": "string", + "defaultValue": "Create", + "allowedValues": [ + "Create", + "Existing" + ], + "metadata": { + "description": "Create the GPU pool, or reference an existing one for app-settings wiring only." + } + }, + "existingBatchPoolId": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "Existing Batch pool id (required when batchPoolMode == Existing)." + } + }, + "batchPoolVmSize": { + "type": "string", + "defaultValue": "STANDARD_NC40ads_H100_v5", + "metadata": { + "description": "VM size for the GPU pool." + } + }, + "batchPoolMaxNodes": { + "type": "int", + "defaultValue": 3, + "metadata": { + "description": "Max dedicated nodes for the autoscale formula." + } + }, + "batchPoolSubnetName": { + "type": "string", + "defaultValue": "batch-subnet", + "metadata": { + "description": "Subnet name (in the env VNet) used by the Batch pool." + } + }, + "sharedBatchSubnetId": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "Shared hub batch-subnet resource id where the SHARED multi-tenant pools are VNet-injected. Set for shared-pool envs (dev/demo) so this env storage allowlists that subnet; empty for single-tenant prod. See spec/features/batch-compute-expansion/networking.md." + } + }, + "trainingImage": { + "type": "string", + "defaultValue": "hastetraining:1.4.1", + "metadata": { + "description": "Training container image (tag included)." + } + }, + "imageryprepImage": { + "type": "string", + "defaultValue": "hasteimageryprep:1.4.1", + "metadata": { + "description": "Imageryprep container image (tag included)." + } + }, + "emailSenderDomainType": { + "type": "string", + "defaultValue": "AzureManaged", + "allowedValues": [ + "AzureManaged", + "Custom" + ], + "metadata": { + "description": "Sender-domain mode for the email backend." + } + }, + "emailCustomDomain": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "Custom sender domain (required when emailSenderDomainType == Custom)." + } + }, + "enableFrontDoor": { + "type": "bool", + "defaultValue": false, + "metadata": { + "description": "Provision Front Door + WAF in front of the Static Web App." + } + }, + "developmentMode": { + "type": "bool", + "defaultValue": false, + "metadata": { + "description": "Dev-only: api/queues auto-provision any authenticated user as admin and use anonymous auth. Keep false for production." + } + }, + "trainingPoolIds": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "Ordered candidate training pool ids (comma-separated). Empty => single training pool." + } + }, + "inferencePoolIds": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "Ordered candidate inference/embedding pool ids (comma-separated)." + } + }, + "imageryprepPoolIds": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "Ordered candidate imageryprep/artifacts pool ids (comma-separated)." + } + }, + "useSas": { + "type": "bool", + "defaultValue": false, + "metadata": { + "description": "Use per-job user-delegation SAS for Batch blob I/O (multi-tenant shared pools)." + } + }, + "managePools": { + "type": "bool", + "defaultValue": true, + "metadata": { + "description": "Runner auto-creates/resizes its pool. False for pre-created autoscale pools." + } + } + }, + "variables": { + "rgName": "[format('{0}-haste-{1}-rg', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "storageAccountName": "[format('{0}haste{1}sa', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "fileStorageAccountName": "[format('{0}haste{1}fs', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "umiName": "[format('{0}-haste-{1}-umi', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "vnetName": "[format('{0}-haste-{1}-vnet', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "nsgName": "[format('{0}-haste-{1}-nsg', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "apimName": "[format('{0}-haste-{1}-apim', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "functionApiName": "[format('{0}haste{1}func', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "functionTitilerName": "[format('{0}hastetitiler{1}func', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "functionQueueName": "[format('{0}hastequeue{1}func', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "staticWebAppName": "[format('{0}haste{1}swa', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "logAnalyticsName": "[format('{0}-haste-{1}-law', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "mapsAccountName": "[format('{0}haste{1}maps', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "wafPolicyName": "[format('{0}haste{1}waf', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "frontDoorName": "[format('{0}haste{1}fd', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "acsName": "[format('{0}-haste-{1}-acs', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "emailServiceName": "[format('{0}-haste-{1}-email', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "functionsSubnetName": "func-subnet", + "createBatchAccount": "[equals(parameters('batchAccountMode'), 'Create')]", + "resolvedSharedRg": "[if(empty(parameters('sharedResourceGroup')), variables('rgName'), parameters('sharedResourceGroup'))]", + "createdBatchAccountName": "[format('{0}haste{1}batch', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "resolvedBatchAccountName": "[if(variables('createBatchAccount'), variables('createdBatchAccountName'), parameters('existingBatchAccountName'))]", + "batchAccountRg": "[if(variables('createBatchAccount'), variables('rgName'), variables('resolvedSharedRg'))]", + "createdBatchPoolName": "[format('{0}-haste-{1}-pool', parameters('resourcePrefix'), parameters('randomSuffix'))]", + "wireAcr": "[not(empty(parameters('sharedAcrName')))]" + }, + "resources": { + "rg": { + "type": "Microsoft.Resources/resourceGroups", + "apiVersion": "2024-03-01", + "name": "[variables('rgName')]", + "location": "[parameters('location')]", + "tags": "[parameters('tags')]" + }, + "batchAccountRef": { + "existing": true, + "type": "Microsoft.Batch/batchAccounts", + "apiVersion": "2024-07-01", + "resourceGroup": "[variables('batchAccountRg')]", + "name": "[variables('resolvedBatchAccountName')]" + }, + "monitoring": { + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "monitoring", + "resourceGroup": "[variables('rgName')]", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "location": { + "value": "[parameters('location')]" + }, + "logAnalyticsName": { + "value": "[variables('logAnalyticsName')]" + }, + "tags": { + "value": "[parameters('tags')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "9380957498639667481" + } + }, + "parameters": { + "location": { + "type": "string", + "metadata": { + "description": "Azure region." + } + }, + "logAnalyticsName": { + "type": "string", + "metadata": { + "description": "Log Analytics workspace name." + } + }, + "tags": { + "type": "object", + "defaultValue": {}, + "metadata": { + "description": "Resource tags." + } + } + }, + "resources": [ + { + "type": "Microsoft.OperationalInsights/workspaces", + "apiVersion": "2023-09-01", + "name": "[parameters('logAnalyticsName')]", + "location": "[parameters('location')]", + "tags": "[parameters('tags')]", + "properties": { + "sku": { + "name": "PerGB2018" + }, + "retentionInDays": 30 + } + } + ], + "outputs": { + "logAnalyticsId": { + "type": "string", + "value": "[resourceId('Microsoft.OperationalInsights/workspaces', parameters('logAnalyticsName'))]" + }, + "logAnalyticsName": { + "type": "string", + "value": "[parameters('logAnalyticsName')]" + } + } + } + }, + "dependsOn": [ + "rg" + ] + }, + "identity": { + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "identity", + "resourceGroup": "[variables('rgName')]", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "location": { + "value": "[parameters('location')]" + }, + "umiName": { + "value": "[variables('umiName')]" + }, + "tags": { + "value": "[parameters('tags')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "16963761547890363884" + } + }, + "parameters": { + "location": { + "type": "string", + "metadata": { + "description": "Azure region." + } + }, + "umiName": { + "type": "string", + "metadata": { + "description": "User-assigned managed identity name." + } + }, + "tags": { + "type": "object", + "defaultValue": {}, + "metadata": { + "description": "Resource tags." + } + } + }, + "resources": [ + { + "type": "Microsoft.ManagedIdentity/userAssignedIdentities", + "apiVersion": "2023-01-31", + "name": "[parameters('umiName')]", + "location": "[parameters('location')]", + "tags": "[parameters('tags')]" + } + ], + "outputs": { + "resourceId": { + "type": "string", + "value": "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', parameters('umiName'))]" + }, + "principalId": { + "type": "string", + "value": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', parameters('umiName')), '2023-01-31').principalId]" + }, + "clientId": { + "type": "string", + "value": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', parameters('umiName')), '2023-01-31').clientId]" + }, + "name": { + "type": "string", + "value": "[parameters('umiName')]" + } + } + } + }, + "dependsOn": [ + "rg" + ] + }, + "network": { + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "network", + "resourceGroup": "[variables('rgName')]", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "location": { + "value": "[parameters('location')]" + }, + "vnetName": { + "value": "[variables('vnetName')]" + }, + "nsgName": { + "value": "[variables('nsgName')]" + }, + "functionsSubnetName": { + "value": "[variables('functionsSubnetName')]" + }, + "batchSubnetName": { + "value": "[parameters('batchPoolSubnetName')]" + }, + "tags": { + "value": "[parameters('tags')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "9880370881471562028" + } + }, + "parameters": { + "location": { + "type": "string", + "metadata": { + "description": "Azure region." + } + }, + "vnetName": { + "type": "string", + "metadata": { + "description": "Virtual network name." + } + }, + "nsgName": { + "type": "string", + "metadata": { + "description": "Network security group name." + } + }, + "functionsSubnetName": { + "type": "string", + "metadata": { + "description": "Functions VNet-integration subnet name." + } + }, + "batchSubnetName": { + "type": "string", + "metadata": { + "description": "Batch pool subnet name." + } + }, + "tags": { + "type": "object", + "defaultValue": {}, + "metadata": { + "description": "Resource tags." + } + } + }, + "resources": [ + { + "type": "Microsoft.Network/networkSecurityGroups", + "apiVersion": "2023-11-01", + "name": "[parameters('nsgName')]", + "location": "[parameters('location')]", + "tags": "[parameters('tags')]", + "properties": { + "securityRules": [] + } + }, + { + "type": "Microsoft.Network/virtualNetworks", + "apiVersion": "2023-11-01", + "name": "[parameters('vnetName')]", + "location": "[parameters('location')]", + "tags": "[parameters('tags')]", + "properties": { + "addressSpace": { + "addressPrefixes": [ + "10.0.0.0/16" + ] + }, + "subnets": [ + { + "name": "default", + "properties": { + "addressPrefix": "10.0.0.0/24", + "networkSecurityGroup": { + "id": "[resourceId('Microsoft.Network/networkSecurityGroups', parameters('nsgName'))]" + }, + "serviceEndpoints": [ + { + "service": "Microsoft.Web" + }, + { + "service": "Microsoft.Storage" + } + ], + "delegations": [ + { + "name": "serverFarms", + "properties": { + "serviceName": "Microsoft.Web/serverFarms" + } + } + ] + } + }, + { + "name": "[parameters('functionsSubnetName')]", + "properties": { + "addressPrefix": "10.0.1.0/24", + "networkSecurityGroup": { + "id": "[resourceId('Microsoft.Network/networkSecurityGroups', parameters('nsgName'))]" + }, + "serviceEndpoints": [ + { + "service": "Microsoft.Storage" + }, + { + "service": "Microsoft.Web" + } + ], + "delegations": [ + { + "name": "appEnvironments", + "properties": { + "serviceName": "Microsoft.App/environments" + } + } + ] + } + }, + { + "name": "[parameters('batchSubnetName')]", + "properties": { + "addressPrefix": "10.0.2.0/24", + "serviceEndpoints": [ + { + "service": "Microsoft.Storage" + } + ] + } + } + ] + }, + "dependsOn": [ + "[resourceId('Microsoft.Network/networkSecurityGroups', parameters('nsgName'))]" + ] + } + ], + "outputs": { + "vnetId": { + "type": "string", + "value": "[resourceId('Microsoft.Network/virtualNetworks', parameters('vnetName'))]" + }, + "vnetName": { + "type": "string", + "value": "[parameters('vnetName')]" + }, + "defaultSubnetId": { + "type": "string", + "value": "[format('{0}/subnets/default', resourceId('Microsoft.Network/virtualNetworks', parameters('vnetName')))]" + }, + "functionsSubnetId": { + "type": "string", + "value": "[format('{0}/subnets/{1}', resourceId('Microsoft.Network/virtualNetworks', parameters('vnetName')), parameters('functionsSubnetName'))]" + }, + "batchSubnetId": { + "type": "string", + "value": "[format('{0}/subnets/{1}', resourceId('Microsoft.Network/virtualNetworks', parameters('vnetName')), parameters('batchSubnetName'))]" + }, + "nsgId": { + "type": "string", + "value": "[resourceId('Microsoft.Network/networkSecurityGroups', parameters('nsgName'))]" + } + } + } + }, + "dependsOn": [ + "rg" + ] + }, + "storage": { + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "storage", + "resourceGroup": "[variables('rgName')]", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "location": { + "value": "[parameters('location')]" + }, + "storageAccountName": { + "value": "[variables('storageAccountName')]" + }, + "fileStorageAccountName": { + "value": "[variables('fileStorageAccountName')]" + }, + "umiPrincipalId": { + "value": "[reference('identity').outputs.principalId.value]" + }, + "vnetName": { + "value": "[variables('vnetName')]" + }, + "defaultSubnetName": { + "value": "default" + }, + "functionsSubnetName": { + "value": "[variables('functionsSubnetName')]" + }, + "batchSubnetName": { + "value": "[parameters('batchPoolSubnetName')]" + }, + "sharedBatchSubnetId": { + "value": "[parameters('sharedBatchSubnetId')]" + }, + "tags": { + "value": "[parameters('tags')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "16630728933207302616" + } + }, + "parameters": { + "location": { + "type": "string", + "metadata": { + "description": "Azure region." + } + }, + "storageAccountName": { + "type": "string", + "metadata": { + "description": "Functions storage account name." + } + }, + "fileStorageAccountName": { + "type": "string", + "metadata": { + "description": "Premium file storage account name." + } + }, + "umiPrincipalId": { + "type": "string", + "metadata": { + "description": "Principal id of the user-assigned managed identity." + } + }, + "vnetName": { + "type": "string", + "metadata": { + "description": "VNet name holding the allowed subnets." + } + }, + "defaultSubnetName": { + "type": "string", + "metadata": { + "description": "Default subnet name." + } + }, + "functionsSubnetName": { + "type": "string", + "metadata": { + "description": "Functions subnet name." + } + }, + "batchSubnetName": { + "type": "string", + "metadata": { + "description": "Batch subnet name." + } + }, + "sharedBatchSubnetId": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "Shared hub batch-subnet resource id (where the SHARED multi-tenant pools are VNet-injected). Empty for single-tenant envs (prod) that run their own in-env pool. See spec/features/batch-compute-expansion/networking.md." + } + }, + "tags": { + "type": "object", + "defaultValue": {}, + "metadata": { + "description": "Resource tags." + } + } + }, + "variables": { + "blobDataOwnerRoleId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'b7e6dc6d-f1e8-4753-8033-0f276bb0955b')]", + "defaultSubnetId": "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('vnetName'), parameters('defaultSubnetName'))]", + "functionsSubnetId": "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('vnetName'), parameters('functionsSubnetName'))]", + "batchSubnetId": "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('vnetName'), parameters('batchSubnetName'))]" + }, + "resources": [ + { + "type": "Microsoft.Storage/storageAccounts", + "apiVersion": "2023-05-01", + "name": "[parameters('storageAccountName')]", + "location": "[parameters('location')]", + "tags": "[parameters('tags')]", + "sku": { + "name": "Standard_LRS" + }, + "kind": "StorageV2", + "properties": { + "accessTier": "Hot", + "allowSharedKeyAccess": true, + "minimumTlsVersion": "TLS1_2", + "supportsHttpsTrafficOnly": true, + "networkAcls": { + "bypass": "Logging, Metrics, AzureServices", + "defaultAction": "Deny", + "virtualNetworkRules": "[concat(createArray(createObject('id', variables('defaultSubnetId'), 'action', 'Allow'), createObject('id', variables('functionsSubnetId'), 'action', 'Allow'), createObject('id', variables('batchSubnetId'), 'action', 'Allow')), if(empty(parameters('sharedBatchSubnetId')), createArray(), createArray(createObject('id', parameters('sharedBatchSubnetId'), 'action', 'Allow'))))]" + } + } + }, + { + "type": "Microsoft.Storage/storageAccounts", + "apiVersion": "2023-05-01", + "name": "[parameters('fileStorageAccountName')]", + "location": "[parameters('location')]", + "tags": "[parameters('tags')]", + "sku": { + "name": "Premium_LRS" + }, + "kind": "FileStorage", + "properties": { + "allowSharedKeyAccess": true, + "minimumTlsVersion": "TLS1_2", + "supportsHttpsTrafficOnly": true, + "networkAcls": { + "bypass": "Logging, Metrics, AzureServices", + "defaultAction": "Deny", + "virtualNetworkRules": [ + { + "id": "[variables('defaultSubnetId')]", + "action": "Allow" + }, + { + "id": "[variables('functionsSubnetId')]", + "action": "Allow" + } + ] + } + } + }, + { + "type": "Microsoft.Storage/storageAccounts/fileServices", + "apiVersion": "2023-05-01", + "name": "[format('{0}/{1}', parameters('fileStorageAccountName'), 'default')]", + "dependsOn": [ + "[resourceId('Microsoft.Storage/storageAccounts', parameters('fileStorageAccountName'))]" + ] + }, + { + "type": "Microsoft.Storage/storageAccounts/fileServices/shares", + "apiVersion": "2023-05-01", + "name": "[format('{0}/{1}/{2}', parameters('fileStorageAccountName'), 'default', 'data')]", + "properties": { + "shareQuota": 1000 + }, + "dependsOn": [ + "[resourceId('Microsoft.Storage/storageAccounts/fileServices', parameters('fileStorageAccountName'), 'default')]" + ] + }, + { + "type": "Microsoft.Storage/storageAccounts/blobServices", + "apiVersion": "2023-05-01", + "name": "[format('{0}/{1}', parameters('storageAccountName'), 'default')]", + "dependsOn": [ + "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]" + ] + }, + { + "type": "Microsoft.Storage/storageAccounts/blobServices/containers", + "apiVersion": "2023-05-01", + "name": "[format('{0}/{1}/{2}', parameters('storageAccountName'), 'default', 'data')]", + "dependsOn": [ + "[resourceId('Microsoft.Storage/storageAccounts/blobServices', parameters('storageAccountName'), 'default')]" + ] + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]", + "name": "[guid(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), parameters('umiPrincipalId'), 'blobDataOwner')]", + "properties": { + "roleDefinitionId": "[variables('blobDataOwnerRoleId')]", + "principalId": "[parameters('umiPrincipalId')]", + "principalType": "ServicePrincipal" + }, + "dependsOn": [ + "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]" + ] + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.Storage/storageAccounts', parameters('fileStorageAccountName'))]", + "name": "[guid(resourceId('Microsoft.Storage/storageAccounts', parameters('fileStorageAccountName')), parameters('umiPrincipalId'), 'blobDataOwner')]", + "properties": { + "roleDefinitionId": "[variables('blobDataOwnerRoleId')]", + "principalId": "[parameters('umiPrincipalId')]", + "principalType": "ServicePrincipal" + }, + "dependsOn": [ + "[resourceId('Microsoft.Storage/storageAccounts', parameters('fileStorageAccountName'))]" + ] + } + ], + "outputs": { + "storageAccountName": { + "type": "string", + "value": "[parameters('storageAccountName')]" + }, + "storageAccountId": { + "type": "string", + "value": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]" + }, + "fileStorageAccountName": { + "type": "string", + "value": "[parameters('fileStorageAccountName')]" + }, + "fileStorageAccountId": { + "type": "string", + "value": "[resourceId('Microsoft.Storage/storageAccounts', parameters('fileStorageAccountName'))]" + } + } + } + }, + "dependsOn": [ + "identity", + "network", + "rg" + ] + }, + "communication": { + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "communication", + "resourceGroup": "[variables('rgName')]", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "acsName": { + "value": "[variables('acsName')]" + }, + "emailServiceName": { + "value": "[variables('emailServiceName')]" + }, + "emailSenderDomainType": { + "value": "[parameters('emailSenderDomainType')]" + }, + "emailCustomDomain": { + "value": "[parameters('emailCustomDomain')]" + }, + "tags": { + "value": "[parameters('tags')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "languageVersion": "2.0", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "2299531501285707142" + } + }, + "parameters": { + "acsName": { + "type": "string", + "metadata": { + "description": "Communication Services resource name." + } + }, + "emailServiceName": { + "type": "string", + "metadata": { + "description": "Email Communication Services resource name." + } + }, + "emailSenderDomainType": { + "type": "string", + "allowedValues": [ + "AzureManaged", + "Custom" + ], + "metadata": { + "description": "Sender-domain mode." + } + }, + "emailCustomDomain": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "Custom sender domain (required when emailSenderDomainType == Custom)." + } + }, + "tags": { + "type": "object", + "defaultValue": {}, + "metadata": { + "description": "Resource tags." + } + } + }, + "variables": { + "isCustom": "[equals(parameters('emailSenderDomainType'), 'Custom')]", + "domainResourceName": "[if(variables('isCustom'), parameters('emailCustomDomain'), 'AzureManagedDomain')]" + }, + "resources": { + "emailService": { + "type": "Microsoft.Communication/emailServices", + "apiVersion": "2023-04-01", + "name": "[parameters('emailServiceName')]", + "location": "global", + "tags": "[parameters('tags')]", + "properties": { + "dataLocation": "United States" + } + }, + "emailDomain": { + "type": "Microsoft.Communication/emailServices/domains", + "apiVersion": "2023-04-01", + "name": "[format('{0}/{1}', parameters('emailServiceName'), variables('domainResourceName'))]", + "location": "global", + "tags": "[parameters('tags')]", + "properties": { + "domainManagement": "[if(variables('isCustom'), 'CustomerManaged', 'AzureManaged')]", + "userEngagementTracking": "Disabled" + }, + "dependsOn": [ + "emailService" + ] + }, + "acs": { + "type": "Microsoft.Communication/communicationServices", + "apiVersion": "2023-04-01", + "name": "[parameters('acsName')]", + "location": "global", + "tags": "[parameters('tags')]", + "properties": { + "dataLocation": "United States", + "linkedDomains": [ + "[resourceId('Microsoft.Communication/emailServices/domains', parameters('emailServiceName'), variables('domainResourceName'))]" + ] + }, + "dependsOn": [ + "emailDomain" + ] + } + }, + "outputs": { + "connectionString": { + "type": "securestring", + "value": "[listKeys('acs', '2023-04-01').primaryConnectionString]" + }, + "senderDomain": { + "type": "string", + "value": "[reference('emailDomain').fromSenderDomain]" + }, + "acsName": { + "type": "string", + "value": "[parameters('acsName')]" + } + } + } + }, + "dependsOn": [ + "rg" + ] + }, + "apim": { + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "apim", + "resourceGroup": "[variables('rgName')]", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "location": { + "value": "[parameters('location')]" + }, + "apimName": { + "value": "[variables('apimName')]" + }, + "publisherEmail": { + "value": "[parameters('apimPublisherEmail')]" + }, + "publisherName": { + "value": "[parameters('apimPublisherName')]" + }, + "umiResourceId": { + "value": "[reference('identity').outputs.resourceId.value]" + }, + "vnetName": { + "value": "[variables('vnetName')]" + }, + "defaultSubnetName": { + "value": "default" + }, + "storageAccountName": { + "value": "[variables('storageAccountName')]" + }, + "tags": { + "value": "[parameters('tags')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "16571433796004974762" + } + }, + "parameters": { + "location": { + "type": "string", + "metadata": { + "description": "Azure region." + } + }, + "apimName": { + "type": "string", + "metadata": { + "description": "API Management service name." + } + }, + "publisherEmail": { + "type": "string", + "metadata": { + "description": "Publisher email." + } + }, + "publisherName": { + "type": "string", + "metadata": { + "description": "Publisher organisation name." + } + }, + "umiResourceId": { + "type": "string", + "metadata": { + "description": "User-assigned managed identity resource id." + } + }, + "vnetName": { + "type": "string", + "metadata": { + "description": "VNet name holding the APIM subnet." + } + }, + "defaultSubnetName": { + "type": "string", + "metadata": { + "description": "Subnet name for APIM VNet integration." + } + }, + "storageAccountName": { + "type": "string", + "metadata": { + "description": "Storage account name to grant the APIM identity blob access on." + } + }, + "tags": { + "type": "object", + "defaultValue": {}, + "metadata": { + "description": "Resource tags." + } + } + }, + "variables": { + "blobDataOwnerRoleId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'b7e6dc6d-f1e8-4753-8033-0f276bb0955b')]" + }, + "resources": [ + { + "type": "Microsoft.ApiManagement/service", + "apiVersion": "2024-06-01-preview", + "name": "[parameters('apimName')]", + "location": "[parameters('location')]", + "tags": "[parameters('tags')]", + "sku": { + "name": "StandardV2", + "capacity": 1 + }, + "identity": { + "type": "SystemAssigned, UserAssigned", + "userAssignedIdentities": { + "[format('{0}', parameters('umiResourceId'))]": {} + } + }, + "properties": { + "publisherEmail": "[parameters('publisherEmail')]", + "publisherName": "[parameters('publisherName')]", + "virtualNetworkType": "External", + "virtualNetworkConfiguration": { + "subnetResourceId": "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('vnetName'), parameters('defaultSubnetName'))]" + }, + "legacyPortalStatus": "Disabled", + "developerPortalStatus": "Disabled", + "releaseChannel": "Default", + "publicNetworkAccess": "Enabled", + "natGatewayState": "Enabled", + "customProperties": { + "Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Protocols.Tls10": "False", + "Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Protocols.Tls11": "False", + "Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Protocols.Ssl30": "False", + "Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Backend.Protocols.Tls10": "False", + "Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Backend.Protocols.Tls11": "False", + "Microsoft.WindowsAzure.ApiManagement.Gateway.Security.Backend.Protocols.Ssl30": "False" + } + } + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]", + "name": "[guid(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), resourceId('Microsoft.ApiManagement/service', parameters('apimName')), 'blobDataOwner')]", + "properties": { + "roleDefinitionId": "[variables('blobDataOwnerRoleId')]", + "principalId": "[reference(resourceId('Microsoft.ApiManagement/service', parameters('apimName')), '2024-06-01-preview', 'full').identity.principalId]", + "principalType": "ServicePrincipal" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service', parameters('apimName'))]" + ] + } + ], + "outputs": { + "resourceId": { + "type": "string", + "value": "[resourceId('Microsoft.ApiManagement/service', parameters('apimName'))]" + }, + "name": { + "type": "string", + "value": "[parameters('apimName')]" + }, + "systemPrincipalId": { + "type": "string", + "value": "[reference(resourceId('Microsoft.ApiManagement/service', parameters('apimName')), '2024-06-01-preview', 'full').identity.principalId]" + } + } + } + }, + "dependsOn": [ + "identity", + "network", + "rg", + "storage" + ] + }, + "functions": { + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "functions", + "resourceGroup": "[variables('rgName')]", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "location": { + "value": "[parameters('location')]" + }, + "functionApiName": { + "value": "[variables('functionApiName')]" + }, + "functionTitilerName": { + "value": "[variables('functionTitilerName')]" + }, + "functionQueueName": { + "value": "[variables('functionQueueName')]" + }, + "storageAccountName": { + "value": "[variables('storageAccountName')]" + }, + "fileStorageAccountName": { + "value": "[variables('fileStorageAccountName')]" + }, + "umiResourceId": { + "value": "[reference('identity').outputs.resourceId.value]" + }, + "vnetName": { + "value": "[variables('vnetName')]" + }, + "functionsSubnetName": { + "value": "[variables('functionsSubnetName')]" + }, + "logAnalyticsId": { + "value": "[reference('monitoring').outputs.logAnalyticsId.value]" + }, + "batchAccountName": { + "value": "[variables('resolvedBatchAccountName')]" + }, + "batchAccountLocation": { + "value": "[parameters('location')]" + }, + "batchPoolName": "[if(equals(parameters('batchPoolMode'), 'Create'), createObject('value', variables('createdBatchPoolName')), createObject('value', parameters('existingBatchPoolId')))]", + "acrLoginServer": "[if(variables('wireAcr'), createObject('value', format('{0}.azurecr.io', parameters('sharedAcrName'))), createObject('value', ''))]", + "trainingImage": { + "value": "[parameters('trainingImage')]" + }, + "imageryprepImage": { + "value": "[parameters('imageryprepImage')]" + }, + "staticWebAppName": { + "value": "[variables('staticWebAppName')]" + }, + "staticAppDomain": { + "value": "[reference('frontend').outputs.staticWebAppHostName.value]" + }, + "emailSenderDomain": { + "value": "[reference('communication').outputs.senderDomain.value]" + }, + "emailConnectionString": { + "value": "[listOutputsWithSecureValues('communication', '2025-04-01').connectionString]" + }, + "batchAccountKey": { + "value": "[listKeys('batchAccountRef', '2024-07-01').primary]" + }, + "developmentMode": { + "value": "[parameters('developmentMode')]" + }, + "trainingPoolIds": { + "value": "[parameters('trainingPoolIds')]" + }, + "inferencePoolIds": { + "value": "[parameters('inferencePoolIds')]" + }, + "imageryprepPoolIds": { + "value": "[parameters('imageryprepPoolIds')]" + }, + "useSas": { + "value": "[parameters('useSas')]" + }, + "managePools": { + "value": "[parameters('managePools')]" + }, + "tags": { + "value": "[parameters('tags')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "1968129085219592897" + } + }, + "parameters": { + "location": { + "type": "string", + "metadata": { + "description": "Azure region." + } + }, + "functionApiName": { + "type": "string", + "metadata": { + "description": "API function app name." + } + }, + "functionTitilerName": { + "type": "string", + "metadata": { + "description": "TiTiler function app name." + } + }, + "functionQueueName": { + "type": "string", + "metadata": { + "description": "Queue-triggers function app name." + } + }, + "storageAccountName": { + "type": "string", + "metadata": { + "description": "Functions storage account name." + } + }, + "fileStorageAccountName": { + "type": "string", + "metadata": { + "description": "Premium file storage account name." + } + }, + "umiResourceId": { + "type": "string", + "metadata": { + "description": "User-assigned managed identity resource id." + } + }, + "vnetName": { + "type": "string", + "metadata": { + "description": "VNet name." + } + }, + "functionsSubnetName": { + "type": "string", + "metadata": { + "description": "Functions subnet name." + } + }, + "logAnalyticsId": { + "type": "string", + "metadata": { + "description": "Log Analytics workspace resource id." + } + }, + "batchAccountName": { + "type": "string", + "metadata": { + "description": "Batch account name the apps submit jobs to." + } + }, + "batchAccountLocation": { + "type": "string", + "metadata": { + "description": "Batch account location (for the batch service URL)." + } + }, + "batchPoolName": { + "type": "string", + "metadata": { + "description": "GPU/imageryprep Batch pool id." + } + }, + "acrLoginServer": { + "type": "string", + "metadata": { + "description": "ACR login server (e.g. myreg.azurecr.io). Empty when no ACR is wired." + } + }, + "trainingImage": { + "type": "string", + "metadata": { + "description": "Training container image (tag included)." + } + }, + "imageryprepImage": { + "type": "string", + "metadata": { + "description": "Imageryprep container image (tag included)." + } + }, + "staticWebAppName": { + "type": "string", + "metadata": { + "description": "Static Web App name (for the invitation/email flow)." + } + }, + "staticAppDomain": { + "type": "string", + "metadata": { + "description": "Static Web App default hostname (for the invitation/email flow)." + } + }, + "emailSenderDomain": { + "type": "string", + "metadata": { + "description": "ACS sender domain (EMAIL_SENDER = DoNotReply@)." + } + }, + "emailConnectionString": { + "type": "securestring", + "metadata": { + "description": "ACS connection string for outbound email." + } + }, + "batchAccountKey": { + "type": "securestring", + "metadata": { + "description": "Batch account primary key." + } + }, + "developmentMode": { + "type": "bool", + "defaultValue": false, + "metadata": { + "description": "Dev-only: auto-provision any authenticated user as admin and drop function-key auth (anonymous). Must be false for production." + } + }, + "trainingPoolIds": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "Ordered candidate training pool ids (comma-separated). Empty => single training pool." + } + }, + "inferencePoolIds": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "Ordered candidate inference/embedding pool ids (comma-separated)." + } + }, + "imageryprepPoolIds": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "Ordered candidate imageryprep/artifacts pool ids (comma-separated)." + } + }, + "useSas": { + "type": "bool", + "defaultValue": false, + "metadata": { + "description": "Use per-job user-delegation SAS for Batch blob I/O (multi-tenant shared pools)." + } + }, + "managePools": { + "type": "bool", + "defaultValue": true, + "metadata": { + "description": "Runner auto-creates/resizes its pool. False for pre-created autoscale pools." + } + }, + "tags": { + "type": "object", + "defaultValue": {}, + "metadata": { + "description": "Resource tags." + } + } + }, + "resources": [ + { + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "fn-api", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "location": { + "value": "[parameters('location')]" + }, + "name": { + "value": "[parameters('functionApiName')]" + }, + "planName": { + "value": "[format('{0}-plan', parameters('functionApiName'))]" + }, + "alwaysReadyCount": { + "value": 10 + }, + "storageAccountName": { + "value": "[parameters('storageAccountName')]" + }, + "fileStorageAccountName": { + "value": "[parameters('fileStorageAccountName')]" + }, + "umiResourceId": { + "value": "[parameters('umiResourceId')]" + }, + "vnetName": { + "value": "[parameters('vnetName')]" + }, + "functionsSubnetName": { + "value": "[parameters('functionsSubnetName')]" + }, + "logAnalyticsId": { + "value": "[parameters('logAnalyticsId')]" + }, + "appSettings": { + "value": [ + { + "name": "env", + "value": "prod" + }, + { + "name": "DEVELOPMENT_MODE", + "value": "[if(parameters('developmentMode'), 'true', 'false')]" + }, + { + "name": "IMAGE_QUEUE_NAME", + "value": "image-layers-queue" + }, + { + "name": "INFERENCE_QUEUE_NAME", + "value": "inference-queue" + }, + { + "name": "STATS_QUEUE_NAME", + "value": "stats-queue" + }, + { + "name": "TRAIN_QUEUE_NAME", + "value": "train-queue" + }, + { + "name": "ZIP_QUEUE_NAME", + "value": "zip-queue" + }, + { + "name": "IMAGERY_STORAGE_TYPE", + "value": "blob" + }, + { + "name": "METADATA_STORAGE_TYPE", + "value": "blob" + }, + { + "name": "ARTIFACT_STORAGE_TYPE", + "value": "blob" + }, + { + "name": "RUNNER_TYPE", + "value": "azure_batch" + }, + { + "name": "TEMP_DATA_PATH", + "value": "/data" + }, + { + "name": "DATA_PATH", + "value": "/data" + }, + { + "name": "TITILER_ENDPOINT", + "value": "/api/titiler/" + }, + { + "name": "BLOB_CONTAINER", + "value": "data" + }, + { + "name": "BLOB_ACCOUNT_URL", + "value": "[reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').primaryEndpoints.blob]" + }, + { + "name": "QUEUE_ACCOUNT_URL", + "value": "[reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').primaryEndpoints.queue]" + }, + { + "name": "BLOB_CONNECTION_STRING", + "value": "[format('DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1};EndpointSuffix={2}', parameters('storageAccountName'), listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').keys[0].value, environment().suffixes.storage)]" + }, + { + "name": "AZURE_BATCH_ACCOUNT_NAME", + "value": "[parameters('batchAccountName')]" + }, + { + "name": "AZURE_BATCH_URL", + "value": "[format('https://{0}.{1}.batch.azure.com', parameters('batchAccountName'), parameters('batchAccountLocation'))]" + }, + { + "name": "AZURE_BATCH_ACCOUNT_KEY", + "value": "[parameters('batchAccountKey')]" + }, + { + "name": "AZURE_BATCH_DOCKER_IMAGE", + "value": "[format('{0}/{1}', parameters('acrLoginServer'), parameters('trainingImage'))]" + }, + { + "name": "AZURE_BATCH_IMAGERYPREP_DOCKER_IMAGE", + "value": "[format('{0}/{1}', parameters('acrLoginServer'), parameters('imageryprepImage'))]" + }, + { + "name": "AZURE_BATCH_OUTPUT_CONTAINER_URL", + "value": "[format('{0}data', reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').primaryEndpoints.blob)]" + }, + { + "name": "AZURE_BATCH_TRAINING_POOL_ID", + "value": "[parameters('batchPoolName')]" + }, + { + "name": "AZURE_BATCH_IMAGERYPREP_POOL_ID", + "value": "[parameters('batchPoolName')]" + }, + { + "name": "AZURE_BATCH_REGISTRY_SERVER_URL", + "value": "[format('https://{0}', parameters('acrLoginServer'))]" + }, + { + "name": "AZURE_BATCH_REGISTRY_IMAGE", + "value": "[format('{0}/{1}', parameters('acrLoginServer'), parameters('trainingImage'))]" + }, + { + "name": "AZURE_BATCH_REGISTRY_IDENTITY_RESOURCE_ID", + "value": "[parameters('umiResourceId')]" + }, + { + "name": "STATIC_APP_SUBSCRIPTION_ID", + "value": "[subscription().subscriptionId]" + }, + { + "name": "STATIC_APP_RESOURCE_GROUP", + "value": "[resourceGroup().name]" + }, + { + "name": "STATIC_APP_NAME", + "value": "[parameters('staticWebAppName')]" + }, + { + "name": "STATIC_APP_DOMAIN", + "value": "[parameters('staticAppDomain')]" + }, + { + "name": "EMAIL_CONNECTION_STRING", + "value": "[parameters('emailConnectionString')]" + }, + { + "name": "EMAIL_SENDER", + "value": "[format('DoNotReply@{0}', parameters('emailSenderDomain'))]" + }, + { + "name": "AZURE_BATCH_TRAINING_POOL_IDS", + "value": "[parameters('trainingPoolIds')]" + }, + { + "name": "AZURE_BATCH_INFERENCE_POOL_IDS", + "value": "[parameters('inferencePoolIds')]" + }, + { + "name": "AZURE_BATCH_IMAGERYPREP_POOL_IDS", + "value": "[parameters('imageryprepPoolIds')]" + }, + { + "name": "AZURE_BATCH_USE_SAS", + "value": "[if(parameters('useSas'), 'true', 'false')]" + }, + { + "name": "AZURE_BATCH_MANAGE_POOLS", + "value": "[if(parameters('managePools'), 'true', 'false')]" + } + ] + }, + "tags": { + "value": "[parameters('tags')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "12545021642088417436" + } + }, + "parameters": { + "location": { + "type": "string", + "metadata": { + "description": "Azure region." + } + }, + "name": { + "type": "string", + "metadata": { + "description": "Function app name." + } + }, + "planName": { + "type": "string", + "metadata": { + "description": "App Service (Flex Consumption) plan name." + } + }, + "alwaysReadyCount": { + "type": "int", + "metadata": { + "description": "Always-ready HTTP instance count." + } + }, + "storageAccountName": { + "type": "string", + "metadata": { + "description": "Functions storage account name." + } + }, + "fileStorageAccountName": { + "type": "string", + "metadata": { + "description": "Premium file storage account name (mounted at /data)." + } + }, + "umiResourceId": { + "type": "string", + "metadata": { + "description": "User-assigned managed identity resource id." + } + }, + "vnetName": { + "type": "string", + "metadata": { + "description": "VNet name for VNet integration." + } + }, + "functionsSubnetName": { + "type": "string", + "metadata": { + "description": "Functions subnet name." + } + }, + "logAnalyticsId": { + "type": "string", + "metadata": { + "description": "Log Analytics workspace resource id." + } + }, + "appSettings": { + "type": "array", + "defaultValue": [], + "metadata": { + "description": "Extra application settings (name/value pairs) merged after the base settings. api/queues pass the hastegeo app config here; titiler passes none." + } + }, + "tags": { + "type": "object", + "defaultValue": {}, + "metadata": { + "description": "Resource tags." + } + } + }, + "variables": { + "blobDataOwnerRoleId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'b7e6dc6d-f1e8-4753-8033-0f276bb0955b')]", + "queueDataContributorRoleId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '974c5e8b-45b9-4653-ba55-5f855dd0fb88')]", + "blobDelegatorRoleId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'db58b8e5-c6ad-4a2a-8342-4190687cbf4a')]", + "deploymentContainerName": "[format('app-package-{0}', parameters('name'))]" + }, + "resources": [ + { + "type": "Microsoft.Storage/storageAccounts/blobServices/containers", + "apiVersion": "2023-05-01", + "name": "[format('{0}/{1}/{2}', parameters('storageAccountName'), 'default', variables('deploymentContainerName'))]" + }, + { + "type": "Microsoft.Insights/components", + "apiVersion": "2020-02-02", + "name": "[format('{0}-ai', parameters('name'))]", + "location": "[parameters('location')]", + "kind": "web", + "tags": "[parameters('tags')]", + "properties": { + "Application_Type": "web", + "WorkspaceResourceId": "[parameters('logAnalyticsId')]" + } + }, + { + "type": "Microsoft.Web/serverfarms", + "apiVersion": "2023-12-01", + "name": "[parameters('planName')]", + "location": "[parameters('location')]", + "tags": "[parameters('tags')]", + "sku": { + "tier": "FlexConsumption", + "name": "FC1" + }, + "kind": "functionapp", + "properties": { + "reserved": true + } + }, + { + "type": "Microsoft.Web/sites", + "apiVersion": "2023-12-01", + "name": "[parameters('name')]", + "location": "[parameters('location')]", + "kind": "functionapp,linux", + "tags": "[parameters('tags')]", + "identity": { + "type": "SystemAssigned, UserAssigned", + "userAssignedIdentities": { + "[format('{0}', parameters('umiResourceId'))]": {} + } + }, + "properties": { + "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', parameters('planName'))]", + "httpsOnly": true, + "virtualNetworkSubnetId": "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('vnetName'), parameters('functionsSubnetName'))]", + "functionAppConfig": { + "deployment": { + "storage": { + "type": "blobContainer", + "value": "[format('{0}{1}', reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').primaryEndpoints.blob, variables('deploymentContainerName'))]", + "authentication": { + "type": "SystemAssignedIdentity" + } + } + }, + "scaleAndConcurrency": { + "maximumInstanceCount": 100, + "instanceMemoryMB": 4096, + "alwaysReady": [ + { + "name": "http", + "instanceCount": "[parameters('alwaysReadyCount')]" + } + ] + }, + "runtime": { + "name": "python", + "version": "3.11" + } + }, + "siteConfig": { + "ftpsState": "Disabled", + "appSettings": "[concat(createArray(createObject('name', 'AzureWebJobsStorage__accountName', 'value', parameters('storageAccountName')), createObject('name', 'AzureWebJobsStorage__blobServiceUri', 'value', reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').primaryEndpoints.blob), createObject('name', 'AzureWebJobsStorage__queueServiceUri', 'value', reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').primaryEndpoints.queue), createObject('name', 'APPLICATIONINSIGHTS_CONNECTION_STRING', 'value', reference(resourceId('Microsoft.Insights/components', format('{0}-ai', parameters('name'))), '2020-02-02').ConnectionString)), parameters('appSettings'))]" + } + }, + "dependsOn": [ + "[resourceId('Microsoft.Insights/components', format('{0}-ai', parameters('name')))]", + "[resourceId('Microsoft.Web/serverfarms', parameters('planName'))]" + ] + }, + { + "type": "Microsoft.Web/sites/config", + "apiVersion": "2023-12-01", + "name": "[format('{0}/{1}', parameters('name'), 'azurestorageaccounts')]", + "properties": { + "data": { + "type": "AzureFiles", + "accountName": "[parameters('fileStorageAccountName')]", + "shareName": "data", + "mountPath": "/data", + "accessKey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('fileStorageAccountName')), '2023-05-01').keys[0].value]" + } + }, + "dependsOn": [ + "[resourceId('Microsoft.Web/sites', parameters('name'))]" + ] + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]", + "name": "[guid(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), resourceId('Microsoft.Web/sites', parameters('name')), 'blobDataOwner')]", + "properties": { + "roleDefinitionId": "[variables('blobDataOwnerRoleId')]", + "principalId": "[reference(resourceId('Microsoft.Web/sites', parameters('name')), '2023-12-01', 'full').identity.principalId]", + "principalType": "ServicePrincipal" + }, + "dependsOn": [ + "[resourceId('Microsoft.Web/sites', parameters('name'))]" + ] + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]", + "name": "[guid(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), resourceId('Microsoft.Web/sites', parameters('name')), 'queueDataContributor')]", + "properties": { + "roleDefinitionId": "[variables('queueDataContributorRoleId')]", + "principalId": "[reference(resourceId('Microsoft.Web/sites', parameters('name')), '2023-12-01', 'full').identity.principalId]", + "principalType": "ServicePrincipal" + }, + "dependsOn": [ + "[resourceId('Microsoft.Web/sites', parameters('name'))]" + ] + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]", + "name": "[guid(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), resourceId('Microsoft.Web/sites', parameters('name')), 'blobDelegator')]", + "properties": { + "roleDefinitionId": "[variables('blobDelegatorRoleId')]", + "principalId": "[reference(resourceId('Microsoft.Web/sites', parameters('name')), '2023-12-01', 'full').identity.principalId]", + "principalType": "ServicePrincipal" + }, + "dependsOn": [ + "[resourceId('Microsoft.Web/sites', parameters('name'))]" + ] + } + ], + "outputs": { + "name": { + "type": "string", + "value": "[parameters('name')]" + }, + "defaultHostName": { + "type": "string", + "value": "[reference(resourceId('Microsoft.Web/sites', parameters('name')), '2023-12-01').defaultHostName]" + }, + "systemPrincipalId": { + "type": "string", + "value": "[reference(resourceId('Microsoft.Web/sites', parameters('name')), '2023-12-01', 'full').identity.principalId]" + } + } + } + } + }, + { + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "fn-titiler", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "location": { + "value": "[parameters('location')]" + }, + "name": { + "value": "[parameters('functionTitilerName')]" + }, + "planName": { + "value": "[format('{0}-plan', parameters('functionTitilerName'))]" + }, + "alwaysReadyCount": { + "value": 5 + }, + "storageAccountName": { + "value": "[parameters('storageAccountName')]" + }, + "fileStorageAccountName": { + "value": "[parameters('fileStorageAccountName')]" + }, + "umiResourceId": { + "value": "[parameters('umiResourceId')]" + }, + "vnetName": { + "value": "[parameters('vnetName')]" + }, + "functionsSubnetName": { + "value": "[parameters('functionsSubnetName')]" + }, + "logAnalyticsId": { + "value": "[parameters('logAnalyticsId')]" + }, + "tags": { + "value": "[parameters('tags')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "12545021642088417436" + } + }, + "parameters": { + "location": { + "type": "string", + "metadata": { + "description": "Azure region." + } + }, + "name": { + "type": "string", + "metadata": { + "description": "Function app name." + } + }, + "planName": { + "type": "string", + "metadata": { + "description": "App Service (Flex Consumption) plan name." + } + }, + "alwaysReadyCount": { + "type": "int", + "metadata": { + "description": "Always-ready HTTP instance count." + } + }, + "storageAccountName": { + "type": "string", + "metadata": { + "description": "Functions storage account name." + } + }, + "fileStorageAccountName": { + "type": "string", + "metadata": { + "description": "Premium file storage account name (mounted at /data)." + } + }, + "umiResourceId": { + "type": "string", + "metadata": { + "description": "User-assigned managed identity resource id." + } + }, + "vnetName": { + "type": "string", + "metadata": { + "description": "VNet name for VNet integration." + } + }, + "functionsSubnetName": { + "type": "string", + "metadata": { + "description": "Functions subnet name." + } + }, + "logAnalyticsId": { + "type": "string", + "metadata": { + "description": "Log Analytics workspace resource id." + } + }, + "appSettings": { + "type": "array", + "defaultValue": [], + "metadata": { + "description": "Extra application settings (name/value pairs) merged after the base settings. api/queues pass the hastegeo app config here; titiler passes none." + } + }, + "tags": { + "type": "object", + "defaultValue": {}, + "metadata": { + "description": "Resource tags." + } + } + }, + "variables": { + "blobDataOwnerRoleId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'b7e6dc6d-f1e8-4753-8033-0f276bb0955b')]", + "queueDataContributorRoleId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '974c5e8b-45b9-4653-ba55-5f855dd0fb88')]", + "blobDelegatorRoleId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'db58b8e5-c6ad-4a2a-8342-4190687cbf4a')]", + "deploymentContainerName": "[format('app-package-{0}', parameters('name'))]" + }, + "resources": [ + { + "type": "Microsoft.Storage/storageAccounts/blobServices/containers", + "apiVersion": "2023-05-01", + "name": "[format('{0}/{1}/{2}', parameters('storageAccountName'), 'default', variables('deploymentContainerName'))]" + }, + { + "type": "Microsoft.Insights/components", + "apiVersion": "2020-02-02", + "name": "[format('{0}-ai', parameters('name'))]", + "location": "[parameters('location')]", + "kind": "web", + "tags": "[parameters('tags')]", + "properties": { + "Application_Type": "web", + "WorkspaceResourceId": "[parameters('logAnalyticsId')]" + } + }, + { + "type": "Microsoft.Web/serverfarms", + "apiVersion": "2023-12-01", + "name": "[parameters('planName')]", + "location": "[parameters('location')]", + "tags": "[parameters('tags')]", + "sku": { + "tier": "FlexConsumption", + "name": "FC1" + }, + "kind": "functionapp", + "properties": { + "reserved": true + } + }, + { + "type": "Microsoft.Web/sites", + "apiVersion": "2023-12-01", + "name": "[parameters('name')]", + "location": "[parameters('location')]", + "kind": "functionapp,linux", + "tags": "[parameters('tags')]", + "identity": { + "type": "SystemAssigned, UserAssigned", + "userAssignedIdentities": { + "[format('{0}', parameters('umiResourceId'))]": {} + } + }, + "properties": { + "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', parameters('planName'))]", + "httpsOnly": true, + "virtualNetworkSubnetId": "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('vnetName'), parameters('functionsSubnetName'))]", + "functionAppConfig": { + "deployment": { + "storage": { + "type": "blobContainer", + "value": "[format('{0}{1}', reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').primaryEndpoints.blob, variables('deploymentContainerName'))]", + "authentication": { + "type": "SystemAssignedIdentity" + } + } + }, + "scaleAndConcurrency": { + "maximumInstanceCount": 100, + "instanceMemoryMB": 4096, + "alwaysReady": [ + { + "name": "http", + "instanceCount": "[parameters('alwaysReadyCount')]" + } + ] + }, + "runtime": { + "name": "python", + "version": "3.11" + } + }, + "siteConfig": { + "ftpsState": "Disabled", + "appSettings": "[concat(createArray(createObject('name', 'AzureWebJobsStorage__accountName', 'value', parameters('storageAccountName')), createObject('name', 'AzureWebJobsStorage__blobServiceUri', 'value', reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').primaryEndpoints.blob), createObject('name', 'AzureWebJobsStorage__queueServiceUri', 'value', reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').primaryEndpoints.queue), createObject('name', 'APPLICATIONINSIGHTS_CONNECTION_STRING', 'value', reference(resourceId('Microsoft.Insights/components', format('{0}-ai', parameters('name'))), '2020-02-02').ConnectionString)), parameters('appSettings'))]" + } + }, + "dependsOn": [ + "[resourceId('Microsoft.Insights/components', format('{0}-ai', parameters('name')))]", + "[resourceId('Microsoft.Web/serverfarms', parameters('planName'))]" + ] + }, + { + "type": "Microsoft.Web/sites/config", + "apiVersion": "2023-12-01", + "name": "[format('{0}/{1}', parameters('name'), 'azurestorageaccounts')]", + "properties": { + "data": { + "type": "AzureFiles", + "accountName": "[parameters('fileStorageAccountName')]", + "shareName": "data", + "mountPath": "/data", + "accessKey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('fileStorageAccountName')), '2023-05-01').keys[0].value]" + } + }, + "dependsOn": [ + "[resourceId('Microsoft.Web/sites', parameters('name'))]" + ] + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]", + "name": "[guid(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), resourceId('Microsoft.Web/sites', parameters('name')), 'blobDataOwner')]", + "properties": { + "roleDefinitionId": "[variables('blobDataOwnerRoleId')]", + "principalId": "[reference(resourceId('Microsoft.Web/sites', parameters('name')), '2023-12-01', 'full').identity.principalId]", + "principalType": "ServicePrincipal" + }, + "dependsOn": [ + "[resourceId('Microsoft.Web/sites', parameters('name'))]" + ] + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]", + "name": "[guid(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), resourceId('Microsoft.Web/sites', parameters('name')), 'queueDataContributor')]", + "properties": { + "roleDefinitionId": "[variables('queueDataContributorRoleId')]", + "principalId": "[reference(resourceId('Microsoft.Web/sites', parameters('name')), '2023-12-01', 'full').identity.principalId]", + "principalType": "ServicePrincipal" + }, + "dependsOn": [ + "[resourceId('Microsoft.Web/sites', parameters('name'))]" + ] + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]", + "name": "[guid(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), resourceId('Microsoft.Web/sites', parameters('name')), 'blobDelegator')]", + "properties": { + "roleDefinitionId": "[variables('blobDelegatorRoleId')]", + "principalId": "[reference(resourceId('Microsoft.Web/sites', parameters('name')), '2023-12-01', 'full').identity.principalId]", + "principalType": "ServicePrincipal" + }, + "dependsOn": [ + "[resourceId('Microsoft.Web/sites', parameters('name'))]" + ] + } + ], + "outputs": { + "name": { + "type": "string", + "value": "[parameters('name')]" + }, + "defaultHostName": { + "type": "string", + "value": "[reference(resourceId('Microsoft.Web/sites', parameters('name')), '2023-12-01').defaultHostName]" + }, + "systemPrincipalId": { + "type": "string", + "value": "[reference(resourceId('Microsoft.Web/sites', parameters('name')), '2023-12-01', 'full').identity.principalId]" + } + } + } + }, + "dependsOn": [ + "[resourceId('Microsoft.Resources/deployments', 'fn-api')]" + ] + }, + { + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "fn-queue", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "location": { + "value": "[parameters('location')]" + }, + "name": { + "value": "[parameters('functionQueueName')]" + }, + "planName": { + "value": "[format('{0}-plan', parameters('functionQueueName'))]" + }, + "alwaysReadyCount": { + "value": 1 + }, + "storageAccountName": { + "value": "[parameters('storageAccountName')]" + }, + "fileStorageAccountName": { + "value": "[parameters('fileStorageAccountName')]" + }, + "umiResourceId": { + "value": "[parameters('umiResourceId')]" + }, + "vnetName": { + "value": "[parameters('vnetName')]" + }, + "functionsSubnetName": { + "value": "[parameters('functionsSubnetName')]" + }, + "logAnalyticsId": { + "value": "[parameters('logAnalyticsId')]" + }, + "appSettings": { + "value": [ + { + "name": "env", + "value": "prod" + }, + { + "name": "DEVELOPMENT_MODE", + "value": "[if(parameters('developmentMode'), 'true', 'false')]" + }, + { + "name": "IMAGE_QUEUE_NAME", + "value": "image-layers-queue" + }, + { + "name": "INFERENCE_QUEUE_NAME", + "value": "inference-queue" + }, + { + "name": "STATS_QUEUE_NAME", + "value": "stats-queue" + }, + { + "name": "TRAIN_QUEUE_NAME", + "value": "train-queue" + }, + { + "name": "ZIP_QUEUE_NAME", + "value": "zip-queue" + }, + { + "name": "IMAGERY_STORAGE_TYPE", + "value": "blob" + }, + { + "name": "METADATA_STORAGE_TYPE", + "value": "blob" + }, + { + "name": "ARTIFACT_STORAGE_TYPE", + "value": "blob" + }, + { + "name": "RUNNER_TYPE", + "value": "azure_batch" + }, + { + "name": "TEMP_DATA_PATH", + "value": "/data" + }, + { + "name": "DATA_PATH", + "value": "/data" + }, + { + "name": "TITILER_ENDPOINT", + "value": "/api/titiler/" + }, + { + "name": "BLOB_CONTAINER", + "value": "data" + }, + { + "name": "BLOB_ACCOUNT_URL", + "value": "[reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').primaryEndpoints.blob]" + }, + { + "name": "QUEUE_ACCOUNT_URL", + "value": "[reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').primaryEndpoints.queue]" + }, + { + "name": "BLOB_CONNECTION_STRING", + "value": "[format('DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1};EndpointSuffix={2}', parameters('storageAccountName'), listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').keys[0].value, environment().suffixes.storage)]" + }, + { + "name": "AZURE_BATCH_ACCOUNT_NAME", + "value": "[parameters('batchAccountName')]" + }, + { + "name": "AZURE_BATCH_URL", + "value": "[format('https://{0}.{1}.batch.azure.com', parameters('batchAccountName'), parameters('batchAccountLocation'))]" + }, + { + "name": "AZURE_BATCH_ACCOUNT_KEY", + "value": "[parameters('batchAccountKey')]" + }, + { + "name": "AZURE_BATCH_DOCKER_IMAGE", + "value": "[format('{0}/{1}', parameters('acrLoginServer'), parameters('trainingImage'))]" + }, + { + "name": "AZURE_BATCH_IMAGERYPREP_DOCKER_IMAGE", + "value": "[format('{0}/{1}', parameters('acrLoginServer'), parameters('imageryprepImage'))]" + }, + { + "name": "AZURE_BATCH_OUTPUT_CONTAINER_URL", + "value": "[format('{0}data', reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').primaryEndpoints.blob)]" + }, + { + "name": "AZURE_BATCH_TRAINING_POOL_ID", + "value": "[parameters('batchPoolName')]" + }, + { + "name": "AZURE_BATCH_IMAGERYPREP_POOL_ID", + "value": "[parameters('batchPoolName')]" + }, + { + "name": "AZURE_BATCH_REGISTRY_SERVER_URL", + "value": "[format('https://{0}', parameters('acrLoginServer'))]" + }, + { + "name": "AZURE_BATCH_REGISTRY_IMAGE", + "value": "[format('{0}/{1}', parameters('acrLoginServer'), parameters('trainingImage'))]" + }, + { + "name": "AZURE_BATCH_REGISTRY_IDENTITY_RESOURCE_ID", + "value": "[parameters('umiResourceId')]" + }, + { + "name": "STATIC_APP_SUBSCRIPTION_ID", + "value": "[subscription().subscriptionId]" + }, + { + "name": "STATIC_APP_RESOURCE_GROUP", + "value": "[resourceGroup().name]" + }, + { + "name": "STATIC_APP_NAME", + "value": "[parameters('staticWebAppName')]" + }, + { + "name": "STATIC_APP_DOMAIN", + "value": "[parameters('staticAppDomain')]" + }, + { + "name": "EMAIL_CONNECTION_STRING", + "value": "[parameters('emailConnectionString')]" + }, + { + "name": "EMAIL_SENDER", + "value": "[format('DoNotReply@{0}', parameters('emailSenderDomain'))]" + }, + { + "name": "AZURE_BATCH_TRAINING_POOL_IDS", + "value": "[parameters('trainingPoolIds')]" + }, + { + "name": "AZURE_BATCH_INFERENCE_POOL_IDS", + "value": "[parameters('inferencePoolIds')]" + }, + { + "name": "AZURE_BATCH_IMAGERYPREP_POOL_IDS", + "value": "[parameters('imageryprepPoolIds')]" + }, + { + "name": "AZURE_BATCH_USE_SAS", + "value": "[if(parameters('useSas'), 'true', 'false')]" + }, + { + "name": "AZURE_BATCH_MANAGE_POOLS", + "value": "[if(parameters('managePools'), 'true', 'false')]" + } + ] + }, + "tags": { + "value": "[parameters('tags')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "12545021642088417436" + } + }, + "parameters": { + "location": { + "type": "string", + "metadata": { + "description": "Azure region." + } + }, + "name": { + "type": "string", + "metadata": { + "description": "Function app name." + } + }, + "planName": { + "type": "string", + "metadata": { + "description": "App Service (Flex Consumption) plan name." + } + }, + "alwaysReadyCount": { + "type": "int", + "metadata": { + "description": "Always-ready HTTP instance count." + } + }, + "storageAccountName": { + "type": "string", + "metadata": { + "description": "Functions storage account name." + } + }, + "fileStorageAccountName": { + "type": "string", + "metadata": { + "description": "Premium file storage account name (mounted at /data)." + } + }, + "umiResourceId": { + "type": "string", + "metadata": { + "description": "User-assigned managed identity resource id." + } + }, + "vnetName": { + "type": "string", + "metadata": { + "description": "VNet name for VNet integration." + } + }, + "functionsSubnetName": { + "type": "string", + "metadata": { + "description": "Functions subnet name." + } + }, + "logAnalyticsId": { + "type": "string", + "metadata": { + "description": "Log Analytics workspace resource id." + } + }, + "appSettings": { + "type": "array", + "defaultValue": [], + "metadata": { + "description": "Extra application settings (name/value pairs) merged after the base settings. api/queues pass the hastegeo app config here; titiler passes none." + } + }, + "tags": { + "type": "object", + "defaultValue": {}, + "metadata": { + "description": "Resource tags." + } + } + }, + "variables": { + "blobDataOwnerRoleId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'b7e6dc6d-f1e8-4753-8033-0f276bb0955b')]", + "queueDataContributorRoleId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '974c5e8b-45b9-4653-ba55-5f855dd0fb88')]", + "blobDelegatorRoleId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'db58b8e5-c6ad-4a2a-8342-4190687cbf4a')]", + "deploymentContainerName": "[format('app-package-{0}', parameters('name'))]" + }, + "resources": [ + { + "type": "Microsoft.Storage/storageAccounts/blobServices/containers", + "apiVersion": "2023-05-01", + "name": "[format('{0}/{1}/{2}', parameters('storageAccountName'), 'default', variables('deploymentContainerName'))]" + }, + { + "type": "Microsoft.Insights/components", + "apiVersion": "2020-02-02", + "name": "[format('{0}-ai', parameters('name'))]", + "location": "[parameters('location')]", + "kind": "web", + "tags": "[parameters('tags')]", + "properties": { + "Application_Type": "web", + "WorkspaceResourceId": "[parameters('logAnalyticsId')]" + } + }, + { + "type": "Microsoft.Web/serverfarms", + "apiVersion": "2023-12-01", + "name": "[parameters('planName')]", + "location": "[parameters('location')]", + "tags": "[parameters('tags')]", + "sku": { + "tier": "FlexConsumption", + "name": "FC1" + }, + "kind": "functionapp", + "properties": { + "reserved": true + } + }, + { + "type": "Microsoft.Web/sites", + "apiVersion": "2023-12-01", + "name": "[parameters('name')]", + "location": "[parameters('location')]", + "kind": "functionapp,linux", + "tags": "[parameters('tags')]", + "identity": { + "type": "SystemAssigned, UserAssigned", + "userAssignedIdentities": { + "[format('{0}', parameters('umiResourceId'))]": {} + } + }, + "properties": { + "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', parameters('planName'))]", + "httpsOnly": true, + "virtualNetworkSubnetId": "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('vnetName'), parameters('functionsSubnetName'))]", + "functionAppConfig": { + "deployment": { + "storage": { + "type": "blobContainer", + "value": "[format('{0}{1}', reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').primaryEndpoints.blob, variables('deploymentContainerName'))]", + "authentication": { + "type": "SystemAssignedIdentity" + } + } + }, + "scaleAndConcurrency": { + "maximumInstanceCount": 100, + "instanceMemoryMB": 4096, + "alwaysReady": [ + { + "name": "http", + "instanceCount": "[parameters('alwaysReadyCount')]" + } + ] + }, + "runtime": { + "name": "python", + "version": "3.11" + } + }, + "siteConfig": { + "ftpsState": "Disabled", + "appSettings": "[concat(createArray(createObject('name', 'AzureWebJobsStorage__accountName', 'value', parameters('storageAccountName')), createObject('name', 'AzureWebJobsStorage__blobServiceUri', 'value', reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').primaryEndpoints.blob), createObject('name', 'AzureWebJobsStorage__queueServiceUri', 'value', reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2023-05-01').primaryEndpoints.queue), createObject('name', 'APPLICATIONINSIGHTS_CONNECTION_STRING', 'value', reference(resourceId('Microsoft.Insights/components', format('{0}-ai', parameters('name'))), '2020-02-02').ConnectionString)), parameters('appSettings'))]" + } + }, + "dependsOn": [ + "[resourceId('Microsoft.Insights/components', format('{0}-ai', parameters('name')))]", + "[resourceId('Microsoft.Web/serverfarms', parameters('planName'))]" + ] + }, + { + "type": "Microsoft.Web/sites/config", + "apiVersion": "2023-12-01", + "name": "[format('{0}/{1}', parameters('name'), 'azurestorageaccounts')]", + "properties": { + "data": { + "type": "AzureFiles", + "accountName": "[parameters('fileStorageAccountName')]", + "shareName": "data", + "mountPath": "/data", + "accessKey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('fileStorageAccountName')), '2023-05-01').keys[0].value]" + } + }, + "dependsOn": [ + "[resourceId('Microsoft.Web/sites', parameters('name'))]" + ] + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]", + "name": "[guid(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), resourceId('Microsoft.Web/sites', parameters('name')), 'blobDataOwner')]", + "properties": { + "roleDefinitionId": "[variables('blobDataOwnerRoleId')]", + "principalId": "[reference(resourceId('Microsoft.Web/sites', parameters('name')), '2023-12-01', 'full').identity.principalId]", + "principalType": "ServicePrincipal" + }, + "dependsOn": [ + "[resourceId('Microsoft.Web/sites', parameters('name'))]" + ] + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]", + "name": "[guid(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), resourceId('Microsoft.Web/sites', parameters('name')), 'queueDataContributor')]", + "properties": { + "roleDefinitionId": "[variables('queueDataContributorRoleId')]", + "principalId": "[reference(resourceId('Microsoft.Web/sites', parameters('name')), '2023-12-01', 'full').identity.principalId]", + "principalType": "ServicePrincipal" + }, + "dependsOn": [ + "[resourceId('Microsoft.Web/sites', parameters('name'))]" + ] + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]", + "name": "[guid(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), resourceId('Microsoft.Web/sites', parameters('name')), 'blobDelegator')]", + "properties": { + "roleDefinitionId": "[variables('blobDelegatorRoleId')]", + "principalId": "[reference(resourceId('Microsoft.Web/sites', parameters('name')), '2023-12-01', 'full').identity.principalId]", + "principalType": "ServicePrincipal" + }, + "dependsOn": [ + "[resourceId('Microsoft.Web/sites', parameters('name'))]" + ] + } + ], + "outputs": { + "name": { + "type": "string", + "value": "[parameters('name')]" + }, + "defaultHostName": { + "type": "string", + "value": "[reference(resourceId('Microsoft.Web/sites', parameters('name')), '2023-12-01').defaultHostName]" + }, + "systemPrincipalId": { + "type": "string", + "value": "[reference(resourceId('Microsoft.Web/sites', parameters('name')), '2023-12-01', 'full').identity.principalId]" + } + } + } + }, + "dependsOn": [ + "[resourceId('Microsoft.Resources/deployments', 'fn-titiler')]" + ] + } + ], + "outputs": { + "apiSystemPrincipalId": { + "type": "string", + "value": "[reference(resourceId('Microsoft.Resources/deployments', 'fn-api'), '2025-04-01').outputs.systemPrincipalId.value]" + }, + "apiName": { + "type": "string", + "value": "[reference(resourceId('Microsoft.Resources/deployments', 'fn-api'), '2025-04-01').outputs.name.value]" + }, + "titilerName": { + "type": "string", + "value": "[reference(resourceId('Microsoft.Resources/deployments', 'fn-titiler'), '2025-04-01').outputs.name.value]" + }, + "queueName": { + "type": "string", + "value": "[reference(resourceId('Microsoft.Resources/deployments', 'fn-queue'), '2025-04-01').outputs.name.value]" + } + } + } + }, + "dependsOn": [ + "batchAccount", + "communication", + "frontend", + "identity", + "monitoring", + "network", + "rg", + "storage" + ] + }, + "frontend": { + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "frontend", + "resourceGroup": "[variables('rgName')]", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "location": { + "value": "[parameters('location')]" + }, + "staticWebAppName": { + "value": "[variables('staticWebAppName')]" + }, + "mapsAccountName": { + "value": "[variables('mapsAccountName')]" + }, + "apimResourceId": { + "value": "[reference('apim').outputs.resourceId.value]" + }, + "tags": { + "value": "[parameters('tags')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "15152592237032588221" + } + }, + "parameters": { + "location": { + "type": "string", + "metadata": { + "description": "Azure region for the Static Web App." + } + }, + "staticWebAppName": { + "type": "string", + "metadata": { + "description": "Static Web App name." + } + }, + "mapsAccountName": { + "type": "string", + "metadata": { + "description": "Azure Maps account name." + } + }, + "apimResourceId": { + "type": "string", + "metadata": { + "description": "APIM resource id to link as the SWA backend." + } + }, + "tags": { + "type": "object", + "defaultValue": {}, + "metadata": { + "description": "Resource tags." + } + } + }, + "resources": [ + { + "type": "Microsoft.Web/staticSites", + "apiVersion": "2024-04-01", + "name": "[parameters('staticWebAppName')]", + "location": "[parameters('location')]", + "tags": "[parameters('tags')]", + "sku": { + "name": "Standard", + "tier": "Standard" + }, + "properties": {} + }, + { + "type": "Microsoft.Web/staticSites/linkedBackends", + "apiVersion": "2024-04-01", + "name": "[format('{0}/{1}', parameters('staticWebAppName'), 'apim-backend')]", + "properties": { + "backendResourceId": "[parameters('apimResourceId')]", + "region": "[parameters('location')]" + }, + "dependsOn": [ + "[resourceId('Microsoft.Web/staticSites', parameters('staticWebAppName'))]" + ] + }, + { + "type": "Microsoft.Maps/accounts", + "apiVersion": "2023-06-01", + "name": "[parameters('mapsAccountName')]", + "location": "global", + "tags": "[parameters('tags')]", + "kind": "Gen2", + "sku": { + "name": "G2" + }, + "properties": {} + } + ], + "outputs": { + "staticWebAppName": { + "type": "string", + "value": "[parameters('staticWebAppName')]" + }, + "staticWebAppHostName": { + "type": "string", + "value": "[reference(resourceId('Microsoft.Web/staticSites', parameters('staticWebAppName')), '2024-04-01').defaultHostname]" + }, + "mapsAccountName": { + "type": "string", + "value": "[parameters('mapsAccountName')]" + }, + "mapsClientId": { + "type": "string", + "value": "[reference(resourceId('Microsoft.Maps/accounts', parameters('mapsAccountName')), '2023-06-01').uniqueId]" + } + } + } + }, + "dependsOn": [ + "apim", + "rg" + ] + }, + "apimApis": { + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "apimApis", + "resourceGroup": "[variables('rgName')]", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "apimName": { + "value": "[variables('apimName')]" + }, + "functionApiName": { + "value": "[variables('functionApiName')]" + }, + "functionTitilerName": { + "value": "[variables('functionTitilerName')]" + }, + "staticWebAppHostName": { + "value": "[reference('frontend').outputs.staticWebAppHostName.value]" + }, + "storageAccountName": { + "value": "[variables('storageAccountName')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "6825195985756767924" + } + }, + "parameters": { + "apimName": { + "type": "string", + "metadata": { + "description": "API Management service name." + } + }, + "functionApiName": { + "type": "string", + "metadata": { + "description": "API function app name (api-id + backend)." + } + }, + "functionTitilerName": { + "type": "string", + "metadata": { + "description": "TiTiler function app name (api-id + backend)." + } + }, + "staticWebAppHostName": { + "type": "string", + "metadata": { + "description": "Static Web App default hostname; its first label is the generated APIM product id." + } + }, + "storageAccountName": { + "type": "string", + "metadata": { + "description": "Storage account name backing the storage-proxy operations." + } + } + }, + "variables": { + "swaProductId": "[split(parameters('staticWebAppHostName'), '.')[0]]", + "subscriptionKeyParameterNames": { + "header": "Ocp-Apim-Subscription-Key", + "query": "subscription-key" + } + }, + "resources": [ + { + "type": "Microsoft.ApiManagement/service/apis", + "apiVersion": "2024-06-01-preview", + "name": "[format('{0}/{1}', parameters('apimName'), parameters('functionApiName'))]", + "properties": { + "displayName": "[parameters('functionApiName')]", + "apiRevision": "1", + "description": "[format('Import from \"{0}\" Function App', parameters('functionApiName'))]", + "subscriptionRequired": true, + "path": "api/haste", + "protocols": [ + "https" + ], + "subscriptionKeyParameterNames": "[variables('subscriptionKeyParameterNames')]", + "isCurrent": true + } + }, + { + "type": "Microsoft.ApiManagement/service/backends", + "apiVersion": "2024-06-01-preview", + "name": "[format('{0}/{1}', parameters('apimName'), parameters('functionApiName'))]", + "properties": { + "description": "[parameters('functionApiName')]", + "url": "[format('https://{0}.azurewebsites.net/api', parameters('functionApiName'))]", + "protocol": "http", + "resourceId": "[format('{0}{1}', environment().resourceManager, substring(resourceId('Microsoft.Web/sites', parameters('functionApiName')), 1))]" + } + }, + { + "type": "Microsoft.ApiManagement/service/products/apis", + "apiVersion": "2024-06-01-preview", + "name": "[format('{0}/{1}/{2}', parameters('apimName'), variables('swaProductId'), parameters('functionApiName'))]", + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis', parameters('apimName'), parameters('functionApiName'))]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/policies", + "apiVersion": "2024-06-01-preview", + "name": "[format('{0}/{1}/{2}', parameters('apimName'), parameters('functionApiName'), 'policy')]", + "properties": { + "format": "xml", + "value": "[replace('\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n', '__BID__', parameters('functionApiName'))]" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis', parameters('apimName'), parameters('functionApiName'))]", + "[resourceId('Microsoft.ApiManagement/service/backends', parameters('apimName'), parameters('functionApiName'))]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis", + "apiVersion": "2024-06-01-preview", + "name": "[format('{0}/{1}', parameters('apimName'), parameters('functionTitilerName'))]", + "properties": { + "displayName": "[parameters('functionTitilerName')]", + "apiRevision": "1", + "description": "[format('Import from \"{0}\" Function App', parameters('functionTitilerName'))]", + "subscriptionRequired": true, + "path": "api/titiler", + "protocols": [ + "https" + ], + "subscriptionKeyParameterNames": "[variables('subscriptionKeyParameterNames')]", + "isCurrent": true + } + }, + { + "type": "Microsoft.ApiManagement/service/backends", + "apiVersion": "2024-06-01-preview", + "name": "[format('{0}/{1}', parameters('apimName'), parameters('functionTitilerName'))]", + "properties": { + "description": "[parameters('functionTitilerName')]", + "url": "[format('https://{0}.azurewebsites.net', parameters('functionTitilerName'))]", + "protocol": "http", + "resourceId": "[format('{0}{1}', environment().resourceManager, substring(resourceId('Microsoft.Web/sites', parameters('functionTitilerName')), 1))]" + } + }, + { + "type": "Microsoft.ApiManagement/service/products/apis", + "apiVersion": "2024-06-01-preview", + "name": "[format('{0}/{1}/{2}', parameters('apimName'), variables('swaProductId'), parameters('functionTitilerName'))]", + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis', parameters('apimName'), parameters('functionTitilerName'))]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/policies", + "apiVersion": "2024-06-01-preview", + "name": "[format('{0}/{1}/{2}', parameters('apimName'), parameters('functionTitilerName'), 'policy')]", + "properties": { + "format": "xml", + "value": "[replace('\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n', '__BID__', parameters('functionTitilerName'))]" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis', parameters('apimName'), parameters('functionTitilerName'))]", + "[resourceId('Microsoft.ApiManagement/service/backends', parameters('apimName'), parameters('functionTitilerName'))]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations", + "apiVersion": "2024-06-01-preview", + "name": "[format('{0}/{1}/{2}', parameters('apimName'), parameters('functionTitilerName'), 'get-tiles')]", + "properties": { + "displayName": "get-tiles", + "method": "GET", + "urlTemplate": "/cog/tiles/WebMercatorQuad/{z}/{x}/{y}", + "templateParameters": [ + { + "name": "z", + "required": true, + "type": "string" + }, + { + "name": "x", + "required": true, + "type": "string" + }, + { + "name": "y", + "required": true, + "type": "string" + } + ] + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis', parameters('apimName'), parameters('functionTitilerName'))]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis", + "apiVersion": "2024-06-01-preview", + "name": "[format('{0}/{1}', parameters('apimName'), 'hastestorageapi')]", + "properties": { + "displayName": "hastestorageapi", + "apiRevision": "1", + "description": "Storage artifact proxy (managed identity).", + "subscriptionRequired": true, + "path": "api/haste/storage", + "protocols": [ + "https" + ], + "subscriptionKeyParameterNames": "[variables('subscriptionKeyParameterNames')]", + "isCurrent": true + } + }, + { + "type": "Microsoft.ApiManagement/service/products/apis", + "apiVersion": "2024-06-01-preview", + "name": "[format('{0}/{1}/{2}', parameters('apimName'), variables('swaProductId'), 'hastestorageapi')]", + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis', parameters('apimName'), 'hastestorageapi')]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations", + "apiVersion": "2024-06-01-preview", + "name": "[format('{0}/{1}/{2}', parameters('apimName'), 'hastestorageapi', 'get-artifacts')]", + "properties": { + "displayName": "get-artifacts", + "method": "GET", + "urlTemplate": "/get-artifacts/{container}/{projectDir}/{modelDir}/{fileName}", + "templateParameters": [ + { + "name": "container", + "required": true, + "type": "string" + }, + { + "name": "projectDir", + "required": true, + "type": "string" + }, + { + "name": "modelDir", + "required": true, + "type": "string" + }, + { + "name": "fileName", + "required": true, + "type": "string" + } + ] + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis', parameters('apimName'), 'hastestorageapi')]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations/policies", + "apiVersion": "2024-06-01-preview", + "name": "[format('{0}/{1}/{2}/{3}', parameters('apimName'), 'hastestorageapi', 'get-artifacts', 'policy')]", + "properties": { + "format": "xml", + "value": "[replace('\r\n \r\n \r\n @{ return \"https://__SA__.blob.core.windows.net/\" + (string)context.Request.MatchedParameters[\"container\"] + \"/\" + (string)context.Request.MatchedParameters[\"projectDir\"] + \"/\" + (string)context.Request.MatchedParameters[\"modelDir\"]+ \"/\" + (string)context.Request.MatchedParameters[\"fileName\"]; }\r\n GET\r\n \r\n 2019-07-07\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n', '__SA__', parameters('storageAccountName'))]" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis/operations', parameters('apimName'), 'hastestorageapi', 'get-artifacts')]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations", + "apiVersion": "2024-06-01-preview", + "name": "[format('{0}/{1}/{2}', parameters('apimName'), 'hastestorageapi', 'get-project-artifacts')]", + "properties": { + "displayName": "get-project-artifacts", + "method": "GET", + "urlTemplate": "/get-project-artifacts/{container}/{projectDir}/{fileName}", + "templateParameters": [ + { + "name": "container", + "required": true, + "type": "string" + }, + { + "name": "projectDir", + "required": true, + "type": "string" + }, + { + "name": "fileName", + "required": true, + "type": "string" + } + ] + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis', parameters('apimName'), 'hastestorageapi')]" + ] + }, + { + "type": "Microsoft.ApiManagement/service/apis/operations/policies", + "apiVersion": "2024-06-01-preview", + "name": "[format('{0}/{1}/{2}/{3}', parameters('apimName'), 'hastestorageapi', 'get-project-artifacts', 'policy')]", + "properties": { + "format": "xml", + "value": "[replace('\r\n \r\n \r\n @{ return \"https://__SA__.blob.core.windows.net/\" + (string)context.Request.MatchedParameters[\"container\"] + \"/\" + (string)context.Request.MatchedParameters[\"projectDir\"] + \"/\" + (string)context.Request.MatchedParameters[\"fileName\"]; }\r\n GET\r\n \r\n 2019-07-07\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n', '__SA__', parameters('storageAccountName'))]" + }, + "dependsOn": [ + "[resourceId('Microsoft.ApiManagement/service/apis/operations', parameters('apimName'), 'hastestorageapi', 'get-project-artifacts')]" + ] + } + ] + } + }, + "dependsOn": [ + "frontend", + "functions", + "rg" + ] + }, + "roles": { + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "roles", + "resourceGroup": "[variables('rgName')]", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "staticWebAppName": { + "value": "[variables('staticWebAppName')]" + }, + "mapsAccountName": { + "value": "[variables('mapsAccountName')]" + }, + "functionSystemPrincipalId": { + "value": "[reference('functions').outputs.apiSystemPrincipalId.value]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "6001236453914961257" + } + }, + "parameters": { + "staticWebAppName": { + "type": "string", + "metadata": { + "description": "Static Web App name to scope the assignment to." + } + }, + "mapsAccountName": { + "type": "string", + "metadata": { + "description": "Azure Maps account name to grant the API identity read access on." + } + }, + "functionSystemPrincipalId": { + "type": "string", + "metadata": { + "description": "System-assigned principal id of the API function app." + } + } + }, + "resources": [ + { + "type": "Microsoft.Authorization/roleDefinitions", + "apiVersion": "2022-04-01", + "name": "[guid(resourceGroup().id, 'HasteWebAppUserManager')]", + "properties": { + "roleName": "[format('HasteWebAppUserManager-{0}', uniqueString(resourceGroup().id))]", + "description": "Lets the HASTE API app manage Static Web App users (invitations).", + "type": "CustomRole", + "permissions": [ + { + "actions": [ + "microsoft.web/staticSites/*" + ], + "notActions": [], + "dataActions": [], + "notDataActions": [] + } + ], + "assignableScopes": [ + "[resourceGroup().id]" + ] + } + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.Web/staticSites', parameters('staticWebAppName'))]", + "name": "[guid(resourceId('Microsoft.Web/staticSites', parameters('staticWebAppName')), parameters('functionSystemPrincipalId'), 'HasteWebAppUserManager')]", + "properties": { + "roleDefinitionId": "[resourceId('Microsoft.Authorization/roleDefinitions', guid(resourceGroup().id, 'HasteWebAppUserManager'))]", + "principalId": "[parameters('functionSystemPrincipalId')]", + "principalType": "ServicePrincipal" + }, + "dependsOn": [ + "[resourceId('Microsoft.Authorization/roleDefinitions', guid(resourceGroup().id, 'HasteWebAppUserManager'))]" + ] + }, + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.Maps/accounts', parameters('mapsAccountName'))]", + "name": "[guid(resourceId('Microsoft.Maps/accounts', parameters('mapsAccountName')), parameters('functionSystemPrincipalId'), 'AzureMapsDataReader')]", + "properties": { + "roleDefinitionId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '423170ca-a8f6-4b0f-8487-9e4eb8f49bfa')]", + "principalId": "[parameters('functionSystemPrincipalId')]", + "principalType": "ServicePrincipal" + } + } + ], + "outputs": { + "roleDefinitionId": { + "type": "string", + "value": "[resourceId('Microsoft.Authorization/roleDefinitions', guid(resourceGroup().id, 'HasteWebAppUserManager'))]" + } + } + } + }, + "dependsOn": [ + "functions", + "rg" + ] + }, + "batchAccount": { + "condition": "[variables('createBatchAccount')]", + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "batchAccount", + "resourceGroup": "[variables('rgName')]", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "location": { + "value": "[parameters('location')]" + }, + "batchAccountName": { + "value": "[variables('createdBatchAccountName')]" + }, + "tags": { + "value": "[parameters('tags')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "16832494055114601667" + } + }, + "parameters": { + "location": { + "type": "string", + "metadata": { + "description": "Azure region." + } + }, + "batchAccountName": { + "type": "string", + "metadata": { + "description": "Batch account name." + } + }, + "tags": { + "type": "object", + "defaultValue": {}, + "metadata": { + "description": "Resource tags." + } + } + }, + "resources": [ + { + "type": "Microsoft.Batch/batchAccounts", + "apiVersion": "2024-07-01", + "name": "[parameters('batchAccountName')]", + "location": "[parameters('location')]", + "tags": "[parameters('tags')]", + "properties": { + "poolAllocationMode": "BatchService" + } + } + ], + "outputs": { + "name": { + "type": "string", + "value": "[parameters('batchAccountName')]" + }, + "id": { + "type": "string", + "value": "[resourceId('Microsoft.Batch/batchAccounts', parameters('batchAccountName'))]" + } + } + } + }, + "dependsOn": [ + "rg" + ] + }, + "batchPool": { + "condition": "[equals(parameters('batchPoolMode'), 'Create')]", + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "batchPool", + "resourceGroup": "[variables('batchAccountRg')]", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "batchAccountName": { + "value": "[variables('resolvedBatchAccountName')]" + }, + "poolName": { + "value": "[variables('createdBatchPoolName')]" + }, + "vmSize": { + "value": "[parameters('batchPoolVmSize')]" + }, + "maxNodes": { + "value": "[parameters('batchPoolMaxNodes')]" + }, + "umiResourceId": { + "value": "[reference('identity').outputs.resourceId.value]" + }, + "subnetId": { + "value": "[resourceId(subscription().subscriptionId, variables('rgName'), 'Microsoft.Network/virtualNetworks/subnets', variables('vnetName'), parameters('batchPoolSubnetName'))]" + }, + "acrName": { + "value": "[parameters('sharedAcrName')]" + }, + "trainingImage": { + "value": "[parameters('trainingImage')]" + }, + "imageryprepImage": { + "value": "[parameters('imageryprepImage')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "14366334891975298585" + } + }, + "parameters": { + "batchAccountName": { + "type": "string", + "metadata": { + "description": "Resolved Batch account name (existing or just-created)." + } + }, + "poolName": { + "type": "string", + "metadata": { + "description": "Pool name." + } + }, + "vmSize": { + "type": "string", + "metadata": { + "description": "Pool VM size." + } + }, + "maxNodes": { + "type": "int", + "metadata": { + "description": "Max nodes (autoscale cap)." + } + }, + "scaleMode": { + "type": "string", + "defaultValue": "Autoscale", + "allowedValues": [ + "Fixed", + "Autoscale" + ], + "metadata": { + "description": "Scale mode: Fixed (dev/prod reserved) or Autoscale (shared-demo burst)." + } + }, + "nodeType": { + "type": "string", + "defaultValue": "Dedicated", + "allowedValues": [ + "Dedicated", + "LowPriority" + ], + "metadata": { + "description": "Node cost tier: Dedicated (guaranteed) or LowPriority (spot, preemptible)." + } + }, + "fixedNodeCount": { + "type": "int", + "defaultValue": 1, + "metadata": { + "description": "Node count when scaleMode == Fixed." + } + }, + "minNodes": { + "type": "int", + "defaultValue": 1, + "metadata": { + "description": "Autoscale floor. 0 = scale-to-zero when idle (shared-demo burst); 1 keeps the legacy always-on behavior." + } + }, + "umiResourceId": { + "type": "string", + "metadata": { + "description": "User-assigned managed identity resource id (for ACR pull)." + } + }, + "subnetId": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "VNet subnet resource id for VNet-injected pools. Empty => no VNet injection (BatchManaged public IPs). Shared-demo pools finalize their subnet + per-demo-storage firewall allowlisting before running workloads." + } + }, + "acrName": { + "type": "string", + "metadata": { + "description": "Shared ACR name (without .azurecr.io)." + } + }, + "trainingImage": { + "type": "string", + "metadata": { + "description": "Training container image (tag included)." + } + }, + "imageryprepImage": { + "type": "string", + "metadata": { + "description": "Imageryprep container image (tag included)." + } + } + }, + "variables": { + "registryServer": "[format('{0}.azurecr.io', parameters('acrName'))]", + "scaleTargetVar": "[if(equals(parameters('nodeType'), 'LowPriority'), '$TargetLowPriorityNodes', '$TargetDedicatedNodes')]", + "autoscaleFormula": "[format('$samples = $ActiveTasks.GetSamplePercent(TimeInterval_Minute * 15);$tasks = $samples < 70 ? max(0, $ActiveTasks.GetSample(1)) : max($ActiveTasks.GetSample(1), avg($ActiveTasks.GetSample(TimeInterval_Minute * 15)));$targetVMs = $tasks > 0 ? $tasks : {0};{1} = max({2}, min($targetVMs, {3}));$NodeDeallocationOption = taskcompletion;', parameters('minNodes'), variables('scaleTargetVar'), parameters('minNodes'), parameters('maxNodes'))]", + "scaleSettings": "[if(equals(parameters('scaleMode'), 'Fixed'), createObject('fixedScale', createObject('targetDedicatedNodes', if(equals(parameters('nodeType'), 'Dedicated'), parameters('fixedNodeCount'), 0), 'targetLowPriorityNodes', if(equals(parameters('nodeType'), 'LowPriority'), parameters('fixedNodeCount'), 0), 'resizeTimeout', 'PT15M')), createObject('autoScale', createObject('formula', variables('autoscaleFormula'), 'evaluationInterval', 'PT5M')))]" + }, + "resources": [ + { + "type": "Microsoft.Batch/batchAccounts/pools", + "apiVersion": "2024-07-01", + "name": "[format('{0}/{1}', parameters('batchAccountName'), parameters('poolName'))]", + "identity": { + "type": "UserAssigned", + "userAssignedIdentities": { + "[format('{0}', parameters('umiResourceId'))]": {} + } + }, + "properties": "[shallowMerge(createArray(createObject('vmSize', parameters('vmSize'), 'interNodeCommunication', 'Disabled', 'taskSlotsPerNode', 1, 'taskSchedulingPolicy', createObject('nodeFillType', 'Pack'), 'deploymentConfiguration', createObject('virtualMachineConfiguration', createObject('imageReference', createObject('publisher', 'microsoft-dsvm', 'offer', 'ubuntu-hpc', 'sku', '2204', 'version', 'latest'), 'nodeAgentSkuId', 'batch.node.ubuntu 22.04', 'osDisk', createObject('caching', 'None', 'diskSizeGB', 1023, 'managedDisk', createObject('storageAccountType', 'Premium_LRS')), 'containerConfiguration', createObject('type', 'DockerCompatible', 'containerImageNames', createArray(format('{0}/{1}', variables('registryServer'), parameters('trainingImage')), format('{0}/{1}', variables('registryServer'), parameters('imageryprepImage'))), 'containerRegistries', createArray(createObject('registryServer', format('https://{0}', variables('registryServer')), 'identityReference', createObject('resourceId', parameters('umiResourceId'))))), 'nodePlacementConfiguration', createObject('policy', 'Regional')))), if(empty(parameters('subnetId')), createObject(), createObject('networkConfiguration', createObject('subnetId', parameters('subnetId'), 'publicIPAddressConfiguration', createObject('provision', 'BatchManaged'), 'dynamicVnetAssignmentScope', 'none', 'enableAcceleratedNetworking', false()))), createObject('scaleSettings', variables('scaleSettings'))))]" + } + ], + "outputs": { + "poolName": { + "type": "string", + "value": "[parameters('poolName')]" + } + } + } + }, + "dependsOn": [ + "batchAccount", + "identity", + "network" + ] + }, + "acrRole": { + "condition": "[variables('wireAcr')]", + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "acrRole", + "resourceGroup": "[variables('resolvedSharedRg')]", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "acrName": { + "value": "[parameters('sharedAcrName')]" + }, + "umiPrincipalId": { + "value": "[reference('identity').outputs.principalId.value]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "15881723511064074666" + } + }, + "parameters": { + "acrName": { + "type": "string", + "metadata": { + "description": "ACR name (without .azurecr.io)." + } + }, + "umiPrincipalId": { + "type": "string", + "metadata": { + "description": "Principal id of the user-assigned managed identity." + } + } + }, + "variables": { + "acrPullRoleId": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '7f951dda-4ed3-4680-a7ca-43fe172d538d')]" + }, + "resources": [ + { + "type": "Microsoft.Authorization/roleAssignments", + "apiVersion": "2022-04-01", + "scope": "[resourceId('Microsoft.ContainerRegistry/registries', parameters('acrName'))]", + "name": "[guid(resourceId('Microsoft.ContainerRegistry/registries', parameters('acrName')), parameters('umiPrincipalId'), 'AcrPull')]", + "properties": { + "roleDefinitionId": "[variables('acrPullRoleId')]", + "principalId": "[parameters('umiPrincipalId')]", + "principalType": "ServicePrincipal" + } + } + ] + } + }, + "dependsOn": [ + "identity" + ] + }, + "frontDoor": { + "condition": "[parameters('enableFrontDoor')]", + "type": "Microsoft.Resources/deployments", + "apiVersion": "2025-04-01", + "name": "frontDoor", + "resourceGroup": "[variables('rgName')]", + "properties": { + "expressionEvaluationOptions": { + "scope": "inner" + }, + "mode": "Incremental", + "parameters": { + "frontDoorName": { + "value": "[variables('frontDoorName')]" + }, + "wafPolicyName": { + "value": "[variables('wafPolicyName')]" + }, + "staticWebAppName": { + "value": "[variables('staticWebAppName')]" + } + }, + "template": { + "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", + "contentVersion": "1.0.0.0", + "metadata": { + "_generator": { + "name": "bicep", + "version": "0.43.8.12551", + "templateHash": "18374080558664767600" + } + }, + "parameters": { + "frontDoorName": { + "type": "string", + "metadata": { + "description": "Front Door (CDN) profile name." + } + }, + "wafPolicyName": { + "type": "string", + "metadata": { + "description": "WAF policy name." + } + }, + "staticWebAppName": { + "type": "string", + "metadata": { + "description": "Static Web App name to use as the origin." + } + } + }, + "resources": [ + { + "type": "Microsoft.Network/FrontDoorWebApplicationFirewallPolicies", + "apiVersion": "2024-02-01", + "name": "[parameters('wafPolicyName')]", + "location": "global", + "sku": { + "name": "Premium_AzureFrontDoor" + }, + "properties": { + "policySettings": { + "enabledState": "Enabled", + "mode": "Prevention" + }, + "managedRules": { + "managedRuleSets": [ + { + "ruleSetType": "DefaultRuleSet", + "ruleSetVersion": "1.0" + }, + { + "ruleSetType": "Microsoft_BotManagerRuleSet", + "ruleSetVersion": "1.0" + } + ] + } + } + }, + { + "type": "Microsoft.Cdn/profiles", + "apiVersion": "2024-02-01", + "name": "[parameters('frontDoorName')]", + "location": "global", + "sku": { + "name": "Premium_AzureFrontDoor" + }, + "identity": { + "type": "SystemAssigned" + } + }, + { + "type": "Microsoft.Cdn/profiles/afdEndpoints", + "apiVersion": "2024-02-01", + "name": "[format('{0}/{1}', parameters('frontDoorName'), format('{0}-endpoint', parameters('frontDoorName')))]", + "location": "global", + "properties": { + "enabledState": "Enabled" + }, + "dependsOn": [ + "[resourceId('Microsoft.Cdn/profiles', parameters('frontDoorName'))]" + ] + }, + { + "type": "Microsoft.Cdn/profiles/originGroups", + "apiVersion": "2024-02-01", + "name": "[format('{0}/{1}', parameters('frontDoorName'), format('{0}-origin-group', parameters('frontDoorName')))]", + "properties": { + "loadBalancingSettings": { + "sampleSize": 4, + "successfulSamplesRequired": 3, + "additionalLatencyInMilliseconds": 50 + }, + "healthProbeSettings": { + "probePath": "/", + "probeRequestType": "GET", + "probeProtocol": "Https", + "probeIntervalInSeconds": 30 + } + }, + "dependsOn": [ + "[resourceId('Microsoft.Cdn/profiles', parameters('frontDoorName'))]" + ] + }, + { + "type": "Microsoft.Cdn/profiles/originGroups/origins", + "apiVersion": "2024-02-01", + "name": "[format('{0}/{1}/{2}', parameters('frontDoorName'), format('{0}-origin-group', parameters('frontDoorName')), format('{0}-origin', parameters('frontDoorName')))]", + "properties": { + "hostName": "[reference(resourceId('Microsoft.Web/staticSites', parameters('staticWebAppName')), '2024-04-01').defaultHostname]", + "originHostHeader": "[reference(resourceId('Microsoft.Web/staticSites', parameters('staticWebAppName')), '2024-04-01').defaultHostname]", + "httpPort": 80, + "httpsPort": 443, + "priority": 1, + "weight": 1000, + "enabledState": "Enabled" + }, + "dependsOn": [ + "[resourceId('Microsoft.Cdn/profiles/originGroups', parameters('frontDoorName'), format('{0}-origin-group', parameters('frontDoorName')))]" + ] + }, + { + "type": "Microsoft.Cdn/profiles/afdEndpoints/routes", + "apiVersion": "2024-02-01", + "name": "[format('{0}/{1}/{2}', parameters('frontDoorName'), format('{0}-endpoint', parameters('frontDoorName')), 'default-route')]", + "properties": { + "originGroup": { + "id": "[resourceId('Microsoft.Cdn/profiles/originGroups', parameters('frontDoorName'), format('{0}-origin-group', parameters('frontDoorName')))]" + }, + "patternsToMatch": [ + "/*" + ], + "forwardingProtocol": "MatchRequest", + "httpsRedirect": "Enabled", + "linkToDefaultDomain": "Enabled", + "supportedProtocols": [ + "Http", + "Https" + ], + "enabledState": "Enabled" + }, + "dependsOn": [ + "[resourceId('Microsoft.Cdn/profiles/afdEndpoints', parameters('frontDoorName'), format('{0}-endpoint', parameters('frontDoorName')))]", + "[resourceId('Microsoft.Cdn/profiles/originGroups/origins', parameters('frontDoorName'), format('{0}-origin-group', parameters('frontDoorName')), format('{0}-origin', parameters('frontDoorName')))]", + "[resourceId('Microsoft.Cdn/profiles/originGroups', parameters('frontDoorName'), format('{0}-origin-group', parameters('frontDoorName')))]" + ] + }, + { + "type": "Microsoft.Cdn/profiles/securityPolicies", + "apiVersion": "2024-02-01", + "name": "[format('{0}/{1}', parameters('frontDoorName'), 'Security')]", + "properties": { + "parameters": { + "type": "WebApplicationFirewall", + "wafPolicy": { + "id": "[resourceId('Microsoft.Network/FrontDoorWebApplicationFirewallPolicies', parameters('wafPolicyName'))]" + }, + "associations": [ + { + "domains": [ + { + "id": "[resourceId('Microsoft.Cdn/profiles/afdEndpoints', parameters('frontDoorName'), format('{0}-endpoint', parameters('frontDoorName')))]" + } + ], + "patternsToMatch": [ + "/*" + ] + } + ] + } + }, + "dependsOn": [ + "[resourceId('Microsoft.Cdn/profiles/afdEndpoints', parameters('frontDoorName'), format('{0}-endpoint', parameters('frontDoorName')))]", + "[resourceId('Microsoft.Cdn/profiles', parameters('frontDoorName'))]", + "[resourceId('Microsoft.Network/FrontDoorWebApplicationFirewallPolicies', parameters('wafPolicyName'))]" + ] + } + ], + "outputs": { + "endpointHostName": { + "type": "string", + "value": "[reference(resourceId('Microsoft.Cdn/profiles/afdEndpoints', parameters('frontDoorName'), format('{0}-endpoint', parameters('frontDoorName'))), '2024-02-01').hostName]" + } + } + } + }, + "dependsOn": [ + "frontend", + "rg" + ] + } + }, + "outputs": { + "AZURE_RESOURCE_GROUP": { + "type": "string", + "value": "[variables('rgName')]" + }, + "AZURE_LOCATION": { + "type": "string", + "value": "[parameters('location')]" + }, + "FUNCTION_API_NAME": { + "type": "string", + "value": "[variables('functionApiName')]" + }, + "FUNCTION_TITILER_NAME": { + "type": "string", + "value": "[variables('functionTitilerName')]" + }, + "FUNCTION_QUEUE_NAME": { + "type": "string", + "value": "[variables('functionQueueName')]" + }, + "STATIC_WEB_APP_NAME": { + "type": "string", + "value": "[variables('staticWebAppName')]" + }, + "VITE_AZURE_MAPS_CLIENT_ID": { + "type": "string", + "value": "[reference('frontend').outputs.mapsClientId.value]" + }, + "APIM_NAME": { + "type": "string", + "value": "[variables('apimName')]" + }, + "STORAGE_ACCOUNT_NAME": { + "type": "string", + "value": "[variables('storageAccountName')]" + }, + "BATCH_ACCOUNT_NAME": { + "type": "string", + "value": "[variables('resolvedBatchAccountName')]" + }, + "BATCH_POOL_NAME": { + "type": "string", + "value": "[if(equals(parameters('batchPoolMode'), 'Create'), variables('createdBatchPoolName'), parameters('existingBatchPoolId'))]" + }, + "ACS_CONNECTION_STRING": { + "type": "securestring", + "value": "[listOutputsWithSecureValues('communication', '2025-04-01').connectionString]" + }, + "EMAIL_SENDER_DOMAIN": { + "type": "string", + "value": "[reference('communication').outputs.senderDomain.value]" + } + } +} \ No newline at end of file diff --git a/infra/modules/storage.bicep b/infra/modules/storage.bicep index f24c1db..69b31f9 100644 --- a/infra/modules/storage.bicep +++ b/infra/modules/storage.bicep @@ -26,6 +26,9 @@ param functionsSubnetName string @description('Batch subnet name.') param batchSubnetName string +@description('Shared hub batch-subnet resource id (where the SHARED multi-tenant pools are VNet-injected). Empty for single-tenant envs (prod) that run their own in-env pool. See spec/features/batch-compute-expansion/networking.md.') +param sharedBatchSubnetId string = '' + @description('Resource tags.') param tags object = {} @@ -54,7 +57,11 @@ resource storageAccount 'Microsoft.Storage/storageAccounts@2023-05-01' = { networkAcls: { bypass: 'Logging, Metrics, AzureServices' defaultAction: 'Deny' - virtualNetworkRules: [ + // The env's own subnets, plus (for shared-pool envs) the shared hub + // batch-subnet so the multi-tenant pools can reach this tenant's blobs + // over the Microsoft.Storage service endpoint. Adding a demo env = this + // one rule; no shared-pool change (see networking.md). + virtualNetworkRules: concat([ { id: defaultSubnetId action: 'Allow' @@ -67,7 +74,12 @@ resource storageAccount 'Microsoft.Storage/storageAccounts@2023-05-01' = { id: batchSubnetId action: 'Allow' } - ] + ], empty(sharedBatchSubnetId) ? [] : [ + { + id: sharedBatchSubnetId + action: 'Allow' + } + ]) } } } diff --git a/infra/shared-pools.bicep b/infra/shared-pools.bicep index d4b3f39..fe1431b 100644 --- a/infra/shared-pools.bicep +++ b/infra/shared-pools.bicep @@ -59,6 +59,12 @@ param h100ScaleMode string = 'Autoscale' @description('H100 reserved node count when h100ScaleMode == Fixed.') param h100FixedNodeCount int = 1 +@description('H100 autoscale floor (when h100ScaleMode == Autoscale). 0 = scale-to-zero; >0 keeps chasing a warm baseline (auto-retries allocation each eval under GPU contention).') +param h100MinNodes int = 0 + +@description('VNet subnet resource id to inject BOTH shared pools into (blob<->batch reach via Microsoft.Storage service endpoint + per-tenant storage VNet rule). Empty => no VNet injection (BatchManaged public IPs). See spec/features/batch-compute-expansion/networking.md.') +param sharedBatchSubnetId string = '' + module h100Pool 'modules/batchPool.bicep' = { name: 'shared-${sharedGroup}-h100' params: { @@ -68,14 +74,13 @@ module h100Pool 'modules/batchPool.bicep' = { scaleMode: h100ScaleMode fixedNodeCount: h100FixedNodeCount nodeType: sharedNodeType - minNodes: 0 + minNodes: h100MinNodes maxNodes: h100MaxNodes umiResourceId: umiResourceId acrName: acrName trainingImage: trainingImage imageryprepImage: imageryprepImage - // subnetId omitted => no VNet injection yet; finalize subnet + per-tenant - // storage firewall allowlisting before running real workloads. + subnetId: sharedBatchSubnetId } } @@ -93,6 +98,7 @@ module t4Pool 'modules/batchPool.bicep' = { acrName: acrName trainingImage: trainingImage imageryprepImage: imageryprepImage + subnetId: sharedBatchSubnetId } } diff --git a/infra/shared-pools.bicepparam b/infra/shared-pools.bicepparam index 61bb37f..d03c006 100644 --- a/infra/shared-pools.bicepparam +++ b/infra/shared-pools.bicepparam @@ -19,8 +19,10 @@ param sharedPrefix = readEnvironmentVariable('HASTE_RESOURCE_PREFIX', 'haste') param sharedGroup = readEnvironmentVariable('HASTE_SHARED_GROUP', 'dev') param sharedNodeType = readEnvironmentVariable('HASTE_SHARED_NODE_TYPE', 'Dedicated') param h100ScaleMode = readEnvironmentVariable('HASTE_SHARED_H100_SCALE_MODE', 'Autoscale') +param h100MinNodes = int(readEnvironmentVariable('HASTE_SHARED_H100_MIN_NODES', '0')) param t4MinNodes = int(readEnvironmentVariable('HASTE_SHARED_T4_MIN_NODES', '0')) param t4MaxNodes = int(readEnvironmentVariable('HASTE_SHARED_T4_MAX_NODES', '2')) +param sharedBatchSubnetId = readEnvironmentVariable('HASTE_SHARED_BATCH_SUBNET_ID', '') param batchAccountName = readEnvironmentVariable('HASTE_EXISTING_BATCH_ACCOUNT', '') param acrName = readEnvironmentVariable('HASTE_SHARED_ACR_NAME', '') param umiResourceId = readEnvironmentVariable('HASTE_SHARED_UMI_ID', '') diff --git a/infra/shared-pools.json b/infra/shared-pools.json index 291f0ca..8f999ab 100644 --- a/infra/shared-pools.json +++ b/infra/shared-pools.json @@ -5,7 +5,7 @@ "_generator": { "name": "bicep", "version": "0.43.8.12551", - "templateHash": "1769188842529029875" + "templateHash": "16520244299666787717" } }, "parameters": { @@ -103,6 +103,20 @@ "metadata": { "description": "H100 reserved node count when h100ScaleMode == Fixed." } + }, + "h100MinNodes": { + "type": "int", + "defaultValue": 0, + "metadata": { + "description": "H100 autoscale floor (when h100ScaleMode == Autoscale). 0 = scale-to-zero; >0 keeps chasing a warm baseline (auto-retries allocation each eval under GPU contention)." + } + }, + "sharedBatchSubnetId": { + "type": "string", + "defaultValue": "", + "metadata": { + "description": "VNet subnet resource id to inject BOTH shared pools into (blob<->batch reach via Microsoft.Storage service endpoint + per-tenant storage VNet rule). Empty => no VNet injection (BatchManaged public IPs). See spec/features/batch-compute-expansion/networking.md." + } } }, "resources": [ @@ -135,7 +149,7 @@ "value": "[parameters('sharedNodeType')]" }, "minNodes": { - "value": 0 + "value": "[parameters('h100MinNodes')]" }, "maxNodes": { "value": "[parameters('h100MaxNodes')]" @@ -151,6 +165,9 @@ }, "imageryprepImage": { "value": "[parameters('imageryprepImage')]" + }, + "subnetId": { + "value": "[parameters('sharedBatchSubnetId')]" } }, "template": { @@ -327,6 +344,9 @@ }, "imageryprepImage": { "value": "[parameters('imageryprepImage')]" + }, + "subnetId": { + "value": "[parameters('sharedBatchSubnetId')]" } }, "template": { diff --git a/spec/features/batch-compute-expansion/README.md b/spec/features/batch-compute-expansion/README.md index 0fc3997..f0a6a2c 100644 --- a/spec/features/batch-compute-expansion/README.md +++ b/spec/features/batch-compute-expansion/README.md @@ -88,6 +88,7 @@ limits and requires mutating a shared, semi-immutable pool per tenant. | [impact-analysis.md](impact-analysis.md) | Risk, dependencies, blast radius, security | approved | | [user-stories.md](user-stories.md) | User stories & acceptance criteria | approved | | [design.md](design.md) | Technical design: topology, isolation, routing, fairness, IaC + app | approved | +| [networking.md](networking.md) | Blob↔Batch network path: VNet injection + per-tenant storage rule | approved | | [test-plan.md](test-plan.md) | Test strategy & coverage matrix | approved | | [rollout.md](rollout.md) | Rollout phases, opt-in flags, rollback, monitoring | approved | | data-model.md | Cosmos/Blob/Data Lake schema changes | n/a — no schema changes | @@ -104,6 +105,7 @@ limits and requires mutating a shared, semi-immutable pool per tenant. | 2026-07-14 | Shared pools deploy into the existing shared framework RG | Where the Batch account + ACR already live; clean logical ownership, no new RG. | | 2026-07-14 | Generic-default IaC (prefix defaults to `haste`, BYO account/ACR) | Reusable by other partners; this deployment overrides via `shared-pools.bicepparam`. | | 2026-07-14 | Pin `azure-batch==14.2.0` | 15.x track-2 rewrite restructures the model API the code uses; migration tracked separately. | +| 2026-07-15 | Blob↔Batch via VNet-injected pools + per-tenant storage VNet rule (not `Allow`/IP band-aid) — see [networking.md](networking.md) | Keeps tenant storage `Deny`; onboarding stays a storage-side self-service step (zero shared-pool change per tenant). One-time pool recreate to migrate the existing subnet-less pools. | ## Remaining gates (before/during build) diff --git a/spec/features/batch-compute-expansion/networking.md b/spec/features/batch-compute-expansion/networking.md new file mode 100644 index 0000000..93f00ba --- /dev/null +++ b/spec/features/batch-compute-expansion/networking.md @@ -0,0 +1,95 @@ +# Design: Blob ↔ Batch network path for shared multi-tenant pools + +**Status:** approved +**Date:** 2026-07-15 + +## Problem + +Data isolation on the shared pools is enforced by per-job user-delegation SAS +(see [design.md](design.md)). SAS is an **auth** boundary — it does not grant +**network reach**. Each tenant storage account is `defaultAction: Deny`, so a +Batch node must also be *network-allowed* to touch that account, or every blob +read/write fails with `403 AuthorizationFailure` (network denial, not a bad SAS). + +The shared `*-shared-demo-*` pools were created with **no VNet injection**, so +their nodes egress from unpredictable BatchManaged public IPs that no tenant +firewall allowlists → blob↔Batch is broken for every shared-pool tenant. + +## The model (proven by prod) + +Prod's single-tenant pool already uses the durable pattern: + +1. A subnet (`batch-subnet`) with the **`Microsoft.Storage` service endpoint**, + no delegation. +2. The Batch pool is **VNet-injected** into it (BatchService allocation mode: + the "Microsoft Azure Batch" SP needs subnet-join on the vnet). +3. The storage account has a **VNet rule allowlisting that subnet** — nodes reach + blobs over the service endpoint; `Deny` still blocks everything else. + +Confirmed live on `ai4glhasteprodsa`: `virtualNetworkRules` = func-subnet + +default + batch-subnet (all `Succeeded`), backing the VNet-injected prod pool. + +## Multi-tenant adaptation + +The shared pools live in the shared account/vnet; **each tenant's** storage +allowlists the **one shared batch subnet**. Storage VNet rules work **cross-vnet +with no peering** as long as the subnet has the `Microsoft.Storage` service +endpoint — so the pool binds to the subnet once, and tenants attach from their +own storage side. + +``` + haste-dev-vnet / batch-subnet (10.0.2.0/24, Microsoft.Storage SE) + │ (pools VNet-injected here, once) + ┌────────────────────────────────┼────────────────────────────────┐ + ai4gl-shared-demo-t4-pool ai4gl-shared-demo-h100-pool + │ node egress via service endpoint │ + ▼ ▼ + demo5 storage (VNet rule: batch-subnet) demo6 storage (VNet rule: batch-subnet) ... demoN +``` + +### Why this scales (the key property) + +- **Pool → subnet is set once and is immutable.** It never changes as tenants + come and go. +- **Onboarding demoN = one VNet rule on demoN's own storage** referencing the + shared batch-subnet. The pool is never touched. This is the storage-side + "adding a tenant needs zero pool/compute changes" principle from + [README.md](README.md#non-negotiable-constraint-data-isolation-on-shared-compute), + extended to the network layer. +- The per-tenant rule is **baked into the per-env storage Bicep**, so a new + env's normal `azd up` allowlists itself — hands-free, no shared-pool change. + +The subnet immutability is a **one-time setup cost** (migrate the existing +subnet-less pools in), never a recurring per-tenant blocker. + +## IaC changes + +| File | Change | +|---|---| +| `infra/shared-pools.bicep` + `.bicepparam` | New `sharedBatchSubnetId` param → passed to both pools' `subnetId` (env: `HASTE_SHARED_BATCH_SUBNET_ID`). Also `h100MinNodes` (env: `HASTE_SHARED_H100_MIN_NODES`) to make the H100 autoscale floor reproducible. | +| `infra/modules/batchPool.bicep` | Already supports `subnetId` (emits `networkConfiguration` + BatchManaged public IPs when set). No change. | +| per-env storage module | Add a `virtualNetworkRules` entry for the shared batch-subnet so each tenant self-allowlists on `azd up`. | + +## Concrete values (this deployment) + +- Shared subnet: `haste-dev-vnet/batch-subnet` = `10.0.2.0/24`, service endpoint + `Microsoft.Storage`, no delegation (mirrors prod's 10.0.2.0/24). +- Batch SP: `Microsoft Azure Batch` (`appId ddbf3205-c6bd-46ae-8127-60eb93363864`), + Network Contributor on the subnet (subnet-join). +- Tenant storage rule: `ai4glhastedemo5sa` → VNet rule for the shared batch-subnet. + +## One-time migration (disruptive step) + +The existing `ai4gl-shared-demo-{t4,h100}-pool` were created without a subnet; +subnet is immutable, so they are **deleted and recreated** into `batch-subnet`. +Safe when no demo Batch jobs are running. For the H100 this is ~free (it holds 0 +nodes under westus2 contention); the T4 briefly drops its idle baseline and +re-acquires. Recreation is via `shared-pools.bicep` with the full config +preserved (T4 autoscale min 2/max 6; H100 autoscale min 1/max 2) plus the subnet. + +## Decision + +Durable VNet-injection + per-tenant storage VNet rule (service-endpoint, +cross-vnet, no peering) over the band-aid (storage `Allow` / rotating-IP +allowlist). Rationale: keeps `Deny` defense-in-depth, and onboarding stays a +storage-side self-service step consistent with the shared-pool isolation model.