shared-compute implementation to support multiple GPU SKUs

CHANGELOG.md

@@ -7,6 +7,18 @@ Versioning follows the Docker image tags defined in the CI workflows (see [.gith
 
 ---
 
+## [Unreleased]
+
+### Added
+- **Shared multi-tenant GPU Batch pools** — For deployments running many environments against scarce GPU quota, HASTE can now share a small set of multi-tenant Batch pools (H100 for training, T4 for inference/imageryprep + spillover) instead of one pool per environment. Data isolation is enforced at the credential boundary: each job mints a short-lived **user-delegation SAS** scoped to its own storage container (the pool identity is used only for ACR pull and holds no storage access), so a tenant's task can never read another tenant's data. Pools autoscale on low-priority nodes and scale to zero when idle. New `hastelib` routing picks a pool from an ordered candidate list **at submit time** (first with an idle node, else the preferred pool). Provisioned by the standalone [`infra/shared-pools.bicep`](infra/shared-pools.bicep) + [`shared-pools.bicepparam`](infra/shared-pools.bicepparam); opted into per environment via `AZURE_BATCH_*_POOL_IDS`, `AZURE_BATCH_USE_SAS`, and `AZURE_BATCH_MANAGE_POOLS` (all default to the legacy single-pool behavior). Full design in [`spec/features/batch-compute-expansion/`](spec/features/batch-compute-expansion). See [docs/configuration.md](docs/configuration.md#shared-multi-tenant-gpu-pools).
+
+### Changed
+- **`infra/modules/batchPool.bicep` parameterized** — one module now serves fixed-dedicated (dev/prod) and autoscale-low-priority (shared) pools via `scaleMode` / `nodeType` / `minNodes` params, with optional VNet injection. Backward-compatible defaults.
+- **Generic-default IaC for reuse by other partners** — `HASTE_RESOURCE_PREFIX` now defaults to the neutral `haste` (overridable per deployment); the shared-pools template keeps its account/ACR as bring-your-own params. The `api`/`queues` Function App identity is granted **Storage Blob Delegator** (in `functionApp.bicep`) so it can mint user-delegation SAS.
+- **Pinned `azure-batch==14.2.0`** — the 15.x track-2 rewrite restructures the batch models this code uses; migration is tracked separately.
+
+---
+
 ## [v2.0.0] — Building labeling workflow & one-step `azd` setup
 
 ### Added

api/hastefuncapi/requirements.txt

@@ -29,5 +29,5 @@ tenacity==9.1.2
 typing-extensions==4.14.1
 docker==7.1.0
 # Use wheel for remote, comment out for local Docker (source is copied directly)
-hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.26-py3-none-any.whl
+hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.28-py3-none-any.whl
 # -e ../../hastelib # for local development only

api/hastefuncqueues/requirements.txt

@@ -29,5 +29,5 @@ tenacity==9.1.2
 typing-extensions==4.14.1
 docker==7.1.0
 # Use wheel for remote, comment out for local Docker (source is copied directly)
-hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.26-py3-none-any.whl
+hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.28-py3-none-any.whl
 # -e ../../hastelib # for local development only

docker/imageryprep/requirements.txt

@@ -31,4 +31,4 @@ pyarrow==23.0.1
 fiona==1.10.1
 fsspec==2024.10.0
 adlfs==2024.12.0
-hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.26-py3-none-any.whl
+hastegeo @ https://researchlabwuopendata.blob.core.windows.net/haste-binaries/hastegeo-1.0.28-py3-none-any.whl

docs/configuration.md

@@ -14,6 +14,7 @@ This guide documents each configuration mode. For the end-to-end workflow, see
 - [Core settings](#core-settings)
 - [Batch (create vs. bring-your-own)](#batch-create-vs-bring-your-own)
 - [Batch image tags and pool immutability](#batch-image-tags-and-pool-immutability)
+- [Shared multi-tenant GPU pools](#shared-multi-tenant-gpu-pools)
 - [Email sender domain](#email-sender-domain)
 - [Front Door](#front-door)
 - [Development mode](#development-mode)
@@ -24,7 +25,7 @@ This guide documents each configuration mode. For the end-to-end workflow, see
 
 | Variable | Default | Purpose |
 |---|---|---|
-| `HASTE_RESOURCE_PREFIX` | `ai4gl` | Prefix for all resource names. |
+| `HASTE_RESOURCE_PREFIX` | `haste` | Prefix for all resource names. Generic default; override per deployment. |
 | `HASTE_RANDOM_SUFFIX` | `dev1` | Per-environment suffix; keeps names unique. |
 | `AZURE_LOCATION` | `westus2` | Azure region. |
 | `HASTE_APIM_PUBLISHER_EMAIL` | — | APIM publisher email (required). |
@@ -98,6 +99,81 @@ reads the existing pool's `containerImageNames` and sets `HASTE_TRAINING_IMAGE`
 only in that mode and never clobbers a tag you set explicitly — set either
 variable yourself to override the auto-resolved value.
 
+## Shared multi-tenant GPU pools
+
+For deployments that run many environments against **scarce GPU quota**, HASTE
+supports a small set of **shared, multi-tenant** Batch pools (H100 for training,
+T4 for inference/imageryprep + spillover) instead of one pool per environment, so
+GPU quota is pooled and rationed centrally rather than fragmented. See
+[`spec/features/batch-compute-expansion/`](https://github.com/microsoft/haste/tree/main/spec/features/batch-compute-expansion)
+for the full design.
+
+**Creating the pools.** The shared pools are provisioned by
+[`infra/shared-pools.bicep`](https://github.com/microsoft/haste/blob/main/infra/shared-pools.bicep)
+— a standalone deployment into the shared Batch account's resource group,
+separate from `azd up` — configured by
+[`infra/shared-pools.bicepparam`](https://github.com/microsoft/haste/blob/main/infra/shared-pools.bicepparam):
+
+```bash
+az deployment group create -g <shared-rg> \
+  --parameters infra/shared-pools.bicepparam
+```
+
+Pools are named `<prefix>-shared-<group>-<tier>-pool` (e.g. the dev-group H100
+pool); `HASTE_SHARED_GROUP` selects the group (`dev`, `demo`, …). They autoscale
+on **dedicated** nodes within a central per-pool core-quota ceiling. A scarce tier
+(H100) can keep a warm floor instead of scaling to zero — under regional GPU
+contention an autoscale floor (or `Fixed` target) **auto-retries allocation each
+evaluation** until capacity frees, whereas a one-shot fixed resize gets stuck in
+its error state. The pool identity is used **only** for ACR pull
+(`haste-shared-acr-umi`) — it holds no storage access.
+
+Shared-pool deploy knobs (env vars read by `shared-pools.bicep`):
+
+| Variable | Default | Purpose |
+|---|---|---|
+| `HASTE_SHARED_GROUP` | `dev` | Pool group (`dev`, `demo`, …) — drives the pool names. |
+| `HASTE_SHARED_H100_SCALE_MODE` | `Autoscale` | `Autoscale`, or `Fixed` reserved baseline for the scarce H100 tier. |
+| `HASTE_SHARED_H100_MIN_NODES` | `0` | H100 autoscale floor (>0 keeps chasing/holding a warm node). |
+| `HASTE_SHARED_T4_MIN_NODES` / `HASTE_SHARED_T4_MAX_NODES` | `0` / `2` | T4 autoscale floor / cap. |
+| `HASTE_SHARED_BATCH_SUBNET_ID` | — | Shared hub batch-subnet to VNet-inject both pools into (see Blob↔Batch networking below). |
+
+**Data isolation.** Tenants share compute but not data: each job mints a
+short-lived **user-delegation SAS** scoped to its own storage container, so a
+tenant's task can never read another tenant's data. This requires the submitting
+Function App identity to hold **Storage Blob Delegator** on its storage account
+(granted in `functionApp.bicep`).
+
+**Blob↔Batch networking.** SAS is an *auth* boundary, not network reach — each
+tenant storage is `Deny`, so the pools must additionally be *network-allowed* to
+it or blob I/O fails with `403 AuthorizationFailure`. The shared pools are
+VNet-injected into a **shared hub batch-subnet** (`haste-hub-vnet/batch-subnet`,
+carrying the `Microsoft.Storage` service endpoint), and **each tenant's storage
+allowlists that subnet** with a VNet rule. Onboarding a new env is *just that one
+storage rule* — wired in `storage.bicep` via `HASTE_SHARED_BATCH_SUBNET_ID` so it
+lands on `azd provision`; the shared pools are never touched. Two one-time
+prerequisites per hub (not per env): create the subnet, and grant the *Microsoft
+Azure Batch* service principal `Network Contributor` on it so Batch can
+VNet-inject. `HASTE_SHARED_BATCH_SUBNET_ID` is read by **both** the shared-pools
+deploy (which subnet to inject the pools into) and each env deploy (which subnet
+its storage allowlists). Full design:
+[`networking.md`](https://github.com/microsoft/haste/blob/main/spec/features/batch-compute-expansion/networking.md).
+
+**Per-environment app settings** (set on the `api`/`queues` Function Apps to opt
+an environment into the shared pools — all default to the legacy single-pool,
+pool-identity behavior, so existing environments are unaffected until opted in):
+
+| Variable | Default | Purpose |
+|---|---|---|
+| `AZURE_BATCH_TRAINING_POOL_IDS` | `AZURE_BATCH_TRAINING_POOL_ID` | Ordered candidate pools for training (preference-first, spillover-second), comma-separated. |
+| `AZURE_BATCH_INFERENCE_POOL_IDS` | training pool | Ordered candidate pools for inference/embedding (e.g. T4-first). |
+| `AZURE_BATCH_IMAGERYPREP_POOL_IDS` | `AZURE_BATCH_IMAGERYPREP_POOL_ID` | Ordered candidate pools for imageryprep/artifacts. |
+| `AZURE_BATCH_USE_SAS` | `false` | Use per-job user-delegation SAS for blob I/O instead of the pool's managed identity. Required for shared pools. |
+| `AZURE_BATCH_MANAGE_POOLS` | `true` | Whether the runner auto-creates/resizes its pool. Set `false` for pre-created autoscale pools (resize fails on an autoscale pool). |
+
+The runner picks a pool from the candidate list **at submit time** — the first
+with an idle node, otherwise the preferred (first) pool, which scales up / queues.
+
 ## Email sender domain
 
 The email backend (Azure Communication Services) is provisioned in-IaC, so its

docs/deployment.md

@@ -27,7 +27,7 @@ az login
 
 # Create an environment and set the required configuration.
 azd env new dev3
-azd env set HASTE_RESOURCE_PREFIX      ai4gl
+azd env set HASTE_RESOURCE_PREFIX      myorg
 azd env set HASTE_RANDOM_SUFFIX        dev3
 azd env set AZURE_LOCATION             westus2
 azd env set HASTE_APIM_PUBLISHER_EMAIL you@example.com
@@ -66,6 +66,15 @@ brings up the full stack with Docker Compose.
 
 ## Production Considerations
 
+### GPU compute at scale
+
+Running many environments against limited GPU quota? HASTE supports **shared,
+multi-tenant Batch pools** (provisioned separately from `azd up` via
+[`infra/shared-pools.bicep`](https://github.com/microsoft/haste/blob/main/infra/shared-pools.bicep)),
+with per-job user-delegation SAS for tenant data isolation and capacity-aware
+routing. See [Shared multi-tenant GPU pools](configuration.md#shared-multi-tenant-gpu-pools)
+in the configuration guide.
+
 ### Security
 
 For production deployments, follow the [Secure Configuration Guidance](security-configuration.md) — it covers identity and authentication setup, secrets management with managed identity and Key Vault, CORS and HTTP security headers, container hardening, logging and monitoring, and known limitations with operational mitigations. The guide also includes a pre-production checklist.

hastelib/pyproject.toml

@@ -34,7 +34,10 @@ dependencies = [
   "azure-storage-file-datalake",
   "azure-storage-queue",
   "azure-mgmt-containerregistry",
-  "azure-batch",
+  # Pinned: azure-batch 15.x is a track-2 rewrite that restructures the models
+  # this code uses (e.g. ComputeNodeIdentityReference). Migration tracked
+  # separately; keep on the last track-1 release until then.
+  "azure-batch==14.2.0",
   "azure-core",
   "azure-identity",
   "psycopg2-binary",

hastelib/src/hastegeo/__about__.py

@@ -5,4 +5,4 @@
 # storage (see ../../../haste_build.py) so that source/Docker installs report
 # the same version that was published. Committed value tracks the latest
 # published release.
-__version__ = "1.0.26"
+__version__ = "1.0.28"

hastelib/src/hastegeo/core/config.py

@@ -423,6 +423,16 @@ def get_azure_batch_config():
         imageryprep_pool_id = os.getenv(
             "AZURE_BATCH_IMAGERYPREP_POOL_ID", "imageryprep-pool"
         )
+
+        def _split_ids(raw, fallback):
+            # Ordered candidate pool ids for capacity-aware routing (v2.1.0).
+            # Comma-separated env override; fall back to the single legacy id.
+            if raw:
+                ids = [p.strip() for p in raw.split(",") if p.strip()]
+                if ids:
+                    return ids
+            return [fallback]
+
         return {
             "account_name": os.getenv(
                 "AZURE_BATCH_ACCOUNT_NAME", "<batch-account-name>"
@@ -438,6 +448,30 @@ def get_azure_batch_config():
             "imageprep_pool_id": os.getenv(
                 "AZURE_BATCH_IMAGERYPREP_POOL_ID", "imageryprep-pool"
             ),
+            # Ordered candidate pools per workload (v2.1.0 capacity-aware
+            # routing): preference-first, spillover-second
+            # (e.g. AZURE_BATCH_TRAINING_POOL_IDS="h100-pool,t4-pool").
+            "training_pool_ids": _split_ids(
+                os.getenv("AZURE_BATCH_TRAINING_POOL_IDS"), training_pool_id
+            ),
+            "inference_pool_ids": _split_ids(
+                os.getenv("AZURE_BATCH_INFERENCE_POOL_IDS"), training_pool_id
+            ),
+            "imageryprep_pool_ids": _split_ids(
+                os.getenv("AZURE_BATCH_IMAGERYPREP_POOL_IDS"),
+                imageryprep_pool_id,
+            ),
+            # Per-job user-delegation SAS instead of pool-identity for blob I/O
+            # (required for multi-tenant shared pools). Default off = legacy
+            # identity_reference path.
+            "use_sas": os.getenv("AZURE_BATCH_USE_SAS", "false").lower()
+            == "true",
+            # Whether the runner auto-creates/resizes its pool. Off for
+            # pre-created IaC/autoscale pools (resize fails on autoscale).
+            "manage_pools": os.getenv(
+                "AZURE_BATCH_MANAGE_POOLS", "true"
+            ).lower()
+            == "true",
             "registry_server": os.getenv(
                 "AZURE_BATCH_REGISTRY_SERVER",
                 "<registry-name>.azurecr.io",

hastelib/src/hastegeo/core/processors/artifacts.py

@@ -63,6 +63,9 @@ def __init__(
             runner_type=self.config.runner_type,
             config=self.config,
             pool_id=self.config.get_azure_batch_config()["imageprep_pool_id"],
+            candidate_pool_ids=self.config.get_azure_batch_config()[
+                "imageryprep_pool_ids"
+            ],
         )
         self.model_artifacts = model_artifacts
         if self.model_data is not None:

hastelib/src/hastegeo/core/processors/embedding.py

@@ -75,6 +75,9 @@ def __init__(
             runner_type=config.runner_type,
             config=self.config,
             pool_id=self.config.get_azure_batch_config()["training_pool_id"],
+            candidate_pool_ids=self.config.get_azure_batch_config()[
+                "training_pool_ids"
+            ],
         )
         self.queue_client = AzureQueueHandler(
             config.queue_config["queue_connection_string"],

hastelib/src/hastegeo/core/processors/imagery.py

@@ -181,6 +181,9 @@ def __init__(self, image_data: ImageLayer, config: Config = None):
             runner_type=config.runner_type,
             config=self.config,
             pool_id=self.config.get_azure_batch_config()["imageprep_pool_id"],
+            candidate_pool_ids=self.config.get_azure_batch_config()[
+                "imageryprep_pool_ids"
+            ],
         )
         self.queue = AzureQueueHandler(
             config.queue_config["queue_connection_string"],

hastelib/src/hastegeo/core/processors/inference.py

@@ -45,6 +45,9 @@ def __init__(
             runner_type=config.runner_type,
             config=self.config,
             pool_id=self.config.get_azure_batch_config()["training_pool_id"],
+            candidate_pool_ids=self.config.get_azure_batch_config()[
+                "inference_pool_ids"
+            ],
         )
 
 

hastelib/src/hastegeo/core/processors/train.py

@@ -51,6 +51,9 @@ def __init__(
             runner_type=config.runner_type,
             config=self.config,
             pool_id=self.config.get_azure_batch_config()["training_pool_id"],
+            candidate_pool_ids=self.config.get_azure_batch_config()[
+                "training_pool_ids"
+            ],
         )