feat(madengine): give the manifest a contract and multi-node runs an honest verdict

.gitignore

@@ -131,6 +131,10 @@ venv/
 docker/
 scripts/
 *.json
+# ... except the schemas madengine ships and loads at runtime. The blanket rule above is
+# for the model JSONs that land in a dev checkout; these are package data, and hatchling
+# picks files by VCS status, so an ignored schema would also be missing from the wheel.
+!src/madengine/schemas/*.json
 .*_env/
 .vscode/
 

docs/cli-reference.md

@@ -522,9 +522,24 @@ madengine uses standard exit codes so scripts and CI (e.g. Jenkins) can detect s
 | `2` | `BUILD_FAILURE` | One or more image builds failed (e.g. Docker build error) |
 | `3` | `RUN_FAILURE` | One or more model executions failed |
 | `4` | `INVALID_ARGS` | Invalid command-line arguments or configuration |
+| `5` | `NO_METRIC` | The workload ran to completion but reported no performance metric |
 
 **Failure recording:** Pre-run failures (e.g. image pull, setup) and run failures are recorded in the performance table (`perf.csv`) with status `FAILURE`, so all attempted models appear in the CSV. The file is created automatically if missing.
 
+**`NO_METRIC` versus `RUN_FAILURE`:** a crashed workload and a workload that finished
+without producing a number are different problems — the first is a broken run, the second a
+broken contract between the model script and madengine — so they get different exit codes
+and the row in `perf.csv` reads `NO_METRIC` rather than `FAILURE`.
+
+**Multi-node verdicts:** every node records its exit code, and the submit node reads them
+all back. A metric outweighs an exit code, the same way it does for a single node: where a
+framework reports throughput from one rank only, the nodes that collected nothing exit
+non-zero on a perfectly healthy run, so a run that produced a metric is reported as a
+success with the node outcomes listed as warnings. With no metric anywhere, the node
+evidence is all there is and it decides: a node that exited non-zero or never reported at
+all gives `RUN_FAILURE` naming the node, while a clean set of exit codes and no metric
+gives `NO_METRIC`.
+
 **Example usage in scripts / CI:**
 
 ```bash

docs/configuration.md

@@ -252,6 +252,11 @@ This allows you to rebuild only changed models while maintaining references to e
 
 ### Environment Variables
 
+Host-side variables (`MODEL_DIR`, cache roots, `MAD_DOCKER_BUILDS`) can be collected in a
+shell env file that the manifest names via `deployment_config.env_file`; see
+[Environment file](deployment.md#environment-file-env_file). The variables below are the
+container's, and are set from the run configuration.
+
 Pass environment variables to containers:
 
 ```json

docs/deployment.md

@@ -252,6 +252,74 @@ The deployment target is automatically detected from the `slurm` key in the conf
 
 See [examples/slurm-configs/](../examples/slurm-configs/) for complete examples.
 
+### Environment file (`env_file`)
+
+A cluster run usually depends on a handful of host variables that say where things live —
+the model directory, the cache roots, the shared image store. Instead of requiring every
+operator to `source mad.env` in the shell that launches madengine, name the file in the
+build manifest and madengine loads it itself, on the submit node and on every worker:
+
+```json
+{
+  "deployment_config": {
+    "target": "slurm",
+    "env_file": "mad.env"
+  }
+}
+```
+
+A relative path is resolved against the manifest's directory, so a run directory holding
+the manifest and its `mad.env` side by side stays movable. The file is sourced with
+`bash`, exactly as `source mad.env` would, so `${VAR:-default}`, command substitution and
+conditionals all work — and, like the manifest that names it, the file is trusted input.
+Values in the file override what madengine inherited from the launching shell. madengine
+logs the names of the variables it applied, never their values, so an env file may carry
+tokens.
+
+The field is optional. Without it madengine behaves exactly as it did before, and nothing
+here requires `MAD_DOCKER_BUILDS` or any other variable to be set. What is fatal is naming
+a file that is not there: that stops the run at startup rather than leaving a variable to
+resolve to the empty string mid-run.
+
+### Shared image store (`MAD_DOCKER_BUILDS`)
+
+For a multi-node run every worker needs the same image. Set `MAD_DOCKER_BUILDS` to a
+directory on shared storage (usually from the `env_file` above) and madengine saves the
+built image there once, then loads it from the tar on workers whose local image ID differs:
+
+```bash
+export MAD_DOCKER_BUILDS=/shared/mad/docker_builds
+```
+
+Leaving it unset is supported but leaves image distribution to the operator: workers that
+do not already have the image cannot reconcile it, and the run fails on those nodes with a
+message saying so.
+
+### How a multi-node run is judged
+
+Every node writes its exit code to `<results_dir>/<model>/<job_id>/node_<rank>/node.status`
+before it copies anything else, so the outcome survives even when the artifacts do not. The
+submit node reads them all back, and weighs them the way a single-node run does — a metric
+outweighs an exit code:
+
+| What came back | Verdict |
+| --- | --- |
+| A metric, all nodes zero | success |
+| A metric, some node non-zero or silent | success, with the node outcomes as warnings |
+| No metric, some node non-zero or silent | `RUN_FAILURE`, naming the node |
+| No metric, all nodes zero | `NO_METRIC` |
+
+The warning row is not a technicality. Where a framework reports throughput from one rank
+only — Primus/Megatron reports from the last global rank — every other node finds no metric
+locally and exits non-zero on a completely healthy run. A verdict that trusted exit codes
+over results would fail every such run, so the exit codes are reported and the numbers are
+kept.
+
+For a workload where every rank really is expected to exit zero, set
+`slurm.kill_on_bad_exit` to tear the step down on the first bad exit instead of letting the
+survivors block on a peer that will never answer. It is off by default for the reason above:
+the node exiting non-zero may be the one whose peer holds the numbers.
+
 ### Multi-Node Training
 
 For distributed training across SLURM nodes:

pyproject.toml

@@ -25,6 +25,7 @@ dependencies = [
   "rich>=13.0.0",
   "click>=8.0.0",
   "jinja2>=3.0.0",
+  "jsonschema>=4.0.0",
   "pyyaml>=6.0",
   "kubernetes>=28.0.0",
   "pytest>=7.0",

src/madengine/cli/commands/run.py

@@ -280,16 +280,35 @@ def run(
             save_summary_with_feedback(execution_summary, summary_output, "Execution")
 
             failed_runs = len(execution_summary.get("failed_runs", []))
-            if failed_runs == 0:
+            no_metric_runs = len(execution_summary.get("no_metric_runs", []))
+            incomplete = execution_summary.get("incomplete")
+            for warning in execution_summary.get("warnings", []):
+                console.print(f"⚠️  [yellow]{warning}[/yellow]")
+            if incomplete:
                 console.print(
-                    "🎉 [bold green]All model executions completed successfully![/bold green]"
+                    f"💥 [bold red]Run did not complete and produced no metric: "
+                    f"{incomplete.get('reason', 'unknown reason')}[/bold red]"
                 )
-                raise typer.Exit(ExitCode.SUCCESS)
-            else:
+                raise typer.Exit(ExitCode.RUN_FAILURE)
+            if failed_runs:
                 console.print(
-                    f"💥 [bold red]Execution failed for {failed_runs} models[/bold red]"
+                    f"💥 [bold red]Execution failed for {failed_runs} "
+                    f"{'run' if failed_runs == 1 else 'runs'}[/bold red]"
                 )
                 raise typer.Exit(ExitCode.RUN_FAILURE)
+            if no_metric_runs:
+                # The workload ran to completion and produced nothing to measure. That is
+                # not a successful benchmark, and it is not the same failure as a crash:
+                # what broke is the contract between the model script and madengine.
+                console.print(
+                    f"📉 [bold yellow]{no_metric_runs} models ran but reported no "
+                    f"performance metric[/bold yellow]"
+                )
+                raise typer.Exit(ExitCode.NO_METRIC)
+            console.print(
+                "🎉 [bold green]All model executions completed successfully![/bold green]"
+            )
+            raise typer.Exit(ExitCode.SUCCESS)
 
         else:
             # MAD_CONTAINER_IMAGE handling is done in RunOrchestrator

src/madengine/cli/constants.py

@@ -17,6 +17,9 @@ class ExitCode(IntEnum):
     BUILD_FAILURE = 2
     RUN_FAILURE = 3
     INVALID_ARGS = 4
+    #: The workload finished but produced no performance metric. Distinct from
+    #: RUN_FAILURE so a caller can tell a crashed run from a broken result contract.
+    NO_METRIC = 5
 
 
 # Valid values for validation

src/madengine/core/env_file.py

@@ -0,0 +1,150 @@
+#!/usr/bin/env python3
+"""
+Load a shell env file (the `mad.env` convention) into the run's environment.
+
+Multi-node runs depend on a set of variables that describe *where things live* on the
+cluster: `MODEL_DIR`, the cache roots, and `MAD_DOCKER_BUILDS`. Until now the only way to
+supply them was to `source mad.env` in the same shell before every `madengine run`, and
+forgetting produced failures far from the cause — an empty `MODEL_DIR` makes the run
+script path resolve to nothing, and a `MAD_DOCKER_BUILDS` that is not on shared storage
+makes every worker rebuild the image or fail to find it.
+
+A manifest can now name the file (`deployment_config.env_file`) and madengine loads it
+itself. The file is executed by `bash`, exactly as sourcing it would, so the usual shell
+constructs (`${VAR:-default}`, `$(cat ~/.token)`, conditionals) behave the same — which
+also means an env file is trusted input, on par with the manifest that names it.
+
+Copyright (c) Advanced Micro Devices, Inc. All rights reserved.
+"""
+
+import os
+import shlex
+import subprocess
+from pathlib import Path
+from typing import Dict, Optional
+
+from madengine.core.errors import ErrorContext, ValidationError
+
+#: A shell that hangs (waiting on a prompt, say) must not hang the run.
+SOURCE_TIMEOUT_SECONDS = 120
+
+#: Separates the before/after environment dumps in the helper shell's output. Both dumps
+#: come from the same shell, so anything bash sets on its own (COLUMNS, SHLVL, ...)
+#: appears in both and is not mistaken for something the env file did.
+_BOUNDARY = "__madengine_env_file_boundary__"
+
+#: Bash's own bookkeeping, which differs between the two dumps for reasons unrelated to
+#: the file's contents.
+_SHELL_BOOKKEEPING = frozenset({"_", "SHLVL", "PWD", "OLDPWD"})
+
+
+def load_env_file(env_file: str, base_dir: Optional[str] = None) -> Dict[str, str]:
+    """
+    Source an env file with bash and return the variables it sets or changes.
+
+    Args:
+        env_file: path to the file; relative paths resolve against `base_dir`
+        base_dir: directory to resolve a relative `env_file` against (typically the
+            manifest's directory), defaults to the current working directory
+
+    Returns:
+        Dict[str, str]: variables the file added or changed, relative to the environment
+            madengine is running with
+
+    Raises:
+        ValidationError: the file is missing, or bash failed to source it
+    """
+    path = Path(env_file)
+    if not path.is_absolute() and base_dir:
+        path = Path(base_dir) / path
+
+    context = ErrorContext(
+        operation="env_file loading", component="core.env_file", file_path=str(path)
+    )
+    if not path.is_file():
+        raise ValidationError(
+            f"env_file not found: {path}",
+            context=context,
+            suggestions=[
+                "deployment_config.env_file is resolved relative to the manifest",
+            ],
+        )
+
+    # `set -a` is what makes plain `KEY=value` lines exported, matching what an operator
+    # gets from `source mad.env` in a shell configured the usual way. Sourcing is checked
+    # explicitly: a syntax error makes `.` fail but would otherwise be masked by the
+    # `env` that follows it.
+    script = (
+        f"env -0; printf '%s\\0' {shlex.quote(_BOUNDARY)}; "
+        f"set -a; . {shlex.quote(str(path))} || exit 42; env -0"
+    )
+    try:
+        result = subprocess.run(
+            ["bash", "-c", script],
+            capture_output=True,
+            text=True,
+            timeout=SOURCE_TIMEOUT_SECONDS,
+            check=False,
+        )
+    except subprocess.TimeoutExpired as exc:
+        raise ValidationError(
+            f"Timed out after {SOURCE_TIMEOUT_SECONDS}s sourcing env_file: {path}",
+            context=context,
+            cause=exc,
+        ) from exc
+
+    if result.returncode != 0:
+        detail = result.stderr.strip() or f"bash exited with {result.returncode}"
+        raise ValidationError(
+            f"Failed to source env_file {path}: {detail}",
+            context=context,
+        )
+
+    before_dump, _, after_dump = result.stdout.partition(f"{_BOUNDARY}\0")
+    before = _parse_env_dump(before_dump)
+
+    loaded: Dict[str, str] = {}
+    for key, value in _parse_env_dump(after_dump).items():
+        # A variable that already held this value is not something the file changed.
+        if key in _SHELL_BOOKKEEPING or before.get(key) == value:
+            continue
+        loaded[key] = value
+    return loaded
+
+
+def _parse_env_dump(dump: str) -> Dict[str, str]:
+    """
+    Parse NUL-delimited `env -0` output into a mapping.
+
+    Args:
+        dump: raw `env -0` output
+
+    Returns:
+        Dict[str, str]: the variables in the dump
+    """
+    parsed: Dict[str, str] = {}
+    for entry in dump.split("\0"):
+        if not entry or "=" not in entry:
+            continue
+        key, value = entry.split("=", 1)
+        parsed[key] = value
+    return parsed
+
+
+def apply_env_file(env_file: str, base_dir: Optional[str] = None) -> Dict[str, str]:
+    """
+    Load an env file and apply it to `os.environ`, as sourcing it would.
+
+    The file wins over the inherited environment, so behaviour matches what the operator
+    gets by sourcing it before the run.
+
+    Args:
+        env_file: path to the file; relative paths resolve against `base_dir`
+        base_dir: directory to resolve a relative `env_file` against
+
+    Returns:
+        Dict[str, str]: the variables that were applied
+    """
+    loaded = load_env_file(env_file, base_dir)
+    os.environ.update(loaded)
+    return loaded

src/madengine/deployment/base.py

@@ -19,6 +19,9 @@
 from jinja2 import Environment, FileSystemLoader
 from rich.console import Console
 
+from madengine.core.env_file import apply_env_file
+from madengine.schemas import perf_csv_header, validate_build_manifest
+
 
 # Regex for parsing "performance: <value> <metric>" log lines.
 # Value: optional sign, integer/decimal, scientific notation (e or E).
@@ -122,8 +125,8 @@ def __init__(self, config: DeploymentConfig):
             config: Deployment configuration
         """
         self.config = config
-        self.manifest = self._load_manifest(config.manifest_file)
         self.console = Console()
+        self.manifest = self._load_manifest(config.manifest_file)
 
     def _load_manifest(self, manifest_file: str) -> Dict:
         """
@@ -152,6 +155,20 @@ def _load_manifest(self, manifest_file: str) -> Dict:
         if missing:
             raise ValueError(f"Invalid manifest, missing: {missing}")
 
+        for warning in validate_build_manifest(manifest, source=str(manifest_path)):
+            self.console.print(f"[yellow]⚠ {warning}[/yellow]")
+
+        env_file = manifest.get("deployment_config", {}).get("env_file")
+        if env_file:
+            # The submit side needs these too: MAD_DOCKER_BUILDS and the cache roots are
+            # read while rendering the job script, before any node starts.
+            applied = apply_env_file(env_file, base_dir=str(manifest_path.resolve().parent))
+            # Names only: an env file legitimately carries secrets.
+            self.console.print(
+                f"[cyan]Loaded env_file {env_file}: "
+                f"{', '.join(sorted(applied)) or '(no new variables)'}[/cyan]"
+            )
+
         return manifest
 
     # Template Method - defines workflow
@@ -590,14 +607,7 @@ def _ensure_perf_csv_exists(self) -> None:
         perf_csv_path = Path("perf.csv")
         if perf_csv_path.exists():
             return
-        standard_header = (
-            "model,n_gpus,nnodes,gpus_per_node,training_precision,pipeline,args,tags,"
-            "docker_file,base_docker,docker_sha,docker_image,git_commit,machine_name,"
-            "deployment_type,launcher,gpu_architecture,performance,metric,relative_change,"
-            "status,build_duration,test_duration,dataname,data_provider_type,data_size,"
-            "data_download_duration,build_number,additional_docker_run_options"
-        )
-        perf_csv_path.write_text(standard_header + "\n", encoding="utf-8")
+        perf_csv_path.write_text(perf_csv_header() + "\n", encoding="utf-8")
 
     def _write_to_perf_csv(self, perf_data: Dict[str, Any]) -> None:
         """