Skip to content

Slurm HPC Integration

Jump to bottom
Paul Rigor edited this page Jun 25, 2026 · 1 revision

Slurm HPC Integration

Background

ADEPT's original HPC integration was the hpc_mcp_server (src/agentic_framework_pkg/hpc_mcp_server/), a built-in MCP server designed for workshops and training. It provided Nextflow workflow execution, video transcription, and repository analysis (GitXray) -- useful for demonstrating the MCP tool pattern but not connected to real HPC schedulers.

The Slurm HPC MCP server (examples/slurm_hpc/slurm_mcp_server/) supersedes it for production HPC workloads. It connects directly to institutional Slurm clusters via SSH, submits real batch jobs, and manages session-scoped workspaces on cluster filesystems. This is ADEPT's primary mechanism for integrating AI agents with supercomputing resources.

Architecture

The Slurm HPC MCP server is a standalone FastMCP server that registers as an external tool stack with the ADEPT gateway. It supports two transport modes:

Transport Use Case Authentication
SSH (primary) Production clusters (Deception, Tahoma) ED25519 key-based via Fabric
REST Local Slurm clusters with slurmrestd JWT tokens (HMAC-signed) 
ADEPT Orchestration Service
    |
    v  (MCP JSON-RPC over HTTP/SSE)
Slurm MCP Server (FastMCP, port 8002)
    |
    +-- SSH Transport --> Deception (deception.pnl.gov)
    +-- SSH Transport --> Tahoma (tahoma.pnl.gov)
    +-- REST Transport --> Local slurmrestd (port 6820)

The MCP server is decoupled from any specific backend -- it connects to any Slurm cluster via configuration.

Tools (27 registered)

Cluster-Specific Job Tools

Tool Description
deception_submit_job Submit a batch job to Deception via SSH + sbatch
deception_get_job_details Query job status via sacct/squeue on Deception
deception_cancel_job Cancel a running or pending job on Deception
deception_list_jobs List jobs in the Deception queue
deception_ping Test SSH connectivity and authentication
deception_get_cluster_summary Full cluster overview with partition info
tahoma_submit_job Submit a batch job to Tahoma via SSH
tahoma_get_job_details Query job status on Tahoma
tahoma_cancel_job Cancel a job on Tahoma
tahoma_list_jobs List jobs in the Tahoma queue
tahoma_ping Test Tahoma connectivity
tahoma_get_cluster_summary Tahoma cluster overview

REST API Tools (for local slurmrestd)

Tool Description
submit_slurm_job Submit via slurmrestd REST API
get_job_details Query job via REST
cancel_slurm_job Cancel via REST
list_cluster_jobs List queue via REST
ping Test REST connectivity
get_cluster_summary Cluster overview via REST

Workspace Management Tools

Tool Description
workspace_list_files List files in a session workspace on any cluster
workspace_upload_file Upload a file from ADEPT to a remote workspace
workspace_download_file Download a file from a remote workspace
workspace_read_file Read file contents from a remote workspace

SSH Key Management Tools

Tool Description
hpc_generate_ssh_key Generate ED25519 key pair for cluster access
hpc_list_ssh_keys List available SSH keys
hpc_get_public_key Retrieve public key for authorized_keys setup

Session-Scoped Workspaces

Every job submission is associated with an MCP session, creating isolated workspace directories on the cluster:

$HOME/adept-workspace/
    {session_id}/           <-- per-session isolation
        scripts/            <-- generated batch scripts
        output/             <-- job stdout/stderr (slurm-*.out)
        artifacts/          <-- intermediate results
    {session_id_2}/         <-- another session (same user)

When a session ID is passed to deception_submit_job, the script is persisted in the workspace and stdout/stderr are redirected there. The agent can then read results with workspace_read_file.

Execution Modes

v1: Single-Agent Job Submission

A single ADEPT agent directly submits and monitors jobs:

User: "Run a COBRApy FBA optimization on Deception"
Agent:
  1. deception_ping() -- verify connectivity
  2. workspace_upload_file() -- send input model
  3. deception_submit_job() -- submit SIF-based script
  4. deception_get_job_details() -- poll until COMPLETED
  5. workspace_read_file() -- retrieve results

v2: Multi-Agent with LLM-as-Judge

A team of specialized agents collaborates on complex HPC workflows:

  • Planner agent: Decomposes the scientific task into Slurm jobs
  • HPC agent: Handles job submission and monitoring
  • Validator agent (LLM-as-Judge): Reviews job outputs for correctness
  • Reporter agent: Synthesizes results into a final report

This mode uses ADEPT's multi-agent orchestration (CreateMultiAgentSession) with RolePersona assignments.

SIF Container Execution

For reproducible scientific workloads, jobs execute inside Apptainer (Singularity) containers:

#!/bin/bash
apptainer exec /path/to/container.sif python3 -c "
import cobra
model = cobra.io.load_json_model('e_coli_core.json')
solution = model.optimize()
print(f'Objective: {solution.objective_value:.4f}')
print(f'Status: {solution.status}')
"

SIF images are provisioned to the cluster workspace by the agent using workspace_upload_file, making the full container lifecycle agent-driven.

COBRApy/Gurobi FBA Validation

A key demonstration workflow uses the Slurm HPC server for constraint-based metabolic modeling:

  1. Agent uploads a COBRA metabolic model (JSON format) to the workspace
  2. Agent submits a job that runs COBRApy inside an Apptainer SIF container
  3. The SIF contains COBRApy + Gurobi solver (optimized for FBA)
  4. Job output is read back via workspace_read_file
  5. In v2 mode, an LLM-as-Judge validates the FBA solution (objective value, flux balance)

Quick Start

cd examples/slurm_hpc

# Full setup: build images, start cluster, bootstrap JWT, register tools
make setup

# Verify connectivity
make validate-all

# Register tools with ADEPT gateway (requires core stack running)
make register-tools

Configuration

Key environment variables for cluster connectivity:

Variable Default Description
DECEPTION_HOST deception.pnl.gov SSH hostname
DECEPTION_USERNAME (from identity mapper) Slurm account
DECEPTION_SSH_KEY_PATH Auto-resolved per user ED25519 private key path
DECEPTION_TRANSPORT_MODE ssh Transport: ssh, rest, or auto
SLURM_API_URL http://slurmrestd:6820 REST API endpoint (local mode)
SLURM_JWT_KEY (from bootstrap) HMAC key for REST JWT auth

Security

  • SSH keys are never exposed to the LLM -- key material stays inside the Fabric transport layer
  • Session isolation prevents cross-user data access on the cluster
  • Identity mapping links Keycloak user identity to HPC cluster accounts
  • SPIFFE/SPIRE federation (planned) for zero-trust cross-domain authentication

Testing

cd examples/slurm_hpc

# Offline tests (no cluster needed)
make test-all          # lint + imports + unit

# Integration (requires MCP server running)
make test-integration

# E2E (requires full stack + ADEPT core)
make test-e2e

Related Resources

Home

Getting Started

Architecture

Deployment

User Guides

Developer Tools

CI/CD

Testing

Contributing

Reference

Clone this wiki locally