v1.98.0

What's Changed

Key New Features 🎉

• feat: Integrate Managed Lustre Dynamic Tier by @parulbajaj01 in #5791
• feat(slurm): Refactor Slurm config generation for multi-version support and enable AsyncRpc for 25.11+ by @AdarshK15 in #5680
• integrate cluster health check into toolkit. by @leiyiz in #5917
• Update A4high blueprints to use pre-built ACI image by @ksaishree in #5932
• feat(job): Enable MTC for Pathways Workloads by @Neelabh94 in #5948
• feat(slurm): support expedited requeue and start-only health checks for Slurm 25.11 by @AdarshK15 in #5934
• feat(job): Support headless Pathways job submission by @Neelabh94 in #5953
• feat(job): add job inspect command for easy diagnostics by @Neelabh94 in #5925
• Add GKE TPU v5e example by @agrawalkhushi18 in #5840
• add nvlink checks to cluster health check daemon by @leiyiz in #5942
• Update A4x blueprint to use pre-built ACI images by @ksaishree in #5933

Module Improvements 🔨

• Feat(gke): optimize daily maintenance window and exclusion start times by @SwarnaBharathiMantena in #5910
• Fix(gke-cluster): use system_node_pool_machine_type for initial default node pool by @SwarnaBharathiMantena in #5908
• feat(slurm): add health check and autohealing to HA controller MIGs by @andybubu in #5827
• Allow specifying mode and owner for mountpoint by @wiktorn in #3613
• Enhance GKE DRANET support for multi-node pool configuration by @agrawalkhushi18 in #5954

Improvements 🛠

• Fixing A3H Issue by @LAVEEN in #5821
• Add HPL benchmark for H4D clusters. by @bk202 in #5377
• fix(job telemetry): remove deprecated jobtelemetry files by @Neelabh94 in #5915
• Update A3ultra blueprint to use pre-built ACI images by @ksaishree in #5786
• Fix(gke): Support shared reservations in job submission by @SwarnaBharathiMantena in #5826
• Update GKE A4X-Max BM example with DRANET config by @agrawalkhushi18 in #5939
• [Telemetry] Extend static node count tracking to support complex mappings and zone multipliers by @kadupoornima in #5952
• feat(job): add smart main-only logs retrieval for JobSets by @Neelabh94 in #5928
• [Telemetry] Add support for Storage Type metric collection by @kadupoornima in #5955
• Add Phase 3 Internal Load Balancer and active-passive health check agent for Slurm HA by @arpit974 in #5959
• [Telemetry] Add support dynamic min/max node count metrics by @kadupoornima in #5961
• Add nccl rdma installer in gke a4x max by @parulbajaj01 in #5962
• fix(slurm-gke): Resolve PSA collisions and GKE nodeset port mismatch by @shubpal07 in #5949
• [Telemetry] Add standalone storage module detection for file-systems by @kadupoornima in #5960
• [Telemetry] Add support for a new AI-Assisted metric by @kadupoornima in #5978

Version Updates ⏫

• Pin DCGM packages to 1:4.6.0-1 for a3mega by @saumyasharma03 in #5930

Bug fixes 🐞

• Use static VPC network names for Managed Lustre integration tests by @sriram-quad in #5893
• fix(destroy): recover when GKE is unreachable by @Neelabh94 in #5897
• Fix(Telemetry): remove extra double quotes from machine_type in static_node_count metric by @kadupoornima in #5947
• fix(Telemetry): Ensure telemetry initialization runs across all subcommands (e.g. cluster, job) by @kadupoornima in #5907
• Pinning Rocky linux 9 version for managed lustre compatibility by @shubpal07 in #5943
• fix(gke): support tpu7x as a valid NAP accelerator label by @Neelabh94 in #5956
• Simplify topology validation for NAP enabled clusters by @Neelabh94 in #5964
• fix: GKE Pathways Kueue auto-configuration and simplify job orchestrator logic by @agrawalkhushi18 in #5965

New Contributors

• @bk202 made their first contribution in #5377
• @leiyiz made their first contribution in #5917
• @arpitjain099 made their first contribution in #5657
• @shivam222 made their first contribution in #5977

Full Changelog: v1.97.0...v1.98.0