v1.98.0

[siddhartha-quad]

What's Changed

Key New Features 🎉

• feat: Integrate Managed Lustre Dynamic Tier by @parulbajaj01 in feat: Integrate Managed Lustre Dynamic Tier #5791
• feat(slurm): Refactor Slurm config generation for multi-version support and enable AsyncRpc for 25.11+ by @AdarshK15 in feat(slurm): Refactor Slurm config generation for multi-version support and enable AsyncRpc for 25.11+ #5680
• integrate cluster health check into toolkit. by @leiyiz in integrate cluster health check into toolkit. #5917
• Update A4high blueprints to use pre-built ACI image by @ksaishree in Update A4high blueprints to use pre-built ACI image #5932
• feat(job): Enable MTC for Pathways Workloads by @Neelabh94 in feat(job): Enable MTC for Pathways Workloads #5948
• feat(slurm): support expedited requeue and start-only health checks for Slurm 25.11 by @AdarshK15 in feat(slurm): support expedited requeue and start-only health checks for Slurm 25.11 #5934
• feat(job): Support headless Pathways job submission by @Neelabh94 in feat(job): Support headless Pathways job submission #5953
• feat(job): add job inspect command for easy diagnostics by @Neelabh94 in feat(job): add job inspect command for easy diagnostics #5925
• Add GKE TPU v5e example by @agrawalkhushi18 in Add GKE TPU v5e example #5840
• add nvlink checks to cluster health check daemon by @leiyiz in add nvlink checks to cluster health check daemon #5942
• Update A4x blueprint to use pre-built ACI images by @ksaishree in Update A4x blueprint to use pre-built ACI images #5933

Module Improvements 🔨

• Feat(gke): optimize daily maintenance window and exclusion start times by @SwarnaBharathiMantena in Feat(gke): optimize daily maintenance window and exclusion start times #5910
• Fix(gke-cluster): use system_node_pool_machine_type for initial default node pool by @SwarnaBharathiMantena in Fix(gke-cluster): use system_node_pool_machine_type for initial default node pool #5908
• feat(slurm): add health check and autohealing to HA controller MIGs by @andybubu in feat(slurm): add health check and autohealing to HA controller MIGs #5827
• Allow specifying mode and owner for mountpoint by @wiktorn in Allow specifying mode and owner for mountpoint #3613
• Enhance GKE DRANET support for multi-node pool configuration by @agrawalkhushi18 in Enhance GKE DRANET support for multi-node pool configuration  #5954

Improvements 🛠

• Fixing A3H Issue by @LAVEEN in Fixing A3H Issue #5821
• Add HPL benchmark for H4D clusters. by @bk202 in Add HPL benchmark for H4D clusters. #5377
• fix(job telemetry): remove deprecated jobtelemetry files by @Neelabh94 in fix(job telemetry): remove deprecated jobtelemetry files #5915
• Update A3ultra blueprint to use pre-built ACI images by @ksaishree in Update A3ultra blueprint to use pre-built ACI images  #5786
• Fix(gke): Support shared reservations in job submission by @SwarnaBharathiMantena in Fix(gke): Support shared reservations in job submission #5826
• Update GKE A4X-Max BM example with DRANET config by @agrawalkhushi18 in Update GKE A4X-Max BM example with DRANET config #5939
• [Telemetry] Extend static node count tracking to support complex mappings and zone multipliers by @kadupoornima in [Telemetry] Extend static node count tracking to support complex mappings and zone multipliers #5952
• updated to support TCPXO version v1.0.17 by @saumyasharma03 in updated to support TCPXO version v1.0.17 #5944
• feat(job): add smart main-only logs retrieval for JobSets by @Neelabh94 in feat(job): add smart main-only logs retrieval for JobSets #5928
• [Telemetry] Add support for Storage Type metric collection by @kadupoornima in [Telemetry] Add support for Storage Type metric collection #5955
• Add Phase 3 Internal Load Balancer and active-passive health check agent for Slurm HA by @arpit974 in Add Phase 3 Internal Load Balancer and active-passive health check agent for Slurm HA #5959
• [Telemetry] Add support dynamic min/max node count metrics by @kadupoornima in [Telemetry] Add support dynamic min/max node count metrics #5961
• Add nccl rdma installer in gke a4x max by @parulbajaj01 in Add nccl rdma installer in gke a4x max #5962
• fix(slurm-gke): Resolve PSA collisions and GKE nodeset port mismatch by @shubpal07 in fix(slurm-gke): Resolve PSA collisions and GKE nodeset port mismatch #5949
• [Telemetry] Add standalone storage module detection for file-systems by @kadupoornima in [Telemetry] Add standalone storage module detection for file-systems #5960
• [Telemetry] Add support for a new AI-Assisted metric by @kadupoornima in [Telemetry] Add support for a new AI-Assisted metric #5978

Version Updates ⏫

• Pin DCGM packages to 1:4.6.0-1 for a3mega by @saumyasharma03 in Pin DCGM packages to 1:4.6.0-1 for a3mega #5930

Bug fixes 🐞

• Use static VPC network names for Managed Lustre integration tests by @sriram-quad in Use static VPC network names for Managed Lustre integration tests #5893
• fix(destroy): recover when GKE is unreachable by @Neelabh94 in fix(destroy): recover when GKE is unreachable  #5897
• Fix(Telemetry): remove extra double quotes from machine_type in static_node_count metric by @kadupoornima in Fix(Telemetry): remove extra double quotes from machine_type in static_node_count metric #5947
• fix(Telemetry): Ensure telemetry initialization runs across all subcommands (e.g. cluster, job) by @kadupoornima in fix(Telemetry): Ensure telemetry initialization runs across all subcommands (e.g. cluster, job) #5907
• Pinning Rocky linux 9 version for managed lustre compatibility by @shubpal07 in Pinning Rocky linux 9 version for managed lustre compatibility #5943
• fix(gke): support tpu7x as a valid NAP accelerator label by @Neelabh94 in fix(gke): support tpu7x as a valid NAP accelerator label #5956
• Simplify topology validation for NAP enabled clusters by @Neelabh94 in Simplify topology validation for NAP enabled clusters #5964
• fix: GKE Pathways Kueue auto-configuration and simplify job orchestrator logic by @agrawalkhushi18 in fix: GKE Pathways Kueue auto-configuration and simplify job orchestrator logic #5965

New Contributors

• @bk202 made their first contribution in Add HPL benchmark for H4D clusters. #5377
• @leiyiz made their first contribution in integrate cluster health check into toolkit. #5917
• @arpitjain099 made their first contribution in ci: declare contents: read on PR description-check and precommit workflows #5657
• @shivam222 made their first contribution in Add shivam222 to toolkit-writers.json #5977

Full Changelog: v1.97.0...v1.98.0

---

This discussion was created from the release v1.98.0.