Add support for latest-generation Google Cloud machine families

[sosoihd]

Add support for latest-generation Google Cloud machine families #6614

Problem

1. Missing support for latest-generation machine families

Google Cloud has introduced several new general-purpose machine families that are not currently supported by Nextflow:

• C4 series: 4th gen Intel Xeon Scalable (Sapphire Rapids) - C4 machine series
• C4A series: AMD EPYC Genoa processors - C4A machine series
• C4D series: AMD EPYC Genoa with local SSD - C4D machine series
• N4 series: 4th gen Intel Xeon Scalable (Sapphire Rapids) - N4 machine series
• N4D series: AMD EPYC Genoa processors - N4D machine series
• N4A series: AMD EPYC Genoa processors - N4A machine series

These families offer significant improvements in:

• Price-performance ratio (up to 10% better than previous generations)
• Memory bandwidth
• Network throughput
• Energy efficiency

Without this support, users cannot leverage:

• Latest Intel Sapphire Rapids processors (C4, N4)
• Latest AMD EPYC Genoa processors (C4A, C4D, N4A, N4D)
• Improved performance characteristics of these newer families

2. Inability to specify boot disk type

Currently, Nextflow only allows configuring the boot disk size via google.batch.bootDiskSize, but not the disk type. This creates several issues:

Compatibility problems:

• The new C4, C4A, C4D, N4, N4A, and N4D families do not support pd-balanced disks (the Google Cloud default)
• These families require alternative disk types like hyperdisk-balanced or pd-ssd
• This makes it impossible to use these new machine families at all

Performance optimization:

• High-I/O workloads may benefit from pd-ssd (higher IOPS)
• Cost-sensitive workflows may prefer pd-standard (lower cost)
• Users cannot optimize disk performance for their specific workloads

Reference: Google Cloud Disk Types Documentation

Solution

This PR addresses both issues with a comprehensive solution:

1. Add support for latest-generation machine families

Machine type recognition:

• Added C4, C4A, C4D, N4, N4A, N4D families
• Updated local SSD handling for C4/C4A/C4D families with -lssd suffix

Testing:

• Added comprehensive tests for all new machine families
• Verified local SSD behavior (supported/not supported) per family

Changes

File	Change
GoogleBatchMachineTypeSelector.groovy	Add HYPERDISK_ONLY_FAMILIES list, isHyperdiskOnly() method, C4/C4A/C4D -lssd handling
GoogleBatchTaskHandler.groovy	Auto-set hyperdisk-balanced boot disk for Hyperdisk-only families
GoogleBatchMachineTypeSelectorTest.groovy	Tests for new family selection, isHyperdiskOnly(), C4/C4A/C4D local SSD
GoogleBatchTaskHandlerTest.groovy	Tests for Hyperdisk boot disk auto-configuration and standard family non-application

Tests

• GoogleBatchMachineTypeSelectorTest — verifies isHyperdiskOnly() returns true for C4/C4A/C4D/N4/N4A/N4D, false for N1/N2/E2/C2
• GoogleBatchMachineTypeSelectorTest — verifies C4/C4A/C4D -lssd types return MemoryUnit(0) (pre-attached, not configurable)
• GoogleBatchMachineTypeSelectorTest — verifies new families are selectable when explicitly requested
• GoogleBatchTaskHandlerTest — verifies boot disk type is set to hyperdisk-balanced for C4 family
• GoogleBatchTaskHandlerTest — verifies boot disk type is not set for standard families (N2)

References

• Google Cloud C4 machine series
• Google Cloud N4 machine series
• Google Cloud Disk Types
• Disk Type Compatibility Matrix
• Google Batch VM Configuration

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	81f6cd8
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/69b96395442d4700081fc61d

[jorgee]

I have added support for Fusion by modifying the management of local-ssd when Fusion is enabled. I have prioritized the local-ssd but if the user specifies a instance that do not allow local SSD, it tries hyperdisk-balanced except for e2 families that is the one that only allows pd-*, it selects pd-balanced.

[sosoihd]

I have added support for Fusion by modifying the management of local-ssd when Fusion is enabled. I have prioritized the local-ssd but if the user specifies a instance that do not allow local SSD, it tries hyperdisk-balanced except for e2 families that is the one that only allows pd-*, it selects pd-balanced.

Thanks a lot for your help on this, @jorgee! Really appreciate

One thing to flag: hyperdisk-balanced disks are not compatible with all machine families. For example, N1, N2, and N2D do not support

Hyperdisk types (see https://docs.cloud.google.com/compute/docs/disks/hd-types/hyperdisk-balanced?hl=en). So the fallback to hyperdisk-balanced when local SSD is not available might fail for those families.

However, N2 and N2D do support local SSDs (see https://docs.cloud.google.com/compute/docs/disks/local-ssd?hl=en#machine-series-lssd), so the local SSD path should work fine for them.

[jorgee]

One thing to flag: hyperdisk-balanced disks are not compatible with all machine families. For example, N1, N2, and N2D do not support

Hyperdisk types (see https://docs.cloud.google.com/compute/docs/disks/hd-types/hyperdisk-balanced?hl=en). So the fallback to hyperdisk-balanced when local SSD is not available might fail for those families.

However, N2 and N2D do support local SSDs (see https://docs.cloud.google.com/compute/docs/disks/local-ssd?hl=en#machine-series-lssd), so the local SSD path should work fine for them.

As far as I saw, the only family that do not support either hyperdisk or local-ssd is E2. But this is only required for the extra disk required when Fusion is enabled.
As summary, there are two cases:
-Boot disk: will default to pd-balanced (default) except for hyperdisk only families that will hyperdisk-balanced, as you did.
-Fusion disk: it will assign local-ssd for all families where ssd is supported. From the list of not supported local ssd, only E2 do not support hyperdisk. Moreover, It seems hyperdisk will be the default for future instances. Thats the reason why I fallback to hyperdisk.

[sosoihd]

One thing to flag: hyperdisk-balanced disks are not compatible with all machine families. For example, N1, N2, and N2D do not support
Hyperdisk types (see https://docs.cloud.google.com/compute/docs/disks/hd-types/hyperdisk-balanced?hl=en). So the fallback to hyperdisk-balanced when local SSD is not available might fail for those families.
However, N2 and N2D do support local SSDs (see https://docs.cloud.google.com/compute/docs/disks/local-ssd?hl=en#machine-series-lssd), so the local SSD path should work fine for them.

As far as I saw, the only family that do not support either hyperdisk or local-ssd is E2. But this is only required for the extra disk required when Fusion is enabled. As summary, there are two cases: -Boot disk: will default to pd-balanced (default) except for hyperdisk only families that will hyperdisk-balanced, as you did. -Fusion disk: it will assign local-ssd for all families where ssd is supported. From the list of not supported local ssd, only E2 do not support hyperdisk. Moreover, It seems hyperdisk will be the default for future instances. Thats the reason why I fallback to hyperdisk.

Agreed, that makes sense. With this approach we should be covering all scenarios. Thanks for the detailed breakdown!

[jorgee]

Just removed a duplicated bootDiskImage management. It is now treated after machineType selection.

[jorgee]

@bentsherman tested with new instances and Fusion setting hyperdisk when local-ssd is not available. It is ready to merge from my side

[pditommaso]

Just a couple of minor comments