Support for Google Cloud new generation machine families (C3, C4, N4) and Hyperdisk

[sosoihd]

New feature

Add support for Google Cloud's new generation machine families:

• C3, C3D
• C4, C4D, C4A
• N4, N4D, N4A

These machine families have specific disk requirements. They only support the following disk types for both boot and mounted disks:

• hyperdisk-balanced
• hyperdisk-balanced-high-availability
• hyperdisk-throughput

Note: C3 and C3D families still support the older disk types (pd-balanced, pd-ssd, pd-standard), but the other new families do not.

Use case

Users want to leverage the improved performance and cost-efficiency of Google Cloud's newest machine generations (C3, C4, N4 series) for their Nextflow pipelines. Currently, Nextflow's Google Cloud Batch integration does not automatically handle the specific disk type requirements (Hyperdisk) for these families, preventing their use or requiring manual and error-prone configuration.

Suggested implementation

• Update GoogleBatchMachineTypeSelector or equivalent logic to recognize these new machine families.
• When a user selects one of these machine types, automatically default to a supported Hyperdisk type (e.g., hyperdisk-balanced) if no specific disk type is provided.
• Validate that the requested disk type is compatible with the selected machine family to prevent launch failures.
• Ensure hyperdisk-* types are recognized as valid options in the disk directive and config.

References:

• Google Cloud General Purpose Machines

[LinearParadox]

I was going to open this exact issue, commenting for visibility!

Can you possibly suggest some configuration options to make this work? I've figured out how to make the non ssd instances work with the disk directive, but it would be nice to leverage the ssd speed. Is there a config option to select the boot disk type? Or do I have to make a template

[bentsherman]

You can set the boot disk size and the boot disk image:

https://nextflow.io/docs/latest/reference/config.html#google

I don't know of a way to set the boot disk type, unless the image implies a particular type (I don't think so).

I'm reminded of this issue -- #4534 -- which offers to support disk image for the disk directive

[sosoihd]

Hi @LinearParadox and @bentsherman ,

I actually already opened a PR to support this: #6616

To clarify, the issue isn't related to the boot disk image, but rather the disk type. The newer Google Cloud machine families (C4, C4A, C4D, N4, N4A, N4D) do not support the legacy disk types like pd-ssd or pd-balanced (the Google Cloud default). These instances strictly require the use of Hyperdisks, both for the boot disk and any additional disks.

I have manually built the plugin with the changes from my PR and it is currently running in production. We have successfully executed over 20,000 jobs with this configuration so far.

[jorgee]

I don't know if something has changed in GCP recently but I was able to run with 25.10.4 setting process.machineType = 'n4-*'. in nextflow.config

It is still failing when enabling fusion because it set a local-ssd disk, and some types of these families do not support it. There is a workaround in this case, that is defining a disk with the supported type in the nextflow.config. Something like process.disk=[request:'100GB', type:'hyperdisk-balanced']