Azure Batch worker pool supports managed identity

[adamrtalbot]

This feature adds an option to tell Azure Batch the worker pool has a managed identity attached.

azcopy needs to be updated, since the current version isn't recent enough.

When using this feature, you can tell Nextflow that a node pool has a managed identity attached:

azure {

    storage {
        accountName = 'seqeralabs'
    }

    batch {
        location = 'eastus'
        accountName = 'seqeralabs'
        copyToolInstallMode = "off"
        pools {
            "hello-world-entra-mi" {
                vmType = "standard_e2ds_v5"
                managedIdentityId = "myManagedIdentityId"
            }

        }
    }
}

it will then use the native azcopy managed identity authentication to download and upload files.

I haven't handled all the old code yet, so Nextflow will generate a SAS which it only uses sometimes.

Importantly, this 'fixes' #5669, because it doesn't need a SAS token to access the files (tested).

To use:

1. Create a node pool in Azure Batch and attach a user-assigned managed identity. Follow the instructions here or try this if you want to use Terraform
2. Change the start task to have the following settings to update azcopy:
   • Command line: bash -c "tar -xzvf azcopy.tar.gz && chmod +x azcopy*/azcopy && mkdir -p $AZ_BATCH_NODE_SHARED_DIR/bin/ && cp azcopy*/azcopy $AZ_BATCH_NODE_SHARED_DIR/bin/"
   • Resource file URL: https://aka.ms/downloadazcopy-v10-linux, path azcopy.tar.gz
3. Retrieve the managed identity client ID
4. Use the following Nextflow config, where the pool name is my-pool:

process.queue = 'my-pool'
azure {

    storage {
        accountName = 'seqeralabs'
    }

    batch {
        location = 'eastus'
        accountName = 'seqeralabs'
        copyToolInstallMode = "off"
        pools {
            "my-pool" {
                managedIdentityId = "myManagedIdentityId"
            }

        }
    }
}

6. Launch pipeline and 🤞

To do:

• See if we can apply the managed ID to pools created by Nextflow. This might need an update from Microsoft
• Documentation
• Remove creation of SAS when using managed identity to make everything more secure and allow this to work if we switch off SAS tokens

[netlify]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	29524a4
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/67f92d58ef25550008426ffa
😎 Deploy Preview	https://deploy-preview-5670--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[Takadonet]

Hello,
This new feature will be very helpful for our project to avoid long duration SaS token.
Just wondering if a timeline or we can provide some assistance.

[adamrtalbot]

This breaks Fusion for everyone who isn't using managed identities. Ideally we take the pool.opts when creating the launcher but I've ran out of time.

[pditommaso]

What's the rationale? remove AZURE_STORAGE_SAS_TOKEN when pool.opts.managedIdentityId is provided?

[adamrtalbot]

We were having authentication problems with Fusion using the SAS token instead of the identity, they might be resolved now.

Ideally, we would not generate a SAS token if we already have a managed identity to reduce our security surface further.

[adamrtalbot]

@bentsherman and anyone who can help, I've spent ages trying to work out how to get the poolOpts into AzFusionEnv to disable creating a SAS token here, but I can't see a good way of doing it without doing a full Azure specific implementation of TaskBean, FusionScriptLauncher, etc. Is there a better way?

I wonder if I can create a class that is task + pool, kinda like the AzJobKey then use that in the Fusion env builder.

Anyway, all of this to say a proper solution for Fusion would require a bit of a refactor top-to-bottom and I don't know how far I should go with this before punting off to actual good developers 😆

Using azcopy is fine, it's basically ready to go.

[pditommaso]

These looks unrelated

[adamrtalbot]

This is what makes azcopy work.

The first iteration of this PR did not include Fusion.

[pditommaso]

This should not be affected

[pditommaso]

What's the rationale? remove AZURE_STORAGE_SAS_TOKEN when pool.opts.managedIdentityId is provided?

[adamrtalbot]

Superseded by #6118

[pditommaso]

Closing in favour of #6118