Skip to content

Releases: OLC-LOC-Bioinformatics/AzureBatch

v0.0.33

Choose a tag to compare

@adamkoziol adamkoziol released this 24 Sep 19:07

AzureBatch v0.0.33

This release fixes Azure Batch task submissions that could be incorrectly reported as failures when the Azure SDK returned a successful task status together with an empty error object.

Fixed

  • Normalize Azure Batch task-add status values across string and enum representations.
  • Recognize success, succeeded, and enum-style success statuses correctly.
  • Ignore empty error models attached to otherwise successful task-add results.
  • Preserve meaningful task-level errors returned by Azure Batch.
  • Prevent successful submissions from triggering automatic pool and job rollback.
  • Return clearer Batch API failure information when task submission genuinely fails.
    Improved
  • Log each task-add result, including task ID, status, normalized status value, and error object.
  • Log submission exceptions with their complete traceback.
  • Log job and pool deletion when submission rollback is performed.
  • Report whether cleanup was performed in failed worker responses.
  • Preserve configurable output containers and destination prefixes.
  • Preserve Nanopore input and output SAS environment variables.
  • Preserve Nanopore runtime path, micromamba, GPU VM, and Trusted Launch configuration.

v0.0.32

Choose a tag to compare

@adamkoziol adamkoziol released this 24 Sep 15:57

AzureBatch v0.0.32 improves support for long-running, manifest-driven
Nanopore workflows.

Key changes include scoped input and output SAS credentials, separate result
containers, run-specific output prefixes, Azure SDK task-result validation,
and best-effort cleanup after partial submission failures.

Worker-mode orchestration now returns plain dictionaries rather than Flask
response objects. Existing workflows remain compatible through default output
container and prefix values.

Focused Azure CLI and Nanopore tests: 61 passed.

v0.0.31

Choose a tag to compare

@adamkoziol adamkoziol released this 04 Sep 15:55

AzureBatch Runtime and Trusted Launch Configuration

Added

  • Added analysis-specific Batch VM-size defaults.
  • Added configurable task runtime paths through FOODPORT_RUNTIME_PATH.
  • Added Nanopore micromamba runtime support.
  • Added MAMBA_ROOT_PREFIX to Nanopore Batch tasks.
  • Added support for placing the PoreSippR environment, micromamba, and Dorado on the Batch task PATH.

Fixed

  • Restored Trusted Launch for COWBAT, AmpliSeq, and COWSNPhR pools created from Trusted Launch-only Compute Gallery images.
  • Preserved the Nanopore Trusted Launch configuration with Secure Boot and vTPM disabled.
  • Prevented the CLI default CPU VM size from overriding the configured Nanopore A10 GPU size.
  • Updated Batch task submission to pass runtime configuration into each task.

Compatibility

  • Existing Miniconda workflows continue to receive:
    • CONDA=/usr/bin/miniconda/bin
    • FOODPORT_RUNTIME_PATH=/usr/bin/miniconda/bin
  • Nanopore workflows receive:
    • CONDA=/opt/micromamba/bin
    • MAMBA_ROOT_PREFIX=/opt/micromamba/root
    • the PoreSippR micromamba environment and Dorado directories on PATH

Testing

  • Expanded unit-test coverage for Trusted Launch, UEFI overrides, VM-size selection, legacy Conda tasks, Nanopore micromamba tasks, and CLI orchestration.
  • All 68 tests pass.

v0.0.30

Choose a tag to compare

@adamkoziol adamkoziol released this 04 Sep 14:24

AzureBatch

Fixed

  • Restored Trusted Launch for all Azure Batch pool types.
  • Fixed allocation failures affecting COWBAT, AmpliSeq, and COWSNPhR pools created from Trusted Launch-only Compute Gallery images.
  • Preserved the Nanopore pool configuration using Trusted Launch with Secure Boot and vTPM disabled for NVIDIA GRID driver compatibility.
  • Expanded unit-test coverage for common and Nanopore-specific security profiles.

v0.0.29

Choose a tag to compare

@adamkoziol adamkoziol released this 02 Sep 15:57

v0.0.29

This patch release adds explicit Azure Batch UEFI security configuration for Nanopore GPU pools.

Added

  • Added support for configuring Secure Boot separately for Nanopore Batch nodes.
  • Added support for configuring the virtual Trusted Platform Module separately for Nanopore Batch nodes.
  • Added strict parsing for boolean settings supplied as strings, booleans, or numeric-style values.
  • Added validation for invalid boolean configuration values.
  • Added comprehensive tests for Nanopore UEFI defaults, explicit overrides, and generated Azure Batch security profiles.

Changed

Nanopore Batch pools continue to use the Trusted Launch security type while explicitly disabling Secure Boot and vTPM by default.

The resulting security configuration is:

Security type: trustedLaunch
Secure Boot: false
vTPM: false

This configuration supports the Trusted Launch-compatible Generation 2 image definition while preparing the Nanopore pool for the NVIDIA GRID driver required by the NVadsA10_v5 VM family.

Configuration

The following settings are now supported:

NANOPORE_SECURITY_TYPE=trustedLaunch
NANOPORE_SECURE_BOOT_ENABLED=false
NANOPORE_VTPM_ENABLED=false

If the Secure Boot and vTPM settings are omitted for a Nanopore analysis, both default to false.

Existing analysis types continue to omit an explicit security profile unless one is configured.

Boolean values

The new boolean parser accepts the following case-insensitive values:

True values:

true
1
yes
on

False values:

false
0
no
off

Native Python boolean values are also accepted.

Unsupported values produce a clear configuration error instead of being interpreted implicitly.

Azure Batch integration

Nanopore pool creation now generates an Azure Batch security profile equivalent to:

security_type = trustedLaunch
secure_boot_enabled = false
v_tpm_enabled = false

The implementation uses the UefiSettings model provided by the pinned azure-batch 14.2.0 package.

Existing COWBAT, AmpliSeq, and COWSNPhR pool behavior remains unchanged.

Tests

The test suite now covers:

  • boolean values supplied as native booleans
  • lowercase and uppercase string values
  • yes and no values
  • numeric-style values
  • on and off values
  • default values
  • invalid configuration values
  • Nanopore Secure Boot defaults
  • Nanopore vTPM defaults
  • explicit UEFI overrides
  • Trusted Launch security-profile construction
  • compatibility with existing non-Nanopore workflows
  • Nanopore VM-size defaults and explicit overrides

Test result:

61 passed

Compatibility

  • No public AzureBatch API changes are included.
  • No changes are required for COWBAT, AmpliSeq, or COWSNPhR configurations.
  • Existing non-Nanopore pools continue to omit an explicit security profile.
  • The Conda package continues to support Python 3.12 and Python 3.13.
  • Automated package builds and tests use Python 3.12.
  • The implementation remains compatible with azure-batch 14.2.0.

Deployment notes

After publishing the updated Conda package:

  1. Update the Batch container image to install azure_batch=0.0.29.
  2. Add the new Nanopore UEFI settings to the FoodPort configuration.
  3. Rebuild and restart the Batch service container.
  4. Confirm that version 0.0.29 is installed in the container.
  5. Continue with the Nanopore image 0.0.2 NVIDIA GRID layer.
  6. Validate the completed image on a Standard_NV18ads_A10_v5 Azure Batch node.

Upgrade notes

Add the following values to the settings supplied to AzureBatch:

NANOPORE_SECURITY_TYPE=trustedLaunch
NANOPORE_SECURE_BOOT_ENABLED=false
NANOPORE_VTPM_ENABLED=false

No other application configuration changes are required when upgrading from v0.0.28.

v0.0.28

Choose a tag to compare

@adamkoziol adamkoziol released this 02 Sep 14:27

v0.0.28

This patch release fixes local Conda channel indexing during post-build package validation.

Fixed

  • Fixed the post-build installation test failing because the conda-index command was unavailable in the build environment.
  • Added conda-index as an explicitly pinned build-tool dependency.
  • Updated local channel indexing to use the documented Python module entry point.
  • Ensured that the newly built AzureBatch package is added to the local channel metadata before installation testing.

Changed

The automated release environment now installs:

Python 3.12
conda-build 25.1.1
conda-index 0.12.1
anaconda-client 1.14.1

Local channel metadata is now generated using:

python -m conda_index <local-channel-root>

The workflow continues to derive the local channel root from the generated package path.

For a generated package such as:

/usr/share/miniconda/envs/azure_batch_build/conda-bld/noarch/azure_batch-0.0.28-py_0.conda

the local channel root is:

/usr/share/miniconda/envs/azure_batch_build/conda-bld

and the channel URL is:

file:///usr/share/miniconda/envs/azure_batch_build/conda-bld

Validation

The release workflow now performs the following post-build checks:

  1. Confirms that the expected .conda package exists.
  2. Displays the package path and SHA-256 checksum.
  3. Determines the local package directory and channel root.
  4. Displays available package and channel metadata files.
  5. Regenerates local channel metadata using conda-index.
  6. Confirms that the requested AzureBatch version is visible through conda search.
  7. Creates a clean Python 3.12 test environment.
  8. Installs the locally built package with full dependency resolution.
  9. Confirms that the azure_batch module can be imported.
  10. Runs AzureBatch -h.
  11. Displays the installed AzureBatch package record.

Channel configuration

The post-build installation test uses the following channel order:

explicit local build channel
conda-forge
bioconda
olcbioinformatics

The locally built AzureBatch package is selected from the explicit file channel, while its runtime dependencies are resolved from the configured remote channels.

Network resilience

The workflow retains timeout and retry handling for transient failures involving:

  • Conda build-tool installation
  • recipe rendering
  • package output-path calculation
  • package building and recipe tests
  • dependency resolution
  • local package installation
  • source archive downloads
  • package publication

Conda index caches are cleared between applicable retry attempts.

Test status

The complete AzureBatch source test suite passes using Python 3.12:

43 passed

The Conda recipe also validates:

  • package dependency resolution
  • wheel generation
  • Conda package creation
  • package installation
  • Python module imports
  • command-line help
  • packaged unit tests

Compatibility

  • No AzureBatch API changes are included in this release.
  • No Azure Batch pool, job, or task behavior is changed.
  • No Azure Storage runtime behavior is changed.
  • No deployment configuration changes are required.
  • Existing AzureBatch workflows remain compatible.
  • The Conda package continues to support Python 3.12 and Python 3.13.
  • Automated package builds and installation tests use Python 3.12.

Upgrade notes

No application configuration changes are required when upgrading from v0.0.27.

This release supersedes v0.0.27 for automated Conda publication. Version 0.0.27 included explicit local-channel testing, but the build environment did not include the separate conda-index package required to regenerate that channel's metadata.

v0.0.27

Choose a tag to compare

@adamkoziol adamkoziol released this 02 Sep 14:12

v0.0.27

This patch release includes the corrected Conda publication workflow for testing installation of the newly built package.

Fixed

  • Fixed post-build installation tests failing because the generated AzureBatch package could not be found through Conda's generic local channel.
  • Replaced implicit --use-local package discovery with an explicit local channel derived from the generated package path.
  • Ensured the corrected publication workflow is included in the release tag.

Changed

The workflow now derives the local Conda channel from PACKAGE_PATH.

For a generated package such as:

/usr/share/miniconda/envs/azure_batch_build/conda-bld/noarch/azure_batch-0.0.27-py_0.conda

the workflow uses the following channel root:

/usr/share/miniconda/envs/azure_batch_build/conda-bld

and supplies it to Conda as:

file:///usr/share/miniconda/envs/azure_batch_build/conda-bld

Validation

The publication workflow now:

  1. Confirms that the expected .conda package exists.
  2. Determines the package directory and local channel root.
  3. Rebuilds the local channel metadata with conda-index.
  4. Confirms that the requested AzureBatch version is visible through conda search.
  5. Creates a clean Python 3.12 test environment.
  6. Installs the locally built package with full dependency resolution.
  7. Confirms that the installed package can be imported.
  8. Runs AzureBatch -h to validate the installed command-line interface.
  9. Displays the installed AzureBatch package record.

Dependency resolution

The post-build installation test uses the following channel order:

explicit local build channel
conda-forge
bioconda
olcbioinformatics

The AzureBatch package is selected from the explicit local channel, while its runtime dependencies are resolved from the configured remote channels.

Test status

The complete source test suite passes using Python 3.12:

43 passed

The Conda recipe also successfully:

  • resolves its build and test environments
  • builds the AzureBatch wheel
  • creates the .conda package
  • installs the package into the recipe test environment
  • imports the azure_batch module
  • runs the AzureBatch command-line help
  • executes all packaged unit tests

Network resilience

The workflow retains retry handling for transient failures involving:

  • Conda build-tool installation
  • recipe rendering
  • package output-path calculation
  • package building and recipe tests
  • dependency resolution
  • local installation testing
  • source archive downloads
  • package publication

Conda index caches are cleared between applicable retry attempts.

Compatibility

  • No AzureBatch API changes are included in this release.
  • No Azure Batch pool, job, or task behavior is changed.
  • No Azure Storage runtime behavior is changed.
  • No deployment configuration changes are required.
  • Existing AzureBatch workflows remain compatible.
  • The Conda package continues to support Python 3.12 and Python 3.13.
  • Automated package builds and installation tests use Python 3.12.

Upgrade notes

No application configuration changes are required when upgrading from v0.0.26.

This release supersedes v0.0.26 for Conda publication. The previous release tag did not contain the committed explicit local-channel workflow changes.

v0.0.26

Choose a tag to compare

@adamkoziol adamkoziol released this 02 Sep 13:43

v0.0.26

This patch release fixes the Conda publication workflow's post-build installation test.

Fixed

  • Fixed local package installation failing with PackagesNotFoundInChannelsError.
  • Fixed the workflow relying on --use-local, which searched Conda's default local build channel rather than the build directory used by the isolated azure_batch_build environment.
  • Updated the workflow to derive the actual local Conda channel from the generated package path.
  • Ensured the newly built azure_batch package is discoverable before the installation test begins.

Changed

  • The workflow now determines the local channel root from PACKAGE_PATH.
  • The local build channel is provided to Conda using an explicit file:// URL.
  • Channel metadata is refreshed with conda-index before package installation.
  • The workflow verifies that the requested AzureBatch version is visible in the local channel using conda search.
  • The installation test continues to resolve all runtime dependencies rather than installing the package archive with dependency checking disabled.
  • Added diagnostic output showing:
    • the generated package path
    • the local package directory
    • the local channel root
    • the local channel URL
    • package and channel metadata files
    • the installed AzureBatch package record

Local channel resolution

For a generated package such as:

/usr/share/miniconda/envs/azure_batch_build/conda-bld/noarch/azure_batch-0.0.26-py_0.conda

the workflow now uses the following local channel root:

/usr/share/miniconda/envs/azure_batch_build/conda-bld

and supplies it to Conda as:

file:///usr/share/miniconda/envs/azure_batch_build/conda-bld

This allows Conda to discover the package under the channel's noarch directory while resolving its dependencies from the configured remote channels.

Dependency resolution

The installation test uses the following channel order:

local build channel
conda-forge
bioconda
olcbioinformatics

The locally built AzureBatch package is selected from the explicit local channel, while its runtime dependencies continue to be resolved normally.

Validation

The workflow now validates the locally built package by:

  1. Confirming that the expected .conda package exists.
  2. Refreshing the local channel metadata.
  3. Confirming that azure_batch=0.0.26 is visible in the local channel.
  4. Creating a clean Python 3.12 test environment.
  5. Installing the locally built package with full dependency resolution.
  6. Importing the azure_batch Python package.
  7. Running AzureBatch -h.
  8. Displaying the installed package record.

Network resilience

Existing retry handling remains enabled for transient failures involving:

  • Conda build-tool installation
  • recipe rendering
  • package output-path calculation
  • package building
  • dependency resolution
  • installation testing
  • source archive downloads
  • package publication

The local package visibility check now fails immediately with useful diagnostics if the generated package is absent from the expected build channel.

Compatibility

  • No AzureBatch application programming interface changes are included.
  • No Azure Batch pool, job, or task behavior is changed.
  • No Azure Storage runtime behavior is changed.
  • No deployment configuration changes are required.
  • Existing AzureBatch workflows remain compatible.
  • The Conda package continues to support Python 3.12 and Python 3.13.
  • Automated package builds and installation tests use Python 3.12.

Upgrade notes

No application configuration changes are required when upgrading from v0.0.25.

This release supersedes v0.0.25 for automated Conda publication. Version 0.0.25 built and passed its recipe tests successfully, but its post-build installation check could not locate the generated package through Conda's generic local channel.

v0.0.25

Choose a tag to compare

@adamkoziol adamkoziol released this 01 Sep 22:53

v0.0.25

This patch release ensures that the corrected credential-independent Azure Storage tests are included in the published source archive and Conda package validation.

Fixed

  • Fixed the published release source containing the earlier credential-dependent test_prep_output_container implementation.
  • Replaced real container SAS signing in the packaged test suite with a mocked generate_container_sas call.
  • Removed the output-container test's dependency on AZURE_ACCOUNT_NAME and AZURE_ACCOUNT_KEY environment variables.
  • Prevented isolated Conda package tests from failing when Azure Storage credentials are unavailable.

Added

  • Added assertions confirming that the expected Blob Storage container is created.
  • Added assertions confirming that SAS generation receives the configured account name, account key, and container name.
  • Added assertions confirming that read and write SAS permissions are requested.
  • Added validation that a SAS expiry time is supplied.
  • Added validation of the final container SAS URL.
  • Added validation that existing non-Nanopore pools omit an explicit security profile.
  • Added validation that Batch pools use one task slot per node.

Test results

The complete test suite passes using Python 3.12:

43 passed

The updated unit tests do not require:

  • live Azure resources
  • Azure Storage credentials
  • network connectivity
  • a local environment file

Compatibility

  • No AzureBatch API changes are included.
  • No Azure Storage runtime behavior is changed.
  • No Azure Batch pool, job, or task behavior is changed.
  • No deployment configuration changes are required.
  • Existing AzureBatch workflows remain compatible.
  • The Conda package continues to support Python 3.12 and Python 3.13.
  • Automated Conda package builds and tests use Python 3.12.

Upgrade notes

No application configuration changes are required when upgrading from v0.0.24.

This release supersedes v0.0.24, whose release tag contained the earlier credential-dependent Azure Storage test.

v0.0.24

Choose a tag to compare

@adamkoziol adamkoziol released this 01 Sep 19:54

v0.0.24

This patch release fixes the Azure Storage unit tests used during Conda package validation.

Fixed

  • Fixed test_prep_output_container failing in isolated Conda test environments when Azure Storage credentials were unavailable.
  • Removed the test's dependency on AZURE_ACCOUNT_NAME and AZURE_ACCOUNT_KEY environment variables.
  • Replaced real SAS token generation with a mocked generate_container_sas call.
  • Updated the legacy pool test to provide the explicit security-profile and VM-size attributes expected by the current pool creation implementation.

Added

  • Added assertions confirming that output-container creation requests the expected container name.
  • Added assertions confirming that SAS generation receives the configured storage account name and account key.
  • Added assertions confirming that the generated SAS token requests read and write permissions.
  • Added validation that an expiry time is supplied when generating the container SAS token.
  • Added validation of the final Blob Storage container SAS URL.
  • Added coverage confirming that existing non-Nanopore workflows omit the VM security profile.
  • Added coverage confirming that Batch pools use one task slot per node.

Changed

  • Azure Storage unit tests now use explicit non-secret test values rather than local environment files or CI credentials.
  • test_prep_output_container now tests application behavior without invoking Azure SDK token-signing logic.
  • The existing create_pool test now reflects configuration-driven security profiles and GPU task-slot behavior.

Test results

The updated test suite validates:

  • output-container creation
  • SAS token argument construction
  • read and write SAS permissions
  • SAS token expiry configuration
  • container SAS URL formatting
  • explicit Batch VM security profiles
  • compatibility with existing workflows
  • one task slot per Batch node

The expected Conda package test result is:

43 passed

Compatibility

  • No AzureBatch application programming interface changes are included in this release.
  • No Azure Storage runtime behavior is changed.
  • No Azure Batch pool, job, or task behavior is changed.
  • No deployment configuration changes are required.
  • Existing AzureBatch workflows remain compatible.
  • The Conda package continues to support Python 3.12 and Python 3.13.
  • Automated package builds and tests use Python 3.12.

Upgrade notes

No application configuration changes are required when upgrading from v0.0.23.

This release is recommended because it removes the final dependency on live Azure Storage credentials from the Conda package unit tests.