Provision, execute, and monitor batch and HPC container workloads on Azure Batch
Latest commit 784393a Sep 20, 2018
Permalink
Failed to load latest commit information.
.github Fix nvidia-docker2 installation Jul 20, 2018
.vsts Migrate to VSTS for builds and deployments Sep 20, 2018
cargo Update Singularity and Alpine Sep 20, 2018
cascade Update Singularity and Alpine Sep 20, 2018
config_templates Federation support Aug 6, 2018
contrib Add CentOS-HPC 7.4 Support Jun 7, 2018
convoy Tag for 3.6.0b1 release Sep 20, 2018
docker Update Singularity and Alpine Sep 20, 2018
docs Migrate to VSTS for builds and deployments Sep 20, 2018
federation Update Singularity and Alpine Sep 20, 2018
heimdall Update Singularity and Alpine Sep 20, 2018
recipes Fix Docker Hub private registry login Jul 27, 2018
schemas Federation support Aug 6, 2018
scripts Update Docker to 18.06.1 Sep 20, 2018
site-extension Tag for 3.5.0b2 release Jun 12, 2018
.gitattributes Add AppVeyor build Aug 10, 2017
.gitignore Allow CentOS 7.3 on NC/NV Jul 6, 2017
.travis.yml Migrate to VSTS for builds and deployments Sep 20, 2018
CHANGELOG.md Tag for 3.6.0b1 release Sep 20, 2018
CODE_OF_CONDUCT.md Update docs Aug 29, 2017
CONTRIBUTING.md Update docs Aug 29, 2017
LICENSE Add dummy README Jul 18, 2016
README.md Migrate to VSTS for builds and deployments Sep 20, 2018
THIRD_PARTY_NOTICES.txt Tag for 3.5.0 release Jun 29, 2018
appveyor.yml Migrate to VSTS for builds and deployments Sep 20, 2018
install.cmd Improve site extension installation robustness May 2, 2018
install.sh Fix shellcheck errors Sep 18, 2018
mkdocs.yml Federation support Aug 6, 2018
req_nodeps.txt Update dependencies Mar 19, 2018
requirements.txt Update to Azure Batch 5.1.0 SDK Sep 18, 2018
shipyard.py Add count commands Aug 9, 2018

README.md

Build Status Build Status Build status Docker Pulls Image Layers

Batch Shipyard

dashboard

Batch Shipyard is a tool to help provision, execute, and monitor container-based batch processing and HPC workloads on Azure Batch. Batch Shipyard supports both Docker and Singularity containers! No experience with the Azure Batch SDK is needed; run your containers with easy-to-understand configuration files. All Azure regions are supported, including non-public Azure regions.

Additionally, Batch Shipyard provides the ability to provision and manage entire standalone remote file systems (storage clusters) in Azure, independent of any integrated Azure Batch functionality.

Major Features

  • Automated Docker Host Engine and Singularity installations tuned for Azure Batch compute nodes
  • Automated deployment of required Docker and/or Singularity images to compute nodes
  • Mixed mode support for Docker and Singularity: run your Docker and Singularity containers within the same job, side-by-side or even concurrently
  • Comprehensive data movement support: move data easily between locally accessible storage systems, remote filesystems, Azure Blob or File Storage, and compute nodes
  • Support for serverless execution binding with Azure Functions
  • Federation support: enables unified, constraint-based scheduling to collections of heterogeneous pools, including across multiple Batch accounts and Azure regions
  • Automated, integrated resource monitoring with Prometheus and Grafana for Batch pools and RemoteFS storage clusters
  • Standalone Remote Filesystem Provisioning with integration to auto-link these filesystems to compute nodes with support for NFS and GlusterFS distributed network file system
  • Automatic shared data volume support for linking to Remote Filesystems as provisioned by Batch Shipyard, Azure File via SMB, Azure Blob via blobfuse, GlusterFS provisioned directly on compute nodes (which can act as a distributed local file system/cache), and custom Linux mount support (fstab)
  • Support for simple, scenario-based pool autoscale and autopool to dynamically scale and control computing resources on-demand
  • Support for Task Factories with the ability to generate tasks based on parametric (parameter) sweeps, randomized input, file enumeration, replication, and custom Python code-based generators
  • Transparent support for GPU-accelerated container applications on both Docker and Singularity on Azure N-Series VM instances
  • Support for multi-instance tasks to accommodate MPI and multi-node cluster applications packaged as Docker or Singularity containers on compute pools with automatic job completion and task termination
  • Transparent assist for running Docker and Singularity containers utilizing Infiniband/RDMA for MPI on HPC low-latency Azure VM instances including A-Series, H-Series, and N-Series
  • Seamless integration with Azure Batch job, task and file concepts along with full pass-through of the Azure Batch API to containers executed on compute nodes
  • Support for Azure Batch task dependencies allowing complex processing pipelines and DAGs
  • Support for merge or final task specification that automatically depends on all other tasks within the job
  • Support for job schedules and recurrences for automatic execution of tasks at set intervals
  • Support for live job and job schedule migration between pools
  • Support for credential management through Azure KeyVault
  • Support for Docker Registries including Azure Container Registry, other Internet-accessible public and private registries, and support for the Singularity Hub Container Registry
  • Support for Low Priority Compute Nodes
  • Support for deploying Batch compute nodes into a specified Virtual Network
  • Automatic setup of SSH or RDP users to all nodes in the compute pool and optional creation of SSH tunneling scripts to Docker Hosts on compute nodes
  • Support for custom host images
  • Support for Windows Containers on compliant Windows compute node pools with the ability to activate Azure Hybrid Use Benefit if applicable
  • Accelerated Docker and Singularity image deployment at scale to compute pools consisting of a large number of VMs via private peer-to-peer distribution of container images among the compute nodes

Installation

Azure Cloud Shell

Batch Shipyard is integrated directly into Azure Cloud Shell and you can execute any Batch Shipyard workload using your web browser or the Microsoft Azure Android and iOS app.

Simply request a Cloud Shell session and type shipyard to invoke the CLI; no installation is required. Try Batch Shipyard now from your browser: Launch Cloud Shell

Local Installation

Please see the installation guide for more information regarding the various local installation options and requirements.

Documentation and Recipes

Please refer to the Batch Shipyard Documentation on Read the Docs.

Visit the Batch Shipyard Recipes section for various sample container workloads using Azure Batch and Batch Shipyard.

Batch Shipyard Compute Node Host OS Support

Batch Shipyard is currently compatible with popular Azure Batch supported Marketplace Linux VMs, compliant Linux custom images, and native Azure Batch Windows Server with Containers VMs. Please see the platform image support documentation for more information specific to Batch Shipyard support of compute node host operating systems.

Change Log

Please see the Change Log for project history.


Please see this project's Code of Conduct and Contributing guidelines.