Skip to content

Nextflow AWS Batch Squared#264

Merged
mehalter merged 7 commits into
mainfrom
batch_squared
Aug 14, 2025
Merged

Nextflow AWS Batch Squared#264
mehalter merged 7 commits into
mainfrom
batch_squared

Conversation

@mehalter
Copy link
Copy Markdown
Member

@mehalter mehalter commented Aug 12, 2025

This repository moves our Nextflow pipeline execution to running entirely in AWS Batch. Where nextflow pipelines are submitted to a workflow queue which in turn spins up more AWS Batch jobs in a job queue for each individual step. This moves us to being able to see when pipelines are running and when they finish as well as avoiding the need to have a Nextflow EC2 instance always running waiting to receive job requests.

Implemented

  • New workflows AWS Batch job queue for submitting Nextflow workflows
  • Creates an ECR container repository for each swimlane to maintain container images
  • Creates a Docker image that installs nextflow and dependencies to execute pipelines

TODO in a followup PR

(need to merge for timeliness sake and pass off deployment rights)

  • Create an AWS Batch Job Definition for kicking off Bactopia runs

@mehalter mehalter force-pushed the batch_squared branch 3 times, most recently from cf6af89 to 9c5bf98 Compare August 13, 2025 12:00
@mehalter mehalter marked this pull request as ready for review August 14, 2025 12:19
@mehalter mehalter merged commit 2edad17 into main Aug 14, 2025
6 checks passed
@mehalter mehalter deleted the batch_squared branch August 14, 2025 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants