Skip to content

v1.1.0: Post-Run Script SQS Integration

Latest
Latest

Choose a tag to compare

View all tags
@drernie drernie released this 04 Dec 01:24
v1.1.0
e786a8e

🚀 New Post-Run Script Architecture

This release introduces a new post-run script approach for SQS integration, replacing the previous workflow.onComplete hook method.

Key Features

📋 Comprehensive Metadata Collection

The new post-run script (scripts/post_run_sqs.py) collects metadata from three sources:

  1. Seqera Platform API - Fetches complete workflow details including:

    • Resolved parameters (outdir, sqs_queue_url, etc.)
    • Run status, session ID, launch info
    • Compute environment ID, work directory, duration
    • Full config text

  2. TOWER_ Environment Variables* - Captures runtime context:

    • Workflow ID, run name, project/workspace IDs
    • User ID, workflow status, timeline
    • Pipeline URL, revision, Nextflow version

  3. WRROC Metadata Files - Parses workflow provenance:

    • Author information (name, ORCID)
    • Runtime platform, programming language
    • Workflow start/end times, run ID
    • License and publication dates

🎯 Benefits Over Previous Approach

  • No workflow code changes required - Works with any Nextflow workflow
  • Richer metadata - Includes API data and WRROC provenance
  • Better error handling - Comprehensive logging and debugging
  • Easier testing - Can be run independently of workflows
  • No size limits - Script hosted on GitHub, bypasses 1024-byte limit

🔧 Setup Required

Before using the post-run script, you must complete the SQS setup:

  1. Configure IAM Permissions - Grant TowerForge roles access to your SQS queue:

    ./setup-sqs.py --profile sales --yes

  2. Verify Setup - Confirm permissions are correct:

    ./check-sqs.sh --profile sales

  3. Configure Post-Run Hook - In Seqera Platform, set the post-run script to:

    curl -sSfL https://raw.githubusercontent.com/data-yaml/seqera-smoke-test/v1.1.0/scripts/post-run-sqs.sh | bash

  4. Launch with SQS Parameters - Workflows must include the SQS queue URL:

    tw launch ... --params.sqs_queue_url='https://sqs.REGION.amazonaws.com/ACCOUNT/QUEUE'

See README-SQS.md for complete setup instructions.

🧪 Quick Test

After setup, verify end-to-end integration:

./test-sqs.sh --yes

📚 Documentation Improvements

  • Consolidated README-SQS-SETUP.md into README-SQS.md
  • Simplified main README with references to detailed SQS docs
  • Removed duplicate and outdated files
  • Fixed markdown linting issues

All SQS setup, configuration, and troubleshooting now in one place: README-SQS.md

🔧 What Changed

Added:

  • scripts/post_run_sqs.py - Main Python integration script
  • scripts/post-run-sqs.sh - Bash launcher
  • scripts/post-run-debug.sh - Debugging utility
  • Enhanced test-sqs.sh with parameter passing

Modified:

  • nextflow.config - Added WRROC provenance support
  • lib/tw-common.sh - Added helper functions
  • README-SQS.md - Consolidated documentation

Removed:

  • README-SQS-SETUP.md - Merged into README-SQS.md
  • main-sqs.nf - Replaced by post-run approach
  • Duplicate README files

📦 Full Changelog

See PR #6 for detailed changes: #6

Note: This release is on the parse-wrroc branch. Merge to main after testing.

Assets 2
Loading