Skip to content

Add scrontab-based flowcell watchdog script#87

Merged
jemma-nelson merged 2 commits into
mainfrom
feat/slurmcron-scheduling
May 8, 2026
Merged

Add scrontab-based flowcell watchdog script#87
jemma-nelson merged 2 commits into
mainfrom
feat/slurmcron-scheduling

Conversation

@jemma-nelson
Copy link
Copy Markdown
Contributor

Replaces the self-chaining wait_for_copycomplete.sh approach with a scrontab job that scans for recently-completed flowcells hourly and auto-launches setup/processing when CopyComplete.txt appears.

Fixes #85.

Once merged, we'll want to create a scrontab entry as solexa (scrontab -e) to run this script periodically.

Replaces the self-chaining wait_for_copycomplete.sh approach with a
scrontab job that scans for recently-completed flowcells hourly and
auto-launches setup/processing when CopyComplete.txt appears.

Fixes the silent chain-death failure mode that caused flowcell 23K3FVLT3
to miss auto-processing (SE-5098).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@jemma-nelson jemma-nelson requested a review from rsandstromUW May 4, 2026 19:59
@jemma-nelson
Copy link
Copy Markdown
Contributor Author

jemma-nelson commented May 4, 2026

Example scrontab entry:

  #SCRON --job-name=watchdog-copycomplete
  #SCRON --partition=hpcz-test,hpcy-test
  #SCRON --time=00:55:00
  #SCRON --nodes=1
  #SCRON --cpus-per-task=1
  #SCRON --mem=1G
  #SCRON --mail-type=FAIL
  #SCRON --mail-user=sequencing@altius.org
  #SCRON --open-mode=append
  #SCRON --output=/home/solexa/stampipes_data/logs/watchdog_copycomplete.log
  0 * * * * /home/solexa/stampipes/scripts/flowcells/watchdog_copycomplete.sh

Copy link
Copy Markdown
Collaborator

@rsandstromUW rsandstromUW left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome! the scrontab will add stability to our automatic launching scheme.

@jemma-nelson jemma-nelson merged commit dc9993f into main May 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

wait_for_copycomplete.sh chain breaks silently on a single TIMEOUT — no watchdog/resurrect

2 participants