prefactor-cwl

CWL version of prefactor

https://github.com/lofar-astron/prefactor/

Note that this project is still very much in alpha stage and very much in flux. It is a prototype implementation and is not intended for end user audience. The goal is to demonstrate various technologies and how they could assist simplifying construction and deployment of big scale data reduction pipelines.

requirements

• Python Virtualenv

optional:

• Slum
• Docker
• Singularity

usage

To run the pipeline with the example dataset just run:

$ make run

To Run using a udocker (user space docker) run:

$ make run-udocker

To try out toil as a schedular run:

$ make toil

HPC cloud

We've made a set of ansbile scripts to automate the deployment of a cluster that can run CWL pipelines: https://github.com/EOSC-LOFAR/ansible

on Cartesius

If you are on cartesius, you first need to activate a python environment:

$ module load python

Since all jobs are quite small, it is better to use the staging partition. You can configure toil to use the staging partition with:

$ export TOIL_SLURM_ARGS="-t 0:30:00 -p staging"

$ make slurm

steps

the pipeline consists of these steps:

step name	status	file	comment
createmap_cal	skip		we don't use map files
ndppp_prep_cal	implemented	steps/ndppp_prep_cal.cwl
combine_data_cal_map	skip		We don't use map files
sky_cal	implemented	steps/sky_cal.cwl
sky_cal_path	skip		we don't use map files
calib_cal	implemented	steps/steps/calib_cal.cwl
h5_imp_cal_map	skip		we don't use map files
h5imp_cal	implemented	steps/losoto_importer.cwl
mk_inspect_dir	skip		we don't need to create dirs
copy_h5imp_cal	skip		this is managed by CWL
fitclock	implemented	steps/fitclock.cwl
ampl	implemented	steps/ampl.cwl	disabled for test since not enough subbands
plots	implemented	steps/plots.cwl	disabled for test since not enough subbands
phase	implemented	steps/phase.cwl
plot_cal_phases	implemented	steps/plot_cal_phases.cwl
createmap_cal_pngs	skip		we don't use map files
copy_cal_pngs	skip		this is managed by CWL
createmap_cal_txts	skip		we don't use map files
copy_cal_txts	skip		this is managed by CWL
mk_cal_values_dir	skip		this is managed by CWL
createmap_cal_npys	skip		we don't use map files
copy_cal_npys	skip		this is managed by CWL