Rescue failed population synthesis runs

[maxbriel]

A POSYDON population synthesis run consists of three distinct phases:

1. Binary population evolution (Job array execution)
2. Creation of evolution.combined.<JOB_ARRAY_ID> files
3. Merging of evolution.combined files into a *_population.h5 file.

However, errors can occur, whether due to memory limitations, walltime constraints, or system interruptions. In response to these issues, we have created two utility scripts: posydon-popsyn check and posydon-popsyn rescue.

Warning

The scripts are not finished, but they provide some basic functionality. Please comment below if you run into issues, and what other functionality you would like!

Validating a population synthesis run

The posydon-popsyn check <run_folder> script is designed to verify the integrity of your population synthesis run. Here, <run_folder> refers to the directory containing your population run.
This script performs the following diagnostics:

• It checks whether all expected metallicity-specific population files (i.e., *_population.h5) are present in the run folder.
• It ensures that the oneline file contains the correct number of binaries, as originally requested by the input parameters.

If any discrepancies are found, they're reported back. If all batch-level (evolution.combined) files are present, the script can resubmit the merge step for you.
Please note that, until absolute paths are fully supported, this script may need to be executed from within the population run folder itself using:

posydon-popsyn check .

Rescuing a job array

The posydon-popsyn rescue <run_folder> script extends the functionality of check, specifically targeting recovery from failed job arrays. This is particularly useful when some array jobs fail due to exceeding memory limits or runtime constraints.

The script does the following steps:

1. Same consistency checks as check.
2. It then identifies any failed or missing array jobs. If no missing array jobs are found, the script fails!
3. A custom rescue script is automatically generated for you.

Similarly to the check script, issues might arise with the scripts when running the script outside the run folder. If such issues occur, please let us know and try running it in the <run_folder>.

posydon-popsyn rescue .

It should create a resubmit.sh script that you can resubmit manually with sh resubmit.sh.