Process exceeded running time limit (4h)

[zl2860]

Hi,

Thank you for providing these wonderful tools.

I am working on prediction tasks using biobank-level data. During my initial run, I encountered an error at the SCORE_AGGREGATE step stating, "Process exceeded running time limit (4h)."

executor > local (47)
[21/7ac43e] process > PGSCATALOG_PGSCCALC:PGSCCALC:INPUT_CHECK:COMBINE_SCOREFILES (1) [100%] 1 of 1 ✔
[skipped ] process > PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_RELABELBIM (Olink chromosome 6) [100%] 22 of 22, stored: 22 ✔
[- ] process > PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_RELABELPVAR -
[- ] process > PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_VCF -
[62/07a331] process > PGSCATALOG_PGSCCALC:PGSCCALC:MATCH:MATCH_VARIANTS (Olink chromosome 10) [100%] 22 of 22 ✔
[2b/30b909] process > PGSCATALOG_PGSCCALC:PGSCCALC:MATCH:MATCH_COMBINE (Olink) [100%] 1 of 1 ✔
[63/9c2be1] process > PGSCATALOG_PGSCCALC:PGSCCALC:APPLY_SCORE:PLINK2_SCORE (Olink chromosome 8 effect type additive 0) [100%] 22 of 22 ✔
[e8/1bf558] process > PGSCATALOG_PGSCCALC:PGSCCALC:APPLY_SCORE:SCORE_AGGREGATE (Olink) [100%] 1 of 1, failed: 1 ✘
[- ] process > PGSCATALOG_PGSCCALC:PGSCCALC:REPORT:SCORE_REPORT -
[- ] process > PGSCATALOG_PGSCCALC:PGSCCALC:DUMPSOFTWAREVERSIONS -
Execution cancelled -- Finishing pending tasks before exit
ERROR ~ Error executing process > 'PGSCATALOG_PGSCCALC:PGSCCALC:APPLY_SCORE:SCORE_AGGREGATE (Olink)'

Caused by:
Process exceeded running time limit (4h)

I suspect this is due to the large number of scores that need to be aggregated. To address this, I attempted to use more powerful computational resources to complete the calculation by adding the argument '-c' to finish computation:

nextflow run pgscatalog/pgsc_calc -r v2.0.0-alpha.5 \
    -profile conda \
    --input "$input_file" \
    --target_build GRCh37 \
    --min_overlap 0.5 \
    --scorefile "$scorefile_pattern" \
    -c /PATH/TO/my_custom.config

When I checked the logs, I noticed that the information on the configuration files was empty.
Could you please help me identify what might be causing this issue and how to ensure that the configuration files are correctly captured by the software?

[nebfield]

Here's an example configuration file I use for UK Biobank on my HPC cluster:

process {
    errorStrategy = 'retry'
    maxRetries = 3
    maxErrors = '-1'
    executor = 'slurm'

    withName: 'SAMPLESHEET_JSON' {
      cpus = { 1 * task.attempt }
      memory = { 1.GB * task.attempt }
      time = { 1.hour * task.attempt }
    }

    withName: 'DOWNLOAD_SCOREFILES' {
      cpus = { 1 * task.attempt }
      memory = { 1.GB * task.attempt }
      time = { 1.hour * task.attempt }
    }

    withName: 'COMBINE_SCOREFILES' {
      cpus = { 1 * task.attempt }
      memory = { 8.GB * task.attempt }
      time = { 2.hour * task.attempt }
    }

    withName: 'PLINK2_MAKEBED' {
      cpus = { 1 * task.attempt }
      memory = { 8.GB * task.attempt }
      time = { 1.hour * task.attempt }
    }

    withName: 'RELABEL_IDS' {
      cpus = { 1 * task.attempt }
      memory = { 16.GB * task.attempt }
      time = { 1.hour * task.attempt }
    }

    withName: 'PLINK2_ORIENT' {
      cpus = { 1 * task.attempt }
      memory = { 8.GB * task.attempt }
      time = { 1.hour * task.attempt }
    }

    withName: 'DUMPSOFTWAREVERSIONS' {
      cpus = { 1 * task.attempt }
      memory = { 1.GB * task.attempt }
      time = { 1.hour * task.attempt }
    }

    withName: 'ANCESTRY_ANALYSIS' {
      cpus = { 1 * task.attempt }
      memory = { 8.GB * task.attempt }
      time = { 1.hour * task.attempt }
    }

    withName: 'SCORE_REPORT' {
      cpus = { 1 * task.attempt }
      memory = { 8.GB * task.attempt }
      time = { 1.hour * task.attempt }
    }

    withName: 'EXTRACT_DATABASE' {
      cpus = { 1 * task.attempt }
      memory = { 8.GB * task.attempt }
      time = { 1.hour * task.attempt }
    }

    withName: 'PLINK2_RELABELPVAR' {
      cpus = { 2 * task.attempt }
      memory = { 16.GB * task.attempt }
      time = { 2.hour * task.attempt }
    }

    withName: 'INTERSECT_VARIANTS' {
      cpus = { 2 * task.attempt }
      memory = { 8.GB * task.attempt }
      time = { 1.hour * task.attempt }
    }

    withName: 'MATCH_VARIANTS' {
      cpus = { 2 * task.attempt }
      memory = { 32.GB * task.attempt }
      time = { 6.hour * task.attempt }
    }

    withName: 'FILTER_VARIANTS' {
      cpus = { 2 * task.attempt }
      memory = { 16.GB * task.attempt }
      time = { 1.hour * task.attempt }
    }

    withName: 'MATCH_COMBINE' {
      cpus = { 4 * task.attempt }
      memory = { 64.GB * task.attempt }
      time = { 6.hour * task.attempt }
    }

    withName: 'FRAPOSA_PCA' {
      cpus = { 2 * task.attempt }
      memory = { 8.GB * task.attempt }
      time = { 1.hour * task.attempt }
    }

    withName: 'PLINK2_SCORE' {
      cpus = 3 // { 2 * task.attempt }
      memory = { 8.GB * task.attempt }
      time = { 12.hour * task.attempt }
    }

    withName: 'SCORE_AGGREGATE' {
      cpus = 2
      memory = { 16.GB * task.attempt }
      time = { 4.hour * task.attempt }
    }
}

You could adopt a similar approach using a configuration file like this, but updating the SCORE_AGGREGATE process. You'd need to save this file and use the -c parameter.

I also recommend updating to the latest version (v2.0.0-beta) of the calculator, which could have faster aggregation. If you do update, you'll need to delete the work folder before starting the workflow again (rm -r work).

[nebfield]

Closing this now. Please feel free to create a new issue or discussion if you have any more questions.