Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ancient keyword causes incorrect scheduling order #946

Closed
kernco opened this issue Apr 8, 2021 · 3 comments · Fixed by #1202
Closed

Ancient keyword causes incorrect scheduling order #946

kernco opened this issue Apr 8, 2021 · 3 comments · Fixed by #1202
Labels
bug Something isn't working

Comments

@kernco
Copy link

kernco commented Apr 8, 2021

Snakemake version
6.1.0

Describe the bug
In some situations, putting ancient() around input files causes rules to be run before the input files they need exist.

Minimal example
Here is a minimal example Snakefile:

rule all: 
        input: 'Step3.done'

rule Preprocess:
        input: 
        output: 'Preprocess.done'
        shell: ''

rule Step1:
    input: ancient('Preprocess.done')
        output: 'Step1_{i}.done'
        shell: ''

rule Step2:
        input: expand('Step1_{g}.done', g=list(range(3)))
        output: 'Step2.done'
        shell: ''

rule Step3:
        input: ancient('Preprocess.done'),'Step2.done'
        output: 'Step3.done'
        shell: ''

Doing a dry run (snakemake -n) results in this output:

[Wed Apr  7 22:45:37 2021]
rule Step1:
    input: Preprocess.done
    output: Step1_1.done
    jobid: 5
    wildcards: i=1

[Wed Apr  7 22:45:37 2021]
rule Preprocess:
    output: Preprocess.done
    jobid: 2

[Wed Apr  7 22:45:37 2021]
rule Step1:
    input: Preprocess.done
    output: Step1_0.done
    jobid: 4
    wildcards: i=0

[Wed Apr  7 22:45:37 2021]
rule Step1:
    input: Preprocess.done
    output: Step1_2.done
    jobid: 6
    wildcards: i=2

[Wed Apr  7 22:45:37 2021]
rule Step2:
    input: Step1_0.done, Step1_1.done, Step1_2.done
    output: Step2.done
    jobid: 3

[Wed Apr  7 22:45:37 2021]
rule Step3:
    input: Preprocess.done, Step2.done
    output: Step3.done
    jobid: 1

[Wed Apr  7 22:45:37 2021]
localrule all:
    input: Step3.done
    jobid: 0

When the DAG is built, it doesn't seem to recognize that Step1 cannot be run before Preprocess, and instead schedules them to run simultaneously. This can be seen by one of the Step1 jobs appearing before the Preprocess job.

Removing the use of ancient from Step1 results in the correct scheduling. This workflow works on older version of Snakemake, but at some point it stopped working. I'm not sure about the exact version where it stopped working, but I know that it works in version 5.12.0.

@kernco kernco added the bug Something isn't working label Apr 8, 2021
@yuefu-jiang
Copy link

yuefu-jiang commented Apr 12, 2021

I observed the same issue under 5.32.2.

@bkohrn
Copy link

bkohrn commented Jul 6, 2021

I've been seeing a similar issue (in an incredibly complex pipeline which may not be appropriate to debugging; see #771), but this example you gave doesn't seem to reproduce the behavior for me when I actually run it. I don't think the dryrun is an appropriate test, since the order of jobs in the dryrun isn't necessarily the order in which jobs will actually be executed. Unfortunately, I'm having trouble replicating it in a simpler pipeline.

@kernco
Copy link
Author

kernco commented Jul 8, 2021

I see, well if it helps the pipeline that trimmed down to get the minimal example is the MERlin software for analyzing MERFISH data developed by Xiaowei Zhuang's lab: https://github.com/emanuega/MERlin

DonFreed added a commit to DonFreed/snakemake that referenced this issue Oct 1, 2021
johanneskoester added a commit that referenced this issue Oct 21, 2021
…uming job (#1202)

Fixes #946

Co-authored-by: Johannes Köster <johannes.koester@uni-due.de>
pvandyken pushed a commit to pvandyken/snakemake that referenced this issue Nov 15, 2021
…uming job (snakemake#1202)

Fixes snakemake#946

Co-authored-by: Johannes Köster <johannes.koester@uni-due.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants