--touch touch all files even missing ones

[yjqiu]

Snakemake version
5.7.0

Describe the bug
My workflow stopped in the middle and I want to rerun the latter part only. Since some intermediate files have been deleted, I used --touch to touch all existing files to prevent them from rerunning. But the workflow simply touch all files even if the files are missing without any errors.

Logs
If applicable, any terminal output to help explain your problem.

Minimal example
Add a minimal example for reproducing the bug.

Additional context
Add any other context about the problem here.

[troycomi]

I can't replicate your issue but also do not see the behavior specified in the documentation

Touch output files (mark them up to date without really changing them) instead of running their commands... Fails if a file does not yet exist.

Using the following Snakefile:

rule all:
    input:
        "test.out", "test1.out"
rule:
    output:
        "test.out"
    shell:
        "echo will not run with touch > {output}"
rule:
    output:
        "test1.out"
    shell:
        "echo exist already > {output}"

Running snakemake --touch appears to go through the workflow without failure but also does not touch the files. Next, I created one of the files with snakemake test1.out to simulate a partial execution. Again, snakemake --touch does not fail, does not update the modified time of test1.out, and does not create test.out. Finally, snakemake test.out followed by snakemake --touch appears to do nothing to either the modified time or the metadata.

Anyone else know what the actual use case is or if this feature is not working? I don't see any tests for when touch is enabled. @johanneskoester

However, for your particular case touch should not be necessary. Intermediate files marked as temp in the Snakefile will be automatically removed when no longer needed and should be ignored when rerunning the workflow. For complex workflows, you may inadvertently remake a temp file, which will cause all dependent rules to be rerun. I have gotten into trouble with this and the best way to get around is to use an input rule to make a stronger form of ancient:

def my_check(wildcards):
  if os.path.exists(output_path.format(wildcards)):
    return ''
  else:
    return input_path.format(wildcards)

rule:
  input: my_check
  output: output_path
  ...

This ignores any dependencies as long as the output exists. Note that you lose the ability to track changes in the input files so be sure you never want to run the rule again. Also, if you are using unpack on the function (because it returns a dict) you should return an empty list instead of an empty string.

Hope that helps!

[johanneskoester]

The semantic behind --touch is that output files that exist and would be recreated upon a rerun will be touched. This means, that your case @troycomi is actually working as expected. When you manually create the file test1.out, snakemake has no reason to recreate it, and hence won't touch it. In order to enforce touching it, you need to combine --touch with one of the force commands (--forceall, --forcerun, --force).

[johanneskoester]

I have extended the --help text, hopefully clarifying your issues. Please feel free to reopen if you disagree.

[yjqiu]

I can't replicate your issue but also do not see the behavior specified in the documentation

Touch output files (mark them up to date without really changing them) instead of running their commands... Fails if a file does not yet exist.

Using the following Snakefile:

rule all:
    input:
        "test.out", "test1.out"
rule:
    output:
        "test.out"
    shell:
        "echo will not run with touch > {output}"
rule:
    output:
        "test1.out"
    shell:
        "echo exist already > {output}"

Running snakemake --touch appears to go through the workflow without failure but also does not touch the files. Next, I created one of the files with snakemake test1.out to simulate a partial execution. Again, snakemake --touch does not fail, does not update the modified time of test1.out, and does not create test.out. Finally, snakemake test.out followed by snakemake --touch appears to do nothing to either the modified time or the metadata.

Anyone else know what the actual use case is or if this feature is not working? I don't see any tests for when touch is enabled. @johanneskoester

However, for your particular case touch should not be necessary. Intermediate files marked as temp in the Snakefile will be automatically removed when no longer needed and should be ignored when rerunning the workflow. For complex workflows, you may inadvertently remake a temp file, which will cause all dependent rules to be rerun. I have gotten into trouble with this and the best way to get around is to use an input rule to make a stronger form of ancient:

def my_check(wildcards):
  if os.path.exists(output_path.format(wildcards)):
    return ''
  else:
    return input_path.format(wildcards)

rule:
  input: my_check
  output: output_path
  ...

This ignores any dependencies as long as the output exists. Note that you lose the ability to track changes in the input files so be sure you never want to run the rule again. Also, if you are using unpack on the function (because it returns a dict) you should return an empty list instead of an empty string.

Hope that helps!

That's super helpful! Thanks for the suggestion.

[yjqiu]

The semantic behind --touch is that output files that exist and would be recreated upon a rerun will be touched. This means, that your case @troycomi is actually working as expected. When you manually create the file test1.out, snakemake has no reason to recreate it, and hence won't touch it. In order to enforce touching it, you need to combine --touch with one of the force commands (--forceall, --forcerun, --force).

But in that case, shouldn't it fail upon missing files? But it runs fine for missing files in @troycomi 's example.