Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Job failure with shadow directive if log files not created #1257

Closed
jaicher opened this issue Nov 12, 2021 · 1 comment
Closed

Job failure with shadow directive if log files not created #1257

jaicher opened this issue Nov 12, 2021 · 1 comment
Labels
bug Something isn't working

Comments

@jaicher
Copy link
Contributor

jaicher commented Nov 12, 2021

Snakemake version: v6.10.0

Describe the bug

For any job that specifies a logfile: if I am not using the shadow directive, it is okay if my job fails to make a logfile itself (because Snakemake creates an empty one at the final location for me, it looks like). But, if I use the shadow directive, I must explicitly make sure that a logfile is created in the specified location, otherwise, my job fails with FileNotFoundError in snakemake.dag.unshadow_output(...) because the logfile does not exist in the shadow directory, so it cannot move it to its final desired location. I think that if it succeeds without the shadow directive, it should succeed with the shadow directive.

Minimal example

As is, this fails. from FileNotFoundError in unshadow_output(...). Comment out the shadow directive, and it works.

# Snakefile
rule fails_if_shadow_enabled:
    shadow: "minimal"  # comment out and it succeeds
    output: touch("results/fails_if_shadow_enabled.done")
    log: "results/log/fails_if_shadow_enabled.log"

This seems like it'd be a pretty easy fix: either automatically touch the logfile in the shadow directory rather than the original location, or ignore FileNotFoundError for logfiles.

@jaicher jaicher added the bug Something isn't working label Nov 12, 2021
jaicher added a commit to jaicher/snakemake that referenced this issue Nov 12, 2021
jaicher added a commit to jaicher/snakemake that referenced this issue Nov 12, 2021
This addresses gh snakemake#1257. If the rule did not create a logfile, an empty
one would be created in its final destination. If the shadow directive
was used, however, Snakemake would try to move the nonexistent logfile
from the shadow directory to its final location. This caused the
observed failures. This commit has Snakemake create the empty file where
Snakemake would have expected the program to create it (i.e. in the
shadow directory). This enables the unshadow operation that takes place
later to succeed.
johanneskoester added a commit that referenced this issue Nov 27, 2021
#1258)

* Test for issue #1257 fails for missing logfile with shadow directory

* If rule did not create logfile, create it in its shadow directory

This addresses gh #1257. If the rule did not create a logfile, an empty
one would be created in its final destination. If the shadow directive
was used, however, Snakemake would try to move the nonexistent logfile
from the shadow directory to its final location. This caused the
observed failures. This commit has Snakemake create the empty file where
Snakemake would have expected the program to create it (i.e. in the
shadow directory). This enables the unshadow operation that takes place
later to succeed.

* move remote handling of log files into the general remote output handling; fix existence check

Co-authored-by: Johannes Köster <johannes.koester@tu-dortmund.de>
Co-authored-by: Johannes Köster <johannes.koester@uni-due.de>
@johanneskoester
Copy link
Contributor

johanneskoester commented Nov 27, 2021

Fixed with PR #1258.

maarten-k added a commit to maarten-k/realignment that referenced this issue May 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants