Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: add quotes to each item of the wait_for_files list #1160

Merged
merged 6 commits into from Oct 21, 2021

Conversation

fdchevalier
Copy link
Contributor

@fdchevalier fdchevalier commented Aug 29, 2021

Hello Snakemake team,

Description

Context: Job submission to SGE cluster using qsub from a folder with a path containing a white space.

Problem: First batch of jobs is successfully submitted but snakemake issue an error regarding missing files. Below is the typical error issued when submitting a job from `$HOME/test test/':

Waiting at most 10 seconds for missing files.
Missing files after 10 seconds:
test/.snakemake/tmp.6tq2tga_

The error is due to the absence of quotes around the tmp directory path when generating the job script. The typical line for the the wait-for-files options is as follows:

--wait-for-files /XXXX/test test/.snakemake/tmp.6tq2tga_ test test2 --latency-wait 10 \

but should be:

--wait-for-files '/XXXX/test test/.snakemake/tmp.6tq2tga_' test test2 --latency-wait 10 \

Solution: Add quotes when appending the path to the wait_for_files list.

Comments

The solution has been tested in cases with white space and and with special character (&).

I am not fluent enough in Python so feel free to edit the code if needed.

Minimal reproducible example

In case this needs to be reproduced.

# Create working directory
mkdir "test test"
cd "test test"

# Create status directory for qsub and output directory
mkdir status
mkdir output

# Create test file
touch test

# Create snakefile
cat  << EOF > snakefile
rule test_rule:
    input:
        "test"
    output:
        "output/a.txt"
    shell:
        'sleep 10 && echo h=$(hostname) "{input}" > "{output}"'
EOF

# Submit job
snakemake --cluster "qsub -V -cwd -o status -j y" --jobs 20 -w 10

# Print status
cat status/*

QC

  • The PR contains a test case for the changes or the changes are already covered by an existing test case.
  • The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

Thank you for considering this patch.

@fdchevalier fdchevalier changed the title Add quotes to tmp dir in wait-for-file options fix: add quotes to tmp dir in wait-for-file options Aug 29, 2021
@fdchevalier fdchevalier changed the title fix: add quotes to tmp dir in wait-for-file options fix: add quotes to each itme of the wait_for_files list Aug 31, 2021
@fdchevalier
Copy link
Contributor Author

fdchevalier commented Aug 31, 2021

I noticed that the bug was also affecting files with white spaces. So I came with a better solution to quote every item of the wait_for_files list.

I also tested the behavior of Snakemake when having more than 20 elements in the wait_for_lists which creates a file with the list of elements: in that case, the content of the file worked as expected (no need of quotes around each element) but the path to the file required quotes. I modified the code accordingly.

Copy link
Contributor

@johanneskoester johanneskoester left a comment

Thanks a lot! Very good idea. Python allows to make the changed code a bit simpler, see below.

snakemake/executors/__init__.py Outdated Show resolved Hide resolved
snakemake/executors/__init__.py Outdated Show resolved Hide resolved
@sonarcloud
Copy link

sonarcloud bot commented Sep 14, 2021

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

@fdchevalier fdchevalier changed the title fix: add quotes to each itme of the wait_for_files list fix: add quotes to each item of the wait_for_files list Sep 14, 2021
@sonarcloud
Copy link

sonarcloud bot commented Oct 21, 2021

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

@johanneskoester johanneskoester merged commit 72856ed into snakemake:main Oct 21, 2021
6 checks passed
pvandyken pushed a commit to pvandyken/snakemake that referenced this pull request Nov 15, 2021
* Add quotes to tmp dir in wait-for-file options

* Add quotes to each item of the wait_for_files list

* Add quotes to the wait_for_files_file path

* Improve quoting strategy

* Format following Black code style

Co-authored-by: Johannes Köster <johannes.koester@uni-due.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants