Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: option flag " --list-conda-envs " throws TypeError when used in snakemake v6.10.0 #1233

Closed

Conversation

iamh2o
Copy link
Contributor

@iamh2o iamh2o commented Oct 24, 2021

Super Brief

When using --list-conda-envs in 6.10.0, the following error is thrown:

    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not LocalSourceFile

Less Brief

  • snakemake --version == 6.10.0

Snakefile

rule all:
    conda:
        "workflow/envs/vanilla_v0.1.yaml"
    output:
        touch('done')
    shell:
        "sleep 1;"

snakemake --conda-frontend mamba --use-conda --conda-prefix /snakemake_envs/ -c 1 --list-conda-envs --verbose -s Snakefile all

  • Returns
Building DAG of jobs...
environment     container       location
Full Traceback (most recent call last):
  File "/locus/home/jmajor/conda/envs/MARIGOLD/lib/python3.9/site-packages/snakemake/__init__.py", line 699, in snakemake
    success = workflow.execute(
  File "/locus/home/jmajor/conda/envs/MARIGOLD/lib/python3.9/site-packages/snakemake/workflow.py", line 958, in execute
    simplify_path(env.file),
  File "/locus/home/jmajor/conda/envs/MARIGOLD/lib/python3.9/site-packages/snakemake/utils.py", line 157, in simplify_path
    relpath = os.path.relpath(path)
  File "/locus/home/jmajor/conda/envs/MARIGOLD/lib/python3.9/posixpath.py", line 456, in relpath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not LocalSourceFile

Traceback (most recent call last):
  File "/locus/home/jmajor/conda/envs/MARIGOLD/lib/python3.9/site-packages/snakemake/__init__.py", line 699, in snakemake
    success = workflow.execute(
  File "/locus/home/jmajor/conda/envs/MARIGOLD/lib/python3.9/site-packages/snakemake/workflow.py", line 958, in execute
    simplify_path(env.file),
  File "/locus/home/jmajor/conda/envs/MARIGOLD/lib/python3.9/site-packages/snakemake/utils.py", line 157, in simplify_path
    relpath = os.path.relpath(path)
  File "/locus/home/jmajor/conda/envs/MARIGOLD/lib/python3.9/posixpath.py", line 456, in relpath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not LocalSourceFile
unlocking
removing lock
removing lock
removed all locks

Running the same command w/out the --list-conda-envs flag. It runs.

snakemake --conda-frontend mamba --use-conda --conda-prefix locus/data/external_data/research_experiments/snakemake_envs/ -c 1 --verbose -s Snakefile all

Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job      count    min threads    max threads
-----  -------  -------------  -------------
all            1              1              1
total        1              1              1

Resources before job selection: {'_cores': 1, '_nodes': 9223372036854775807}
Ready jobs (1):
        all
Select jobs to execute...
Using greedy selector because only single job has to be scheduled.
Selected jobs (1):
        all
Resources after job selection: {'_cores': 0, '_nodes': 9223372036854775806}

[Sun Oct 24 04:32:29 2021]
rule all:
    output: done
    jobid: 0
    resources: tmpdir=/tmp

Activating conda environment: /locus/data/external_data/research_experiments/snakemake_envs/0af30cac396757609305612ab73f9d1e
Touching output file done.
[Sun Oct 24 04:32:30 2021]
Finished job 0.
1 of 1 steps (100%) done
Complete log: /locus/home/jmajor/EXPERIMENTS/EX4816/RU0/analyses/secondary/human/20211020_v0.9.8.9/INV_EX4816_L339025663-EXT1-HYFJCDMXX_0_sent/marigold-pipes/.snakemake/log/2021-10-24T043227.309543.snakemake.log
unlocking
removing lock
removing lock
removed all locks

Running with flag after minor changes applied -- environments list as expected.

snakemake --conda-frontend mamba --use-conda --conda-prefix /locus/data/external_data/research_experiments/snakemake_envs/ -c 1 --list-conda-envs -s Snakefile all

Building DAG of jobs...
environment     container       location
workflow/envs/vanilla_v0.1.yaml         /locus/data/external_data/research_experiments/snakemake_envs/0af30cac396757609305612ab73f9d1e

Other jobs

  • I ran a few other workflows that actually did something more than touch a file, they all worked with this change.

QC

I Ran some workflows using conda, and there were no problems. And the flag now works as advertised.

  • [ it contains a fix to enable the stated functionality- I'm not sure what else I could add. ] The PR contains a test case for the changes or the changes are already covered by an existing test case.
  • [ not necessary as this restores functionality already described ] The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

…me contexts, specifically if using --list-conda-envs
@iamh2o iamh2o requested a review from johanneskoester as a code owner Oct 24, 2021
@sonarcloud
Copy link

sonarcloud bot commented Oct 24, 2021

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@iamh2o
Copy link
Contributor Author

iamh2o commented Oct 25, 2021

Shoot- a closely related bug is also causing --containerize to break. Let me see if I can sort out a palate-able more general fix. (at the very least , this is a bug report)

@iamh2o iamh2o marked this pull request as draft Oct 25, 2021
@johanneskoester
Copy link
Contributor

johanneskoester commented Oct 26, 2021

Thanks a lot for looking into this! I've fixed this (and the singularity issue) in a more generic way now in #1236. I hope you don't mind. Nevetheless, your work was very useful in quickly hunting down the issue.

johanneskoester added a commit that referenced this pull request Oct 26, 2021
…#1233 and #1211). (#1236)

* fix: more informative nothing to be done message

* fmt

* fmt, fixes and additional tests

* implement fspath
pvandyken pushed a commit to pvandyken/snakemake that referenced this pull request Nov 15, 2021
…snakemake#1233 and snakemake#1211). (snakemake#1236)

* fix: more informative nothing to be done message

* fmt

* fmt, fixes and additional tests

* implement fspath
@iamh2o
Copy link
Contributor Author

iamh2o commented Nov 16, 2021

no problem- Glad it was helpful! I apologize for loosing the thread, but I'm drowning in potentially more snamekame excitement :-) inexplicable pipeline failures the last weeks which recovering from has been occupying all of my time - not quite enough to properly debug. I hope to have some focus time this week- and appropriate you might be hearing from me again :-) (this is a failure mode which stikes infrequently, but for workflows of 100000+ nodes, its certain to crash you out completely a few times)- more to come it relevant - thank

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants