Using environment variables as configuration parameters

[sebastian-luna-valero]

Snakemake version

7.24.0

Describe the bug

Cannot use environment variables as configuration parameters. See below.

Logs

Minimal example

Snakefile

rule test:
    input:
    output:
        touch("test.done")
    params:
        folder = config['folder']
    shell:
        """ 
        echo {params.folder}
        """

config.yml

folder: "${HOME}/"

Error message:

WildcardError in rule test in file /tmp/tmp.VKzVjRn5Dq/use-case-hisea/workflow/Snakefile-2, line 2:
Wildcards in params cannot be determined from output files. Note that you have to use a function to deactivate automatic wildcard expansion in params strings, e.g., `lambda wildcards: '{test}'`. Also see https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules:
'HOME'

Additional context

[backeb]

Hi @sebastian-luna-valero,

I've tried the below as a potential work around and it gave a slightly different error:

Snakefile

configfile: "workflow/config.yml"

rule test:
    input:
    output:
        touch("test.done")
    params:
        folder = expand("{folder}", folder=config['folder'])
    shell:
        """ 
        echo {params.folder}
        """

config.yml

folder: "${HOME}/"

Error message:

Building DAG of jobs...
WildcardError in file /home/centos/test/workflow/Snakefile, line 3:
Wildcards in params cannot be determined from output files. Note that you have to use a function to deactivate automatic wildcard expansion in params strings, e.g., `lambda wildcards: '{test}'`. Also see https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#non-file-parameters-for-rules:
'HOME'

[dariober]

I don't think this is a bug, it's the expected behavior of snakemake in interpreting curly braces. I think a solution could be to escape curly braces by doubling them. E.g in the config file use:

folder: "${{HOME}}/"

alternatively, at the beginning of the snakefile you could add something like:

folder = config["folder"]

if os.path.normpath(folder) == "${HOME}":
    folder = os.environ["HOME"]
    ## Or:
    folder = "${{HOME}}"

Then use folder instead of config["folder"].

[sebastian-luna-valero]

Thanks @dariober

Attempt 1

Only replacing:

folder: "${HOME}/"

with:

folder: "${{HOME}}/"

in config.yml gave the same error as above.

Attempt 2

The files below gave the same error as above.

config.yml

folder: "${HOME}/"

Snakefile:

folder = config["folder"]

if os.path.normpath(folder) == "${HOME}":
    folder = "${{HOME}}"

rule test:
    input:
    output:
        touch("test.done")
    params:
        folder = folder
    shell:
        """ 
        echo {params.folder}
        """

Attempt 3

The following worked:

config.yml

folder: "${HOME}/"

Snakefile:

folder = config["folder"]

if os.path.normpath(folder) == "${HOME}":
    folder = os.environ["HOME"]

rule test:
    input:
    output:
        touch("test.done")
    params:
        folder = folder
    shell:
        """ 
        echo {params.folder}
        """

[sebastian-luna-valero]

However, what would you suggest if I need to work with something like ${HOME}/folder or similar? i.e. an environment variable plus relative paths?

[dariober]

I vaguely remembered a similar issue and here it is https://stackoverflow.com/questions/61194156/use-special-symbols-in-snakemake-parameter-section. Maybe that is the least clunky solution as there is no need for testing strings or fiddling with curly braces. It should address also cases like ${HOME}/folder. Basically:

rule test:
    input:
    output:
        touch("test.done")
    params:
        folder = lambda wc: config['folder'],
    shell:
        """ 
        echo {params.folder}
        """

[sebastian-luna-valero]

Many thanks @dariober

It worked! I am pasting the full example below for reference:

config.yml:

folder: "${PWD}/data"

Snakefile:

rule test:
    input:
    output:
        touch("test.done")
    params:
        folder = lambda wc: config['folder']
    shell:
        """ 
        echo {params.folder}
        """