Misformatting of lambda functions in threads: (causing "Unexpected keyword wildcards in rule definition")

[karel-brinda]

Problem description

This is an issues that we encountered when working on MOF-Search.

Snakefmt seems to be unable to parse correctly lambda functions used in the threads: section of rule definitions, due to which it then introduces line breaks in a way that invalidates the original Snakefile.

Example

Original Snakefile

rule decompress_and_run_cobs:
    output: "test"
    threads: lambda wildcards, input: get_number_of_COBS_threads(wildcards, input, predefined_cobs_threads, streaming)
    shell:
        """
        touch {output}
        """

Reformated Snakefile

If I run snakefmt Snakefile, I end up with the following Snakefile:

rule decompress_and_run_cobs:
    output:
        "test",
    threads: lambda wildcards, input: get_number_of_COBS_threads(
    wildcards, input, predefined_cobs_threads, streaming
)
    shell:
        """
        touch {output}
        """

Parsing errors

After which Snakemake fails with the following error:

$ snakemake -j1
SyntaxError in file /private/tmp/~20231128150941/Snakefile, line 5:
Unexpected keyword wildcards in rule definition (Snakefile, line 5)

And Snakefmt cannot be rerun either.

$ snakefmt  Snakefile 
snakefmt.exceptions.InvalidParameterSyntax: 4lambda wildcards , input: get_number_of_COBS_threads(

Version

$ snakefmt --version
snakefmt, version 0.8.5

[mbhall88]

Thanks for reporting Karel. Lambda functions have been really hard to parse. Brice and I have also found it very difficult to find time to address snakefmt bugs as they do take a lot of time to debug.

Ultimately the whole parser needs to be rewritten with something like Lark as each bug fix requires us to make the codebase a little "hackier". But again, finding the time to do this has been hard.

I'll add it to the list of bugs and see when I can get around to it. Also, do feel free to have a crack at fixing it. My normal process is to add a failing test, like the example above, then use the debugger in PyCharm to find where in the code the bad formatting comes from.

[karel-brinda]

Hi Michael, thanks for the answer.

Just one thing to add – I think it was failing even when I replaced lambda with functools.partial, so I assume it's somehow an issue specifically with the threads keyword.

As a workaround, I think the following solution would be sufficient in most usecases:
#86

If I know Snakefmt is failing for certain lines, I'll just switch it off for them.

Btw. thanks for all the work on Snakefmt – it's making many people's life easier.