On windows, the new storage plugin functionality seems to fail to properly match s3 objects between input and output of rules

[johanneskoester]

Snakemake version
Snakemake 8.0 (in development)

Describe the bug
On windows, the new storage plugin functionality seems to fail to properly match s3 objects between input and output of rules

Logs

# snakemake.exceptions.MissingInputException: Missing input files for rule all:
# affected files:
#   s3://snakemake-ec6c5e4c267d4a1e890976b57c99ee55/test2.out (storage)
#

Minimal example
This can for example be observed when activating the test case test_storage on windows.

@melund, your help would be greatly appreciated! (at the moment, the test fails in PR #5440, but I will merge that PR ASAP even before this is fixed in order to proceed with the finalization of Snakemake 8.0).

[melund]

Yes. Just merge it. I will be happy to help fix the issue on Windows if I can. However, I will not have time to really look into it until next weekend.

[johanneskoester]

Thank you so much! PR is merged, the problematic test is

snakemake/tests/tests.py

Line 655 in fb65c33

def test_storage(s3_storage):

[melund]

@johanneskoester. What do you suspect. Is it snakemake itself or the new storage-plugin that is causing the bug?

[melund]

@johanneskoester. I gave it a look while the kids were sleeping. But it took a me a while to get into the code base again. Just getting the test running locally proved difficult.

Turns out that "Git for Windows" have reorganized its internal unix tools so they are no longer on path (Things like "cp", "touch", "cat" etc.) I guess it works in the cloud because the GitHub actions VM comes prepackaged with all the tools MSYS2 tools on path.

I fixed that and got the tests suite running, but for some reason a few tests still fail for me:

FAILED tests/tests.py::test_glpk_solver - AssertionError: expected successful execution
FAILED tests/tests.py::test_output_file_cache - AssertionError: expected successful execution
FAILED tests/tests.py::test_module_complex - AssertionError: expected successful execution
FAILED tests/tests.py::test_module_complex2 - AssertionError: expected successful execution
FAILED tests/tests.py::test_module_with_script - AssertionError: expected successful execution

I will disregard that for now and begin looking at the test_storage test.

However, is there any documentation or discussion on how the new storage plugin works from a user perspective?