Is this suggestion related to a problem? Please describe.
This has spawned from modelblocks-org/module_geo_boundaries#81
The problem is that I want to store my download data in one place, separate from all other data. There are several reasons:
- so I can share that data across multiple calls to
snakemake
- so I can easily clean out derived data on-the-fly without worrying about potentially deleting download data
- so I can bundle download data into archives in case the upstream data becomes unavailable or I need to work offline
Describe the solution you'd like
I'd like a new pathvars entry that is used for any rule outputs that are related to downloaded data. They could be processed in the same step as being downloaded, so long as the user-facing configuration has not impacted that processing at all. This could be a downloads or retrieved pathvar.
Describe alternatives you've considered
Alternatives include:
- overriding download rule outputs within downstream workflows, for each modelblock. I don't like this as it requires much more set-up and maintenance on the workflow side. Updating to a new modelblock version could be a real pain.
- Considering all
resources as equivalent to initially downloaded data, including any resources that might be differentiated by a wildcard, e.g. scenario. I don't like this as it already mixes quite a lot of derived data (including processes that are heavily influenced by user config) with the initially "clean" downloaded data.
Additional context
This has been triggered by how configs are handled in https://github.com/pypsa/pypsa-eur, in which config files often refer to different run names, each of which spawns its own subdirectory in resources. These would all want to share the same downloaded data (provided their configs are aligned on that downloaded data), but they then want to place derived data in their own subdirectories. This split works with the way pypsa-eur is current set up, with all retrieved data sitting in a top-level data directory and all derived data sitting in resources/<run-name>. It may not be the best way to structure snakemake projects (I am not a massive fan of it, tbh!) but it is now used quite widely within the community as a way to manage different configurations within the same snakemake project.
Is this suggestion related to a problem? Please describe.
This has spawned from modelblocks-org/module_geo_boundaries#81
The problem is that I want to store my download data in one place, separate from all other data. There are several reasons:
snakemakeDescribe the solution you'd like
I'd like a new
pathvarsentry that is used for any rule outputs that are related to downloaded data. They could be processed in the same step as being downloaded, so long as the user-facing configuration has not impacted that processing at all. This could be adownloadsorretrievedpathvar.Describe alternatives you've considered
Alternatives include:
resourcesas equivalent to initially downloaded data, including anyresourcesthat might be differentiated by a wildcard, e.g.scenario. I don't like this as it already mixes quite a lot of derived data (including processes that are heavily influenced by user config) with the initially "clean" downloaded data.Additional context
This has been triggered by how configs are handled in https://github.com/pypsa/pypsa-eur, in which config files often refer to different
runnames, each of which spawns its own subdirectory inresources. These would all want to share the same downloaded data (provided their configs are aligned on that downloaded data), but they then want to place derived data in their own subdirectories. This split works with the way pypsa-eur is current set up, with all retrieved data sitting in a top-leveldatadirectory and all derived data sitting inresources/<run-name>. It may not be the best way to structure snakemake projects (I am not a massive fan of it, tbh!) but it is now used quite widely within the community as a way to manage different configurations within the same snakemake project.