-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a use_relative_output_paths parameter that flattens output directories. #606
Conversation
One question: I see that all test files are properly cleaned up. Except the files that I added to |
Pull Request Test Coverage Report for Build 3128243776
💛 - Coveralls |
@rhpvorderman Apologies for the long delay picking up on this, been moving house and other interruptions. On first reading, two comments on the code
Regarding test file cleanup, all the files created by |
I agree this is a bit of a hack that should be solved better. I'd rather not hardcode this, in case for some reason the name changes from "work" to "execution" for example and then suddenly a lot of code breaks.
But I really do care about that structure. Otherwise I will get a lot of collissions and it is quite nice to have it in an ordered fashion and to be able to set that in the pipeline. Nextflow has this with "PublishDir" and snakemake can do it too.
In the current working directory, at least when I ran it last time. I will check it again. At least the test code tests the desired result correctly. So there is at least one thing in this PR that does not have to be massively refactored 😅 . |
Motivation
#604
Approach
As suggested, modify the link_outputs function. I wrote a small map_paths_relative function that is much simpler and for
File
andDirectory
types just defers to map_paths. That way the handling of hardlinks etc. is correctly handled and there is no code duplication. I added a few small code snippets to theFile
/Directory
section of map_paths to handle the relativity part.I added a test, and also tested using BioWDL's germline-DNA pipeline manually to verify that multiple nested subworkflows do not influence the result.
Checklist
make pretty
to reformat the code with blackmake check
to statically check the code using Pyre and Pylint