Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shared results-folder between different pipeline runs? #1028

Open
asp8200 opened this issue May 19, 2023 · 0 comments
Open

Shared results-folder between different pipeline runs? #1028

asp8200 opened this issue May 19, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@asp8200
Copy link
Contributor

asp8200 commented May 19, 2023

Description of feature

It is possible to run Sarek repeatedly for the same sample or different samples using the same output-folder (results-folder), and some users do run Sarek like that.

Currently there are a number of issues with reusing the results-folder.

  1. The multiqc-data gets overwritten,
  2. joint-germline-data gets overwritten,
  3. csv/variantcalled.csv gets overwritten, (there is probably more files that get overwritten)
  4. if starting more pipelines at the exact same time, then different pipelines may construct pipeline-info-files with exactly the same name. (Tower actually did this at DNGC; bug reported to Rob Syme and Harshil Patel.)
  5. Using the same results-folder for different pipeline-runs makes it very difficult to subsequently, say, find and delete all files from some particular pipeline-run.

Should we try to solve those issue?

Perhaps we just want to advise against using the same results-folder for different pipeline-runs?

N.B. What about work-folders? On re-runs (-resume) one, of course, wants to use the same work-folder, but what about work-folders for pipeline-runs of different samples? I see no reason that they should share work-folders. (I know that the subfolders with the hash-strings make any clashes/conflicts between pipelines sharing the same work-folder very unlikely, but still?)

@asp8200 asp8200 added the enhancement New feature or request label May 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant