-
Notifications
You must be signed in to change notification settings - Fork 42
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Variant calling using sfm
does not adhere to thread limit
#56
Comments
Hi Matthias, How are you observing that more threads are used by elprep? (I double checked the source code, which seems fine, and I cannot reproduce the behaviour). Thanks! |
I was monitoring CPU usage through I'll rerun the analysis and see of this occurs again. |
If possible, can you run your command with /usr/bin/time -v (e.g. /usr/bin/time -v elprep sfm input.bam output.bam) and send me the output? Thanks! |
Good thing I'm benchmarking right now!
|
Hmm that seems ok. Any chance the extra threads you see are not from the elprep command? |
Elprep was the only thing running on the node at that moment. |
Elprep log
|
I think the problem might be there are too many system calls blocked in this specific case. It seems that the go runtime continues to launch threads in that case, though only 10 goroutines (threads) should actually ever be running in parallel. So if I understand it correctly, it should not be an issue so many threads are spawned, and it is why we don't see a bigger cpu usage than expected. I am unsure if I can prevent the go runtime to do this. I have to look into this a bit deeper. |
I'm currently re-running with tmp path set to a faster FS, that should help the blocked calls, no? |
No dice, same result, but to a lesser extent. A slow(ish) FS seems to be the culprit in this case |
That makes sense. Are you restricting the nr of threads to run multiple jobs on the server? |
Currently, yes. I'm running the same sample with different parameters to find the sweet spot between performance and speed specific to our hardware. Atm I've noticed that throwing more resources at elprep does not scale linearly, with only about 2min speed gains with double the resources. |
Hi,
I noticed that when using 'sfm' mode, elprep uses more resources than allowed (e.g. all available threads instead of the given thread count). When using
filter
, this does not occur.Cheers
Matthias
The text was updated successfully, but these errors were encountered: