Uncycler 0.4.6.0 failing at PCS Bridges, Trinity runs fine #185
Comments
jennaj
added
bug
functionality usegalaxy.org
|
ping @natefoo -- any updates about what may be going wrong? |
|
Still getting a server error. Any updates? |
|
@eringaf We just added this issue to our next weekly list of priorities. I would expect a solution near-term, possibly even sometime next week. We'll post updates here. Thanks! |
|
Status: still failing |
|
xref prior issue ticket (PCS Bridges problems impacting both Trinity & Unicycler) to avoid end-user confusion about Unicycler's most current status: #176 |
|
EDIT: struck some stuff that's incorrect lest I mislead future readers. See followup comments. This is an issue with the resources configured for SPAdes by Unicycler, how SPAdes uses those limits, The Shortly afterward, it This works up until the kernel returns
Pulling apart Unicycler, you can run its call to SPAdes by hand and set Which gets it a little further, but it ends in a segfault: Digging through the syscalls at this point shows it's roughly the same thing, this time the child processes are If you increase the amount of memory given to The real solution is to figure out why SPAdes is doing such large anonymous 1 Not sure why it's 11, as per the Bridges User Guide, the LM partition should be allocating 1 core per 48 GB of memory requested, so I expect it to be 10. |
|
Thank you for your precious help and support. Really hope this problem could be fixed soon. I have a lot of sequences to be analyzed by Unicycler... |
|
thanks @natefoo for looking into this @suzukimicro Please see the workaround at the very top of this ticket. In short, the tool is functional at Galaxy EU https://usegalaxy.eu based on tests with smaller sample data. Please consider running your jobs there for now. |
|
Update: It looks like my previous conclusion was partially incorrect: cgroups are unrelated to the problem. With the cgroup memory limits as they are set on Bridges, the OS will happily allocate an amount of virtual memory larger than the limit. The issue is in fact with SPAdes' usage of lm002 (3TB, 64 cores): login005 (128GB, 28 cores): In both cases, I thought maybe the issue was with SPAdes' use of jemalloc, since it bases the number of "arenas" on the number of cores (although not the total system memory, as far as I can tell), but after compiling SPAdes without jemalloc ( So, I still need to figure out what is responsible for the size of those |
It's login005: $ ulimit -s
24576
$ strace -e trace=getrlimit,mmap spades-hammer ...
getrlimit(RLIMIT_STACK, {rlim_cur=24576*1024, rlim_max=4194304*1024}) = 0
mmap(NULL, 25169920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fd007bfe000l002: $ ulimit -s
31457280
$ strace -e trace=getrlimit,mmap spades-hammer ...
getrlimit(RLIMIT_STACK, {rlim_cur=31457280*1024, rlim_max=31457280*1024}) = 0
mmap(NULL, 32212258816, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = -1 ENOMEM (Cannot allocate memory)
$ ulimit -s 24576
$ strace -e trace=getrlimit,mmap spades-hammer ...
getrlimit(RLIMIT_STACK, {rlim_cur=24576*1024, rlim_max=24576*1024}) = 0
mmap(NULL, 25169920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fb226783000We can fix this by setting I'll test this out and update once there's a fix in place. |
Workaround for end-users: Use the Galaxy EU https://usegalaxy.eu server until the Galaxy Main https://usegalaxy.org server is fixed and this ticket closed out.
Important part of error seems to be after prep steps, when the assembly is actually starting
Tool ID: toolshed.g2.bx.psu.edu/repos/iuc/unicycler/unicycler/0.4.6.0
Test histories usegalaxy.ORG
Test history usegalaxy.EU (for comparison) -- Update: Tool works at EU
GUI "Bug" message
GUI "Info" message
test1 full error report
test2 full error report
The text was updated successfully, but these errors were encountered: