-
Notifications
You must be signed in to change notification settings - Fork 78
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
remove_many
causes prefetch to hang on large-ish metagenome samples
#1552
Comments
Great find! Maybe the question here is "do we want to report this?", or only report it if The true fix is making related to #1517 (in the sense of "we have issues with gigantic signatures" =]) |
I'd be happy to change prefetch to either not report this, OR to use python sets underneath for the moment. I think the latter preserves nice functionality without triggering this particular problem, and doesn't cause new problems. |
@bluegenes I think you can try again after merging #1571 - I'm curious to see the effect 😄 |
Thanks for your efforts, @mr-eyes!! Unfortunately, it's still quite long -- less than a minute for the
This is likely only a huge issue when using a very large query sig with quite a small search database, but this seems like a use case we want to support... |
@ctb could we have prefetch not report (or just optionally report)? |
I'd rather fix the problem, which we know how to fix :). Just needs a little TLC. |
ah, didn't realize the fix above wasn't the whole picture |
Ahh, that's because I didn't put it in this issue... let me see if it's in the issues anywhere... OK, it's this one, over in genome-grist: dib-lab/genome-grist#32 We should remove the unknown hashes as soon as we can, after the prefetch and before the gather. The challenge is doing the right accounting in the |
Note to all involved (myself included) - we actually needed to change the code in |
example command:
When running via slurm, I noticed a prefetch on this sample wrote csv results in the first ~ minute, but then kept running (killed at ~ 1hr). When I ran via the command line, I noticed the job hung after this output, and could not be exited with
Ctrl-C
With a dev install, I confirmed that commenting out
remove_many
(https://github.com/dib-lab/sourmash/blob/latest/src/sourmash/commands.py#L1169) enablesprefetch
to finish normally.Note that prefetch does not hang when used within
gather
(though I do see now thatremove_many
is only used when callingprefetch
directly from the command line.Making the sig tiny or the searched database tiny (by downsampling) eliminated this issue, enabling the hash counting notifications to print, e.g.
For now, the sig and related databases can be found on our
farm
hpc at/home/ntpierce/prefetch_hangtest
.The text was updated successfully, but these errors were encountered: