Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: allow multithreaded fmindex construction #208

Merged
merged 3 commits into from
May 9, 2023
Merged

feat: allow multithreaded fmindex construction #208

merged 3 commits into from
May 9, 2023

Conversation

SGSSGene
Copy link
Collaborator

  • updates to newest fmindex
  • uses fmindex_collection now uses libsais instead of libdivsufsort
  • support for openmp
  • adds option to control the thread count for suffix array construction

h-2
h-2 reviewed Apr 25, 2023
View reviewed changes
Copy link
Member

@h-2 h-2 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks for the PR!

I think you still need to add libsais to the your library's cmake (or ours).

src/mkindex_algo.hpp Outdated Show resolved Hide resolved
@h-2
Copy link
Member

h-2 commented Apr 25, 2023

Do you know if memory usage changes between libdivsufsort and libsais? Currently, the memory requirements of lambda3's index construction are almost twice as high as lambda2's 😑

@SGSSGene SGSSGene force-pushed the feat/threaded_fmindex_construction branch from cfaf5e3 to eddb7c3 Compare April 26, 2023 14:59
@SGSSGene
Copy link
Collaborator Author

Do you know if memory usage changes between libdivsufsort and libsais? Currently, the memory requirements of lambda3's index construction are almost twice as high as lambda2's expressionless

No, I don't have any clue. I also don't know what influence the thread number will have on the memory consumption.

@SGSSGene SGSSGene force-pushed the feat/threaded_fmindex_construction branch 5 times, most recently from 31fbc60 to 35c9bf9 Compare April 27, 2023 14:31
@SGSSGene SGSSGene marked this pull request as ready for review April 27, 2023 14:51
@SGSSGene SGSSGene requested a review from h-2 April 27, 2023 17:53
SGSSGene and others added 3 commits May 3, 2023 16:41
@SGSSGene
fix: update to newest fmindex_collection
7c85a9b 
- This also changes the way that a search with an reversed index is
working, so no reversing of the text is required anymore.

- Requires to set the numbers of thread used for suffix array
  construction.
@SGSSGene @h-2
Update src/mkindex_algo.hpp
ed7a5a2 
Co-authored-by: Hannes Hauswedell <h2+github.com@fsfe.org>
@SGSSGene
fix: update fmindex_collection submodule
7b08976
@SGSSGene SGSSGene force-pushed the feat/threaded_fmindex_construction branch from ffccda7 to 7b08976 Compare May 3, 2023 14:41
@SGSSGene SGSSGene mentioned this pull request May 9, 2023
Merged
@sarahet sarahet merged commit 7b08976 into seqan:lambda3 May 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@h-2 h-2 Awaiting requested review from h-2

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@SGSSGene @h-2 @sarahet