Merged
Conversation
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
cache_payloads was single-threaded despite the -t flag; with 164M nodes on an HPRC graph it hung for hours. Two fixes: 1. Pass `true` to for_each_handle to enable OpenMP parallelism; guard the non-thread-safe writes (oversized_zipcodes vector and node_id_to_payload map) with named omp critical sections. 2. Call distance_index->preload(true) immediately before cache_payloads in build_minimizer_index. find_frequent_kmers runs for ~3300 s before this point and evicts the mmap'd index pages, causing a page fault on every snarl-tree lookup in fill_in_zipcode_from_pos. Reloading here ensures the index is warm when the parallel loop starts. Also add a depth guard (abort at >10000) in fill_in_zipcode_from_pos to catch any future infinite loops in the snarl tree traversal. Also use distance_index.get_snarl_child_count() (O(1) record read) instead of for_each_child iteration in get_regular/irregular_snarl_code. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Member
Author
|
This isn't really going to pass tests unless we can use |
Member
Author
|
OK the only problem now is the lack of |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds @electricEpilith's parallel payload caching to #4857 (which shouldn't merge until after this).