Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restructure geometry reordering to try to reduce virtual memory thrashing #202

Merged
merged 24 commits into from
Apr 5, 2016
Merged

Restructure geometry reordering to try to reduce virtual memory thrashing #202

merged 24 commits into from
Apr 5, 2016

Conversation

e-n-f
Copy link
Contributor

@e-n-f e-n-f commented Apr 4, 2016

No description provided.

e-n-f added 24 commits March 31, 2016 15:29
@e-n-f
OS-specific code to determine memory size
9acecdf
@e-n-f
Reunify the string pool and metadata earlier to free up file descriptors
24deae5
@e-n-f
More calculation about how many fds are available
560add2
@e-n-f
Merge branch 'master' into radix
851d7b5
@e-n-f
Split indices and geometries up by radix
9793be1
@e-n-f
Provide top-level output files for geometry and index
6013953
@e-n-f
Slow, but works for some cases now
cf806ba
@e-n-f
Stop recursing when the indices are exactly the same
55f9396
@e-n-f
Restore mergesort for the intermediate sort. Tests pass.
e2b36a8
@e-n-f
Rework sorting/merging progress indicator
986719f
@e-n-f
Keep meta count inline to avoid thrash if no attributes
c9e542e
@e-n-f
Fix the any-valid-geometries check again
7b6cd2d
@e-n-f
Remove extra newline output from old progress format
2fe841c
@e-n-f
Remove dead code
31c4d48
@e-n-f
Fix the progress indicator when there is an inner radix sort
d4b9f79
@e-n-f
Fix the status message about how much geometry there is
2b0ac89
@e-n-f
Sprinkle madvise calls to hint about how files should be paged in
b1d3849
@e-n-f
Also advise when unmapping pages
72e485f
@e-n-f
Can't tell Linux to free pages, so just say we don't need them
c627620
@e-n-f
Parallel reading is effectively random order, not sequential
2a4be4f
@e-n-f
Use large sort chunks, since that part can be parallelized
c0a0aef
@e-n-f
Check for errors when closing files
e394501
@e-n-f
Add a way to test recursive radix sorting. Bump version number.
b10b436
@e-n-f
Close some file descriptors that were left dangling before
4a572b8
@e-n-f e-n-f merged commit d1456c0 into master Apr 5, 2016
@e-n-f e-n-f deleted the radix branch April 5, 2016 22:13
@e-n-f e-n-f mentioned this pull request Apr 5, 2016
Closed
@e-n-f e-n-f mentioned this pull request Apr 14, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@e-n-f