Please sign in to comment.
EXPERIMENTAL: midx3 format with separate prefix/postfix searching.
This new midx format doesn't take any extra space, but it reduces the number of page faults we'll take in the common case (common, at least, when doing a fresh new backup) because we can pack more sha1 prefixes into each page, and we can stop searching if a particular sha1 prefix isn't actually found in the list. That means we won't have to ever read from disk the remaining sha1 bytes for final comparison. On the other hand, if a sha1 *does* exist, we now have to fault in three pages (fanout + prefix + postfix) instead of two (fanout + sha1), which *may* cause more churn in low memory situations. On the other hand, in bup, churning through large numbers of already-existing sha1s should be a much less common case (since 'bup save' short circuits at the top level of any directory tree that's unchanged). Moreover, 'bup server' should benefit almost always, since clients will almost never send objects it already has. That means, on average, this patch should save us a whole bunch of RAM. 'bup memtest' results show that memory usage decreases to about 1/3 of the old results, which is pretty good. Of course, 'bup memtest --existing' results are essentially unchanged, since it still has to swap in the whole midx file.
- Loading branch information...
Showing with 106 additions and 35 deletions.