-
-
Notifications
You must be signed in to change notification settings - Fork 368
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RFC] Split reflists to share their contents across snapshots #1282
base: master
Are you sure you want to change the base?
Conversation
70035b8
to
7abea61
Compare
7abea61
to
4adca84
Compare
@refi64 could you have a look at https://github.com/aptly-dev/aptly/actions/runs/9429650201/job/25976276670?pr=1282#step:9:438 ?
|
4adca84
to
df6e811
Compare
In some local tests w/ a slowed down filesystem, this massively cut down on the time to clean up a repository by ~3x, bringing a total 'publish update' time from ~16s to ~13s. Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
In current aptly, each repository and snapshot has its own reflist in the database. This brings a few problems with it: - Given a sufficiently large repositories and snapshots, these lists can get enormous, reaching >1MB. This is a problem for LevelDB's overall performance, as it tends to prefer values around the confiruged block size (defaults to just 4KiB). - When you take these large repositories and snapshot them, you have a full, new copy of the reflist, even if only a few packages changed. This means that having a lot of snapshots with a few changes causes the database to basically be full of largely duplicate reflists. - All the duplication also means that many of the same refs are being loaded repeatedly, which can cause some slowdown but, more notably, eats up huge amounts of memory. - Adding on more and more new repositories and snapshots will cause the time and memory spent on things like cleanup and publishing to grow roughly linearly. At the core, there are two problems here: - Reflists get very big because there are just a lot of packages. - Different reflists can tend to duplicate much of the same contents. *Split reflists* aim at solving this by separating reflists into 64 *buckets*. Package refs are sorted into individual buckets according to the following system: - Take the first 3 letters of the package name, after dropping a `lib` prefix. (Using only the first 3 letters will cause packages with similar prefixes to end up in the same bucket, under the assumption that packages with similar names tend to be updated together.) - Take the 64-bit xxhash of these letters. (xxhash was chosen because it relatively good distribution across the individual bits, which is important for the next step.) - Use the first 6 bits of the hash (range [0:63]) as an index into the buckets. Once refs are placed in buckets, a sha256 digest of all the refs in the bucket is taken. These buckets are then stored in the database, split into roughly block-sized segments, and all the repositories and snapshots simply store an array of bucket digests. This approach means that *repositories and snapshots can share their reflist buckets*. If a snapshot is taken of a repository, it will have the same contents, so its split reflist will point to the same buckets as the base repository, and only one copy of each bucket is stored in the database. When some packages in the repository change, only the buckets containing those packages will be modified; all the other buckets will remain unchanged, and thus their contents will still be shared. Later on, when these reflists are loaded, each bucket is only loaded once, short-cutting loaded many megabytes of data. In effect, split reflists are essentially copy-on-write, with only the changed buckets stored individually. Changing the disk format means that a migration needs to take place, so that task is moved into the database cleanup step, which will migrate reflists over to split reflists, as well as delete any unused reflist buckets. All the reflist tests are also changed to additionally test out split reflists; although the internal logic is all shared (since buckets are, themselves, just normal reflists), some special additions are needed to have native versions of the various reflist helper methods. In our tests, we've observed the following improvements: - Memory usage during publish and database cleanup, with `GOMEMLIMIT=2GiB`, goes down from ~3.2GiB (larger than the memory limit!) to ~0.7GiB, a decrease of ~4.5x. - Database size decreases from 1.3GB to 367MB. *In my local tests*, publish times had also decreased down to mere seconds but the same effect wasn't observed on the server, with the times staying around the same. My suspicions are that this is due to I/O performance: my local system is an M1 MBP, which almost certainly has much faster disk speeds than our DigitalOcean block volumes. Split reflists include a side effect of requiring more random accesses from reading all the buckets by their keys, so if your random I/O performance is slower, it might cancel out the benefits. That being said, even in that case, the memory usage and database size advantages still persist. Signed-off-by: Ryan Gonzalez <ryan.gonzalez@collabora.com>
needed by: - deb/reflist.go:431:15: min requires go1.21 or later - deb/reflist.go:720:31: cannot convert digest (variable of type []byte) to type reflistDigestArray: conversion of slices to arrays requires go1.20 or later
df6e811
to
c58a634
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1282 +/- ##
==========================================
+ Coverage 74.49% 74.77% +0.28%
==========================================
Files 146 146
Lines 16540 17096 +556
==========================================
+ Hits 12321 12784 +463
- Misses 3241 3311 +70
- Partials 978 1001 +23 ☔ View full report in Codecov by Sentry. |
0c32ca5
to
6e5b3fd
Compare
tests fixed, made compatible with go 1.19 (and current debian/bookworm) |
Sorry for the delays, I've been a bit occupied elsewhere lately 😅 I had looked into the nil errors in the tests, but it seemed to specifically come from the way the tests were initializing the repos; normally they're created and then have something loaded into them, but the tests create them and immediately start calling methods w/o any load. Changing that around also makes the tests pass and looks like this:
I'm...not sure which approach is better, really? My concern with just making |
Replaces #1235
This builds on top of #1222, #1227, and #1233, and is thus in draft state until those are merged in. The only commit that's actually new is the very last one, whose commit message I copied below. (Even that single commit is admittedly quite big, but a sizable chunk of the changes are just plumbing a new
RefListCollection
around across the code.)Description of the Change
In current aptly, each repository and snapshot has its own reflist in
the database. This brings a few problems with it:
get enormous, reaching >1MB. This is a problem for LevelDB's overall
performance, as it tends to prefer values around the confiruged block
size (defaults to just 4KiB).
full, new copy of the reflist, even if only a few packages changed.
This means that having a lot of snapshots with a few changes causes
the database to basically be full of largely duplicate reflists.
loaded repeatedly, which can cause some slowdown but, more notably,
eats up huge amounts of memory.
time and memory spent on things like cleanup and publishing to grow
roughly linearly.
At the core, there are two problems here:
Split reflists aim at solving this by separating reflists into 64
buckets. Package refs are sorted into individual buckets according to
the following system:
lib
prefix. (Using only the first 3 letters will cause packages with
similar prefixes to end up in the same bucket, under the assumption
that packages with similar names tend to be updated together.)
relatively good distribution across the individual bits, which is
important for the next step.)
buckets.
Once refs are placed in buckets, a sha256 digest of all the refs in the
bucket is taken. These buckets are then stored in the database, split
into roughly block-sized segments, and all the repositories and
snapshots simply store an array of bucket digests.
This approach means that repositories and snapshots can share their
reflist buckets. If a snapshot is taken of a repository, it will have
the same contents, so its split reflist will point to the same buckets
as the base repository, and only one copy of each bucket is stored in
the database. When some packages in the repository change, only the
buckets containing those packages will be modified; all the other
buckets will remain unchanged, and thus their contents will still be
shared. Later on, when these reflists are loaded, each bucket is only
loaded once, short-cutting loaded many megabytes of data. In effect,
split reflists are essentially copy-on-write, with only the changed
buckets stored individually.
Changing the disk format means that a migration needs to take place, so
that task is moved into the database cleanup step, which will migrate
reflists over to split reflists, as well as delete any unused reflist
buckets.
All the reflist tests are also changed to additionally test out split
reflists; although the internal logic is all shared (since buckets are,
themselves, just normal reflists), some special additions are needed to
have native versions of the various reflist helper methods.
In our tests, we've observed the following improvements:
GOMEMLIMIT=2GiB
, goes down from ~3.2GiB (larger than the memorylimit!) to ~0.7GiB, a decrease of ~4.5x.
In my local tests, publish times had also decreased down to mere
seconds but the same effect wasn't observed on the server, with the
times staying around the same. My suspicions are that this is due to I/O
performance: my local system is an M1 MBP, which almost certainly has
much faster disk speeds than our DigitalOcean block volumes. Split
reflists include a side effect of requiring more random accesses from
reading all the buckets by their keys, so if your random I/O
performance is slower, it might cancel out the benefits. That being
said, even in that case, the memory usage and database size advantages
still persist.
Checklist
AUTHORS
It would be awesome if anyone could also test this out and report how it affects their performance & memory usage.