CRC32C for bookkeeper #590
Labels
Comments
athanatos
pushed a commit
to athanatos/bookkeeper
that referenced
this issue
Jan 25, 2019
…e watches scenario The current HashSet based WatcherManager will consume more than 40GB memory when creating 300M watches. This patch optimized the memory and time complexity for concentrate watches scenario, compared to WatchManager, both the memory consumption and time complexity improved a lot. I'll post more data later with micro benchmark result. Changed made compared to WatchManager: * Only keep path to watches map * Use BitSet to save the memory used to store watches * Use ConcurrentHashMap and ReadWriteLock instead of synchronized to reduce lock retention * Lazily clean up the closed watchers Author: Fangmin Lyu <allenlyu@fb.com> Reviewers: Andor Molnár <andor@apache.org>, Norbert Kalmar <nkalmar@yahoo.com>, Michael Han <hanm@apache.org> Closes apache#590 from lvfangmin/ZOOKEEPER-1177
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
JIRA: https://issues.apache.org/jira/browse/BOOKKEEPER-951
Reporter: Venkateswararao Jujjuri (JV) @jvrao
CRC32C is hardware based CRC32 and it substantially improves performance and practically offloads CPU.
[~merlimat] at yahoo implemented this for Pulsar, opening this item to port that change.
https://github.com/yahoo/pulsar/tree/master/pulsar-checksum
Java 9 has CRC32C native support but we need this JNI patch to achieve it on Java 8.
The text was updated successfully, but these errors were encountered: