[CE] Refactoring of the topic tree structures and a performance improvement #373

Remit · 2022-12-19T14:08:32Z

Tickets

Motivation

Improves performance when dispatching PUBLISH packets to shared subscriptions. Refactors structures used by topic tree for better developer experience.

Changes

Refactored structure of the topic tree.

Performance improvement when dispatching publishes to shared subscriptions.

src/main/java/com/hivemq/metrics/HiveMQMetrics.java

src/main/java/com/hivemq/mqtt/topic/SubscriberWithQoS.java

src/main/java/com/hivemq/extensions/services/publish/PublishServiceImpl.java

src/main/java/com/hivemq/extensions/services/subscription/SubscriptionStoreImpl.java

src/main/java/com/hivemq/persistence/clientsession/SharedSubscriptionServiceImpl.java

src/test/java/com/hivemq/extensions/services/publish/PublishServiceImplTest.java

src/test/java/com/hivemq/extensions/services/subscription/SubscriptionStoreImplTest.java

src/test/java/com/hivemq/mqtt/InternalPublishServiceImplTest.java

src/test/java/com/hivemq/mqtt/services/PublishPollServiceImplTest.java

src/test/java/com/hivemq/mqtt/topic/tree/TopicTreeStartupTest.java

src/test/java/com/hivemq/persistence/clientqueue/ClientQueuePersistenceImplTest.java

.../java/com/hivemq/persistence/clientsession/ClientSessionSubscriptionPersistenceImplTest.java

src/test/java/com/hivemq/persistence/clientsession/SharedSubscriptionServiceTest.java

src/main/java/com/hivemq/mqtt/topic/tree/MatchingNodeSubscriptions.java

src/main/java/com/hivemq/mqtt/topic/tree/TopicTreeImpl.java

src/main/java/com/hivemq/metrics/HiveMQMetrics.java

src/main/java/com/hivemq/mqtt/topic/tree/TopicTreeImpl.java

…est practice for indirect null checks.

…ns when populating using a filter.

… the tree.

…and going through the nodes.

…eIfAbsent().

…rray as it was not returned.

@mock

* update version to 2022.2-SNAPSHOT * Update dependency versions (hivemq#346) * update Netty and Jackson version * update dependency check plugin * update dependencies (hivemq#347) * update not critical dependencies * refactored FutureUtils * Refactor DiagnosticModeTest to use mock(...) instead of @mock. * Add SystemProperties.DIAGNOSTIC_MODE replacing diagnosticMode * Fix SIGSEGV during shutdown with enabled diagnostics mode This was caused by DiagnosticMode's ConsoleReporter which periodically writes metrics to disk and must not access any RocksDB instance after it has been closed already. Therefore, stop DiagnosticMode and its ConsoleReporter's executore first via ShutdownHooks. * fixed extension priority comparator for multiple entries with same priorities. * Reduce Garbage Collection during incoming subscribe by removing wrapper classes like ClientToken around ClientConnection. * fix duplicate extension starts (hivemq#361) * fix naming of extension task executor (hivemq#362) * add XSD for broker config.xml * add missing license header * copy xsd file to conf folder on packaging * move defaults to documentation to prevent confusing empty values * Refactor SslContextStore * Fix not sending will on clean start during re-connect If a client re-connects with a clean start, the specification defines that this "ends" and existing sessions. Whenever a session ends, the specification defines that the ending session's will must be sent, if available and still pending due to a session expiry interval or a will delay interval that has not been reached yet. This is achieved by sending any still pending will between disconnecting and (re-)connecting. Also improved method names of PendingWillMessages. * Send any pending wills on session expiry Don't rely on the CheckWillsTask and ClientSessionCleanUpTask to be scheduled well enough that there can never be a situation where there is still a will send pending while the session, which contains the will data, has already been cleaned up. * Log a warning if a will can't be sent because the session is gone For testing purposes, assert that this never happens because it is unexpected: We must guarantee that a will is sent before the session is gone/cleaned up. To detect unknown race conditions in production, log a warning and don't throw a NullPointerException, as this might block client connections and the sending of other wills. * Fix will count metrics on client re-connect or takeover with cleanStart=true * Fix will message disk leak on client re-connect or takeover Added commit hooks to update will message references whenever a client session is removed or overwritten. * Reduce used message IDs memory footprint Switched from HashSet<Integer> to Eclipse Collections' ShortHashSet to reduce the memory footprint. We can use ShortHashSet instead of IntHashSet because the range of IDs, 1..65535, is unsigned short anyway, and we can shift those int values into signed short range. * Removed obsolete @author comment * Fix leak of CONNECT in ClientConnection if no plugin initializers exist * [CE] Refactoring of the topic tree structures and a performance improvement (hivemq#373) * Refactoring of the topic tree and performance improvement for mapping PUBLISHes to the shared subscriptions. Co-authored-by: A-Imal <Abdullah.Imal@hivemq.com> * Choose shared subscriber via randomly-ordered iteration Avoid shuffling a copy of the collection to reduce CPU load. * Remove desiredExtensionClass parameter from extension loading * Catch errors thrown by ExtensionMain classes during extension loading * Reduce BucketLock contention by RemoveEntryTask Added a guard so that concurrently running RemoveEntryTasks don't try to process entries of the shared removablePayloads queue. Since this is more of a workaround, the structure should be improved in the scope of the ticket #11000. * Remove TODO Ticket number is enough. * Add TRACE logs for 'no matching subscriber' case * Refactoring of the utils (hivemq#378) Removed utils: ChannelAttributes ChannelUtils ClientSessions DefaultSslEngineUtil MacAddressFormatter PhysicalMemoryUtil PublishUtil ResourceUtils TemporaryFileUtils UcaughtExceptionHandler Validators * Remove payloadCache Leave the caching to RocksDB. * Refactor code, remove warnings * Fix and extend Exception handling test cases * renamed a class (hivemq#382) * Code refactoring, formatting * Catch Throwable to guarantee re-scheduling of CleanUpTask * Add timeout to CleanUpTask, schedule next once timed out Avoids situations where the cleanup seems stuck in case any persistence cleanup takes a long time or is actually stuck because of a bug. * Debug-log CancellationException of timeouts * fix reason string in event log * Version to 2023.1 * Version to 2023.1 in README.adoc * Update third party licenses * Remove sdk checkout from publish workflow (use release sdk version) * Remove redundant addition, use preconditions * Remove warnings * Fix a race of polling duplicate messages, overwriting futures Might happen if previous sends finish concurrently and decrement the in-flight counter to 0. This would trigger message polling which overwrites existing futures of previous polling(s). * Share UnsubscribeHandler between channels/client connections * Remove warnings, reformat code * refactoring * fixing circular dependency and broken unit test * Fix shields.io build badge * Update README.adoc Co-authored-by: Silvio Giebl <silvio.giebl@hivemq.com> * Introduced and applied .editorconfig. * Optimize imports. * Remove connectionAttributesMutex which can be avoided by making the set method synchronized from the ClientConnection. * Pass-through the cleanStart flag directly to EventLog. * Introduced ClientConnectionContext which is used for client entry points like Netty handlers where the state of the client can be undefined (connecting). * Avoid creating multiple Netty attributes for ClientConnectionContext. * Annotate fields in ClientConnection with NotNull when we are sure that they are always available. * Remove unused methods in UndefinedClientConnection. * Formatting. * Remove unused CONNACKs from ConnackMessages. * Move the builder of CONNACK out of the class for a better overview and isolation of the classes. * Use for any instance creation of CONNACK the builder class. Removes all other constructors. * Move the precondition check of CONNACK to the build method. * Fix failing tests. * Formatting. * Version to 2023.2 * Fix race where a QoS 0 payload gets deleted concurrently during polling (hivemq#394) Previously, the payload reference counter was decremented immediately after polling the QoS 0 PUBLISH from its queue, but before assigning the payload to the PUBLISH again. This introduced a race condition with the payload cleanup which was then permitted to delete the payload concurrently. Therefore, assign the payload during lookup to fix the race. Additionally, drop messages without payload immediately during iteration; don't schedule another single writer task. * LocalTopicTree > Avoid iterating over the root wildcards when there is nothing to remove. * Version to 2023.3 * [CE] Feature-id 14176: TLS log messages port (hivemq#400) * added information to tls logs * fix license header * remove wrong since * Refactoring: Remove warnings * Fix IntelliJ typo warnings * Use exclusive and readonly transactions - Exclusive transactions can be used by these methods because all calls are serialized by the single writer - Some transactions are only read-only * Move metrics, payload and event log updates to transaction hooks This makes changes only visible after the transaction succeeds. Avoids intermediate state during e.g. loops in a transaction. * Refactoring: Remove warnings * Fix IntelliJ typo warnings * Make removeWithTimestamp only visible for testing There is no public calling/use case. * Use single Runnable commit hook where possible * Update session counters via accumulated delta * Fix wrong session count on start with persistent session * Version to 2023.4 * Update dependencies (hivemq#404) * Update gradle to 7.4.2 * Update io.github.sgtsilvio.gradle.utf8 -> defaults * Cleanup test logging * Replace license-third-party-tool with buildSrc plugin * Update third party licenses * CVE-2023-2976, Guava update to recommended version 32.0.1-jre * CVE: CVE-2023-34462, Netty update to recommended version 4.1.94.Final * reauth > fix missing client state update * Update version (hivemq#413) * Formatting and cleanup for better readability. * RocksDBLocalPersistence > Reduce number of returns. * Fix memTable typo. * RetainedMessagePersistence > Remove unnecessary try-catch guards. * ClientQueueLocalPersistence > Removed unused qos0Size method. * reference counting > Remove unnecessary delta arguments. * RetainedMessageLocalPersistence > Cleanup. * Fix resource leakage. * reference counting > Add and decrement payloads for queue entries in the ClientQueuePersistence. * reference counting > Remove workarounds for retained message payloads which required before some retries until the payloads were persisted on the disk. Currently, we are persisting the payloads in the same single writer tasks to the persistence which guarantees us to find the payload when we can find the retained message itself. * PublishPayloadXodusLocalPersistence > Use exclusive transaction for write operations. * Remove unused PayloadCacheRemovalListener. * PayloadPersistence > Cleanup and formatting. * PayloadPersistence > Make methods in PublishPayloadXodusSerializer static. * PublishPayloadPersistence > Remove test only method from the interface. * PublishPayloadPersistence > Rework the payload cleanup job: Threads are assigned to buckets for which they are responsible. * reference counting > Remove usage of PayloadPersistenceException when payload was not found. * reference counting > Removed any calls to the payload persistence for retained messages, all adds and removes of payloads are only performed in the RetainedMessageLocalPersistence. * Fix RetainedMessageRocksDBLocalPersistence not adding the payload when replacing an existing retained message. * Fix EmbeddedHiveMQ leakage in test code. * Update gradle to 7.6.2 * Update io.github.sgtsilvio.gradle plugins * Update bouncycastle jdk15on -> jdk18on * Bugfix for expired inflight PUBREL not getting removed from the queue (hivemq#417) Expired inflight PUBREL gets default values for publish timestamp and message expiry to avoid stuck message queue; added inflight PUBREL expiry config. * Version to 2023.6 * Update base image to eclipse temurin (hivemq#419) * Version to 2023.7 * [CE] An NPE is thrown if port 1883 is in use when HiveMQ CE is started (hivemq#424) Co-authored-by: Henning Lohse <henning.lohse@hivemq.com> * Version to 2023.8 * Use version catalog (hivemq#425) * Ranged IDs implementation for packet IDs (hivemq#426) * Ranged IDs implementation for assigning packet IDs * [1429] A failed CONNACK should be sent, when a CONNECT packet exceed the max packet size version 2 (hivemq#423) Co-authored-by: Mikhail Chernyakov <mikhail.chernyakov@hivemq.com> Co-authored-by: Vladimir Podolskiy <v.e.podolskiy@gmail.com> * NoMessageIdAvailableException is thrown even if there are free ids available in the pool (hivemq#428) * FreePacketIdRanges > Making class thread-safe and fixing an edge case when an empty id range could become a root range. * Update version to 2023.9 (hivemq#430) * Update third party licenses (hivemq#431) * Logging > The current TurboFilterList implementation of logback-classic which calls the TurboFilters is a CopyOnWriteArrayList. This means every time any logger method is called the array of TurboFilters is copied. As we are using the logger extensively this small copy is visible in the flamegraph. A call to isXLevelEnabled check does not avoid this path. * make JMX port configurable * JMX_PORT -> HIVEMQ_JMX_PORT * Update logback, update apache-commmons-compress * Version to 2023.10 * Logging > Register our TurboFilter only once. During the HiveMQ start we load our logback.xml which will reset the current context and call the reset which will register the TurboFilter a second time. * Logging > corrected and added logged statements about logging configuration * LoggingBootstrap > formatting * LoggingBootstrap > formatting * SingleWriterService > Handle all exceptions that can be thrown during the task execution. This allows all futures to finish and issues being visible. * fixed missing empty string handling for MQTT3 Publish topic and MQTT5 Publish response topic * added empty string handling for MQTT5 will response topic and improved logging and reason strings. * Version to 2024.1 * Fixed mac build * Updated version --------- Co-authored-by: Daniel Krueger <daniel.krueger@hivemq.com> Co-authored-by: Daniel Krüger <45030029+DC2-DanielKrueger@users.noreply.github.com> Co-authored-by: Lukas Brandl <lukas.brandl@dc-square.de> Co-authored-by: Henning Lohse <henning.lohse@hivemq.com> Co-authored-by: flimpoec <florian.limpoeck@hivemq.com> Co-authored-by: A-Imal <Abdullah.Imal@hivemq.com> Co-authored-by: Christoph Schaebel <christoph.schaebel@hivemq.com> Co-authored-by: Mario Schwede <mario.schwede@hivemq.com> Co-authored-by: Vladimir Podolskiy <v.e.podolskiy@gmail.com> Co-authored-by: David Sondermann <david.sondermann@hivemq.com> Co-authored-by: Michael Walter <michael.walter@dc-square.de> Co-authored-by: gitseti <till.seeberger@hivemq.com> Co-authored-by: Silvio Giebl <silvio.giebl@hivemq.com> Co-authored-by: Peter Giacomo Lombardo <pglombardo@hey.com> Co-authored-by: Yonathan Negussie Mengesha <yonathan.mengesha@hivemq.com> Co-authored-by: Antonio Fernandez Alhambra <antonio.alhambra@hivemq.com> Co-authored-by: tgracchus <85102174+tgracchus@users.noreply.github.com> Co-authored-by: Mikhail Chernyakov <mikhail.chernyakov@hivemq.com> Co-authored-by: yannick <yannick.weber@hivemq.com> Co-authored-by: Christoph Kempfer <christoph.kempfer@adesso.de>

topic tree refactoring and bugfix

1fb3e77

cla-bot bot added the cla-signed label Dec 19, 2022

fix

d922533

Remit commented Dec 19, 2022

View reviewed changes

src/main/java/com/hivemq/metrics/HiveMQMetrics.java Outdated Show resolved Hide resolved

fix

2bb7c39

Remit requested review from LBrandl and hlohse December 19, 2022 14:29

Remit self-assigned this Dec 19, 2022

LBrandl reviewed Dec 19, 2022

View reviewed changes

src/main/java/com/hivemq/mqtt/topic/SubscriberWithQoS.java Show resolved Hide resolved