AMQ-7091 - O(n) Memory consumption when broker has inactive durable s… #315
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ubscribes causing OOM
roelofjansen
added a commit
to portofrotterdam/activemq
that referenced
this pull request
Nov 22, 2019
* AMQ-7067 - add missing license header * NO-JIRA - add missing license headers * AMQ-7086 - make kahadb gc/cleanup on shutdown optional to trade availability over disk usage for fast failover * AMQ-7088 - fix deadlock on remove add interaction from mqtt virtual topic sub use case * no-jira - fix intermittent test failure when cron falls on next minute * AMQ-7082 - Make sure that the recovery will only mark pages as free if they were created in a previous execution * AMQ-7082 We should make sure that pages managed during recovery are not recovered in error variation of patch from Alan Protasio <alanprot@gmail.com> closes apache#317 * AMQ-7094 - track the objectName with an annotated mbean such that the jmx audit log event can extract that target of an mbean operation, fix and test * Small change to verify repo access. * AMQ-7091 - O(n) Memory consumption when broker has inactive durable subscribes causing OOM * This closes apache#315 - with thanks to Alan Protasio * AMQ-7009 - apply fix to memorymessagestore setBatch with thanks to Nezih BEN FREDJ for test and suggestion * NO-JIRA: Add 5.15.8 schema version * AMQ-7102 - don't track objectNames that have not been registered due to suppressMBean filter, fix and test * AMQ-7097 Update AMQP dependencies to latest Update Qpid JMS, Proton-J and Netty to the latest release. * AMQ-7106 - fix pending stop support by avoiding sync through single shared status var - fix and test * AMQ-7107 - Make QueueBrowsingTest and UsageBlockedDispatchTest succeed with ConcurrentStoreAndDispachQueues=false * Enable secure processing on all DocumentBuilderFactory.newInstance() * [AMQ-7076] Does not define spring-legacy repository by default in ActiveMQ Karaf features repo, and let the user add it * NO-JIRA: CronParserTest is failing in Indian StandardTime * AMQ-7115 - ensure transport error handling is a one shot for a connection - avoid contention and possible dead lock - fix and test * Set the secure processing feature on the TransformerFactory * AMQ-7118 This closes apache#327 - with thanks to Heath Kesler * AMQ-7097 Update Qpid JMS and proton-j to laetst Qpid JMS 0.39.0 and proton-j 0.31.0 released * Update OWasp dependency check plugin version * [AMQ-7126] Prevent OOM when recovering KahaDB and memory space is insufficient to load full page. Thank you Heath Kesler and Johan Edstrom for investigation and patch. * AMQ-7118 - Override hostname for naming to allow tests to apss on all platforms * [AMQ-7126] Improvement to perf of 5266Test * AMQ-7097 Update Qpid JMS and Netty to latest Qpid JMS to v0.40.0 Netty to 4.1.32.Final * AMQ-7129 - Properly recover messages from KahaDB for a durable when there are messages to recover before the stored lastAck value With individual ack mode we need to check the durable ackPosition sequence set in the KahaDB index on subsription load to see if there are earlier messages before the lastAck value that still haven't been acked. While this normally wouldn't happen it is possible in individual ack mode * AMQ-7129 - minor junit test fix * Update scm section * AMQ-7129 - fix durable message size statistics with individual ack Make sure that the pending message size for a durable sub only includes messages part of the ack range * AMQ-7129 - code cleanup * AMQ-7125 Broker does not send error frame when an empty destination is sent with a stomp subscribe frame * AMQ-7132 - ActiveMQ reads lots of index pages upon startup (after a graceful or ungraceful shutdown) * AMQ7135 : do a purge before deleting the destination (cherry picked from commit 9f513f8) * AMQ-7136 - Improve recovery of durable subscription metrics in KahaDB Updated metrics recovery to only have to iterate over the order index 1 time to recovery the pending metrics for the subscriptions instead of making a pass over the index once per subscription * AMQ-7132 - user periodic fsync in RecoveryStatsBrokerTest to improve performance * AMQ-7137 - Implement abort() properly in the LoginModules. Also fix a bug in LdapLoginModule relating to Logout * AMQ-7132 - update RecoveryStatsBrokerTest back to using always journal sync but a smaller journal size and less test messages to speed up test * AMQ-7085 - Properly start TempUsage inside Queue start This fix allows temp usage changes to the broker temp usage to propagate properly to a Queue's temp usage settings Patch applied with thanks to David Sitsky * replace hardcoded path to etc directory with ${karaf.etc} * Update copyright year * [AMQ-7135] Update AMQ2832 unit test expectation post 7135 fix * AMQ-7143 - Temporary transaction file (PageFile) being opened and closed many times, causing poor performance on high latency FS as NFS * Disable directory listing for the webconsole * Enable X-XSS-Protection + X-Content-Type-Options headers for the webconsole * AMQ-7151 - Remove dependency on Guava from activemq-broker Guava is supposed to only be used for the leveldb store and as a test dependency elsewhere but there is one spot it is used in activemq-broker for filtering collections. With Java 8 this can be easily rewritten to avoid the use of Guava. This commit is based on the patch submitted by Filip Hrisafov * AMQ-7153 - Publish BrokerService as OSGi service * AMQ-5790 - rework the fix from AMQ-4486 to tie the inbound xaResource connection lifectyle to the rar start/stop * close DataByteArrayOutputStream buff of WriteBatch * AMQ-7159 - Adding a new attribute on PersistenceAdapterViewMBean to show information about Storage write/read latency * AMQ-7163 - If the broker had an unclean shutdown and number of free pages is Zero after the recovery, the next shutdown will also be 'unclean' * AMQ-7166 - upgrade mqtt client library * AMQ-7165 - ensure failover updated uris are additive such that statically configured uris are respected * no jira - fix unit test failure * no jira - init throwable cause to identify error on shutdown * AMQ-7167 - ensure remote IP is visible in acceptor error messages from the transport connector - test and fix * AMQ-7112 - Show duplex network bridges in the remote broker console * AMQ-7103 - Dependency Updates (cherry picked from commit dd74f46) * NO-JIRA - Update spring schemas and missing license files * AMQ-7165 - randomize needs to be disabled to make test deterministic * AMQ-7179 Update Qpid JMS and Proton-j to latest and sync netty * AMQ-7185 - track durable sub prepared acks in preCommit callback and release/remove on xa outcome, avoid duplicate delivery; fix and test * no jira - remove extraneous printStackTrace, ex is propagated up * AMQ-7189: ensure messages with defauled durable field are internally classed non-persistent by the 'native' transformer * AMQ-7191 - Prevent extra creation of openwire verifier objects per new connection when using the auto transport * Updating Spring + Camel versions * AMQ-7193 - resolve intermittent NPE on duplex network connector start; wait for start completion before dispatching inbound messages, fix and test * AMQ-7196 - During startup ActiveMq load all the scheduleDB.data on memory causing OOM * [AMQ-7199] Replace Math.Random with Random.nextDouble * AMQ-7201 added shebang to generated broker admin script * Update CreateCommand.java * no jira - fix intermittent failure of JdbcXARecoveryBrokerTest#testQueuePersistentPreparedAcksAvailableAfterRollback * AMQ-7209 suppress stack trace in stomp error frame for SecurityExceptions to avoid any implementation detail leakage, fix and test * Switch to using a https URL in the mqtt pom * AMQ-7179: update to proton-j 0.33.0 and qpid-jms 0.42.0 * AMQ-6949: SocketTimeoutException when using HTTP transport connector A default socket timeout of 30 secods for the HTTP client was introduced with the following ticket: https://issues.apache.org/jira/browse/AMQ-6397 This timeout somewhow seems to conflict with the keep alive requests that stays open for arround 30 seconds. I increased the default timeout to 90 seconds and this seems to resolve the problem. * AMQ-7218 Fix loss of Ack id from tracking after TX commit / abort Ensure that we properly track Ack IDs across TX commit and abort operations and only clear out values enlisted in the TX on commit and re-acquire the Ack Ids on TX abort. * AMQ-7219 - ActiveMQ replays journal file on a clean/unclean shutdown with transacted session + Non persistent Messages * AMQ-7221 - Delete Scheduled messages causes ActiveMQ create/write a unnecessary huge transaction file * AMQ-7179: update to proton-j 0.33.1 and qpid-jms 0.43.0 * AMQ-7225 - defer cleanup task operation till recovery processing complete, track prepared location in recovered ops to ensure they are retained on recovery failure. Fix and test * AMQ-7225 - fix intermittent failure, avoid gc of partial tx pending commit * AMQ-7227 - ensure db.free file is moved in scheduler store upgrade process * AMQ-7221 - Fix InMemoryJobSchedulerManagementTest that was broken after patch * AMQ-7228 - Avoid unnecessary lock contention when getting pending metrics * AMQ-7230 - Add support for regex based certificate authentication * AMQ-7233 - ensure sessions are cleaned up on pooledconnection.close before temp dests to ensure temp dests actually get deleted, fix and test * AMQ-7234 - fix up memory usage wait timeout such that topic pfc in a transaction can see connection context state changes, fix and test * AMQ-7235 - Remove hard-coded credentials from OsgiConfiguration * AMQ-7216 - Update Rome dependency * AMQ-7238 - Ensure remoteId subscription map is also cleared when local map is cleared inside DemandForwardingBridgeSupport * AMQ-7239 - Update Jettison * Updating Karaf + also the surefire maven plugin * Make ActiveMQConfiguration.loadClass private * Updating Shiro plus some other minor updates * Remove default "secret" password from the LDAPAuthorizationMap * AMQ-7242 - REST Content Type fixes * Update HTTPClient and replace deprecated calls * AMQ-7236 - Update Spring and xstream to latest * AMQ-7208 - update Guava to 28.0-jre * NO-JIRA: update maven-plugin-plugin version to fix build * AMQ-7247 - Update maven plugin API + Pax URL versions * AMQ-7179 Update Qpid JMS test dependency to v0.44.0 Updates to latest client release used for testing activemq-amqp * [AMQ-7253] Upgrade to commons-dbcp2 2.6.0 and commons-pool 2.6.2 * AMQ-7102 - Improve ManagedRegionBroker performance by replacing registeredMBeans CopyOnWriteSet with a Concurrent hashmap backed set * [AMQ-7254] Fix OSGi headers * AMQ-7244 - Update Apache Directory + FileServer * AMQ-7266 - Update Jackson databind to 2.9.9.3 * [AMQ-7258] Update spring.schemas in activemq-osgi * AMQ-7268 - Update Zookeeper version * AMQ-7270 - remove toggle on maxPageSize, request page in of entire queue only when cache is in play or broker is non persistent, fix and test * AMQ-7271 - Update XBean * [AMQ-7249] Upgrade to Camel 2.24.1 * AMQ-7274 Update proton-j and fix footer key type on outbound transform Footer annotations maps should be using Symbol keys not strings. * AMQ-7275 - Update Commons BeanUtils * Revert "[AMQ-7258] Update spring.schemas in activemq-osgi" This reverts commit 3c71b5e. * [AMQ-7249] Upgrade to Jetty 9.4.19.v20190610 * NO-JIRA: Add gpg key for jbonofre * AMQ-7289 - Fix failing HTTP tests * AMQ-7290 Update Qpid JMS and Netty to lastest releases * AMQ-7292 - WebConsole problem due to missing ecj jar * AMQ-7298 - rework redelivery message tracking to ensure no duplicate suppression (and dlq) in error for local transaction batches that failover * AMQ-7298 - use final modifier in test to preserve older jdk builds * no jira - fix intermittent test failure with two derby instanced in play in error * no jira - fix intermittent failure by not differenciating between dispatch of delivery loops, it can be either depending on timing * AMQ-7196 - fix broken test; recovery not applicable to inmemory store * AMQ-7300 - Improve performance of LongSequenceGenerator LongSequenceGenerator now uses an AtomicLongFieldUpdater instead of synchronized * AMQ-7300 - Remove unnecessary call to sequence updater as read is already volatile * maven-compiler-plugin 3.8.1 * AMQ-7302 - make jmx ops that pageIn aware of cursor memory limits to avoid excessive looping, fix and test * AMQ-7118 ensure message size and usage is constant for the test, fix failure * AMQ-7308 - ensure kahadb message add does not auto create the message store in error, expect an existing store. fix and test * AMQ-7185 - rework to leave tx-inflight messages pending in the cursor to avoid duplicates on completion, fix and test * no jira - fix intermittent test failure * AMQ-7311 - track recovered prepared ack locations on a per subscriber basis, fix and test * no jira - fix test to aligh with implementation limitations of deprecated leveldb store * Improve activemq init JAVACMD auto detection If JAVACMD is set to "auto" (the default from the "env" file) and the current working directory contains a directory named "auto", the activemq init script will incorrectly detect the "auto" directory as the "java" binary thus leaving JAVACMD set to "auto" which is incorrect and will obviously fail to execute the Java VM. To fix this, in the second attempt to detect the "java" binary, repeat the tests for a zero-length JAVACMD or a JAVACMD equal to "auto" before testing whether JAVACMD does not exist or is not executable. This is necessary because when JAVA_HOME's length is zero, the first attempt to detect the location of the "java" binary will not set JAVACMD (so it will still be "auto" when it reaches the second auto-detection attempt). * AMQ-7312 virtualSelectorCacheBrokerPlugin addConsumer issue * AMQ-6494 is related, fix intermittent failure of RedeliveryPolicyTest related to vm transport server being shutdown while in use via async onException handler * AMQ-7319 - Update Jackson to 2.9.10 * AMQ-7295 - Update JTA spec * Remove out-of-bounds index comparison * AMQ-7130 Update AMQ doap * AMQ-7322 - Add HTTPOnly flag to the webconsole + REST API Cookies * [AMQ-7323] Upgrade to rome 1.12.2 * [AMQ-7330] Upgrade to Jetty 9.4.22.v20191022 * [AMQ-7324] Upgrade to commons-dbcp2 2.7.0 and commons-pool2 2.7.0 * [AMQ-7331] Upgrade to commons-daemon 1.2.2 * [AMQ-7332] Upgrade to ant 1.10.7 * [AMQ-7333] Upgrade to Camel 2.24.2 * [AMQ-7321] Reduce the number of messages in MQTT SSL offline test to limit timing issue * Update owasp plugin (and enable it to run correctly) * AMQ-7335 Update Qpid JMS to 0.47.0 netty to 4.1.43.Final * [AMQ-7327] Use maxFrameSize to limit HTTP content length * [AMQ-7327] Updated unit test around maxFrameSize support on the http transport connector * Revert "[AMQ-7333] Upgrade to Camel 2.24.2" This reverts commit 5568a09. * AMQ-7339 - Fix possible XSS attack in the HttpTunnelServlet * [AMQ-7301] Use warn for connection failure instead of error. * no jira - fix intermittent test failure due to incorrect assertion * [AMQ-7252] Upgrade to Velocity 2.1 * [AMQ-7310] Upgrade to Tomcat API 9.0.27 * [AMQ-7068] Advisory messages are empty when received with a AMQP subscription (apache#312) * AMQ-6908 allow use of ActiveMQ console login credential (for authorization) when browsing messages through the console (cherry picked from commit 822cf62) * [AMQ-6556] Support system property proxy settings for HTTP(S) client (cherry picked from commit 746a839) * [AMQ-7345] Upgrade to jackson-databind 2.9.10.1 * [AMQ-7329] Fix exclusion in activemq-osgi * [AMQ-7008] Add index on messages table to speed up browsing * Fix build of ActiveMQ OSGi bundle * AMQ-7106 - fix intermittent failure in the test * AMQ-5734 - additional test that verifies wildcard sub does not get access to unauthorised dests retain messages * AMQ-7302 - rename test such that is is run with surefire pattern match
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…ubscribes causing OOM