Router lifecycle events occurring when large numbers of addresses exist causes Agent heap spike #3090
Comments
One problematic case is |
Reading around on the internet, I'm lead to the conclusion that creating thousands of promises at one time is a bad thing and leads to memory spikes. I plan to throttle the creation of the promises representing creation and deletion of address. At the moment, https://www.npmjs.com/package/p-limit seems to be may favourite - it is quite unobtrusive. |
…creating/deleting entities on the routers
…s on the brokers Fix EnMasseProject#3090
…s on the brokers Fix EnMasseProject#3090
* Backport address allocation equality fix Backport of commit 228fa4e * Turn down noisy per reconcilation per address logging (#3085) * Restrict the number of active promises when creating/deleting entities on the routers (#3095) Fix #3090 * Fix #3086: Adjust node max_old_space_size to a percentage of available container memory (#3099) * Correct address status equality check (#3094) * Fix #3089: Correct address status equality check * address review comments * made same_allocation same implementation symmetric * Fix #3092: Turn off EventEmitter#maxListeners check for router management connection (#3093) * Restrict the number of active promises when creating/deleting entities on the brokers Fix #3090 * Ensure that addresses are not synced until addresses have been defined As proposed in #3101 * Fix unit tests * Start the watcher after the listeners are in place Signed-off-by: Vanessa <vbusch@redhat.com> * Fix test Signed-off-by: Vanessa <vbusch@redhat.com>
* Turn down noisy per reconcilation per address logging (#3085) * Restrict the number of active promises when creating/deleting entities on the brokers Fix #3090 * Restrict the number of active promises when creating/deleting entities on the routers (#3095) Fix #3090 * Fix #3086: Adjust node max_old_space_size to a percentage of available container memory (#3099) * Correct address status equality check (#3094) * Fix #3089: Correct address status equality check * address review comments * made same_allocation same implementation symmetric * Fix #3092: Turn off EventEmitter#maxListeners check for router management connection (#3093) * Ensure that addresses are not synced until addresses have been defined As proposed in #3101 * Fix unit tests * Start the watcher after the listeners are in place Signed-off-by: Vanessa <vbusch@redhat.com> * Improve api server address create performance (#3114) This change improves api server address create performance by almost an order of magnitude when 1000 addresses are defined. The changes moves the validation of spec.address to standard-controller for the standard address space. For the brokered address space, the validation remains in the api-server, as it would require adding write-back capability to the agent. Future refactoring/consolidation of agent/standard-controller should incorporate this validation. Fixes #3111
* Correct address allocation equality. (#3083) This defect meant that all existing addresses were considered modified even if they weren't. This lead to unnecessary reconciliations against all routers and brokers, consuming extra resources across the system. Fix: #3081 * Turn down noisy per reconcilation per address logging (#3085) * Restrict the number of active promises when creating/deleting entities on the brokers Fix #3090 * Restrict the number of active promises when creating/deleting entities on the routers (#3095) Fix #3090 * Fix #3086: Adjust node max_old_space_size to a percentage of available container memory (#3099) * Correct address status equality check (#3094) * Fix #3089: Correct address status equality check * address review comments * made same_allocation same implementation symmetric * Fix #3092: Turn off EventEmitter#maxListeners check for router management connection (#3093) * Ensure that addresses are not synced until addresses have been defined As proposed in #3101 * Fix unit tests * Start the watcher after the listeners are in place Signed-off-by: Vanessa <vbusch@redhat.com> * Improve api server address create performance (#3114) This change improves api server address create performance by almost an order of magnitude when 1000 addresses are defined. The changes moves the validation of spec.address to standard-controller for the standard address space. For the brokered address space, the validation remains in the api-server, as it would require adding write-back capability to the agent. Future refactoring/consolidation of agent/standard-controller should incorporate this validation. Fixes #3111
If a router event occurs (such as a router restart, or the addition of a new router owing plan rules) whilst large numbers of addresses are defined, the Agent's heap memory usage may spike possible leading to an OOM condition.
The text was updated successfully, but these errors were encountered: