Skip to content
This repository has been archived by the owner on Dec 4, 2023. It is now read-only.

Router lifecycle events occurring when large numbers of addresses exist causes Agent heap spike #3090

Closed
k-wall opened this issue Aug 15, 2019 · 2 comments · Fixed by #3102
Closed

Comments

@k-wall
Copy link
Member

k-wall commented Aug 15, 2019

If a router event occurs (such as a router restart, or the addition of a new router owing plan rules) whilst large numbers of addresses are defined, the Agent's heap memory usage may spike possible leading to an OOM condition.

@k-wall
Copy link
Member Author

k-wall commented Aug 15, 2019

One problematic case is router_config#ensure_elements() - here promises are used to cause the deletion of n stale addresses and the addition of m new addresses. In the case, where say a router restarts and the number of addresses large, this leads to an equal number of promises being created. Even though the management requests are throttled further down the chai we still have the overhead of the promise itself. I believe this will explain a spike in memory in some circumstances.

@k-wall
Copy link
Member Author

k-wall commented Aug 15, 2019

Reading around on the internet, I'm lead to the conclusion that creating thousands of promises at one time is a bad thing and leads to memory spikes. I plan to throttle the creation of the promises representing creation and deletion of address. At the moment, https://www.npmjs.com/package/p-limit seems to be may favourite - it is quite unobtrusive.

@k-wall k-wall changed the title Router lifecycle events when large numbers of addresses exist causes Agent heap spike Router lifecycle events occurring when large numbers of addresses exist causes Agent heap spike Aug 15, 2019
k-wall added a commit to k-wall/enmasse that referenced this issue Aug 15, 2019
@k-wall k-wall closed this as completed in cb7589f Aug 15, 2019
k-wall added a commit to k-wall/enmasse that referenced this issue Aug 16, 2019
vbusch pushed a commit that referenced this issue Aug 19, 2019
lulf pushed a commit to lulf/enmasse that referenced this issue Aug 20, 2019
lulf pushed a commit to lulf/enmasse that referenced this issue Aug 20, 2019
lulf pushed a commit that referenced this issue Aug 21, 2019
* Backport address allocation equality fix

Backport of commit 228fa4e

* Turn down noisy per reconcilation per address logging (#3085)

* Restrict the number of active promises when creating/deleting entities on the routers (#3095)

Fix #3090

* Fix #3086: Adjust node max_old_space_size to a percentage of available container memory (#3099)

* Correct address status equality check (#3094)

* Fix #3089: Correct address status equality check

* address review comments
* made same_allocation same implementation symmetric

* Fix #3092: Turn off EventEmitter#maxListeners check for router management connection (#3093)

* Restrict the number of active promises when creating/deleting entities on the brokers

Fix #3090

* Ensure that addresses are not synced until addresses have been defined

As proposed in #3101

* Fix unit tests

* Start the watcher after the listeners are in place

Signed-off-by: Vanessa <vbusch@redhat.com>

* Fix test

Signed-off-by: Vanessa <vbusch@redhat.com>
lulf pushed a commit that referenced this issue Aug 23, 2019
lulf pushed a commit that referenced this issue Aug 23, 2019
lulf pushed a commit that referenced this issue Aug 23, 2019
lulf pushed a commit that referenced this issue Aug 23, 2019
lulf pushed a commit that referenced this issue Aug 23, 2019
* Turn down noisy per reconcilation per address logging (#3085)

* Restrict the number of active promises when creating/deleting entities on the brokers

Fix #3090

* Restrict the number of active promises when creating/deleting entities on the routers (#3095)

Fix #3090

* Fix #3086: Adjust node max_old_space_size to a percentage of available container memory (#3099)

* Correct address status equality check (#3094)

* Fix #3089: Correct address status equality check

* address review comments
* made same_allocation same implementation symmetric

* Fix #3092: Turn off EventEmitter#maxListeners check for router management connection (#3093)

* Ensure that addresses are not synced until addresses have been defined

As proposed in #3101

* Fix unit tests

* Start the watcher after the listeners are in place

Signed-off-by: Vanessa <vbusch@redhat.com>

* Improve api server address create performance (#3114)

This change improves api server address create performance by almost an order of magnitude when 1000 addresses are defined.
The changes moves the validation of spec.address to standard-controller for the standard address space. For the brokered address space, the validation remains in the api-server, as it would require adding write-back capability to the agent. Future refactoring/consolidation of agent/standard-controller should incorporate this validation.

Fixes #3111
lulf pushed a commit that referenced this issue Aug 23, 2019
* Correct address allocation equality. (#3083)

This defect meant that all existing addresses were considered modified even if they weren't.
This lead to unnecessary reconciliations against all routers and brokers, consuming extra resources across the system.

Fix: #3081

* Turn down noisy per reconcilation per address logging (#3085)

* Restrict the number of active promises when creating/deleting entities on the brokers

Fix #3090

* Restrict the number of active promises when creating/deleting entities on the routers (#3095)

Fix #3090

* Fix #3086: Adjust node max_old_space_size to a percentage of available container memory (#3099)

* Correct address status equality check (#3094)

* Fix #3089: Correct address status equality check

* address review comments
* made same_allocation same implementation symmetric

* Fix #3092: Turn off EventEmitter#maxListeners check for router management connection (#3093)

* Ensure that addresses are not synced until addresses have been defined

As proposed in #3101

* Fix unit tests

* Start the watcher after the listeners are in place

Signed-off-by: Vanessa <vbusch@redhat.com>

* Improve api server address create performance (#3114)

This change improves api server address create performance by almost an order of magnitude when 1000 addresses are defined.
The changes moves the validation of spec.address to standard-controller for the standard address space. For the brokered address space, the validation remains in the api-server, as it would require adding write-back capability to the agent. Future refactoring/consolidation of agent/standard-controller should incorporate this validation.

Fixes #3111
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant