Fixes #18981: Slow computation of dynamic groups on large system #3577
Conversation
| } | ||
| updateManager ! GroupUpdateMessage.DynamicUpdateResult(processId, modId, startTime, DateTime.now, results.toMap) | ||
|
|
There was a problem hiding this comment.
There should be a debug-level log message here telling how much parallelism is used for groups without dependencies
| @@ -441,6 +445,7 @@ class LDAPBasedConfigService( | |||
| rudder.generation.delay=0s | |||
| rudder.generation.trigger=all | |||
| node.accept.duplicated.hostname=false | |||
| rudder.compute.dyngroups.max.parallelism=1 | |||
There was a problem hiding this comment.
I think that for consistancy sack, it should be like rudder.generation.max.parallelism, ie accepting either an integer like here, or xF, with F a float and x the character x, meaning F time the number of CPU available. The same parsing logic can be used (and can even be factored out in an utility object, which would also ensure to always return an integer at least equals to 1).
And with that, setting it to x0.5 (ie, number CPU / 2) seems to be a good default.
There was a problem hiding this comment.
it's not really generation, as it is to compute dynamic groups (so also done outside of policy generation)
i'll change the xF
There was a problem hiding this comment.
I'm not sure i'd like to change the default to something larger than 1, as it depends on the size of the LDAP connection pool which is 2 by default
|
PR updated with a new commit |
| val error = (e.fullMsg + s"Error when updating dynamic group '${id.value}'") | ||
| logger.error(error) | ||
| // e.fullMsg.foreach(ex => | ||
| // logger.error("Exception was:", ex) |
ncharles
force-pushed
the
bug_18981/slow_computation_of_dynamic_groups_on_large_system
branch
from
4078275
to
d65eda3
Compare
|
PR updated with a new commit |
|
This PR is not mergeable to upper versions. |
|
OK, squash merging this PR |
fanf
force-pushed
the
bug_18981/slow_computation_of_dynamic_groups_on_large_system
branch
from
f016e6a
to
074cb46
Compare
https://issues.rudder.io/issues/18981