Fixes #19151: add caching for NodeExpectedReports #3728

ncharles · 2021-07-26T14:57:21Z

https://issues.rudder.io/issues/19151

This PR adds a cache for NodeExpectedReports. This cache works as a map of [NodeId, Option]. The cache is inited with all nodes, and None for each NodeExpectedReports.
Then once we search for current expected reports, cache is explored, and fetched expected reports are added to the cache.
When a policy generation happens, it also update the current expected reports for modified nodes

The rational is that it is really expensive to fetch these from the database, and keeping them in the cache does improve perfs.

ncharles · 2021-07-26T15:03:06Z

PR updated with a new commit

ncharles · 2021-07-26T15:06:51Z

...der-core/src/main/scala/com/normation/rudder/services/reports/NodeConfigurationService.scala

+   */
+  def getCurrentExpectedReports(nodeIds: Set[NodeId]): Box[Map[NodeId, Option[NodeExpectedReports]]] = {
+    // First, get all nodes from cache (even the none)
+    val dataFromCache = cache.filter{  case (nodeId, _) => nodeIds.contains(nodeId) }


i'm unsure if this should be in a semaphore or not

ncharles · 2021-07-26T15:07:25Z

.../rudder-core/src/main/scala/com/normation/rudder/services/reports/ReportingServiceImpl.scala

@@ -96,6 +97,7 @@ object CacheComplianceQueueAction {
  final case class RemoveNodeInCache      (nodeId: NodeId                                          ) extends CacheComplianceQueueAction
  final case class InitializeCompliance   (nodeId: NodeId, nodeCompliance: Option[NodeStatusReport]) extends CacheComplianceQueueAction // do we need this?
  final case class UpdateCompliance       (nodeId: NodeId, nodeCompliance: NodeStatusReport        ) extends CacheComplianceQueueAction
+  final case class InitNodeConfiguration  (nodeId: NodeId, nodeConfiguration: NodeExpectedReports  ) // unsure about keeping this one


it's unused, i' pretty sure i should not keep it

clean it, it's cognitively more costly to have something and wonder why it's there than needing to add it if needed later ont

ncharles · 2021-07-26T15:08:14Z

.../rudder-core/src/main/scala/com/normation/rudder/services/reports/ReportingServiceImpl.scala

  , val batchSize                       : Int
  , val complianceRepository            : ComplianceRepository
 ) extends ReportingService with RuleOrNodeReportingServiceImpl with CachedFindRuleNodeStatusReports {
  val confExpectedRepo = defaultFindRuleNodeStatusReports.confExpectedRepo
  val directivesRepo   = defaultFindRuleNodeStatusReports.directivesRepo
  val rulesRepo        = defaultFindRuleNodeStatusReports.rulesRepo

+  val nodeConfigrationService = nodeConfigService


THIS is bothering me. i need to register the nodeConfigrationService because it in the CachedFindRuleNodeStatusReports

perhaps you need to create a facade/new trait on top of FindExpectedReportRepository and NodeConfigurationService to only expose what is needed?

Good idea !

actually, it's never used, so I removed it

...der-core/src/main/scala/com/normation/rudder/services/reports/NodeConfigurationService.scala

ncharles · 2021-07-26T20:01:05Z

PR updated with a new commit

fanf · 2021-07-27T08:51:40Z

...der-core/src/main/scala/com/normation/rudder/services/reports/NodeConfigurationService.scala

+   */
+  def findNodeExpectedReports(
+         nodeConfigIds: Set[NodeAndConfigId]
+      ): Box[Map[NodeAndConfigId, Option[NodeExpectedReports]]]


Can you return IOResult in place of Box? We try to get ride of them, andIOResult` compose better

fanf · 2021-07-27T08:55:24Z

.../rudder/rudder-core/src/main/scala/com/normation/rudder/services/nodes/NodeInfoService.scala

+   * Get all nodes id
+   */
+
+  def getAllNodeIds(): Box[Set[NodeId]]


prefer use of IOResult, Box is deprecated in rudder

fanf · 2021-07-27T09:05:49Z

webapp/sources/rudder/rudder-web/src/main/scala/bootstrap/liftweb/RudderConfig.scala

+    cached.init().runNow
+    cached
+  }
+
  /*


is it possible to completly hide findExpectedRepo from public use with a facade that delegate to either it or cache ? We would be sure to never call the bad service

it is a private[this] - i'm unsure how to do better than that (unless putting a big comment around it to not use it)

or it is used twice, once per cache, so I could instanciate one per cache, overhead should be minimal

fanf

I think the general logic is ok. Please, limit the use of Box to compat (and if possible, remove some more of them). I'm don't understand fully enought all the usage to be sure it's sound. If you can, you should hide from public use (ie, in RudderConfig) the non-cached version of the feature, to avoid people (in plugins etc, for ex) to not take benefits of them

ncharles · 2021-07-28T11:48:20Z

PR updated with a new commit

fanf · 2021-07-28T20:26:59Z

...der-core/src/main/scala/com/normation/rudder/services/reports/NodeConfigurationService.scala

+                          case delete: RemoveNodeInCache => IOResult.effectNonBlocking {  cache = cache.removed(delete.nodeId) }
+                          case update: UpdateNodeConfiguration => IOResult.effectNonBlocking {  cache = cache + (update.nodeId -> Some(update.nodeConfiguration)) }
+                          case something =>
+                            Inconsistency(s"NodeConfiguration service cache received unknown command : ${something}").fail


I'm going to try to make a subtype for action to be sure that that property holds

It works (with some boilerplate), so we can make it checkable: ncharles#10

fanf · 2021-07-28T20:28:47Z

...der-core/src/main/scala/com/normation/rudder/services/reports/NodeConfigurationService.scala

+  // * if the nodeId is in cache, and value is None, augment cache
+  // * if the nodeid is in cache, and value is not None, pick the most recent value to augment cache
+  // Returns the values merged with cache
+  private[this] def mergeDataFromDBWithCache(fromDb: Map[NodeId, Option[NodeExpectedReports]]):IOResult[ Map[NodeId, Option[NodeExpectedReports]] ] = {


I'm not sure that that function should be split appart from the "getCurrentExpectedReports" part. It makes harder to see what need to be atomic, and it hides the fact that nodes "fromDb" should all have beed not initialized in cache (since it's called for that reason).
Let me try to merge them

i think it needs to be appart. It'll be probably used in other methods

but for now, it is not, and it hads substential complexity.

fanf · 2021-07-28T20:41:33Z

...der-core/src/main/scala/com/normation/rudder/services/reports/NodeConfigurationService.scala

+    semaphore.withPermit(
+      IOResult.effect({
+      val mergedEntries = fromDb.map {  case (nodeId, expFromDb) => cache.get(nodeId) match {
+        case None => None // nodeid not in cache, dropping


for example, here I don't understand how the nodeId could be not in cache, since we restricted the call to fromDb to only node in cache but not initialized

adding/removing nodes can happen at the same time
also, a node may exist without compliance (ignored, or policy not yet generated)

For the case without compliance, it's ok: it was none, it's still none after.
For the case "adding/removing nodes can happen at the same time", I don't understand: here, we are just augmenting the cache knwon entries, we don't remove entries. And there's a semaphore to avoid side effects from performAction

fanf · 2021-07-28T20:43:17Z

...der-core/src/main/scala/com/normation/rudder/services/reports/NodeConfigurationService.scala

+          case Some(entry) => // now we must compare with the one from DB
+            expFromDb match {
+              case None => Some((nodeId -> value))
+              case Some(db) if db.beginDate.isAfter(entry.beginDate) => Some((nodeId -> expFromDb)) // data fom db is newer


and here, only node with "none" in cache were called for db, so that should not happen

query can take several seconds, cache might have been updated in between with a fresh nodeconfiguration pushed by policy generation

I have an alternative proposal: ncharles#10
Cache should not be updated in between, that doesn't make sense to look for empty nodeIds and then consider that that data is not the truth - or it was not usefull to check for that at first. I can't see how the DB query can't be in the semaphore if you need consistency (for ex, if you don't want to miss the fact that a cache reset was done in between).

ncharles · 2021-07-29T12:22:42Z

PR updated with a new commit

fanf · 2021-07-29T20:05:48Z

...der-core/src/main/scala/com/normation/rudder/services/reports/NodeConfigurationService.scala

+    // In a semaphore, nothing should change the cache
+    semaphore.withPermit(for {
+      timeInSemaphore <- currentTimeMillis
+      _               = logger.trace(s"Entered the semaphore after ${timeInSemaphore - before_semaphoreTime} ms")


it needs to be <- logger ...

ncharles · 2021-07-29T20:11:36Z

PR updated with a new commit

ncharles · 2021-07-29T20:32:16Z

PR updated with a new commit

Normation-Quality-Assistant · 2021-07-29T20:56:38Z

OK, squash merging this PR

fixup! Fixes #19151: add caching for NodeExpectedReports Fixes #19151: add caching for NodeExpectedReports fixup! fixup! Fixes #19151: add caching for NodeExpectedReports Fixes #19151: add caching for NodeExpectedReports fixup! fixup! fixup! Fixes #19151: add caching for NodeExpectedReports Fixes #19151: add caching for NodeExpectedReports Proposal for update in nodeconfigcache split cache action to allows exhaustive checks fixup! Merge pull request #10 from fanf/pr/3728 Fixes #19151: add caching for NodeExpectedReports

ncharles · 2021-08-02T12:27:52Z

.../sources/rudder/rudder-web/src/main/scala/com/normation/rudder/web/components/RuleGrid.scala

@@ -180,19 +182,19 @@ class RuleGrid(
      val start = System.currentTimeMillis

      ( for {
-          rules           <- roRuleRepository.getAll(false).toBox.map { allRules => onlyRules match {


this causes https://issues.rudder.io/issues/19651

ncharles · 2021-11-18T12:19:05Z

...er/rudder-core/src/main/scala/com/normation/rudder/services/policies/DeploymentService.scala

-  override def invalidateComplianceCache(actions: Seq[(NodeId, CacheComplianceQueueAction)]): Unit = {
-    complianceCache.invalidateWithAction(actions).runNow
+  override def invalidateComplianceCache(actions: Seq[(NodeId, CacheExpectedReportAction)]): IOResult[Unit] = {
+    cachedNodeConfigurationService.invalidateWithAction(actions) *>


this is probably not doing what we want

ncharles requested a review from fanf July 26, 2021 14:57

ncharles added Trigger test and removed Trigger test labels Jul 26, 2021

ncharles commented Jul 26, 2021

View reviewed changes

...der-core/src/main/scala/com/normation/rudder/services/reports/NodeConfigurationService.scala Outdated Show resolved Hide resolved

ncharles added Trigger test and removed Trigger test labels Jul 26, 2021

ncharles force-pushed the arch_19151/add_caching_for_nodeexpectedreports branch from 2adedf5 to 6122d3b Compare July 26, 2021 20:03

fanf reviewed Jul 27, 2021

View reviewed changes

ncharles added Trigger test and removed Trigger test labels Jul 28, 2021

fanf reviewed Jul 28, 2021

View reviewed changes

ncharles added Trigger test and removed Trigger test labels Jul 29, 2021

ncharles force-pushed the arch_19151/add_caching_for_nodeexpectedreports branch from f33a730 to 07043a1 Compare July 29, 2021 20:02

fanf reviewed Jul 29, 2021

View reviewed changes

ncharles removed the Trigger test label Jul 29, 2021

ncharles added the Trigger test label Jul 29, 2021

fanf approved these changes Jul 29, 2021

View reviewed changes

ncharles added Trigger test and removed Trigger test labels Jul 29, 2021

amousset removed the Trigger test label Jul 29, 2021

Normation-Quality-Assistant force-pushed the arch_19151/add_caching_for_nodeexpectedreports branch from 7fb6313 to ea691a4 Compare July 29, 2021 20:56

Normation-Quality-Assistant merged commit ea691a4 into Normation:master Jul 29, 2021

ncharles commented Aug 2, 2021

View reviewed changes

ncharles commented Nov 18, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes #19151: add caching for NodeExpectedReports #3728

Fixes #19151: add caching for NodeExpectedReports #3728

ncharles commented Jul 26, 2021 •

edited

Loading

ncharles commented Jul 26, 2021

ncharles Jul 26, 2021

ncharles Jul 26, 2021

fanf Jul 27, 2021

ncharles Jul 26, 2021

fanf Jul 27, 2021

ncharles Jul 27, 2021

ncharles Jul 28, 2021

ncharles commented Jul 26, 2021

fanf Jul 27, 2021

fanf Jul 27, 2021

fanf Jul 27, 2021

ncharles Jul 27, 2021

ncharles Jul 27, 2021

fanf left a comment

ncharles commented Jul 28, 2021

fanf Jul 28, 2021

fanf Jul 28, 2021

fanf Jul 28, 2021

ncharles Jul 28, 2021

fanf Jul 28, 2021

fanf Jul 28, 2021

ncharles Jul 28, 2021

fanf Jul 28, 2021

fanf Jul 28, 2021

ncharles Jul 28, 2021

fanf Jul 28, 2021

ncharles commented Jul 29, 2021

fanf Jul 29, 2021

ncharles commented Jul 29, 2021

ncharles commented Jul 29, 2021

Normation-Quality-Assistant commented Jul 29, 2021

ncharles Aug 2, 2021

ncharles Nov 18, 2021

Fixes #19151: add caching for NodeExpectedReports #3728

Fixes #19151: add caching for NodeExpectedReports #3728

Conversation

ncharles commented Jul 26, 2021 • edited Loading

ncharles commented Jul 26, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ncharles commented Jul 26, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fanf left a comment

Choose a reason for hiding this comment

ncharles commented Jul 28, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ncharles commented Jul 29, 2021

Choose a reason for hiding this comment

ncharles commented Jul 29, 2021

ncharles commented Jul 29, 2021

Normation-Quality-Assistant commented Jul 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ncharles commented Jul 26, 2021 •

edited

Loading