Fixes #23967: Rudder score backend and API #5296

VinceMacBuche · 2023-12-29T00:37:52Z

https://issues.rudder.io/issues/23967

VinceMacBuche · 2023-12-29T11:35:55Z

Commit modified

VinceMacBuche · 2023-12-29T15:47:13Z

Commit modified

VinceMacBuche · 2024-01-07T22:57:41Z

PR updated with a new commit

VinceMacBuche · 2024-01-09T15:32:57Z

Commit modified

VinceMacBuche · 2024-01-10T18:00:17Z

Commit modified

VinceMacBuche · 2024-01-11T11:43:48Z

Commit modified

VinceMacBuche · 2024-01-11T15:10:34Z

Commit modified

fanf · 2024-01-11T15:30:08Z

webapp/sources/rudder/rudder-core/src/main/resources/reportsSchema.sql

+Create table GlobalScore (
+  nodeId  text primary key
+, score   text
+, message text


what is that message for? I would prefer to use name (for a short human name), description (for a one line description for ex on mouse over) or documentation (for a longer, possibly semi-structured/mkd explanation)

It's the score explanation message. I think it is really a message. It is not a configuraiton structure and i do think it should not follow the same rules than other items

fanf · 2024-01-11T15:31:04Z

webapp/sources/rudder/rudder-core/src/main/resources/reportsSchema.sql

+Create table scoreDetails (
+  nodeId  text
+, score   text
+, name    text


name should be scoreId I think

if you want

fanf · 2024-01-11T15:38:15Z

.../rudder/rudder-core/src/main/scala/com/normation/rudder/domain/reports/ComplianceLevel.scala

+object ComplianceSerializable {
+  def fromPercent(compliancePercent: CompliancePercent) = {
+    ComplianceSerializable(
+      if (compliancePercent.pending == 0) None else Some(compliancePercent.pending),


you should have a private def to avoid duplicating that logic and avoid risking micopying the name in both place

fanf · 2024-01-11T15:52:26Z

.../rudder/rudder-core/src/main/scala/com/normation/rudder/domain/reports/ComplianceLevel.scala

+    unexpectedMissingComponent: Option[Double],
+    noReport:                   Option[Double],
+    reportsDisabled:            Option[Double],
+    badPolicyMode:              Option[Double]


the elements are not in the same order than CompliancePercent, that seems to be an extremely easy way to have errors.
For the naming, it seems better to use the event name like you did, but I wonder why we don't use them in CompliancePercent too.

fanf · 2024-01-11T15:55:45Z

...der/rudder-core/src/main/scala/com/normation/rudder/facts/nodes/NodeFactServiceProxies.scala

 ) extends PipelinedInventorySaver[Unit] {

  override def commitChange(inventory: Inventory): IOResult[Unit] = {
    implicit val cc = ChangeContext.newForRudder()
-    backend.updateInventory(FullInventory(inventory.node, Some(inventory.machine)), Some(inventory.applications)).unit
+    backend.updateInventory(FullInventory(inventory.node, Some(inventory.machine)), Some(inventory.applications)).unit *>
+    scoreServiceManager.handleEvent(InventoryScoreEvent(inventory.node.main.id, inventory)).unit


Why puting it here and not in a PostCommit handler? They should be able to match that use case, and I don't see the need for hard implementation coupling here. And I don't think it gives any additional consistency benefits.

VinceMacBuche · 2024-01-11T16:51:45Z

Commit modified

fanf · 2024-01-11T16:58:42Z

webapp/sources/rudder/rudder-core/src/main/resources/reportsSchema.sql

+
+Create table scoreDetails (
+  nodeId  text
+, score   text


I think it could be an enumerated type, it seems to be the perfect case for that, and it would likely help regarding perf and size. But I don't know what is the migration story if for ex we want to add new values.

fanf · 2024-01-11T16:59:27Z

webapp/sources/rudder/rudder-core/src/main/resources/reportsSchema.sql

+, message text
+, details jsonb
+, PRIMARY KEY (nodeId, name)
+);


you can add not null not empty constraints for nodeId, score, name

fanf · 2024-01-11T17:30:36Z

webapp/sources/rudder/rudder-core/src/main/scala/com/normation/rudder/score/Score.scala

+  }
+}
+
+trait Score[T] {


I don't think we want to have that parametrization. We never use a generic access to details, and List[Score[_]] is a signal that we maybe are not doing things like we should.

Plus, that parametrization risk to create some unpleasant situation since Score[ComplianceSerializable] is not the same as ComplianceScore (at type level), but in our code, we really really want to have that equality.

Something like the first commit in that PR: VinceMacBuche#29

In fact we could remove all ComplianceScore, and juste avec a Score that has a Json as details. Just everything translated into a jsonScore directly from the score event

I like the idea to have different score implementation for the different cases, it makes the score computation very clean and really feels like each module can do whatever they want from a structured data type, and not Json. But perhaps I misunderstood ?

yes ScoreEvent will be different witth diffrent structure.

fanf · 2024-01-11T17:32:58Z

...p/sources/rudder/rudder-core/src/main/scala/com/normation/rudder/score/ScoreSerializer.scala

+}
+
+trait ScoreSerializerService {
+  def parse(score:  Score[Json]): IOResult[Option[Score[_]]]


I think it should be a pure result, I don't see why we should have IO here

maybe i need some services that may not be pure

I didn't see any and parsing should really always be pure (and perhaps you need some impure one before geting the data, but having IO involved in the parsing feels really like not what we want to have to deal with)

Also it's uised mostly in ZIO context and i have to transform my PureResult into IORestult to i guess it does not gain much

it gains in intent conveyed, and in unit test easyness. IOResult is a bit like global variables, they are easy and powerfull, but we should always try to restraint as much as possible to use them because they are too powerful.

(not a subject anymore since that part of the code disapeared, it's for info)

fanf · 2024-01-11T17:34:13Z

...p/sources/rudder/rudder-core/src/main/scala/com/normation/rudder/score/ScoreSerializer.scala

+
+trait ScoreSerializerService {
+  def parse(score:  Score[Json]): IOResult[Option[Score[_]]]
+  def toJson(score: Score[_]):    IOResult[Option[Score[Json]]]


I don't see how it can fail. I understand to potential None case if there is no serializer for that score, but if there is one, it should not ever fail (and so just be a Option[Score[Json]])

I wonder if that can't be simplier.
It's used in two place:

in DB to read/write from a JSON column (so a string), and using the intermediary Json means that we do db -> get String -> parse to Json -> serialize to String -> parse to T - we could certainly directly use the string, even if that means having a StringScore to simplify doobie interaction,

in node API, to serialize these info. But here, we should really just need the JsonEncoder which we already have (and again avoid the T -> to String -> parse Json -> to String)

General idea draft in the second commit of that PR: VinceMacBuche#29 (not sure it floats till the end, but we can in the worst case keep JsonScore for NodeAPI)

I prefer to remove invalid score (that are not json) at the beginning. and it will end up parsed as json (either in api, could be also used to init score if we wanted to recompute a score ) I don't think there is a usage in using a String and not a Json, which is the underlying type

We already have everything to manage json from database. and here you have to translate into json at the end which can be already done

You can still check for invalid data structure at the begining, it doesn't change that. You certainly can use directly the JsonEncoder and JsonDecoder in the score repository, I just didn't take the time to implem it.
Here, each time you use as or toJsonAST, you really go through a string parsing in the middle.

And I really believe we should not have one unique JsonScore, it will make the score calculation part less clear.

VinceMacBuche · 2024-01-12T15:03:22Z

Commit modified

VinceMacBuche · 2024-01-12T15:04:49Z

Commit modified

VinceMacBuche · 2024-01-12T15:09:22Z

Commit modified

fanf · 2024-01-12T17:16:51Z

...p/sources/rudder/rudder-core/src/main/scala/com/normation/rudder/score/ComplianceScore.scala

+
+object ComplianceScoreEventHandler extends ScoreEventHandler {
+  implicit val compliancePercentEncoder: JsonEncoder[ComplianceSerializable]     = DeriveJsonEncoder.gen
+  def handle(event: ScoreEvent):         PureResult[List[(NodeId, List[Score])]] = {


it looks like we always have that structure:

a Score computation from a subtype of ScoreEvent

a partial function from a ScoreEvent to a Score (ie a "can handle that type" + "if yes, use the computation function with the correct subtype of score event"

I think you need to split appart these two bits so that the score from score event computation is unit-testable by itself on two aspects:

the correctness of the computation (easy here, but it will be a kind of spec so users will question it a lot, we need to be able to accumulate all the strange corner case they will find)

the JSON format, because it becomes a storage format that will need to never change without a migration

Plus, once you have that separation, the implementation of the handle will be almost automatic, so more easy for other people needing to do so to understand what's going on.

fanf

LGTM for me, once spotless/tests pass.
We tracked all the non blocking change elsewhere for latter PR.

clarktsiory · 2024-01-12T17:41:17Z

webapp/sources/rudder/rudder-web/src/main/elm/sources/SystemUpdateScore.elm

+main =
+  Browser.element
+    { init = init
+    , view = always (Html.text "")


Is it intended to not display anything ?

Yep! these apps only exist to provide details for generic node details api that does not know how to handle them (in fact it could handle system-updates and compliance by itself but i wanted them to be like the plugins so that we know the recipe to make a plugin from Rudder

clarktsiory · 2024-01-12T17:45:40Z

.../rudder-core/src/main/scala/com/normation/rudder/services/reports/ReportingServiceImpl.scala

@@ -393,6 +397,7 @@ trait CachedFindRuleNodeStatusReports
      // * no report from the node (compliance expires): recompute compliance
      {
        (for {
+          _ <- ReportLoggerPure.warn(s"Hello ! ${actions} ")


Please do not forget to remove this !

clarktsiory · 2024-01-12T18:01:29Z

...p/sources/rudder/rudder-core/src/main/scala/com/normation/rudder/score/ScoreRepository.scala

+  implicit val scoreWithIdRead: Read[(NodeId, Score)] = {
+    Read[(String, ScoreValue, String, String, Json)].map { d: (String, ScoreValue, String, String, Json) =>
+      (NodeId(d._1), Score(d._3, d._2, d._4, d._5))
+    }
+  }
+
+  import doobie._
+  override def getAll(): IOResult[Map[NodeId, List[Score]]] = {
+    val q = sql"select nodeId, score, name, message, details from scoreDetails "
+    transactIOResult(s"error when getting scores for node")(xa => q.query[(NodeId, Score)].to[List].transact(xa))
+      .map(_.groupMap(_._1)(_._2))
+  }


Those tuple indices look really confusing between both implicits (but they look correct)
Normally doobie can generate the Read[(NodeId, Score)] by re-using the Read[Score] so we can remove the noise from scoreWithIdRead here

VinceMacBuche · 2024-01-15T08:35:35Z

Commit modified

VinceMacBuche · 2024-01-15T09:36:44Z

Commit modified

VinceMacBuche · 2024-01-15T09:44:29Z

PR updated with a new commit

clarktsiory · 2024-01-15T10:08:16Z

...p/sources/rudder/rudder-core/src/main/scala/com/normation/rudder/score/ScoreRepository.scala

+    }
+
+    transactIOResult(s"error when inserting global score for node '${nodeId.value}''")(xa =>
+      query.updateWithLabel("test").run.transact(xa)


clarktsiory · 2024-01-15T10:17:34Z

webapp/sources/rudder/rudder-core/src/main/resources/reportsSchema.sql

+, score   text
+, message text
+, details jsonb
+, PRIMARY KEY (nodeId, scoreIdé)


scoreIdé !

VinceMacBuche · 2024-01-15T10:36:02Z

PR updated with a new commit

clarktsiory

LGTM, it only misses the database migration and the not null constraint ! (And of course to fix the compilation errors)

VinceMacBuche · 2024-01-15T11:10:35Z

Commit modified

VinceMacBuche · 2024-01-15T15:41:42Z

Commit modified

VinceMacBuche · 2024-01-15T15:42:45Z

PR rebased

Normation-Quality-Assistant · 2024-01-15T16:29:52Z

OK, merging this PR

VinceMacBuche requested a review from fanf December 29, 2023 00:37

VinceMacBuche force-pushed the ust_23967/rudder_score_backend_and_api branch 2 times, most recently from 88904b5 to a74abb4 Compare December 29, 2023 15:47

VinceMacBuche force-pushed the ust_23967/rudder_score_backend_and_api branch from 385e026 to db81b09 Compare January 9, 2024 15:32

VinceMacBuche force-pushed the ust_23967/rudder_score_backend_and_api branch from db81b09 to 517e3c5 Compare January 9, 2024 16:02

clarktsiory self-requested a review January 10, 2024 13:14

VinceMacBuche force-pushed the ust_23967/rudder_score_backend_and_api branch from 517e3c5 to cff2251 Compare January 10, 2024 18:00

VinceMacBuche changed the base branch from master to branches/rudder/8.1 January 11, 2024 07:56

VinceMacBuche force-pushed the ust_23967/rudder_score_backend_and_api branch from cff2251 to 5b03248 Compare January 11, 2024 11:43

VinceMacBuche force-pushed the ust_23967/rudder_score_backend_and_api branch from 5b03248 to 70d93b4 Compare January 11, 2024 15:10

fanf reviewed Jan 11, 2024

View reviewed changes

VinceMacBuche force-pushed the ust_23967/rudder_score_backend_and_api branch 2 times, most recently from caa95a6 to b7f81cc Compare January 11, 2024 16:51

fanf reviewed Jan 11, 2024

View reviewed changes

VinceMacBuche force-pushed the ust_23967/rudder_score_backend_and_api branch from 504926a to 452574c Compare January 12, 2024 15:03

VinceMacBuche force-pushed the ust_23967/rudder_score_backend_and_api branch from 452574c to a629025 Compare January 12, 2024 15:04

VinceMacBuche force-pushed the ust_23967/rudder_score_backend_and_api branch from a629025 to 8cdb57e Compare January 12, 2024 15:09

fanf reviewed Jan 12, 2024

View reviewed changes

clarktsiory reviewed Jan 12, 2024

View reviewed changes

VinceMacBuche force-pushed the ust_23967/rudder_score_backend_and_api branch 2 times, most recently from 75e02da to 9445f91 Compare January 15, 2024 08:36

VinceMacBuche force-pushed the ust_23967/rudder_score_backend_and_api branch from 9445f91 to ea1469d Compare January 15, 2024 09:36

clarktsiory reviewed Jan 15, 2024

View reviewed changes

VinceMacBuche force-pushed the ust_23967/rudder_score_backend_and_api branch from 4e54e58 to c053316 Compare January 15, 2024 11:10

VinceMacBuche force-pushed the ust_23967/rudder_score_backend_and_api branch from c053316 to f91eacc Compare January 15, 2024 15:41

Fixes #23967: Rudder score backend and API

df8b61d

VinceMacBuche force-pushed the ust_23967/rudder_score_backend_and_api branch from f91eacc to df8b61d Compare January 15, 2024 15:42

VinceMacBuche added the Ready for merge label Jan 15, 2024

Normation-Quality-Assistant merged commit df8b61d into Normation:branches/rudder/8.1 Jan 15, 2024
16 checks passed

Fixes #23967: Rudder score backend and API #5296

Fixes #23967: Rudder score backend and API #5296

Conversation

VinceMacBuche commented Dec 29, 2023

VinceMacBuche commented Dec 29, 2023

VinceMacBuche commented Dec 29, 2023

VinceMacBuche commented Jan 7, 2024

VinceMacBuche commented Jan 9, 2024

VinceMacBuche commented Jan 10, 2024

VinceMacBuche commented Jan 11, 2024

VinceMacBuche commented Jan 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VinceMacBuche commented Jan 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fanf Jan 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VinceMacBuche commented Jan 12, 2024

VinceMacBuche commented Jan 12, 2024

VinceMacBuche commented Jan 12, 2024

Choose a reason for hiding this comment

fanf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VinceMacBuche commented Jan 15, 2024

VinceMacBuche commented Jan 15, 2024

VinceMacBuche commented Jan 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VinceMacBuche commented Jan 15, 2024

clarktsiory left a comment

Choose a reason for hiding this comment

VinceMacBuche commented Jan 15, 2024

VinceMacBuche commented Jan 15, 2024

VinceMacBuche commented Jan 15, 2024

Normation-Quality-Assistant commented Jan 15, 2024

fanf Jan 11, 2024 •

edited

Loading