[backend/frontend] Add user and group confidence level (#4304) #5436

labo-flg · 2024-01-09T10:24:16Z

Proposed changes

This PR addresses the following issues:

Have a look at each issue description for details.

Code review highlights

The key points to review are backend additions and changes, the model definition.
A lot of stuff in the frontend besides the core job (refactor in TS, solving react warnings) make the review difficult, I'm sorry.

How can you test this PR?

Here are some scenarios to test:

start the platform on this branch -> you shall see the critical alert dialog, inviting you to edit all your groups
Create a user without groups -> they should be created with default groups
Create a user with groups -> they should be created with the specified groups, and not the default group
Create a user without a confidence level and no group
- it should be displayed as in error in the various view (list, details, list of users in a group details view) -> icon + tooltip
- edit the user : the message above the confidence level field shall be explicit (no group to inherit from)
- edit the user, tab groups : the confidence of each group shall be displayed, with red icon and tooltip if not defined
- edit the user : set a group in the groups tab, go back to overview -> the message above the confidence level field shall be explicit (inherited value)
- edit the user : set the user's confidence level -> -> the message above the confidence level field shall be explicit (user value)
Create a user without a confidence level and no group -> it should be displayed as in error in the various view (list, details, list of users in a group details view)
Create a user with a confidence level -> it should be displayed in the various view (list, details, list of users in a group details view) -> tooltip shall state it's coming from user
Create a user without a confidence level but with a group -> it should be displayed in the various view (list, details, list of users in a group details view) -> tooltip shall state it's coming from a specific group - link clickable in tooltip
Create a group, you must select a confidence level -> value is then displayed correctly in list and details
Open a group without confidence level -> explicit alert in details -> disappears when edited
all groups are set with a confidence level -> refresh page -> no more alert dialog

If you wish to artificially nuke your group confidence level for testing purpose, a GQL query on playground is possible (there is no validation implemented yet - see #5696).

mutation GroupEditionOverviewFieldPatchMutation(
  $id: ID!
  $input: [EditInput]!
) {
  groupEdit(id: $id) {
    fieldPatch(input: $input) {
      id
      group_confidence_level {
        max_confidence
      }
    }
  }
}

{
  "id": "4b762ee9-b2da-49fe-99d7-dfd6b9e1fe33",
  "input": {
    "key": "group_confidence_level",
    "value": [null]
  }
}

Checklist

I consider the submitted work as finished
I tested the code for its functionality
I wrote test cases for the relevant uses case
I added/update the relevant documentation (either on github or on notion)
Where necessary I refactored code to improve the overall quality

Further comments

Now that the new schema definition is in place, the job done on the subject of user's confidence level has been fully reworked.
Thus this PR invalidates the previous PR on the subject: #5283 #5292 #5323

opencti-platform/opencti-front/src/private/components/settings/users/UserEditionOverview.tsx

opencti-platform/opencti-graphql/src/migrations/1704792752287-user-confidence-level.js

...cti-platform/opencti-graphql/src/modules/attributes/internalObject-registrationAttributes.ts

codecov · 2024-01-09T10:56:56Z

Codecov Report

Attention: 31 lines in your changes are missing coverage. Please review.

Comparison is base (61c0432) 66.12% compared to head (1344932) 66.15%.

Files	Patch %	Lines
...ti-platform/opencti-graphql/src/domain/settings.js	15.38%	22 Missing ⚠️
...pencti-platform/opencti-graphql/src/domain/user.js	91.42%	6 Missing ⚠️
...encti-platform/opencti-graphql/src/domain/grant.js	75.00%	2 Missing ⚠️
...cti-platform/opencti-graphql/src/resolvers/user.js	90.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5436      +/-   ##
==========================================
+ Coverage   66.12%   66.15%   +0.03%     
==========================================
  Files         513      513              
  Lines       60641    60793     +152     
  Branches     4434     4448      +14     
==========================================
+ Hits        40097    40218     +121     
- Misses      20544    20575      +31

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

labo-flg · 2024-01-10T08:12:56Z

I'm able to send through the API a partial update like

{
  "id": "<id>",
  "input": {
  "key": "user_confidence_level",
    "value": {
    "max_confidence": 96
  }
}

Which will update the user_confidence_level with overrides to null, causing GQL error at query time as this field is supposedly never null.
I'll check the model, maybe around bad mandatoryType

lndrtrbn

Seems good to me, but I'll let someone else have a look at migration and attributes registrations 😅

opencti-platform/opencti-front/src/private/components/settings/users/UserConfidenceLevel.tsx

opencti-platform/opencti-graphql/config/default.json

labo-flg · 2024-01-10T15:39:25Z

Seems good to me, but I'll let someone else have a look at migration and attributes registrations 😅

Thanks !

labo-flg · 2024-01-12T08:46:46Z

back to draft : I need to make sure partial edit payload are not accepted

labo-flg · 2024-01-16T09:28:58Z

We need to add schema validation when patching a field.
This is significant work that deserve a dedicated branch/PR.

Everything in this PR is ok, but we need this validation before merging to master.
I will use this branch as a feature branch, and merge into it the validation.

EDIT: validation will be done in another issue (#5696) , this branch will still serve as feature branch

labo-flg · 2024-01-25T17:39:08Z

opencti-platform/opencti-front/src/components/InputSliderField.tsx

-        <Grid container={true} spacing={3} alignItems="end">
+        <Grid container={true} spacing={3} >


this align the 2 inputs (input number + scale Selector) to the top, keeping alignments when the input number shows an error label below.

labo-flg · 2024-01-25T19:22:10Z

opencti-platform/opencti-front/src/private/Index.tsx

      <SettingsMessagesBanner />
+      <PlatformCriticalAlertDialog alerts={settings.platform_critical_alerts}/>


New popup dialog for admins when the backend has some "critical alerts" to show.
This is for warning about groups with null confidence levels, but could be extended to other types of alerts

labo-flg · 2024-01-25T19:23:03Z

opencti-platform/opencti-front/src/private/Root.tsx

+    platform_critical_alerts {
+      message
+      type
+      details {
+        groups {
+          id 
+          name 
+        }
+      }
+    }


alert details shall depends on alert type... current implem is a bit rough on the edges but we had to move quickly about this.

labo-flg · 2024-01-25T19:24:21Z

opencti-platform/opencti-front/src/private/components/common/form/GroupField.tsx

+            const max_confidence = n?.node.group_confidence_level
+              ? `${t_i18n('Max Confidence Level:')} ${n.node.group_confidence_level.max_confidence}`
+              : t_i18n('No confidence level');
+            const newLabel = showConfidence
+              ? `${n?.node.name} (${max_confidence})`
+              : n?.node.name ?? '';


old GroupField Select, refactored to typescript, and added opt prop showConfidence to add the group confidence to the label after the name of the group.

labo-flg · 2024-01-25T19:25:38Z

opencti-platform/opencti-front/src/private/components/settings/Users.jsx

+      effective_confidence_level: {
+        label: 'confidence',
+        width: '10%',
+        isSortable: false,


effective confidence level is a computed value backend side, not an attribute in DB (depends on user's confidence + his groups confidence)

we cannot craft a query to do this computation, so we cannot paginate and sort properly this value.

labo-flg · 2024-01-25T19:37:50Z

opencti-platform/opencti-front/src/private/components/settings/groups/GroupEditionOverview.tsx

+        if (name === 'group_confidence_level') {
+          if (group.group_confidence_level) {
+            commitFieldPatch({
+              variables: {
+                id: group.id,
+                input: {
+                  key: 'group_confidence_level',
+                  object_path: '/group_confidence_level/max_confidence',
+                  value: parseInt(value, 10),
+                },
+              },
+            });
+          } else {
+            commitFieldPatch({
+              variables: {
+                id: group.id,
+                input: {
+                  key: 'group_confidence_level',
+                  value: {
+                    max_confidence: parseInt(value, 10),
+                    overrides: [],
+                  },
+                },
+              },
+            });
+          }


the else statement could work in both cases (existing confidence or not) right now, but we will implement soon the overrides definition and edition (6.1). By then we'll need to patch with object_path anyway.

labo-flg · 2024-01-25T19:46:30Z

opencti-platform/opencti-front/src/private/components/settings/users/UserCreation.jsx

+  user_confidence_level_enabled: Yup.boolean(),
+  user_confidence_level: Yup.number()
+    .min(0, t('The value must be greater than or equal to 0'))
+    .max(100, t('The value must be less than or equal to 100'))
+    .when('user_confidence_level_enabled', {
+      is: true,
+      then: (schema) => schema.required(t('This field is required')).nullable(),
+      otherwise: (schema) => schema.nullable(),
+    }),


a technical field (not to submit) associated with a when. Very useful for validation conditioned on other values!

labo-flg · 2024-01-25T19:56:43Z

opencti-platform/opencti-graphql/src/domain/user.js

+  return computeUserEffectiveConfidenceLevel(user);
+};
+
+export const computeUserEffectiveConfidenceLevel = (user) => {


function split for easy unit tests

labo-flg · 2024-01-25T20:09:30Z

opencti-platform/opencti-graphql/config/schema/opencti.graphql

+type PlatformCriticalAlertDetails {
+  groups: [Group!]!
+}


this is where the critical alert system is a bit limited, not well generalized. details shall depends on type.

I did not have time to overengineer this system, honestly it's a quick bandaid to help admins prepare for 6.0.

Not a problem, thats purely dynamic and so easy to change.

SouadHadjiat · 2024-01-26T08:24:07Z

opencti-platform/opencti-front/lang/en.json

@@ -1704,7 +1704,6 @@
  "INGESTION_MANAGER": "Ingestion manager",
  "PLAYBOOK_MANAGER": "Playbook manager",
  "FILE_INDEX_MANAGER": "File index manager",
-  "INDICATOR_DECAY_MANAGER": "Indicator decay manager",


why has this translation been removed ?

In must have executed an extract script in my branch, pre-rebase.
I'll check and restore the missing keys.

I've done a review, all keys not related to confidence level have been restored (there was a couple)

aHenryJard

I just did a "quick read" review so I share it, I will take take time for a deeper one later.

opencti-platform/opencti-front/lang/en.json

opencti-platform/opencti-front/lang/fr.json

opencti-platform/opencti-front/src/private/components/settings/groups/GroupLine.tsx

opencti-platform/opencti-front/src/private/components/settings/groups/GroupConfidenceLevel.tsx

opencti-platform/opencti-front/src/private/components/settings/users/UserConfidenceLevel.tsx

lndrtrbn · 2024-01-26T13:10:14Z

opencti-platform/opencti-front/src/private/components/settings/users/UserConfidenceLevel.tsx

+  const renderSource = () => {
+    const source = (confidenceLevel as Data_EffectiveConfidenceLevel)?.source;
+
+    // FIXME: if watching the current user's detailed view, the source is {}, hence the check if (source.entity_type && ...) below


FIXME later ?

I did not have time to investigate, but it's probably related to how we poorly handle the graphql cache records

Warning: RelayModernRecord: Invalid record update, expected both versions of record 88ec0c6a-13ce-5e39-b486-354fe4a7084f to have the same __typename but got conflicting types MeUser and User. The GraphQL server likely violated the globally unique id requirement by returning the same id for different objects.

Another whole issue to address, certainly not in this PR.

I'm editing myself, which collides with MeUser query for instance. The record gets "updated" wrongly somehow.

…r details

Co-authored-by: Laurent Bonnet <laurent.bonnet@filigran.io>

Co-authored-by: Sarah Bocognano <98532359+SarahBocognano@users.noreply.github.com>

Co-authored-by: BocognanoSarah <sarah.bocognano@filigran.io>

labo-flg requested a review from richard-julien January 9, 2024 10:24

labo-flg marked this pull request as ready for review January 9, 2024 10:24

labo-flg commented Jan 9, 2024

View reviewed changes

opencti-platform/opencti-front/src/private/components/settings/users/UserEditionOverview.tsx Outdated Show resolved Hide resolved

labo-flg commented Jan 9, 2024

View reviewed changes

opencti-platform/opencti-graphql/src/migrations/1704792752287-user-confidence-level.js Outdated Show resolved Hide resolved

labo-flg commented Jan 9, 2024

View reviewed changes

...cti-platform/opencti-graphql/src/modules/attributes/internalObject-registrationAttributes.ts Outdated Show resolved Hide resolved

labo-flg mentioned this pull request Jan 9, 2024

[backend] add user_confidence_level attribute to Users (#4303) #5283

Closed

5 tasks

labo-flg changed the title ~~Add user confidence level - over new schema definition (#4304)~~ [backend/frontend] Add user confidence level - over new schema definition (#4304) Jan 9, 2024

labo-flg self-assigned this Jan 9, 2024

labo-flg added the filigran team use to identify PR from the Filigran team label Jan 9, 2024

This was referenced Jan 9, 2024

[frontend] add the ability to edit user_confidence_level (#4304) #5323

Closed

[frontend] view user's confidence level (#4304) #5292

Closed

lndrtrbn self-requested a review January 10, 2024 08:28

lndrtrbn reviewed Jan 10, 2024

View reviewed changes

labo-flg marked this pull request as draft January 12, 2024 08:46

richard-julien force-pushed the master branch from b42d752 to d189a00 Compare January 12, 2024 23:15

labo-flg force-pushed the issue/4304-confidence-new-schema-def branch from f5d2686 to f376065 Compare January 16, 2024 09:24

labo-flg force-pushed the issue/4304-confidence-new-schema-def branch 2 times, most recently from 32b5a01 to 00ac6e9 Compare January 24, 2024 10:23

labo-flg changed the title ~~[backend/frontend] Add user confidence level - over new schema definition (#4304)~~ [backend/frontend] Add user and group confidence level (#4304) Jan 25, 2024

labo-flg force-pushed the issue/4304-confidence-new-schema-def branch from ce1c332 to 76e784f Compare January 25, 2024 16:11

labo-flg marked this pull request as ready for review January 25, 2024 16:52

labo-flg commented Jan 25, 2024

View reviewed changes

SouadHadjiat reviewed Jan 26, 2024

View reviewed changes

aHenryJard reviewed Jan 26, 2024

View reviewed changes

lndrtrbn reviewed Jan 26, 2024

View reviewed changes

lndrtrbn approved these changes Jan 26, 2024

View reviewed changes

labo-flg and others added 12 commits January 29, 2024 09:14

[backend/frontend] add user_confidence_level in model, display in Use…

b21e903

…r details

[frontend/backend] optional user confidence level (create/edit)

4e91598

Co-authored-by: Laurent Bonnet <laurent.bonnet@filigran.io>

[frontend] fix intl after rebase

e0a982c

[backend] add effective_confidence_level computed field to users (#4304)

e6cd366

Co-authored-by: Sarah Bocognano <98532359+SarahBocognano@users.noreply.github.com>

[frontend/backend] confidence level at the group level (#4304)

ebfb8cf

Co-authored-by: BocognanoSarah <sarah.bocognano@filigran.io>

[frontend/backend] multiple code tweaks after code review

91ab9c0

[frontend] repair intl files corrupted after merge

4515811

[backend/frontend] tweaks after review

925b9e1

[backend] add tests on confidence level

d43b041

[frontend] restore intl from previous PR

741bc10

[frontend] update intl after product review

eb8efb3

[frontend] tweak intl after rebase

1344932

labo-flg force-pushed the issue/4304-confidence-new-schema-def branch from 9f0452f to 1344932 Compare January 29, 2024 08:29

Kedae merged commit e3dd23b into master Jan 29, 2024
8 checks passed

labo-flg deleted the issue/4304-confidence-new-schema-def branch January 30, 2024 09:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[backend/frontend] Add user and group confidence level (#4304) #5436

[backend/frontend] Add user and group confidence level (#4304) #5436

labo-flg commented Jan 9, 2024 •

edited

codecov bot commented Jan 9, 2024 •

edited

labo-flg commented Jan 10, 2024

lndrtrbn left a comment

labo-flg commented Jan 10, 2024

labo-flg commented Jan 12, 2024

labo-flg commented Jan 16, 2024 •

edited

labo-flg Jan 25, 2024

labo-flg Jan 25, 2024

labo-flg Jan 25, 2024

labo-flg Jan 25, 2024

labo-flg Jan 25, 2024

labo-flg Jan 25, 2024

labo-flg Jan 25, 2024

labo-flg Jan 25, 2024

labo-flg Jan 25, 2024

richard-julien Jan 25, 2024

SouadHadjiat Jan 26, 2024

labo-flg Jan 26, 2024

labo-flg Jan 26, 2024

aHenryJard left a comment

lndrtrbn Jan 26, 2024

labo-flg Jan 26, 2024

labo-flg Jan 26, 2024

		<Grid container={true} spacing={3} alignItems="end">
		<Grid container={true} spacing={3} >

		<SettingsMessagesBanner />
		<PlatformCriticalAlertDialog alerts={settings.platform_critical_alerts}/>

[backend/frontend] Add user and group confidence level (#4304) #5436

[backend/frontend] Add user and group confidence level (#4304) #5436

Conversation

labo-flg commented Jan 9, 2024 • edited

Proposed changes

Code review highlights

How can you test this PR?

Checklist

Further comments

codecov bot commented Jan 9, 2024 • edited

Codecov Report

labo-flg commented Jan 10, 2024

lndrtrbn left a comment

Choose a reason for hiding this comment

labo-flg commented Jan 10, 2024

labo-flg commented Jan 12, 2024

labo-flg commented Jan 16, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aHenryJard left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

labo-flg commented Jan 9, 2024 •

edited

codecov bot commented Jan 9, 2024 •

edited

labo-flg commented Jan 16, 2024 •

edited