[backend/frontend] Use admiralty scale for reliability and confidence (#2362) #3719

SouadHadjiat · 2023-07-11T15:04:02Z

Proposed changes

MVP ready

Add reliability open vocabulary (migration)
Add reliability field for Report, Individual and System (same as for Organization)
Client python PR : [client] add reliability in Report (#opencti/2362) client-python#423
Display reliability of author for all entities, for reports we display the report reliability if set
Display confidence along with reliability on the left
Set default confidence scale to admiralty scale
Update tests : since confidence default value is now 0, and 6 vocabulary values have been added (+6 creation events)

Nice to have done

Improve confidence level style (to be able to select a confidence level directly)
Add confidence templates to be applied in confidence scale configuration
Update confidence filter to use the confidence scale for each entity type
Add reliability filter for Reports, Organizations, Individuals and Systems
TODO : add "reliability of author" filter for all stix domain entities
TODO : fix "custom" issue when editing a confidence scale configuration and add a dialog if configuration is not saved

Related issues

Consider adding a custom confidence scale for sources and contents #2362

Checklist

I consider the submitted work as finished
I tested the code for its functionality
I wrote test cases for the relevant uses case
I added/update the relevant documentation (either on github or on notion)
Where necessary I refactored code to improve the overall quality

Further comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc...

SouadHadjiat · 2023-07-19T12:27:01Z

PR status
MVP : ready for review
Nice to have (confidence templates) : still to do, will be added in the PR soon.

opencti-platform/opencti-front/src/components/InputSliderField.tsx

opencti-platform/opencti-front/src/components/ItemConfidence.tsx

opencti-platform/opencti-front/src/components/ItemOpenVocab.tsx

opencti-platform/opencti-front/src/private/components/analyses/reports/ReportCreation.tsx

...ti-platform/opencti-front/src/private/components/entities/individuals/IndividualCreation.tsx

...opencti-front/src/private/components/common/stix_domain_objects/StixDomainObjectOverview.jsx

...platform/opencti-front/src/private/components/entities/organizations/OrganizationDetails.jsx

opencti-platform/opencti-front/src/utils/hooks/useScale.ts

RomuDeuxfois · 2023-07-24T06:42:25Z

We have two reliability fields in individual entity now (same as System and Organization)

In edit confidence attribute (ex: for a report), If I change from custom to standard without update, It's not possible to retrieve my custom template
screen-capture (6).webm

We can't edit the confidence level in some entities (ex: grouping, region, country, ...). The field is not present in form edit.

marieflorescontact · 2023-07-26T12:25:26Z

Locally tested, seems all good.

Some UX comments:
- in settings > customization > entity_type, if we modify the reliability of an entity (for example, an individual), we need to remember to click on the Update button to memorize the Selected scale template Custom , except that you have to scroll down to see the Update button.
- In a report, the reliability of the report is in the same place as the reliability of the author for the entities. it's a bit confusing.

marieflorescontact · 2023-07-25T15:48:54Z

opencti-platform/opencti-graphql/tests/02-integration/01-database/stix-test.js

-      if (remainingData.confidence) {
-        expect(remainingData.confidence).toEqual(15);
+      if (remainingData.confidence !== undefined) {
+        // expect(remainingData.confidence).toEqual(15); // can't test the confidence value, could be 15 ou 0


Why can't we test confidence value?

maybe you should remove the test line if it's no longer useful?

SarahBocognano

Tested : ✅
The only question that remains for me is : Isn't that weird that in some entities (like locations ones) we can see a level of confidence in the stix domain object overview established at the lowest level (truth cannot be judged) and it's impossible to update it ?
Like maybe we can change it to "don't have any info" or something like that for entities that have not the possibility to update their confidence level.
I didn't check the stix policy on that, if it's part of it, ignore my question ^^

SarahBocognano · 2023-07-27T08:45:59Z

opencti-platform/opencti-front/src/utils/filters/useSearchEntities.tsx

@@ -990,6 +991,25 @@ const useSearchEntities = ({
            );
          });
        break;
+      case 'x_opencti_reliability':


Is there a reason why there's nothing here ?

it's because it's the same case as the next one, there is no break. (the two cases are the same)

SouadHadjiat · 2023-07-27T09:32:08Z

Tested : ✅ The only question that remains for me is : Isn't that weird that in some entities (like locations ones) we can see a level of confidence in the stix domain object overview established at the lowest level (truth cannot be judged) and it's impossible to update it ? Like maybe we can change it to "don't have any info" or something like that for entities that have not the possibility to update their confidence level. I didn't check the stix policy on that, if it's part of it, ignore my question ^^

you're right, I'm having a discussion about this display issue with @Jipegien at the moment :)

SouadHadjiat · 2023-07-27T09:36:23Z

PR status

All raised isssues should be fixed now :

confidence scale configuration update UX : selector with custom that disapears when selecting a template => Done ✅
confidence and reliability display for Systems / individuals / organization => Done ✅
Working on adding the filter "Reliability of author" for all stix domain entities (it has been done for Malware only for now) => Done ✅

opencti-platform/opencti-front/src/private/components/threats/IntrusionSets.jsx

opencti-platform/opencti-front/src/private/components/threats/ThreatActorsGroup.jsx

...ti-front/src/private/components/settings/sub_types/scaleConfiguration/ScaleConfiguration.tsx

opencti-platform/opencti-front/src/components/ItemOpenVocab.tsx

…#2362)

… selection

…ity types

Archidoit · 2023-08-02T07:23:07Z

When you put a negative value as confidence scale, the value is accepted and the confidence is unknown:

Archidoit · 2023-08-02T07:27:35Z

Create a reliability vocabulary with a float number (ex: 3,4). When you click on the 'create' button, the vocabulary is created with an integer (3). We should maybe inform the user that the value is not correct instead of modifying the entered value.

SouadHadjiat · 2023-08-02T08:04:30Z

When you put a negative value as confidence scale, the value is accepted and the confidence is unknown:

yes all confidence values are accepted, the scale is only used to display the label. When the value is lower than the minimum defined in the scale, we display "none".

SouadHadjiat · 2023-08-02T08:05:24Z

Create a reliability vocabulary with a float number (ex: 3,4). When you click on the 'create' button, the vocabulary is created with an integer (3). We should maybe inform the user that the value is not correct instead of modifying the entered value.

I guess this is the case for all vocabularies, not spectifically Reliability ?

SouadHadjiat mentioned this pull request Jul 11, 2023

[client] add reliability in Report (#opencti/2362) OpenCTI-Platform/client-python#423

Merged

5 tasks

SouadHadjiat force-pushed the issue/2362 branch from 602c5c2 to 4fee3d4 Compare July 13, 2023 15:20

Kedae added the filigran team use to identify PR from the Filigran team label Jul 17, 2023

SouadHadjiat force-pushed the issue/2362 branch 2 times, most recently from 05e7623 to abaf79c Compare July 19, 2023 08:28

SouadHadjiat marked this pull request as ready for review July 19, 2023 12:25

RomuDeuxfois reviewed Jul 21, 2023

View reviewed changes

SouadHadjiat force-pushed the issue/2362 branch from 0e5febe to 1fddd0d Compare July 21, 2023 08:58

SamuelHassine force-pushed the master branch 2 times, most recently from 725e4e4 to 4cca592 Compare July 21, 2023 09:21

SouadHadjiat force-pushed the issue/2362 branch from 18e8187 to 7fa78fd Compare July 25, 2023 09:28

marieflorescontact reviewed Jul 26, 2023

View reviewed changes

SarahBocognano reviewed Jul 27, 2023

View reviewed changes

SouadHadjiat force-pushed the issue/2362 branch 2 times, most recently from 6f17f07 to d27bebf Compare July 28, 2023 15:51

RomuDeuxfois reviewed Aug 1, 2023

View reviewed changes

SouadHadjiat added 9 commits August 1, 2023 14:23

[backend/frontend] Use admiralty scale for reliability and confidence (…

1d517c4

…#2362)

[frontend] add confidence scale templates

3eb2c47

[frontend] add missing translations

ae8530e

[backend/frontend] PR review fixes

7f00670

[frontend] improve display of reliability and confidence

8ba1443

[backend/frontend] add reliability filter and update confidence filter

f4361cd

[frontend] fix item vocab introduced bug + formatting

6365a63

[backend/frontend] add filter reliability of author

f3e39f4

[frontend] cleanup

e53fe87

SouadHadjiat added 3 commits August 1, 2023 14:24

[frontend] fix confidence scale configuration issue with custom scale…

f47ef1d

… selection

[frontend] fix display of confidence and reliability according to ent…

92abac7

…ity types

[backend/frontend] add source_reliability filter for all entities

29c61ae

SouadHadjiat force-pushed the issue/2362 branch from d27bebf to 29c61ae Compare August 1, 2023 12:25

SouadHadjiat added 2 commits August 1, 2023 15:36

[backend/frontend] PR review fixes

05e69f2

[backend] fix deepscan issue

98129d6

RomuDeuxfois approved these changes Aug 2, 2023

View reviewed changes

SouadHadjiat merged commit b4d1bf4 into master Aug 2, 2023
6 checks passed

SouadHadjiat deleted the issue/2362 branch August 2, 2023 08:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[backend/frontend] Use admiralty scale for reliability and confidence (#2362) #3719

[backend/frontend] Use admiralty scale for reliability and confidence (#2362) #3719

SouadHadjiat commented Jul 11, 2023 •

edited

SouadHadjiat commented Jul 19, 2023

RomuDeuxfois commented Jul 24, 2023

marieflorescontact commented Jul 26, 2023

marieflorescontact Jul 25, 2023

marieflorescontact Jul 26, 2023

SarahBocognano left a comment

SarahBocognano Jul 27, 2023

SouadHadjiat Jul 27, 2023

SouadHadjiat commented Jul 27, 2023

SouadHadjiat commented Jul 27, 2023 •

edited

Archidoit commented Aug 2, 2023

Archidoit commented Aug 2, 2023

SouadHadjiat commented Aug 2, 2023

SouadHadjiat commented Aug 2, 2023

[backend/frontend] Use admiralty scale for reliability and confidence (#2362) #3719

[backend/frontend] Use admiralty scale for reliability and confidence (#2362) #3719

Conversation

SouadHadjiat commented Jul 11, 2023 • edited

Proposed changes

MVP ready

Nice to have done

Related issues

Checklist

Further comments

SouadHadjiat commented Jul 19, 2023

RomuDeuxfois commented Jul 24, 2023

marieflorescontact commented Jul 26, 2023

marieflorescontact Jul 25, 2023

Choose a reason for hiding this comment

marieflorescontact Jul 26, 2023

Choose a reason for hiding this comment

SarahBocognano left a comment

Choose a reason for hiding this comment

SarahBocognano Jul 27, 2023

Choose a reason for hiding this comment

SouadHadjiat Jul 27, 2023

Choose a reason for hiding this comment

SouadHadjiat commented Jul 27, 2023

SouadHadjiat commented Jul 27, 2023 • edited

PR status

All raised isssues should be fixed now :

Archidoit commented Aug 2, 2023

Archidoit commented Aug 2, 2023

SouadHadjiat commented Aug 2, 2023

SouadHadjiat commented Aug 2, 2023

SouadHadjiat commented Jul 11, 2023 •

edited

SouadHadjiat commented Jul 27, 2023 •

edited