Implement performances test infrastructure

[SamuelHassine]

Problem to Solve

Be able to track ingestion performances in various use cases in time or from one version to the next one.

Current Workaround

Manual tests.

Proposed Solution

Deploy a performance test infrastructure with beautiful reports. Ask @SamuelHassine for various datasets (small, big, lot of embedded relations, lot of stix relations, lot of observables, etc.)

Additional Information

None.

[cvdsouza]

this would be really useful especially for those adopting openCTI in prod, to gauge performance of deployment.

[richard-julien]

Starting to work on it.

• Global performance timing for specific amount of data to inject on different target architecture.
• Specific metrics
  -- Memory allocation for opencti nodejs process.
  -- GraphQL response times
  -- Dependencies metrics??

[SamuelHassine]

• Memory allocation of Python processes
• CPU consumption?
• When the Python library is doing several calls (reading, then writing, then adding some embedded relations such as external references), it could be useful to have the timing of each of the calls.

[cvdsouza]

Just wondering here, I have seen errors like this constantly :

{"message":"[ELASTIC] 2afdd443-0405-4e89-a169-7c2d2bc85f80 missing, cant update the element, you need to reindex","level":"error"}
{"message":"[ELASTIC] d4c1410c-4ef4-48eb-8f1f-be1cdd66f505 missing, cant update the element, you need to reindex","level":"error"}

Questions / Recommendations :

• Would there be a way to determine how long it would take to re-index ?
• Metrics to determine if re-index is needed , i.e. elastic not in sync with grakn for some reason?

My thought around this was, it would help users understand if / when they should run a reindex.