Better handling of concurrent integration #217
Assignees
Labels
feature
use for describing a new feature to develop
solved
use to identify issue that has been solved (must be linked to the solving PR)
technical improvement
Technical refactor or improvement is needed
Milestone
Comments
SamuelHassine
added
feature
SamuelHassine
pushed a commit
that referenced
this issue
Oct 17, 2019
SamuelHassine
added
the
solved
SamuelHassine
pushed a commit
that referenced
this issue
Oct 18, 2019
SamuelHassine
pushed a commit
that referenced
this issue
Oct 18, 2019
SamuelHassine
pushed a commit
that referenced
this issue
Oct 18, 2019
SamuelHassine
pushed a commit
that referenced
this issue
Oct 18, 2019
richard-julien
pushed a commit
that referenced
this issue
Oct 19, 2019
SamuelHassine
pushed a commit
that referenced
this issue
Oct 22, 2019
…chitecture (#130) * Starting work around minio * Introduce File manager component and use it in report * Adapt file manager UI * Prepare the export api * Refactor to move import and export files at stix entity level * Adapt view and backend to log file actions * Add file management in thread actors * Refactor functions to move everything we need at StixDomainEntity level * Remove unused info and add capacity to delete in progress export after 5 min * Adapt deletion + fix dependencies errors from yarn control * Start transformation of the worker. Provide API round queue management * Introducing Jobs and failure reporting * Re activate auto refresh * Add jobs management, massive update * Fix relayjs query typo * Introduce global import menu. Adapt the entity import and export * Fixing tests by replacing the matchAll function * Handle deletion of global imports * Remove unused import * Adapt usage of interface, update of dependencies * [frontend/api] Introduce key for internal_id and stix_id for avoid race conditions (#217) * [api] Fix migations if starting fresh (#217) * [api] Fix migations if starting fresh (#217) * [api] Fix migations if starting fresh (#217) * [api] Fix migations if starting fresh (#217) * Adapt work and jobs to use elastic. Worker changed to report jobs * Cleanup the grakn schema * Fix comments * Add delete button in observable enrichment * Refactoring of createdByRef, markingDef and KillChain functions * Fix duplicate and adapt readtx * Fix deepScan warning * Fix deepScan warning * Adapt schema for internal_id_key * Adapt for internal_key * [api/frontend] Enhance display and fix some bugs
SamuelHassine
pushed a commit
that referenced
this issue
Oct 22, 2019
…chitecture (#130) * Starting work around minio * Introduce File manager component and use it in report * Adapt file manager UI * Prepare the export api * Refactor to move import and export files at stix entity level * Adapt view and backend to log file actions * Add file management in thread actors * Refactor functions to move everything we need at StixDomainEntity level * Remove unused info and add capacity to delete in progress export after 5 min * Adapt deletion + fix dependencies errors from yarn control * Start transformation of the worker. Provide API round queue management * Introducing Jobs and failure reporting * Re activate auto refresh * Add jobs management, massive update * Fix relayjs query typo * Introduce global import menu. Adapt the entity import and export * Fixing tests by replacing the matchAll function * Handle deletion of global imports * Remove unused import * Adapt usage of interface, update of dependencies * [frontend/api] Introduce key for internal_id and stix_id for avoid race conditions (#217) * [api] Fix migations if starting fresh (#217) * [api] Fix migations if starting fresh (#217) * [api] Fix migations if starting fresh (#217) * [api] Fix migations if starting fresh (#217) * Adapt work and jobs to use elastic. Worker changed to report jobs * Cleanup the grakn schema * [frontend] Working on diagram! * Fix comments * Add delete button in observable enrichment * [frontend] Working on graph * Upgrade dependencies * Refactoring of createdByRef, markingDef and KillChain functions * Fix duplicate and adapt readtx * Fix deepScan warning * Fix deepScan warning * [frontend] Handle relation to relation in the knowledge graph * Adapt schema for internal_id_key * Adapt for internal_key * [frontend] Handle relation to relation * [frontend] Graph is now working (#34) * [frontend] Graph is now working (#34) * [frontend] Add marking definitions to StixDOmainEntity edition * [frontend] Add marking definitions to StixDOmainEntity edition * [frontend] Fix relation actions icons and entities update * [api/frontend] Enhance display and fix some bugs * [frontend] Refactor the views of files management
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem to Solve
With the current splitting of stix2 in the client-python and the way the worker connector absorb the data, we can create some element duplications.
Current Workaround
Limit the number of worker to 1. That is not really cool for absorption speed.
Proposed Solution
STIX_ID is a key
Move STIX_ID to key in grakn schema.
MISP Connector
Adapt the UUID generator to generate the same UUID for the same element name (lowercase).
CVE Connector
Adapt the UUID generator to generate the same UUID for the same CVE Key.
All connectors (extra)
Implements the local state storage to make connector smarter.
Connector docker must have a volume accessible to store his state. Base on this state, the connector should not retrieve everything.
Worker
Limitations
With multiple different connectors working on the same data at the same time but with a different STIX_ID it will be possible to have a duplication.
Additional Information
Should also fix OpenCTI-Platform/client-python#18
Hints
STIX_ID migration https://discuss.grakn.ai/t/key-attribute-and-migration/1293
The text was updated successfully, but these errors were encountered: