Full refactor of files storage and fetching to avoid querying S3 when not necessary

[SamuelHassine]

Use case

Today, a lot of components in the OpenCTI frontend (UI, Python library) are requesting the files of an entity each time we read it. This leads a lot of organizations to have extra-cost on their S3 buckets and this has a lot of limitation:

• No real sorting / pagination
• No search on file
• Performance and latency
• etc.

To be clear: all features implemented in the ElasticSearch engine are not available when it comes to handle files (including mass deletion, etc.). This is a high priority for a lot of OpenCTI users and customers.

Solution discussed with @Kedae and @richard-julien: implement files representation in ElasticSearch, maintain synchronization between S3 and Elastic and fetch S3 content only when needed (file read or download).

[ckane]

I'll also add that another downside to the present design is that the additional S3 querying introduces additional latency, in addition to the cost overhead mentioned. When I've refactored some of the connectors to address this (most recently, the yara connector) it speeds up the processing rate of the connector, too.

[SamuelHassine]

Definitely @ckane!