Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why is there a limit of 5000 entities in a csv feed? #4636

Closed
RegturH opened this issue Oct 17, 2023 · 2 comments · Fixed by #5061
Closed

Why is there a limit of 5000 entities in a csv feed? #4636

RegturH opened this issue Oct 17, 2023 · 2 comments · Fixed by #5061
Assignees
Labels
feature use for describing a new feature to develop solved use to identify issue that has been solved (must be linked to the solving PR)

Comments

@RegturH
Copy link

RegturH commented Oct 17, 2023

Description

Limit of 5.000 entities in csv feed. Why is there a limit on entities in a csv feed?

This is probably for performance but I would like to know if we can increase this?

Environment

  1. OpenCTI version: 5.11.2
  2. OpenCTI client: front-end

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Data-> Data Sharing->CSv-feed
  2. Create a feed with more than 5.000 entities
  3. it will only show till 5.000

Solution

Introduce an option to increase the size

data_sharing:max_csv_feed_result

@RegturH RegturH added needs triage use to identify issue needing triage from Filigran Product team question Further information is requested labels Oct 17, 2023
@richard-julien
Copy link
Member

The concept of the CSV feeds is have information rolling on a time period. In order to prevent some performance problem we decide to put a limit to 5000 elements on the rolling period. I recently introduced an option but it require to change also another option to make it work and it was not expected. I will rework that part to correctly handle the option soon.

@RegturH
Copy link
Author

RegturH commented Oct 18, 2023

Hey Julien,

This is wonderful! Thank you for making this possible in the future :)

@SamuelHassine SamuelHassine added feature use for describing a new feature to develop and removed question Further information is requested needs triage use to identify issue needing triage from Filigran Product team labels Oct 18, 2023
@SamuelHassine SamuelHassine added this to the Release 5.11.5 milestone Oct 18, 2023
@richard-julien richard-julien self-assigned this Oct 18, 2023
@richard-julien richard-julien added the solved use to identify issue that has been solved (must be linked to the solving PR) label Dec 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature use for describing a new feature to develop solved use to identify issue that has been solved (must be linked to the solving PR)
Projects
None yet
3 participants