Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[file indexing] infinit retry on parsing #5629

Closed
pierremahot opened this issue Jan 22, 2024 · 1 comment · Fixed by #5631
Closed

[file indexing] infinit retry on parsing #5629

pierremahot opened this issue Jan 22, 2024 · 1 comment · Fixed by #5631
Assignees
Labels
bug use for describing something not working as expected critical use to identify critical bug to fix ASAP solved use to identify issue that has been solved (must be linked to the solving PR)

Comments

@pierremahot
Copy link
Member

Description

The indexing process retry on error when elastic respond an error on parsing the file

Environment

  1. docker / kube
  2. OpenCTI version: 5.12.20
  3. OpenCTI client: frontend

Reproducible Steps

1 start idexing
2 get error with elastic (don't know the exact cause can be no available heap related)

Expected Output

do not infinite retry

Actual Output

infinite retry.
this also restart the manager and update startup date of the manager.
get big logs because of the base 64 of the file send to the logs.

@pierremahot pierremahot added bug use for describing something not working as expected needs triage use to identify issue needing triage from Filigran Product team labels Jan 22, 2024
@richard-julien
Copy link
Member

Multiple problems here:

  • When a file fail indexing, the process retry every minute on the same file
    As we cant do anything, we only need to log the error instead of throw.

  • When a file fail indexing in elastic, we throw and log too much
    Instead of throw we will log (point 1) but we need to log the error without the body content as it contains the file in base64.

@richard-julien richard-julien added critical use to identify critical bug to fix ASAP and removed needs triage use to identify issue needing triage from Filigran Product team labels Jan 22, 2024
@SamuelHassine SamuelHassine added this to the Release 5.12.22 milestone Jan 22, 2024
@SouadHadjiat SouadHadjiat linked a pull request Jan 22, 2024 that will close this issue
5 tasks
@richard-julien richard-julien added the solved use to identify issue that has been solved (must be linked to the solving PR) label Jan 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug use for describing something not working as expected critical use to identify critical bug to fix ASAP solved use to identify issue that has been solved (must be linked to the solving PR)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants