Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

checkpoint: avoid data loss from loguid collisions #6483

Merged
merged 1 commit into from Jun 7, 2023
Merged

checkpoint: avoid data loss from loguid collisions #6483

merged 1 commit into from Jun 7, 2023

Conversation

efd6
Copy link
Contributor

@efd6 efd6 commented Jun 6, 2023

What does this PR do?

Because Checkpoint reuse loguids, it is possible for two logged events to be given the same fingerprint. So use the segment_time as well in the fingerprint args. Also document that if collisions are important, the user should enable semi-unified logging on the device's dashboard.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@efd6 efd6 self-assigned this Jun 6, 2023
@efd6
checkpoint: avoid data loss from loguid collisions
4fe4b88 
Because Checkpoint reuse loguids, it is possible for two logged events
to be given the same fingerprint. So use the segment_time as well in
the fingerprint args. Also document that if collisions are important,
the user should enable semi-unified logging on the device's dashboard.
@elasticmachine
Copy link

elasticmachine commented Jun 6, 2023 •
edited

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-06-06T03:53:46.452+0000

  • Duration: 15 min 0 sec

Test stats 🧪

Test Results
Failed 0
Passed 16
Skipped 0
Total 16

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (1/1) 💚
Files 100.0% (1/1) 💚 1.266
Classes 100.0% (1/1) 💚 1.266
Methods 100.0% (17/17) 💚 5.582
Lines 86.969% (901/1036) 👎 -6.594
Conditionals 100.0% (0/0) 💚

@efd6 efd6 marked this pull request as ready for review June 6, 2023 04:26
@efd6 efd6 requested a review from a team as a code owner June 6, 2023 04:26
@elasticmachine
Copy link

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

kcreddy
kcreddy approved these changes Jun 7, 2023
View reviewed changes
Copy link
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor clarification. LGTM 👍🏼

packages/checkpoint/_dev/build/docs/README.md
Comment on lines +42 to +43
In some instances firewall events may have the same Checkpoint `loguid` and arrive during the same timestamp resulting in a fingerprint collision. To avoid this [enable semi-unified logging](https://sc1.checkpoint.com/documents/R81/WebAdminGuides/EN/CP_R81_LoggingAndMonitoring_AdminGuide/Topics-LMG/Log-Exporter-Appendix.htm?TocPath=Log%20Exporter%7C_____9) in the Checkpoint dashboard.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to add this bit to Troubleshooting section instead? or does it look good here?
https://github.com/elastic/integrations/blob/main/docs/documentation_guidelines.md#troubleshooting

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intention is that this be considered at set-up so that users don't need to get to the stage of trouble-shooting.

@efd6 efd6 merged commit c9d9974 into elastic:main Jun 7, 2023
3 checks passed
@elasticmachine
Copy link

Package checkpoint - 1.22.0 containing this change is available at https://epr.elastic.co/search?package=checkpoint

sodhikirti07 pushed a commit that referenced this pull request Jun 15, 2023
@efd6 @sodhikirti07
checkpoint: avoid data loss from loguid collisions (#6483)
b6e0e3b 
Because Checkpoint reuse loguids, it is possible for two logged events
to be given the same fingerprint. So use the segment_time as well in
the fingerprint args. Also document that if collisions are important,
the user should enable semi-unified logging on the device's dashboard.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kcreddy kcreddy kcreddy approved these changes

Assignees

@efd6 efd6

Labels
enhancement New feature or request Integration:Checkpoint
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

integration checkpoint: fingerprint processor creates duplications
3 participants
@efd6 @elasticmachine @kcreddy