Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Google Workspace ] Fix order of processes in ingest pipeline #5300

Merged
merged 4 commits into from
Feb 16, 2023
Merged

[Google Workspace ] Fix order of processes in ingest pipeline #5300

merged 4 commits into from
Feb 16, 2023

Conversation

bhapas
Copy link
Contributor

@bhapas bhapas commented Feb 16, 2023 •
edited
Loading

What does this PR do?

Fixes the order of processes in ingest-pipeline. The fingerprint process had colliding documents because rename process ran first and changed the value of json.events.name which potentially could create same fingerprint for documents with different name but other matching fields.

Found this bug as part of system testing in https://github.com/elastic/security-team/issues/5869

Checklist

  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

  • Build elastic-package locally - make build
  • Navigate to google_workspace package
  • Add assert: hit_count : 22 to admin data_stream's System test config
  • The system test fails without this PR as the number of documents ingested are less than expected , reason being duplicate fingerprint created for multiple documents.

Related issues

@bhapas bhapas added the bug Something isn't working, use only for issues label Feb 16, 2023
@bhapas
Copy link
Contributor Author

bhapas commented Feb 16, 2023

/test

@bhapas bhapas marked this pull request as ready for review February 16, 2023 11:51
@bhapas bhapas requested a review from a team as a code owner February 16, 2023 11:51
@elasticmachine
Copy link

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@elasticmachine
Copy link

elasticmachine commented Feb 16, 2023 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-02-16T13:15:08.723+0000

  • Duration: 20 min 51 sec

Test stats 🧪

Test Results
Failed 0
Passed 76
Skipped 0
Total 76

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (8/8) 💚
Files 100.0% (8/8) 💚
Classes 100.0% (8/8) 💚
Methods 100.0% (106/106) 💚
Lines 96.271% (2969/3084) 👍 2.413
Conditionals 100.0% (0/0) 💚

@bhapas bhapas merged commit d8512dc into elastic:main Feb 16, 2023
bhapas added a commit to bhapas/integrations that referenced this pull request Feb 16, 2023
@bhapas
[Google Workspace ] Fix order of processes in ingest pipeline (elasti…
73611fc 
…c#5300)

* Fix order of processes in ingest pipeline

* Provide correct PR number to changelog

* fix version

* Fix version in manifest
@elasticmachine
Copy link

Package google_workspace - 2.2.2 containing this change is available at https://epr.elastic.co/search?package=google_workspace

@bhapas bhapas deleted the duplicate-fingerprint branch February 23, 2023 14:45
agithomas pushed a commit to agithomas/integrations that referenced this pull request Mar 20, 2023
@bhapas @agithomas
[Google Workspace ] Fix order of processes in ingest pipeline (elasti…
a9494e5 
…c#5300)

* Fix order of processes in ingest pipeline

* Provide correct PR number to changelog

* fix version

* Fix version in manifest
agithomas pushed a commit to agithomas/integrations that referenced this pull request Mar 21, 2023
@bhapas @agithomas
[Google Workspace ] Fix order of processes in ingest pipeline (elasti…
fada8fa 
…c#5300)

* Fix order of processes in ingest pipeline

* Provide correct PR number to changelog

* fix version

* Fix version in manifest
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@marc-gr marc-gr marc-gr approved these changes

Assignees
No one assigned
Labels
bug Something isn't working, use only for issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug] Duplicate Fingerprint while ingesting documents with different field attributes
3 participants
@bhapas @elasticmachine @marc-gr