Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[google_workspace] Fix pagination and filters for Report API data streams. #4982

Merged
merged 3 commits into from
Jan 16, 2023
Merged

[google_workspace] Fix pagination and filters for Report API data streams. #4982

merged 3 commits into from
Jan 16, 2023

Conversation

marc-gr
Copy link
Contributor

@marc-gr marc-gr commented Jan 12, 2023

What does this PR do?

  • Fixes the cursor logic to keep the first event date instead of last. The API returns records from newest to oldest, so effectively the startDate was always kept the same, making every interval process since the beginning, making it slower over time.
  • Fixes pagination logic to be able to resume if the agent is stopped in the middle of a pagination cycle. Before, given the order of the records, any non-processed events for that cycle would be lost on resume. Now it will reprocess the last interval if caught in the middle of pagination (repeated events will be ignored by the pipeline deduplication).

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Closes #4796

@marc-gr marc-gr requested a review from efd6 January 12, 2023 15:32
@marc-gr marc-gr requested a review from a team as a code owner January 12, 2023 15:32
@elasticmachine
Copy link

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@elasticmachine
Copy link

elasticmachine commented Jan 12, 2023 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-01-16T11:08:35.652+0000

  • Duration: 21 min 48 sec

Test stats 🧪

Test Results
Failed 0
Passed 76
Skipped 0
Total 76

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Jan 12, 2023 •
edited
Loading

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (8/8) 💚
Files 100.0% (8/8) 💚 2.509
Classes 100.0% (8/8) 💚 2.509
Methods 100.0% (106/106) 💚 8.302
Lines 96.271% (2969/3084) 👍 4.903
Conditionals 100.0% (0/0) 💚

efd6
efd6 approved these changes Jan 16, 2023
View reviewed changes
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has this been tested against a real endpoint?

packages/google_workspace/data_stream/admin/agent/stream/httpjson.yml.hbs
# Use this value to be able to resume from an interrupted pagination cycle.
value: '[[.last_response.url.params.Get "startTime"]]'
next_start_date:
# The API returns records sorted from newest to oldest,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a way to catch a change in this behaviour in the API if it happens?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened this #5009 to not forget about doing that

@marc-gr
Copy link
Contributor Author

marc-gr commented Jan 16, 2023

Yes I tested with an environment of our own and I am waiting for confirmation from another one with more activity to validate bigger event numbers scenarios.

@marc-gr marc-gr merged commit 3d2ace9 into elastic:main Jan 16, 2023
@marc-gr marc-gr deleted the fix/gw_pagination branch January 16, 2023 12:57
@marc-gr marc-gr mentioned this pull request Jan 16, 2023
Merged
6 tasks
@elasticmachine
Copy link

Package google_workspace - 2.2.1 containing this change is available at https://epr.elastic.co/search?package=google_workspace

@andrewkroh andrewkroh mentioned this pull request Jul 14, 2023
Merged
4 tasks
@kcreddy kcreddy mentioned this pull request Dec 5, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@efd6 efd6 efd6 approved these changes

Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Google Workspace] Failure during pagination can cause data loss
3 participants
@marc-gr @elasticmachine @efd6