Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x-pack/filebeat/input/httpjson: Apply rate limiting to all responses #38161

Merged
merged 2 commits into from Mar 8, 2024
Merged

x-pack/filebeat/input/httpjson: Apply rate limiting to all responses #38161

merged 2 commits into from Mar 8, 2024

Conversation

chrisberkhout
Copy link
Contributor

Draft discussion

I didn't find a solution to the failing request_honors_rate_limit test. The context is okay when processing the first response, and remains okay after the wait time is applied, then it appears as cancelled after the retry request is run (that is, after this line).

Proposed commit message

x-pack/filebeat/input/httpjson: Apply rate limiting to all responses (#)

- Drain the body before rate limiting.
- Apply rate limiting to all responses, waiting immediately. Retry if
  the response was not successful and there was a rate limit wait (even
  if immediately expired), otherwise return.
- Improve names of variables `epoch` and `activeLimit`.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

@chrisberkhout chrisberkhout added Team:Service-Integrations Label for the Service Integrations team bugfix labels Mar 3, 2024
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Mar 3, 2024
@mergify Mergify
Copy link
Contributor

mergify bot commented Mar 3, 2024

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @chrisberkhout? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@elasticmachine
Copy link
Collaborator

elasticmachine commented Mar 3, 2024 •
edited

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Duration: 135 min 50 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@efd6
x-pack/filebeat/input/httpjson: optimistically accept missing rate li…
5d49c94 
…mits as no limit

If a rate limit header is not present, do not fail the request, but
rather optimistically assume that we are within limits.
@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @chrisberkhout @efd6

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @chrisberkhout @efd6

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @chrisberkhout @efd6

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @chrisberkhout @efd6

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @chrisberkhout @efd6

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @chrisberkhout @efd6

@jamiehynds jamiehynds added Team:Security-Service Integrations Security Service Integrations Team and removed Team:Service-Integrations Label for the Service Integrations team labels Mar 5, 2024
@efd6 efd6 marked this pull request as ready for review March 5, 2024 21:07
@efd6 efd6 requested a review from a team as a code owner March 5, 2024 21:07
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@efd6 efd6 added Filebeat Filebeat backport-skip Skip notification from the automated backport with mergify labels Mar 5, 2024
@ShourieG
Copy link
Contributor

ShourieG commented Mar 7, 2024

@chrisberkhout Can we get a Changelog update, otherwise looks pretty good to me, I'm approving to unblock, please add the Changelog before merging.

ShourieG
ShourieG approved these changes Mar 7, 2024
View reviewed changes
Copy link
Contributor

@ShourieG ShourieG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving to unblock, LGTM

@efd6 efd6 merged commit ca07b8e into elastic:main Mar 8, 2024
37 of 38 checks passed
@efd6 efd6 added backport-v8.13.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Mar 8, 2024
mergify bot pushed a commit that referenced this pull request Mar 8, 2024
@chrisberkhout @mergify
x-pack/filebeat/input/httpjson: Apply rate limiting to all responses (#…
5a5b23d 
…38161)

- Drain the body before rate limiting.
- Apply rate limiting to all responses, waiting immediately. Retry if
  the response was not successful and there was a rate limit wait (even
  if immediately expired), otherwise return.
- Improve names of variables `epoch` and `activeLimit`.

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
(cherry picked from commit ca07b8e)
@mergify mergify bot mentioned this pull request Mar 8, 2024
Merged
@efd6
Copy link
Contributor

efd6 commented Mar 8, 2024

Missed the changelog. Will add in new PR.

@efd6 efd6 mentioned this pull request Mar 8, 2024
Merged
6 tasks
efd6 pushed a commit that referenced this pull request Mar 8, 2024
@mergify @chrisberkhout
x-pack/filebeat/input/httpjson: Apply rate limiting to all responses (#…
8c12ceb 
…38161) (#38236)

- Drain the body before rate limiting.
- Apply rate limiting to all responses, waiting immediately. Retry if
  the response was not successful and there was a rate limit wait (even
  if immediately expired), otherwise return.
- Improve names of variables `epoch` and `activeLimit`.

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
(cherry picked from commit ca07b8e)

Co-authored-by: Chris Berkhout <chris.berkhout@elastic.co>
@chrisberkhout chrisberkhout deleted the rate-limit-fix branch March 25, 2024 03:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ShourieG ShourieG ShourieG approved these changes

Assignees

@chrisberkhout chrisberkhout

@efd6 efd6

Labels
backport-v8.13.0 Automated backport with mergify bugfix Filebeat Filebeat Team:Security-Service Integrations Security Service Integrations Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

x-pack/filebeat/input/httpjson/rate_limiter.go: Rate limit evaluation only happens on 429 responses
5 participants
@chrisberkhout @elasticmachine @ShourieG @efd6 @jamiehynds