Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding aws-s3 metric for delayed time #34306

Merged
merged 22 commits into from Jan 27, 2023
Merged

Adding aws-s3 metric for delayed time #34306

merged 22 commits into from Jan 27, 2023

Conversation

kgeller
Copy link
Contributor

@kgeller kgeller commented Jan 18, 2023

What does this PR do?

Adding the metric to answer how delayed are the SQS messages being received

For messages that are being received the first time (ApproximateReceiveCount==0) what is the moving average delta between the message's SentTimestamp and now. This can indicate it the input is falling behind without needed to capture queue metrics.

Why is it important?

This will let us observe how much lag there is in receiving SQS events by measuring delta between the time the SQS message was written to the queue to the time Filebeat receives it.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Verify name of the new metric

How to test this PR locally

http://localhost:5066/dataset?pretty now includes the following

"sqs_lag_time": {
      "histogram": {
        "count": 1,
        "max": 123893321000,
        "mean": 123893321000,
        "median": 123893321000,
        "min": 123893321000,
        "p75": 123893321000,
        "p95": 123893321000,
        "p99": 123893321000,
        "p999": 123893321000,
        "stddev": 0
      }
    },

Related issues

Relates https://github.com/elastic/security-team/issues/5526

@kgeller kgeller requested a review from a team January 18, 2023 18:39
@kgeller kgeller self-assigned this Jan 18, 2023
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jan 18, 2023
@mergify
Copy link
Contributor

mergify bot commented Jan 18, 2023

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @kgeller? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

@kgeller kgeller added the backport-skip Skip notification from the automated backport with mergify label Jan 18, 2023
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jan 18, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-01-27T17:53:13.001+0000

  • Duration: 129 min 27 sec

Test stats 🧪

Test Results
Failed 0
Passed 5064
Skipped 350
Total 5414

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@andrewkroh andrewkroh changed the title Adding aws-s3 metic for delayed time Adding aws-s3 metric for delayed time Jan 18, 2023
x-pack/filebeat/input/awss3/metrics.go Outdated Show resolved Hide resolved
x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/input/awss3/sqs_s3_event.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/awss3/sqs_s3_event.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/awss3/sqs_s3_event.go Outdated Show resolved Hide resolved
@mergify
Copy link
Contributor

mergify bot commented Jan 24, 2023

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b s3-metrics upstream/s3-metrics
git merge upstream/main
git push upstream s3-metrics

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved
x-pack/filebeat/input/awss3/input_integration_test.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/awss3/input_integration_test.go Outdated Show resolved Hide resolved
x-pack/filebeat/input/awss3/metrics.go Outdated Show resolved Hide resolved
@kgeller kgeller marked this pull request as ready for review January 25, 2023 20:15
@kgeller kgeller requested a review from a team as a code owner January 25, 2023 20:15
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@kgeller kgeller merged commit 48d3f04 into elastic:main Jan 27, 2023
@kgeller kgeller deleted the s3-metrics branch January 27, 2023 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-skip Skip notification from the automated backport with mergify enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants