Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AWS] Add MSK integration #9133

Merged
merged 11 commits into from Mar 4, 2024
Merged

[AWS] Add MSK integration #9133

merged 11 commits into from Mar 4, 2024

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Feb 13, 2024

Proposed commit message

This PR is to add a new integration for Amazon MSK.

Metrics available for MSK serverless cluster: https://docs.aws.amazon.com/msk/latest/developerguide/serverless-monitoring.html

Metrics available for general MSK cluster: https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details.html

This integration collects all metrics available at the DEFAULT monitoring level and the basic dashboard only uses metrics available for MSK serverless clusters.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Related issues

Screenshots

Screenshot 2024-02-21 at 4 40 57 PM Screenshot 2024-02-21 at 4 44 33 PM

@kaiyan-sheng kaiyan-sheng self-assigned this Feb 13, 2024
@kaiyan-sheng kaiyan-sheng marked this pull request as ready for review February 21, 2024 23:46
@kaiyan-sheng kaiyan-sheng requested review from a team as code owners February 21, 2024 23:46
Copy link
Contributor

@drewdaemon drewdaemon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few suggestions to consider for the dashboard

  • There seems to be a floating icon (Grammerly?) captured in the screenshot
  • Consider changing the metrics to the new Lens metric, stacked vertically. You could increase the width of the top chart to make up for lost horizontal space
  • Adding panel titles would make it much easier to scan for the information I'm looking for. And/or you could add markdown panels in between the first and second row and second and third row with a larger title that describes both visualizations.
metricbeat-aws-kafka-overview

@elasticmachine
Copy link

elasticmachine commented Feb 23, 2024

🚀 Benchmarks report

Package aws 👍(11) 💚(4) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
route53_public_logs 13513.51 10869.57 -2643.94 (-19.57%) 💔
waf 5747.13 4255.32 -1491.81 (-25.96%) 💔

To see the full report comment with /test benchmark fullreport

@kaiyan-sheng
Copy link
Contributor Author

@drewdaemon Thank you so much for your review!! We are planning on enhancing the dashboard later in a separate PR anyway so Im gonna address your comment then! Thanks again!! (I did crop the screenshot so no Grammarly logo there 😬 )

@elasticmachine
Copy link

💚 Build Succeeded

History

cc @kaiyan-sheng

Copy link

Quality Gate failed Quality Gate failed

Failed conditions

66.7% 66.7% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

@kaiyan-sheng kaiyan-sheng merged commit 797fb3f into elastic:main Mar 4, 2024
4 of 5 checks passed
@kaiyan-sheng kaiyan-sheng deleted the aws_msk branch March 4, 2024 16:08
@elasticmachine
Copy link

Package aws - 2.13.0 containing this change is available at https://epr.elastic.co/search?package=aws

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants