Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(telemetry): enable statsd and dogstatsd telemetry sinks #18646

Merged
merged 21 commits into from
Dec 9, 2023

Conversation

itsdevbear
Copy link
Contributor

@itsdevbear itsdevbear commented Dec 6, 2023

Description

Closes: #XXXX


Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

  • included the correct type prefix in the PR title
  • confirmed ! in the type prefix if API or client breaking change
  • targeted the correct branch (see PR Targeting)
  • provided a link to the relevant issue or specification
  • reviewed "Files changed" and left comments if necessary
  • included the necessary unit and integration tests
  • added a changelog entry to CHANGELOG.md
  • updated the relevant documentation or specification, including comments for documenting Go code
  • confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

  • confirmed the correct type prefix in the PR title
  • confirmed all author checklist items have been addressed
  • reviewed state machine logic, API design and naming, documentation is accurate, tests and test coverage

Summary by CodeRabbit

  • New Features

    • Introduced new telemetry configuration options for selecting metrics backends (in-memory, statsd, dogstatsd).
    • Added support for statsd and dogstatsd telemetry sinks.
  • Enhancements

    • Improved metrics collection and display interfaces.
    • Enhanced x/gov module for direct proposal queries.
  • Configuration Changes

    • Added metrics-backend, statsd-addr, and datadog-hostname settings to telemetry configuration.
  • Bug Fixes

    • Addressed issues with non in-memory metrics sinks not supporting generic format.
  • Documentation

    • Updated CHANGELOG.md with recent feature additions and improvements.
  • Refactor

    • Refactored telemetry code to support multiple metrics sinks.
  • Tests

    • Updated tests to include new MetricsSink configuration field.
  • Chores

    • Modified script behavior for error handling by commenting out set -euo pipefail.

@itsdevbear itsdevbear requested a review from a team as a code owner December 6, 2023 19:41
@itsdevbear itsdevbear changed the title feat(telemetry): enable statd and dogstatsd telemetry sinks feat(telemetry): enable statsd and dogstatsd telemetry sinks Dec 6, 2023
Copy link
Contributor

coderabbitai bot commented Dec 6, 2023

Walkthrough

Walkthrough

The changes involve enhancing the telemetry and metrics system to support different backends, specifically statsd and dogstatsd, in addition to the existing in-memory sink. New configuration options have been added to specify the metrics backend type, statsd server address, and Datadog hostname. The code now handles the initialization of the specified metrics sink and integrates it into the telemetry system. Additionally, there are updates to tests and configuration defaults to align with these new features.

Changes

File Path Change Summary
telemetry/metrics.go Added support for statsd and dogstatsd metrics sinks, including new configuration options and logic to initialize and integrate these sinks.
server/config/toml.go Introduced new configuration options for metrics backend type, statsd server address, and Datadog hostname.
CHANGELOG.md Documented new telemetry features, updates to sdk.DecCoins, improvements in x/gov module, and additions to client.Context{} methods.
scripts/go-mod-tidy-all.sh Modified error handling by commenting out set -euo pipefail.
server/config/config.go Updated DefaultConfig to include MetricsSink in the Telemetry configuration.
telemetry/metrics_test.go Updated tests to include the new MetricsSink configuration option.

Tips

Chat with CodeRabbit Bot (@coderabbitai)

  • If you reply to a review comment from CodeRabbit, the bot will automatically respond.
  • To engage with CodeRabbit bot directly around the specific lines of code in the PR, mention @coderabbitai in your review comment
  • Note: Review comments are made on code diffs or files, not on the PR overview.
  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Note: For conversation with the bot, please use the review comments on code diffs or files.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • The JSON schema for the configuration file is available here.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 3

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between 52bbf81 and c9c05d3.
Files ignored due to filter (4)
  • go.mod
  • go.sum
  • simapp/go.mod
  • simapp/go.sum
Files selected for processing (1)
  • telemetry/metrics.go (6 hunks)
Additional comments: 1
telemetry/metrics.go (1)
  • 206-206: Verify that the DisplayMetrics method implementations can handle nil values for http.ResponseWriter and http.Request.

telemetry/metrics.go Outdated Show resolved Hide resolved
telemetry/metrics.go Show resolved Hide resolved
telemetry/metrics.go Show resolved Hide resolved
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between c9c05d3 and da47c25.
Files selected for processing (1)
  • server/config/toml.go (1 hunks)
Additional comments: 2
server/config/toml.go (2)
  • 121-133: The configuration template has been correctly updated to include the new telemetry sink options for statsd and dogstatsd. This aligns with the PR objectives and the AI-generated summaries.

  • 121-133: Verify that the new configuration options are accompanied by updates to the documentation and that there are tests covering these new features.

itsdevbear and others added 3 commits December 6, 2023 14:49
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Copy link
Contributor

@alexanderbez alexanderbez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK. Thanks!!! Left a minor comment. Also, let's get a small changelog entry in for the new support.

telemetry/metrics.go Outdated Show resolved Hide resolved
telemetry/metrics.go Outdated Show resolved Hide resolved
@itsdevbear
Copy link
Contributor Author

@alexanderbez addressed nits

@itsdevbear
Copy link
Contributor Author

@alexanderbez are you cool to backport into 0.50.x?

@alexanderbez
Copy link
Contributor

@alexanderbez are you cool to backport into 0.50.x?

Yeah, I don't see why not. This change is simple and purely additive. No objections from me. @tac0turtle?

server/config/toml.go Outdated Show resolved Hide resolved
telemetry/metrics.go Outdated Show resolved Hide resolved
itsdevbear and others added 2 commits December 6, 2023 17:25
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between da47c25 and 7a11ee2.
Files selected for processing (3)
  • CHANGELOG.md (1 hunks)
  • server/config/toml.go (1 hunks)
  • telemetry/metrics.go (8 hunks)
Files skipped from review as they are similar to previous changes (1)
  • server/config/toml.go
Additional comments: 2
CHANGELOG.md (1)
  • 59-59: The changes in the CHANGELOG.md correctly reflect the updates made to the system, including the addition of telemetry features, updates to sdk.DecCoins, improvements to the x/gov module, and enhancements to the client.Context{}. The links to pull requests provide traceability for the changes.
telemetry/metrics.go (1)
  • 207-213: Please verify that passing nil for both http.ResponseWriter and *http.Request to DisplayMetrics is safe and will not cause any issues in all implementations of DisplayableSink.

@itsdevbear
Copy link
Contributor Author

@julienrbrt updated v0.50 app toml as well

@julienrbrt
Copy link
Member

Can you run ./scripts/go-mod-tidy-all.sh?

@julienrbrt julienrbrt mentioned this pull request Dec 7, 2023
12 tasks
@julienrbrt julienrbrt added the backport/v0.50.x PR scheduled for inclusion in the v0.50's next stable release label Dec 7, 2023
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between 7a11ee2 and 4731792.
Files ignored due to filter (42)
  • client/v2/go.mod
  • client/v2/go.sum
  • tests/go.mod
  • tests/go.sum
  • tests/starship/tests/go.mod
  • tests/starship/tests/go.sum
  • tools/confix/data/v0.50-app.toml
  • tools/confix/data/v0.51-app.toml
  • x/accounts/go.mod
  • x/accounts/go.sum
  • x/auth/go.mod
  • x/auth/go.sum
  • x/authz/go.mod
  • x/authz/go.sum
  • x/bank/go.mod
  • x/bank/go.sum
  • x/circuit/go.mod
  • x/circuit/go.sum
  • x/distribution/go.mod
  • x/distribution/go.sum
  • x/evidence/go.mod
  • x/evidence/go.sum
  • x/feegrant/go.mod
  • x/feegrant/go.sum
  • x/gov/go.mod
  • x/gov/go.sum
  • x/group/go.mod
  • x/group/go.sum
  • x/mint/go.mod
  • x/mint/go.sum
  • x/nft/go.mod
  • x/nft/go.sum
  • x/params/go.mod
  • x/params/go.sum
  • x/protocolpool/go.mod
  • x/protocolpool/go.sum
  • x/slashing/go.mod
  • x/slashing/go.sum
  • x/staking/go.mod
  • x/staking/go.sum
  • x/upgrade/go.mod
  • x/upgrade/go.sum
Files selected for processing (1)
  • telemetry/metrics.go (8 hunks)
Additional comments: 9
telemetry/metrics.go (9)
  • 4-14: The addition of the net/http import aligns with the introduction of the DisplayableSink interface, which uses http.ResponseWriter and http.Request.

  • 23-30: The new constants for different metric sink types (MetricSinkInMem, MetricSinkStatsd, MetricSinkDogsStatsd) are added to support the configuration of various telemetry sinks.

  • 32-35: The DisplayableSink interface is introduced to define a method for displaying metrics, which is a part of the telemetry system refactoring to support different metric sinks.

  • 63-79: > Note: This review was outside the patches, so it was mapped to the patch with the greatest overlap. Original lines [37-77]

The Config struct is expanded with new fields (MetricsSink, StatsdAddr, DatadogHostname) to configure the metrics backend type and the addresses for statsd and dogstatsd.

  • 82-87: The Metrics struct has been refactored to include a generic sink field of type metrics.MetricSink, replacing the specific memSink. This change supports the use of different metric sinks.

  • 103-141: The New function is modified to support the instantiation of different metric sinks based on the MetricsSink configuration, including in-memory, statsd, and dogstatsd sinks.

  • 180-184: The gatherPrometheus function includes a check to ensure Prometheus metrics are enabled before attempting to gather them, which is part of the improved error handling.

  • 206-216: The gatherGeneric function is updated to use the DisplayableSink interface for displaying metrics, enabling a more flexible display of metrics.

  • 122-125: Please verify if the issues raised in previous comments regarding the defer statement within the case "mem": block and the nil check for sink before adding it to the FanoutSink have been addressed in the current code.

Also applies to: 138-138

@itsdevbear
Copy link
Contributor Author

@tac0turtle are you good with this? seems like tests are just weird flakiness

@tac0turtle
Copy link
Member

tac0turtle commented Dec 8, 2023

@itsdevbear made the inmemsink the default on the switch case, let me know if youre fine with that.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 3

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between 8664b7d and 9ef410e.
Files ignored due to filter (4)
  • x/mint/go.mod
  • x/nft/go.mod
  • x/slashing/go.mod
  • x/staking/go.mod
Files selected for processing (3)
  • server/config/config.go (1 hunks)
  • telemetry/metrics.go (8 hunks)
  • telemetry/metrics_test.go (2 hunks)
Additional comments: 2
server/config/config.go (1)
  • 234-238: The addition of MetricsSink to the Telemetry configuration in DefaultConfig is consistent with the PR's objective to enable statsd and dogstatsd telemetry sinks. Ensure that telemetry.MetricSinkInMem is the intended default value for MetricsSink.
telemetry/metrics_test.go (1)
  • 20-25: The addition of the MetricsSink field to the Config struct in the TestMetrics_InMem function is consistent with the PR's objective to enhance telemetry features. This change allows the test to specify the in-memory sink for metrics.

telemetry/metrics_test.go Show resolved Hide resolved
telemetry/metrics.go Show resolved Hide resolved
telemetry/metrics.go Show resolved Hide resolved
@tac0turtle tac0turtle added this pull request to the merge queue Dec 9, 2023
Merged via the queue into cosmos:main with commit 3ba1c5b Dec 9, 2023
55 of 58 checks passed
@tac0turtle tac0turtle deleted the datadog-metrics branch December 9, 2023 22:01
mergify bot pushed a commit that referenced this pull request Dec 9, 2023
)

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: marbar3778 <marbar3778@yahoo.com>
Co-authored-by: Marko <marko@baricevic.me>
(cherry picked from commit 3ba1c5b)

# Conflicts:
#	CHANGELOG.md
#	client/v2/go.mod
#	go.mod
#	simapp/go.mod
#	simapp/gomod2nix.toml
#	tests/go.mod
#	tests/starship/tests/go.mod
#	tests/starship/tests/go.sum
#	tools/confix/data/v0.51-app.toml
#	x/accounts/go.mod
#	x/accounts/go.sum
#	x/auth/go.mod
#	x/auth/go.sum
#	x/authz/go.mod
#	x/authz/go.sum
#	x/bank/go.mod
#	x/bank/go.sum
#	x/circuit/go.mod
#	x/distribution/go.mod
#	x/distribution/go.sum
#	x/evidence/go.mod
#	x/feegrant/go.mod
#	x/gov/go.mod
#	x/gov/go.sum
#	x/group/go.mod
#	x/group/go.sum
#	x/mint/go.mod
#	x/mint/go.sum
#	x/nft/go.mod
#	x/params/go.mod
#	x/params/go.sum
#	x/protocolpool/go.mod
#	x/protocolpool/go.sum
#	x/slashing/go.mod
#	x/slashing/go.sum
#	x/staking/go.mod
#	x/staking/go.sum
#	x/upgrade/go.mod
julienrbrt added a commit that referenced this pull request Dec 10, 2023
…kport #18646) (#18673)

Co-authored-by: Julien Robert <julien@rbrt.fr>
marcello33 added a commit to 0xPolygon/cosmos-sdk that referenced this pull request Dec 13, 2023
* feat: secp256k1 public key constant time (cosmos#18026)

Signed-off-by: bizk <santiago.yanzon1999@gmail.com>

* chore: Fixed changelog duplicated items (cosmos#18628)

* adr: Un-Ordered Transaction Inclusion (cosmos#18553)

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>

* docs: lint ADR-070 (cosmos#18634)

* fix(baseapp)!: postHandler should run regardless of result (cosmos#18627)

* docs: fix typos in adr-007-specialization-groups.md (cosmos#18635)

* chore: alphabetize labels (cosmos#18640)

* docs(x/circuit): add note on ante handler (cosmos#18637)

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>

* fix: telemetry metric label variable (cosmos#18643)

* chore: typos fix (cosmos#18642)

* refactor(store/v2): updates from integration (cosmos#18633)

* build(deps): Bump actions/setup-go from 4 to 5 (cosmos#18647)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>

* feat(store/v2): snapshot manager (cosmos#18458)

* chore(client/v2): fix typos in the README.md (cosmos#18657)

* fix(baseapp):  protocompat.go gogoproto.Merge does not work with custom types (cosmos#18654)

Co-authored-by: unknown unknown <unknown@unknown>

* chore: fix several minor typos (cosmos#18660)

* chore(tools/confix/cmd): fix typo in view.go (cosmos#18659)

* refactor(x/staking): check duplicate addresses in StakeAuthorization's params (cosmos#18655)

* feat(accounts): use gogoproto API instead of protov2.  (cosmos#18653)

Co-authored-by: unknown unknown <unknown@unknown>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* fix(store/commitment/iavl): honor tree.Remove error firstly (cosmos#18651)

* build(deps): Bump actions/stale from 8 to 9 (cosmos#18656)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(docs): fix typos & wording in docs (cosmos#18667)

* chore: fix several typos.   (cosmos#18666)

* feat(telemetry): enable `statsd` and `dogstatsd` telemetry sinks (cosmos#18646)

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: marbar3778 <marbar3778@yahoo.com>
Co-authored-by: Marko <marko@baricevic.me>

* feat(store/v2): add SetInitialVersion in SC (cosmos#18665)

* feat(client/keys): support display discreetly for `keys add` (cosmos#18663)

Co-authored-by: Julien Robert <julien@rbrt.fr>

* ci: add misspell action (cosmos#18671)

* chore: typos fix by misspell-fixer (cosmos#18683)

Co-authored-by: github-merge-queue <github-merge-queue@users.noreply.github.com>
Co-authored-by: Julien Robert <julien@rbrt.fr>

* chore: add v0.50.2 changelog to main (cosmos#18682)

* build(deps): Bump github.com/jhump/protoreflect from 1.15.3 to 1.15.4 in /tests (cosmos#18678)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* refactor(bank): remove .String() calls  (cosmos#18175)

Co-authored-by: Facundo <facundomedica@gmail.com>

* ci: use codespell instead of misspell-fixer (cosmos#18686)

Co-authored-by: Marko <marbar3778@yahoo.com>

* feat(gov): add proposal types and spam votes (cosmos#18532)

* feat(accounts): use account number as state prefix for account state (cosmos#18664)

Co-authored-by: unknown unknown <unknown@unknown>

* chore: typos fixes by cosmos-sdk bot (cosmos#18689)

Co-authored-by: github-merge-queue <github-merge-queue@users.noreply.github.com>
Co-authored-by: Julien Robert <julien@rbrt.fr>
Co-authored-by: marbar3778 <marbar3778@yahoo.com>

* feat(client/keys): support display discreetly for keys mnemonic (cosmos#18688)

* refactor: remove panic usage in keeper methods (cosmos#18636)

* ci: rename pr name in misspell job (cosmos#18693)

Co-authored-by: Marko <marko@baricevic.me>

* build(deps): Bump github.com/pelletier/go-toml/v2 from 2.1.0 to 2.1.1 in /tools/confix (cosmos#18702)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>

* feat(client/keys): support display discreetly for keys export (cosmos#18684)

* feat(x/gov): better gov genesis validation (cosmos#18707)

---------

Signed-off-by: bizk <santiago.yanzon1999@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Carlos Santiago Yanzon <27785807+bizk@users.noreply.github.com>
Co-authored-by: yihuang <huang@crypto.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Facundo Medica <14063057+facundomedica@users.noreply.github.com>
Co-authored-by: Akaonetwo <107335783+Akare123@users.noreply.github.com>
Co-authored-by: Marko <marbar3778@yahoo.com>
Co-authored-by: Julien Robert <julien@rbrt.fr>
Co-authored-by: dreamweaverxyz <153101746+dreamweaverxyz@users.noreply.github.com>
Co-authored-by: Pioua <136521243+dzizazda@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: cool-developer <51834436+cool-develope@users.noreply.github.com>
Co-authored-by: leonarddt05 <139609434+leonarddt05@users.noreply.github.com>
Co-authored-by: testinginprod <98415576+testinginprod@users.noreply.github.com>
Co-authored-by: unknown unknown <unknown@unknown>
Co-authored-by: Sukey <35202440+sukey2008@users.noreply.github.com>
Co-authored-by: axie <152680487+azukiboy@users.noreply.github.com>
Co-authored-by: Luke Ma <867273263@qq.com>
Co-authored-by: Emmanuel T Odeke <emmanuel@orijtech.com>
Co-authored-by: 0xn4de <109149873+0xn4de@users.noreply.github.com>
Co-authored-by: hattizai <150505746+hattizai@users.noreply.github.com>
Co-authored-by: Devon Bear <itsdevbear@berachain.com>
Co-authored-by: Marko <marko@baricevic.me>
Co-authored-by: Halimao <1065621723@qq.com>
Co-authored-by: Cosmos SDK <113218068+github-prbot@users.noreply.github.com>
Co-authored-by: github-merge-queue <github-merge-queue@users.noreply.github.com>
Co-authored-by: Facundo <facundomedica@gmail.com>
Co-authored-by: Likhita Polavarapu <78951027+likhita-809@users.noreply.github.com>
@marcello33 marcello33 mentioned this pull request Jan 22, 2024
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport/v0.50.x PR scheduled for inclusion in the v0.50's next stable release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants