Skip to content

chore: change otel collector policies #2436

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

chore: change otel collector policies #2436

wants to merge 1 commit into from

Conversation

@MasterPtato
Copy link
Contributor

@MasterPtato MasterPtato commented May 10, 2025

Changes

@MasterPtato MasterPtato requested a review from NathanFlurry May 10, 2025 00:55
@MasterPtato Graphite App
Copy link
Contributor Author

MasterPtato commented May 10, 2025

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

greptile-apps[bot]
greptile-apps bot reviewed May 10, 2025
View reviewed changes
Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Summary

Modified OpenTelemetry Collector configuration in cluster service to replace latency-based sampling with targeted OTTL condition-based sampling policies.

  • Changed sampling in /packages/core/services/cluster/src/workflows/server/install/install_scripts/files/otel_collector.sh to focus on specific span events ('subscribe' and 'message') with 'pegboard_actor_ready' attribute
  • Removed latency-based sampling (>15s) for 'routing_fn' spans which may impact visibility of slow operations
  • Added ERROR status code sampling policy to ensure error traces are captured
  • Configured tail sampling with 60s decision wait time and 50,000 trace buffer

1 file(s) reviewed, 1 comment(s)
Edit PR Review Bot Settings | Greptile

...s/core/services/cluster/src/workflows/server/install/install_scripts/files/otel_collector.sh Outdated
Comment on lines 42 to 47
- name: policy-2
type: and
and:
and_sub_policy:
- name: latency-policy-1
type: latency
latency:
threshold_ms: 15000
- name: span-name-policy-1
type: ottl_condition
ottl_condition:
span:
- 'name == "routing_fn"'
type: ottl_condition
ottl_condition:
span:
- 'name == "subscribe" and attributes["message"] == "pegboard_actor_ready"'
- name: policy-3
type: ottl_condition
ottl_condition:
span:
- 'name == "message" and attributes["message"] == "pegboard_actor_ready"'
Copy link

@greptile-apps greptile-apps bot May 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: Policies 2 and 3 appear to be sampling for identical conditions (pegboard_actor_ready) with only slightly different span names. Consider consolidating these into a single policy with an OR condition.

@MasterPtato MasterPtato force-pushed the 05-10-chore_change_otel_collector_policies branch from f17b27a to 21b246a Compare May 10, 2025 00:59
@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented May 10, 2025 •
edited
Loading

Deploying rivet with  Cloudflare Pages  Cloudflare Pages

Latest commit: 3c8db79
Status: ✅  Deploy successful!
Preview URL: https://24dd18a4.rivet.pages.dev
Branch Preview URL: https://05-10-chore-change-otel-coll.rivet.pages.dev

View logs

@MasterPtato MasterPtato force-pushed the 05-10-chore_change_otel_collector_policies branch from 21b246a to 3c8db79 Compare May 12, 2025 19:04
@MasterPtato MasterPtato force-pushed the 05-10-chore_change_otel_collector_policies branch from 3c8db79 to 56647b9 Compare May 12, 2025 20:55
@graphite-app
Copy link
Contributor

graphite-app bot commented May 12, 2025 •
edited
Loading

Merge activity

  • May 12, 4:56 PM EDT: MasterPtato added this pull request to the Graphite merge queue.
  • May 12, 4:56 PM EDT: CI is running for this pull request on a draft pull request (#2437) due to your merge queue CI optimization settings.
  • May 12, 4:57 PM EDT: Merged by the Graphite merge queue via draft PR: #2437.

graphite-app bot pushed a commit that referenced this pull request May 12, 2025
@MasterPtato
chore: change otel collector policies (#2436)
bb8ccd5 
<!-- Please make sure there is an issue that this PR is correlated to. -->

## Changes

<!-- If there are frontend changes, please include screenshots. -->
@graphite-app graphite-app bot closed this May 12, 2025
@graphite-app graphite-app bot deleted the 05-10-chore_change_otel_collector_policies branch May 12, 2025 20:57
@github-actions github-actions bot mentioned this pull request May 12, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@NathanFlurry NathanFlurry Awaiting requested review from NathanFlurry

1 more reviewer

@greptile-apps greptile-apps[bot] greptile-apps[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@MasterPtato