Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Beyla upgrade breaks sample #61

Closed
dashpole opened this issue Dec 8, 2023 · 2 comments · Fixed by #74
Closed

Beyla upgrade breaks sample #61

dashpole opened this issue Dec 8, 2023 · 2 comments · Fixed by #74
Assignees
Labels
bug Something isn't working

Comments

@dashpole
Copy link
Contributor

dashpole commented Dec 8, 2023

Updating the image to: image: grafana/beyla:1.0.2 causes the sample to stop emitting traces and metrics after about 2 minutes, based on the last metric produced:

StartTimestamp: 2023-12-08 18:48:33.302193405 +0000 UTC
Timestamp: 2023-12-08 18:50:03.301807418 +0000 UTC

The beyla logs contain:

time=2023-12-08T18:48:28.125Z level=INFO msg="system wide instrumentation. Creating a single instrumenter" component=discover.TraceAttacher
2023/12/08 18:49:55 failed to upload metrics: gRPC exporter is shutdown
2023/12/08 18:49:58 failed to upload metrics: gRPC exporter is shutdown
2023/12/08 18:49:59 failed to upload metrics: gRPC exporter is shutdown
2023/12/08 18:50:01 failed to upload metrics: gRPC exporter is shutdown
2023/12/08 18:50:03 failed to upload metrics: gRPC exporter is shutdown
2023/12/08 18:50:04 failed to upload metrics: gRPC exporter is shutdown
2023/12/08 18:50:05 failed to upload metrics: gRPC exporter is shutdown
time=2023-12-08T18:50:05.418Z level=WARN msg="error shutting down metrics provider" component=otel.MetricsReporter error="failed to upload metrics: gRPC exporter is shutdown"
2023/12/08 18:50:05 failed to upload metrics: gRPC exporter is shutdown
time=2023-12-08T18:50:15.417Z level=WARN msg="error shutting down metrics provider" component=otel.MetricsReporter error="failed to upload metrics: gRPC exporter is shutdown"
@dashpole dashpole added the bug Something isn't working label Dec 8, 2023
@dashpole dashpole self-assigned this Dec 8, 2023
@dashpole
Copy link
Contributor Author

Since we want to change the metrics beyla provides anyways, the plan is to stay on this version, and try using span-based metrics for the time being. That way, we can get resource attributes (like the application's IP) to use with the k8sattributes processor.

@dashpole
Copy link
Contributor Author

Might be able to work around this with: BEYLA_METRICS_REPORT_CACHE_LEN=some large number. Same applies to traces.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant