Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Ability to correlate logs with other data sources #42

Open
sorantis opened this issue Jul 18, 2023 · 3 comments
Open
Labels
enhancement New feature or request

Comments

@sorantis
Copy link

sorantis commented Jul 18, 2023

The recently announced correlation feature in Grafana enables data correlation between any enabled data sources.

We want to be able to correlate logs from Google Cloud Logging plugin with other observability data (not in GCP) like metrics and traces.

In practice this means adding Links to a Log Panel in Grafana Explore that use correlation variables:

Screenshot 2023-07-13 at 16 48 54

In this example I used a Loki data source with some dummy data to configure log correlation with traces and metrics.
One thing that's required for a correlation is to know what field the log line is in. This can be checked in Grafana Explore by clicking the Inspector and then viewing the Data tab:

Screenshot 2023-07-13 at 17 00 32

In this case the required field is Line which is used as an input for configuring source data in the correlation configuration page:

Screenshot 2023-07-13 at 17 02 06

When I use the Inspector for logs in Google Cloud Logging plugin I get a different field for each log entry (see the csv examples attached), which makes it impossible to define correlations for our logs.

Screenshot 2023-07-13 at 17 34 05

I also tried using regex to extract resoure.labels.pod_name from the content field, but unfortunately it doesn't seem to work. The regex I used: resource.labels.pod_name="(.+?)"

Screenshot 2023-07-18 at 14 08 54

I suspect that Grafana treats (content + payload) as a field name and a log entry as the field value. So any regex would be checked against the field value, which is why it's not possible to make it work with the current field + payload structure.

It would be great if the plugin could work with the new correlation feature as it enables us to improve our troubleshooting experience.

@xiangshen-dk xiangshen-dk added the enhancement New feature or request label Jul 18, 2023
@sorantis
Copy link
Author

@xiangshen-dk any update on this issue?

@lmuhlha
Copy link

lmuhlha commented Jan 8, 2024

Any updates?

@xiangshen-dk
Copy link
Collaborator

A partner has proposed a solution and we are evaluating it. Also, it appears correlation is not GA yet. So, we want to wait a little.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants