Trace ID in logs not linking properly

[Brandon-Kimberly]

What happened:
Followed the steps to add links to Xray trace IDs in CloudWatch logs here (X-Ray trace links section) but there is no link appearing in any log line, even though @xrayTraceId is present.

The CloudWatch data source is configured properly and can retrieve data. The Xray data source appears to be working. And the CloudWatch data source does have the "X-ray trace link" set to the Xray data source.

You can see in this screenshot that it does not link the trace ID correctly:

What you expected to happen:
I expected a link to show up like in this screenshot from the above docs:

How to reproduce it (as minimally and precisely as possible):

1. Create Xray data source as shown in screenshot above.
2. Create CW data source as shown above.
3. Pull logs from CW into Grafana in Explore tab that contain an @xrayTraceId field.
4. Expand one of the log lines that has the @xrayTraceId field, see that there is no link to AWS X-Ray.

Screenshots
See above

Anything else we need to know?:

Environment:

• Grafana version: 8.4.7
• X-Ray Plugin version: 2.1.0
• OS Grafana is installed on: Linux
• Running on Amazon Managed Grafana

[sarahzinger]

@Brandon-Kimberly Sorry I'm having trouble reproducing this bug, can you give me any other details? Here's what I have done so far:

• I created a node test lambda which had console.log(process.env._X_AMZN_TRACE_ID) in it
• I then added a trigger for it and triggered it
• I then went to cloudwatch logs for my lambda function and ran the following logs query:

fields @timestamp, @message, @xrayTraceId
| limit 20

• I was able to see that when xrayTraceId was present it linked to a trace view

I then tried the same thing in grafana by doing the following:

• I randocker run -d -p 3000:3000 -e "GF_INSTALL_PLUGINS=https://github.com/grafana/x-ray-datasource/releases/download/v2.1.0/grafana-x-ray-datasource-2.1.0.zip;grafana-x-ray-datasource" grafana/grafana-oss:8.4.7 which I believe should run grafana 8.4.7 and the xray datasource plugin at 2.1.0
• I then configured an instance of the xray plugin and verified that I could find a trace with one of the traceIds from the cloudwatch logs query above.
• I then configured an instance of the grafana cloudwatch plugin, and selected the xray instance we just created.
• I then went to the explore page in grafana and selected cloudwatch, cloudwatch logs, and then made the same exact cloudwatch logs query above

from there I was able to see logs with links to xray traces.

which I could then click and was able to open in the xray datasource

Perhaps you can let us know if there are any additional repro steps we should be aware of to see if we can recreate this on our end? Also because the code is open source you may wish to try pulling and running the code locally and adding additional logging to see where the trouble might be. I believe the data links part of the code is mostly located here: https://github.com/grafana/grafana/blob/main/public/app/plugins/datasource/cloudwatch/utils/datalinks.ts#L14

Let us know if you find out anything more!

[sarahzinger]

One interesting thing I notice is that in that screenshot the xrayTraceId field is in quotations where as in my screenshot and the docs it is not, maybe that's the issue?

[taleena]

@Brandon-Kimberly please review the above and let us know if this works, or if there are more things to consider. :)

[Brandon-Kimberly]

That did resolve the issue. Thanks!