Include APM Trace Id in the X-Request-ID header in all Agentless API requests

[opauloh]

Summary

This ticket aims to include the X-Request-ID Header on all HTTP calls from the Kibana server to the Agentless API.

The X-Request-ID should be captured from the currentTraceparent provided by the APM service.

If a request is initiated as part of a background job, all Kibana transactions with the Agentless Api should happen after calling Apm's startTransaction.

Motivation

If a customer receives an error it can include the ID in a bug report, allowing the SDH operators to look up the corresponding log statements (without having to rely on timestamps, IPs, etc).

Also, this generated ID does not contain any sensitive information (client_id, cluster_id, etc), and won't violate the user's privacy as a unique ID is created per request it does help with tracking all the transactions related to a request when debugging an issue.

Definition of done

• X-Request-ID Header included on all HTTP calls from the Kibana server to the Agentless API.
• X-Request-ID is the currentTraceparent string captured from Apm Service and is unique per request.
• Request initiated as part of a background job (i.e. delete inactive agents), the Kibana transactions with the Agentless Api should happen after calling Apm's startTransaction.
• X-Request-ID is included on the error logs thrown on the try/catch blocks.
• logging documents created on Elasticsearch should use the http.request.id field to store the value of X-Request-ID.
• logging documents should also include trace.id and transaction.id

• Modifications on the agentless API TBD in a separate ticket.

Related tasks/epics

• Initial discussion
• Transaction doc
• currentTraceparent

[elasticmachine]

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

[opauloh]

I have a draft PR including the X-Request-ID using APM's traceparent, however, I had no success in testing the PR on the ESS environment with the QAF tool and the ci:cloud-deployment is not working, I'm blocking the issue until I can successfully test the changes with the QAF tool or when using the ci:cloud-deployment tag on the PR.