Clustered: observability and query tracing #5143
Description
Teach customers (esp. Dedicated and Clustered) how to add trace-ids to Flight queries when using client libraries.
- Document the
debugoption - Document (link to example of) the Flight middle-ware option
FlightReader accepts middle-ware so we can define a simple middleware factory and have that inject a trace header (and dump the output header). See the Flight repo (https://github.com/apache/arrow/blob/ac7e9a4c41b73242b2f7a15f13f3c8fde843416d/python/examples/flight/middleware.py#L68)
Define the middleware:
class TracingClientMiddleWareFactory(flight.ClientMiddleware):
def start_call(self, info):
print("Starting new call:", info)
return TracingClientMiddleware()
class TracingClientMiddleware(flight.ClientMiddleware):
def sending_headers(self):
print("Sending trace ID:", "BensTraceHeader")
return {
"x-tracing-id": "BensTraceHeader",
}
def received_headers(self, headers):
if "trace-id" in headers:
trace_id = headers["trace-id"][0]
print("Found trace header with value:", trace_id)
# Don't overwrite our trace ID```
Then, pass the factory into the V3 client via `flight_client_options`:
``` client = InfluxDBClient3(token = TOOLS_TOKEN,
host = TOOLS_URL,
org = TOOLS_ORG,
database="influxql_query_log",
flight_client_options={"middleware": (TracingClientMiddleWareFactory(),)}
)```
Run yer query and get lovely trace-id goodness
```ben@ratchett:~/Repos/c2_usage_extraction_scripts$ QUERY_INTERVAL='30 minutes' influxql/ticket_114205.py 58fa619b8b55b03e
SELECT request FROM influxql_query_log WHERE time > now() - interval '30 minutes' AND "orgID" = '58fa619b8b55b03e'
Fetching queries from Tools
Starting new call: CallInfo(method=<FlightMethod.DO_GET: 5>)
Sending trace ID: BensTraceHeader
Found trace header with value: b6c3a9e01d758a13```
[Slack Message](https://influxdata.slack.com/archives/C04768675QD/p1694788872943159?thread_ts=1694788872.943159&cid=C04768675QD)