[Feature] collect traces of CnosDB

[yukkit]

The current issue is used to record all trace related features

• [Feature] collect traces of query #1234
• [Feature] collect traces of tskv #1235

[yukkit]

CnosDB 全链路追踪

采集的 trace 信息支持多种展示方式

• log 文件
• jaeger（推荐使用，更清晰明了）

使用方式

CnosDB 服务端开启 trace 采集

数据库服务端配置

客户端指定 span 上下文

目前我们支持 https://www.jaegertracing.io/docs/1.46/client-libraries/#propagation-format 格式

http 方式访问数据库

在请求头中添加 span context

e.g.

uber-trace-id: 3a3a43:432e345:0:1

flight sql

e.g.

let mut client = FlightSqlServiceClient::new(channel);
// <trace_id>:<span_id>:<parent_span_id>:<sampled>
client.set_header("uber-trace-id", "3a3a43:432e345:0:1");

具体的flight sql使用方式请参见 连接器 Arrow Flight SQL

span

不同请求的主要访问链路

client
├── rest
│   ├── rest sql request: 通过 http 接口执行的 sql 请求
│   │   ├── authenticate: 校验用户登陆权限
│   │   └── sql handle: 执行 sql 任务，并构建 http 响应结果
│   │       ├── execute: 执行 sql 任务
│   │       │   ├── init session ctx: 初始化会话
│   │       │   ├── parse sql: sql 解析
│   │       │   ├── statement to logical plan: ast 转逻辑计划
│   │       │   ├── check privilege: 权限校验
│   │       │   ├── analyze plan: 语义检查
│   │       │   ├── optimize logical plan: 逻辑计划优化
│   │       │   ├── logical plan to physical plan: 逻辑计划转物理计划
│   │       │   ├── optimize physical plan: 物理计划优化
│   │       │   ├── add traced proxy: 给每个物理节点添加 trace 代理
│   │       │   └── **<execution stream trace>**: 流式调度执行
│   │       └── build response: 构建 http 响应结果
│   ├── rest line protocol write
│   │   └── ......
│   ├── rest prom remote read
│   │   └── ......
│   ├── rest prom remote write
│   │   └── ......
│   ├── rest open tsdb write
│   │   └── ......
│   ├── rest open tsdb put
│   │   └── ......
└── flight sql
    ├── flight sql do_handshake
    │   └── ......
    ├── flight sql get_flight_info_statement
    │   └── ......
    ├── flight sql do_action_create_prepared_statement
    │   └── ......
    ├── flight sql do_get_prepared_statement
    │   └── ......
    ├── flight sql do_get_statement
    │   └── ......
    ├── flight sql do_put_statement_update
    │   └── ......
    ├── flight sql get_flight_info_catalogs
    │   └── ......
    ├── flight sql get_flight_info_catalogs
    │   └── ......
    ├── flight sql get_flight_info_tables
    │   └── ......
    └── flight sql get_flight_info_table_types
        └── ......

execution stream trace

用于采集执行流的trace信息，在CnosDB中，物理执行计划根据数据 partition 被组织成一个树状的stream，每个物理算子+partition唯一对应stream中的一个节点。
此处有两个节点需要特别说明，分别是 TskvExec 对应的 TableScanStream 和 TableWriteExec 对应的 TskvRecordBatchSink，分别是从tskv中读数据和写数据。

TableWriteExec (N)

├── ......
└── TskvRecordBatchSink (N): 将 partition N 的输入流分批次写入 tskv
    ├── Batch (0): 写入第一个 batch
    │   ├── record batch to points: 将 RecordBatch 转为 Points
    │   ├── limit check
    │   ├── map point: 将points进行分区
    │   ├── build requests: 构建写入请求
    │   ├── write to vnode <vnode id> on node <node id>: 向 vnode 写入数据
    │   │   └── **<write to remote node>** / **<write to local node>**: vnode 在同一进程则直接进行本地写入，否则进行远程写入
    │   └── ......
    ├── ......
    └── Batch (N)

TskvExec (N)

TableScanStream (N): 读取 partition N 的数据
└── **<tskv remote read trace>** / **<tskv local read trace>**: vnode 在同一进程则直接进行本地读取，否则进行远程读

tskv remote read trace

grpc query_record_batch: 通过 grpc 进行远程读取
└── RecordBatch encorder stream: 对数据流进行序列化
    └── **<tskv local read trace>**: 本地读数据

tskv local read trace

LocalTskvTableScanStream (<vnode id>): 读取本地 vnode 的数据
└── RowIterator: 行迭代器
    ├── get super version: 获取本地读请求的 super version
    ├── get series ids by filter: 过滤 series
    ├── SeriesGroupRowiterator [<start>, <end>): series 分组并行读取数据
    └── SeriesGroupRowiterator [..., ...): 

write to remote node

write to remote node: 
└── **<tskv remote write trace>**: 

write to local node

write to local node: 
└── **<tskv local write trace>**: 

tskv remote write trace

grpc write_vnode_points: 
└── **<tskv local write trace>**: 

tskv local write trace

tskv engine write: 
├── build write group: 根据 series id 对数据进行分组
├── write wal: 写 wal
└── put points: 写 points 到 memtable

[yukkit]

wait for test @Benxiaohai001

[Benxiaohai001]

rest api:

curl -i -u "root:" -H "Accept: application/json" -H "uber-trace-id: 3a3a43:432e345:0:1" -XPOST "http://127.0.0.1:12002/api/v1/sql?db=usage_schema&pretty=true" -d "select * from user_queries order by time desc limit 5;"
curl -i -u "root:" -H "Accept: application/json" -H "uber-trace-id: 3a3a43:432e345:0:1" -XPOST "http://127.0.0.1:12002/api/v1/sql?db=usage_schema&pretty=true" -d "CREATE TABLE air (visibility DOUBLE,temperature DOUBLE,presssure DOUBLE,TAGS(station));"
curl -i -u "root:" -H "Accept: application/json" -H "uber-trace-id: 3a3a43:432e345:0:1" -XPOST "http://127.0.0.1:12002/api/v1/sql?db=usage_schema&pretty=true" -d "CREATE TABLE air (visibility DOUBLE,temperature DOUBLE,presssure DOUBLE,TAGS(station));"
curl -i -u "root:" -H "Accept: application/json" -H "uber-trace-id: 3a3a43:432e345:0:1"  -XPOST "http://127.0.0.1:12002/api/v1/write?db=public&pretty=true" -d "ma,station=XiaoMaiDao visibility=50,temperature=63,pressure=52"

prometheus remote read/write

remote_write:
- url: "http://192.168.0.23:12002/api/v1/prom/write?db=prometheus"

 headers:
 uber-trace-id: '3a3a43:432e345:0:1'
 basic_auth:
 username: 'root'
 password: ''
remote_read cnosdb
remote_read:
- url: "http://192.168.0.23:12002/api/v1/prom/read?db=prometheus"
headers:
 uber-trace-id: '3a3a43:432e345:0:1'
basic_auth:
 username: 'root'
 password:

open tsdb write

curl -i -u "root:" -H "Accept: application/json" -H "uber-trace-id: 3a3a43:432e345:0:1" -XPOST "http://127.0.0.1:12002/api/v1/opentsdb/write?db=public" -d 'opentsdb 1689054412124000000 1 tag1=t1 field1="f1" value1=1'

curl -i -u "root:" -H "Accept: application/json" -H "uber-trace-id: 3a3a43:432e345:0:1" -XPOST "http://127.0.0.1:12002/api/v1/opentsdb/put?db=public" -d '[{"metric":"tsdbput","timestamp": 1689069407572000000,"value": 9,"tags":{"tag1":"t1","tag2":"t2"}}]'

flight sql ,add the follow code, then exec 'cargo test --package e2e_test --lib -- flight_sql::test::test_sql_client_get_catalogs --exact --nocapture'

# e2e_test/src/flight_sql.rs
...
client.set_header("uber-trace-id", "3a3a43:432e345:0:1");::
...