Requirement Request: Timeline Tracing

[breezewish]

Description

Related project: TiDB, TiKV, raft-rs, Dashboard

For now it is a bit hard to discover what causes a SQL to run slow in some scenarios. Existing facilities are:

• “EXPLAIN ANALYZE”: Only useful to know which executor takes a long time, cannot know RocksDB time, raft time, snapshot time, etc. Cannot inspect a SQL in the past.
• Tracing: Only available in TiDB, cannot trace TiKV, and enabling it will greatly affect the performance so that it is rarely useful to inspect a SQL in the past.
• Metrics: It displays execution information of all SQLs, when payload is hybrid it is very difficult to figure out. Also metrics are not grouped in a way that we are easy to follow by SQL lifetime.
• Log / slow log: Different components output their own logs and it is very hard to link them together. Some logs only contain aggregated data and are not precise enough.

Timeline Tracing adds a new tracing layer to both TiDB and TiKV. The existing low-performance tracing facility in TiDB will be dropped. TiKV traces the time of each specific event in the request and packs them along with the response. Timeline tracing is enabled by default and has <1% performance impact.

Timeline tracing displays trace spans like:

Timeline tracing can be integrated with a visualization of Execution Plan. The execution plan is usually visualized as:

For each executor, users can click and view child spans of the executor. Note that not all spans belong to an executor. Thus an overview of all Timeline tracing spans is still needed.

Category

• Feature
• Usability

Value

• Know where specific SQL performance jitters comes from
• Explore how much time it takes for each step for each SQL

• Score: 5

Workload Estimation

60 Person Day

[breezewish]

Related: pingcap/tidb#17693

[breezewish]

This feature is accepted and progress should be tracked in #710