*: Introduce tracing framework #8981

zhongzc · 2020-11-06T01:26:10Z

Signed-off-by: zhongzc zhongzc_arch@outlook.com

What problem does this PR solve?

Lack of a mechanism on TiKV to explore tracing results.

Related feature requests: pingcap/tidb#17693
Related TiDB side & usage: pingcap/tidb#19557
Roadmap: pingcap/tidb-dashboard#710

What is changed and how it works?

Display tracing results directly via Jaeger UI.

What's Changed:

A trait tracing::Reporter is introduced with its two implementations JaegerReporter and NullReporter:
- JaegerReporter reports tracing results to a UDP agent asynchronously via a tokio runtime.
- NullReporter drops all results passed to it.
Some configurations are introduced for reporting tracing results:
- jaeger_thrift_compact_agent: Jaeger agent, commonly set to "127.0.0.1:6831".
- duration-threshold: the lower limit of duration of an individual procedure to report. A tracing result with a duration less than this value will be discarded.
- spans-max-length: the upper limit of the number of spans of a tracing result to report. The excess spans will be discarded.
A macro requests_to_trace to register requests to trace. If a request is not within this list (i.e. it's not intended to trace), even though lots of spans are left for the request in the codebase, we don't need to pay for it.

Release note

TBD

sre-bot · 2020-11-06T01:26:18Z

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

zhongzc · 2020-11-06T01:27:06Z

/bench +sysbench +tpcc +tpch

sre-bot · 2020-11-06T01:39:55Z

The tpcc workload is being maintained...

zhongzc · 2020-11-06T03:21:08Z

/bench +sysbench

sre-bot · 2020-11-06T05:49:09Z

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

zhongzc · 2020-11-06T05:58:37Z

/bench +sysbench

zhongzc · 2020-11-06T06:34:02Z

/bench +sysbench

zhongzc · 2020-11-09T06:33:22Z

/bench +sysbench

zhongzc · 2020-11-09T07:51:43Z

/bench +sysbench +tpch

zhongzc · 2020-11-09T07:52:15Z

/run-integration-common-test

zhongzc · 2020-11-10T05:34:34Z

/bench +sysbench +tpch

zhongzc · 2020-11-10T07:08:26Z

/bench sysbench

zhongzc · 2020-11-10T07:41:44Z

/bench sysbench

sre-bot · 2020-11-10T08:57:38Z

Benchmark Report

Run Sysbench Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
tidb: 1702608a6f597a306a601867e195796193c554e0
--- tikv: ad3eb9724f38d7f2c3022ae4c1a30e92cfc35e63
+++ tikv: 52b3d6f38c3f37c0af8dfe86a28300f9d9313721
pd: bf8aa5801cae7f435db979ee0c1be94a7e900e61
================================================================================
oltp_read_write:
    * QPS: 29072.20 ± 0.07% (std=14.42) delta: -1.66% (p=0.000)
    * Latency p50: 89.06 ± 0.07% (std=0.04) delta: 1.69%
    * Latency p99: 153.02 ± 0.00% (std=0.00) delta: 1.82%
            
oltp_point_select:
    * QPS: 93811.21 ± 0.23% (std=166.98) delta: -4.91% (p=0.000)
    * Latency p50: 1.37 ± 0.37% (std=0.01) delta: 5.00%
    * Latency p99: 4.78 ± 0.84% (std=0.04) delta: 4.60%
            
oltp_update_non_index:
    * QPS: 21779.32 ± 0.02% (std=3.18) delta: -6.18% (p=0.000)
    * Latency p50: 5.88 ± 0.21% (std=0.01) delta: 6.67%
    * Latency p99: 11.65 ± 0.00% (std=0.00) delta: 5.53%
            
oltp_update_index:
    * QPS: 16524.06 ± 0.02% (std=2.89) delta: -5.54% (p=0.000)
    * Latency p50: 7.75 ± 0.35% (std=0.02) delta: 5.94%
    * Latency p99: 15.27 ± 0.00% (std=0.00) delta: 4.62%

zhongzc · 2020-11-10T10:02:23Z

/bench sysbench

Signed-off-by: zhongzc <zhongzc_arch@outlook.com> trace raftstore Signed-off-by: zhongzc <zhongzc_arch@outlook.com> trace read Signed-off-by: zhongzc <zhongzc_arch@outlook.com> config for bench Signed-off-by: zhongzc <zhongzc_arch@outlook.com> fix test compilation Signed-off-by: zhongzc <zhongzc_arch@outlook.com> fix test compilation Signed-off-by: zhongzc <zhongzc_arch@outlook.com> set span id prefix Signed-off-by: zhongzc <zhongzc_arch@outlook.com> bump minitrace Signed-off-by: zhongzc <zhongzc_arch@outlook.com> optimize enabling multiple scopes Signed-off-by: zhongzc <zhongzc_arch@outlook.com> background worker Signed-off-by: zhongzc <zhongzc_arch@outlook.com> start scopes Signed-off-by: zhongzc <zhongzc_arch@outlook.com> turn off tracing Signed-off-by: zhongzc <zhongzc_arch@outlook.com> bench no serialization Signed-off-by: zhongzc <zhongzc_arch@outlook.com> bump minitrace Signed-off-by: zhongzc <zhongzc_arch@outlook.com> fix conflict Signed-off-by: zhongzc <zhongzc_arch@outlook.com> bump minitrace Signed-off-by: zhongzc <zhongzc_arch@outlook.com> tmp Signed-off-by: zhongzc <zhongzc_arch@outlook.com> pb + jaeger Signed-off-by: zhongzc <zhongzc_arch@outlook.com> polish Signed-off-by: zhongzc <zhongzc_arch@outlook.com> return on demand Signed-off-by: zhongzc <zhongzc_arch@outlook.com> make test compiled Signed-off-by: zhongzc <zhongzc_arch@outlook.com> make test compiled Signed-off-by: zhongzc <zhongzc_arch@outlook.com> Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> simplify Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

ti-chi-bot · 2021-03-02T04:56:47Z

@zhongzc: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

sre-bot · 2021-07-21T06:49:06Z

download tikv at http://fileserver.pingcap.net/download/builds/pingcap/tikv/pr/60b1ffc56c76c73a8d4dceb7d97007a288b3f331/centos7/tikv-server.tar.gz

sre-bot · 2021-07-22T06:29:23Z

download tikv at http://fileserver.pingcap.net/download/builds/pingcap/tikv/pr/60b1ffc56c76c73a8d4dceb7d97007a288b3f331/centos7/tikv-server.tar.gz

sre-bot · 2021-07-22T06:35:12Z

bench result at http://fileserver.pingcap.net/download/builds/pingcap/tikv/pr/60b1ffc56c76c73a8d4dceb7d97007a288b3f331/centos7/tikv-server.tar.gz

sre-bot · 2021-07-22T09:28:24Z

bench result at http://fileserver.pingcap.net/download/builds/pingcap/tikv/pr/60b1ffc56c76c73a8d4dceb7d97007a288b3f331/centos7/tikv-server.tar.gz

sre-bot · 2021-07-22T09:35:09Z

bench result at http://fileserver.pingcap.net/download/builds/pingcap/tikv/pr/60b1ffc56c76c73a8d4dceb7d97007a288b3f331/centos7/tikv-server.tar.gz

sre-bot · 2021-07-22T09:36:11Z

bench result at http://fileserver.pingcap.net/download/builds/pingcap/tikv/pr/60b1ffc56c76c73a8d4dceb7d97007a288b3f331/centos7/tikv-server.tar.gz

sre-bot · 2021-07-26T05:23:15Z

bench result at http://fileserver.pingcap.net/download/builds/pingcap/tikv/pr/60b1ffc56c76c73a8d4dceb7d97007a288b3f331/centos7/tikv-server.tar.gz

sre-bot · 2021-07-26T05:54:12Z

bench result at http://fileserver.pingcap.net/download/builds/pingcap/tikv/pr/60b1ffc56c76c73a8d4dceb7d97007a288b3f331/centos7/tikv-server.tar.gz

sre-bot · 2021-08-06T07:45:11Z

Benchmark Report

> Run Sysbench Performance Test on VMs

sticnarf · 2021-11-16T03:56:25Z

@zhongzc May I ask is there any update on this feature?

zhongzc added the status/DNM label Nov 6, 2020

zhongzc force-pushed the new-minitrace branch from a389d5e to a55772e Compare November 6, 2020 05:48

zhongzc force-pushed the new-minitrace branch from 8404562 to 1238ee9 Compare November 6, 2020 08:31

This comment has been minimized.

Sign in to view

zhongzc force-pushed the new-minitrace branch 3 times, most recently from caf42b3 to f227a2b Compare November 9, 2020 06:16

breezewish mentioned this pull request Nov 9, 2020

*: Report tracing results to Jaeger #8440

Closed

This comment has been minimized.

Sign in to view

zhongzc force-pushed the new-minitrace branch from a41adcb to 2d9c131 Compare November 10, 2020 06:33

zhongzc changed the title *: bench tracing *: set up tracing framework Nov 10, 2020

zhongzc force-pushed the new-minitrace branch from d2d17e9 to 52b3d6f Compare November 10, 2020 07:37

zhongzc changed the title *: set up tracing framework *: introduce tracing framework Nov 10, 2020

zhongzc changed the title *: introduce tracing framework *: Introduce tracing framework Nov 10, 2020

zhongzc force-pushed the new-minitrace branch from 8977622 to fac6e64 Compare January 15, 2021 06:35

ti-chi-bot added the needs-rebase label Jan 15, 2021

zhongzc force-pushed the new-minitrace branch from fac6e64 to 9ff12f1 Compare January 19, 2021 12:18

ti-chi-bot added needs-rebase and removed needs-rebase labels Jan 19, 2021

zhongzc force-pushed the new-minitrace branch from 9ff12f1 to 9448150 Compare January 25, 2021 04:11

ti-chi-bot removed the needs-rebase label Jan 25, 2021

breezewish self-assigned this Jan 25, 2021

breezewish added the sig/diagnosis SIG: Diagnosis label Jan 25, 2021

zhongzc force-pushed the new-minitrace branch 2 times, most recently from 3a5ff76 to 22335af Compare January 28, 2021 07:12

ti-chi-bot added the needs-rebase label Jan 28, 2021

zhongzc force-pushed the new-minitrace branch from 22335af to 019f2e9 Compare February 2, 2021 06:23

ti-chi-bot removed the needs-rebase label Feb 2, 2021

ti-chi-bot added the needs-rebase label Mar 2, 2021

external span as parent span

60b1ffc

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

sticnarf mentioned this pull request Nov 16, 2021

Trace client-side and server-side latency to diagnose RPC latency #11378

Open

tikv deleted a comment from sre-bot Dec 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: Introduce tracing framework #8981

*: Introduce tracing framework #8981

zhongzc commented Nov 6, 2020 •

edited

sre-bot commented Nov 6, 2020

zhongzc commented Nov 6, 2020

sre-bot commented Nov 6, 2020

zhongzc commented Nov 6, 2020

sre-bot commented Nov 6, 2020

zhongzc commented Nov 6, 2020

zhongzc commented Nov 6, 2020

This comment has been minimized.

zhongzc commented Nov 9, 2020

zhongzc commented Nov 9, 2020

zhongzc commented Nov 9, 2020

This comment has been minimized.

This comment has been minimized.

zhongzc commented Nov 10, 2020

zhongzc commented Nov 10, 2020

zhongzc commented Nov 10, 2020

sre-bot commented Nov 10, 2020

zhongzc commented Nov 10, 2020

ti-chi-bot commented Mar 2, 2021

sre-bot commented Jul 21, 2021

sre-bot commented Jul 22, 2021

sre-bot commented Jul 22, 2021

sre-bot commented Jul 22, 2021

sre-bot commented Jul 22, 2021

sre-bot commented Jul 22, 2021

sre-bot commented Jul 26, 2021

sre-bot commented Jul 26, 2021

sre-bot commented Aug 6, 2021

sticnarf commented Nov 16, 2021

*: Introduce tracing framework #8981

Are you sure you want to change the base?

*: Introduce tracing framework #8981

Conversation

zhongzc commented Nov 6, 2020 • edited

What problem does this PR solve?

What is changed and how it works?

Release note

sre-bot commented Nov 6, 2020

zhongzc commented Nov 6, 2020

sre-bot commented Nov 6, 2020

zhongzc commented Nov 6, 2020

sre-bot commented Nov 6, 2020

zhongzc commented Nov 6, 2020

zhongzc commented Nov 6, 2020

This comment has been minimized.

zhongzc commented Nov 9, 2020

zhongzc commented Nov 9, 2020

zhongzc commented Nov 9, 2020

This comment has been minimized.

This comment has been minimized.

zhongzc commented Nov 10, 2020

zhongzc commented Nov 10, 2020

zhongzc commented Nov 10, 2020

sre-bot commented Nov 10, 2020

Benchmark Report

zhongzc commented Nov 10, 2020

ti-chi-bot commented Mar 2, 2021

sre-bot commented Jul 21, 2021

sre-bot commented Jul 22, 2021

sre-bot commented Jul 22, 2021

sre-bot commented Jul 22, 2021

sre-bot commented Jul 22, 2021

sre-bot commented Jul 22, 2021

sre-bot commented Jul 26, 2021

sre-bot commented Jul 26, 2021

sre-bot commented Aug 6, 2021

Benchmark Report

sticnarf commented Nov 16, 2021

zhongzc commented Nov 6, 2020 •

edited