Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Record some system events #5799

Open
rleungx opened this issue Nov 4, 2019 · 6 comments
Open

Record some system events #5799

rleungx opened this issue Nov 4, 2019 · 6 comments
Assignees
Labels

Comments

@rleungx
Copy link
Member

@rleungx rleungx commented Nov 4, 2019

Feature Request

Describe the feature you'd like:

Record some system events in TiKV just like the CH does. Then we can support SQL query in TiDB to help diagnose the TiKV by utilizing these events.
Some examples of how CH does this:
record page fault events through rusage syscall
record open file events through open syscall
SQL query from the system table

@winkyao

This comment has been minimized.

Copy link
Contributor

@winkyao winkyao commented Nov 4, 2019

@overvenus

This comment has been minimized.

Copy link
Contributor

@overvenus overvenus commented Nov 4, 2019

Prometheus already records them, we can just read these metrics.

@siddontang

This comment has been minimized.

Copy link
Contributor

@siddontang siddontang commented Nov 4, 2019

@rleungx

I think this is awesome, but how can a contributor do it? where to start?

please list the detailed steps here. Or we should build a framework at first, then the contributors can add them one by one easily.

@siddontang

This comment has been minimized.

Copy link
Contributor

@siddontang siddontang commented Nov 4, 2019

Prometheus already records them, we can just read these metrics.

If the user hasn't deployed Prometheus, how can we know the running system?

Another thing is that don't let TiKV, TiDB, PD read Prometheus directly.

@siddontang

This comment has been minimized.

Copy link
Contributor

@siddontang siddontang commented Nov 5, 2019

I suggest we can build a crate to do it so other Rust programs can get benefit from it.

@lonng lonng self-assigned this Nov 5, 2019
@nolouch nolouch added the C: Util label Nov 11, 2019
@wangwangwar

This comment has been minimized.

Copy link
Contributor

@wangwangwar wangwangwar commented Dec 4, 2019

How about adding some DTrace probes as MySQL does and collecting metrics by eBPF or other dynamic tracing tools? I think this way is more extensible and can benefit from the community of dynamic tracing.


MySQL DTrace probes: https://dev.mysql.com/doc/refman/5.7/en/dba-dtrace-mysqld-ref.htm
Export eBPF metrics: #4183
Rust BCC: https://github.com/rust-bpf/rust-bcc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.