Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pika-exporter：Add RocksDB column family metrics(#1566) #1696

Merged
merged 4 commits into from
Jul 24, 2023
Merged

pika-exporter：Add RocksDB column family metrics(#1566) #1696

merged 4 commits into from
Jul 24, 2023

Conversation

chenbt-hz
Copy link
Collaborator

@chenbt-hz chenbt-hz commented Jul 8, 2023 •
edited
Loading

Fixes: #1566

新增以下的监控指标：
已zset为例，以下是info all的结果

zsets_cf-l0-file-count-limit-delays-with-ongoing-compaction: 0
zsets_cf-l0-file-count-limit-stops-with-ongoing-compaction: 0
zsets_compaction.L0.AvgSec: 0.043061
zsets_compaction.L0.CompCount: 1.000000
zsets_compaction.L0.CompMergeCPU: 0.001078
zsets_compaction.L0.CompSec: 0.043061
zsets_compaction.L0.CompactedFiles: 0.000000
zsets_compaction.L0.KeyDrop: 0.000000
zsets_compaction.L0.KeyIn: 0.000000
zsets_compaction.L0.MovedGB: 0.000000
zsets_compaction.L0.NumFiles: 1.000000
zsets_compaction.L0.RblobGB: 0.000000
zsets_compaction.L0.ReadGB: 0.000000
zsets_compaction.L0.ReadMBps: 0.000000
zsets_compaction.L0.RnGB: 0.000000
zsets_compaction.L0.Rnp1GB: 0.000000
zsets_compaction.L0.Score: 0.250000
zsets_compaction.L0.SizeBytes: 1182.000000
zsets_compaction.L0.WblobGB: 0.000000
zsets_compaction.L0.WnewGB: 0.000001
zsets_compaction.L0.WriteAmp: 1.000000
zsets_compaction.L0.WriteGB: 0.000001
zsets_compaction.L0.WriteMBps: 0.026177
zsets_compaction.Sum.AvgSec: 0.043061
zsets_compaction.Sum.CompCount: 1.000000
zsets_compaction.Sum.CompMergeCPU: 0.001078
zsets_compaction.Sum.CompSec: 0.043061
zsets_compaction.Sum.CompactedFiles: 0.000000
zsets_compaction.Sum.KeyDrop: 0.000000
zsets_compaction.Sum.KeyIn: 0.000000
zsets_compaction.Sum.MovedGB: 0.000000
zsets_compaction.Sum.NumFiles: 1.000000
zsets_compaction.Sum.RblobGB: 0.000000
zsets_compaction.Sum.ReadGB: 0.000000
zsets_compaction.Sum.ReadMBps: 0.000000
zsets_compaction.Sum.RnGB: 0.000000
zsets_compaction.Sum.Rnp1GB: 0.000000
zsets_compaction.Sum.Score: 0.000000
zsets_compaction.Sum.SizeBytes: 1182.000000
zsets_compaction.Sum.WblobGB: 0.000000
zsets_compaction.Sum.WnewGB: 0.000001
zsets_compaction.Sum.WriteAmp: 1.000000
zsets_compaction.Sum.WriteGB: 0.000001
zsets_compaction.Sum.WriteMBps: 0.026177
zsets_l0-file-count-limit-delays: 0
zsets_l0-file-count-limit-stops: 0
zsets_memtable-limit-delays: 0
zsets_memtable-limit-stops: 0
zsets_pending-compaction-bytes-delays: 0
zsets_pending-compaction-bytes-stops: 0
zsets_total-delays: 0
zsets_total-stops: 0

采集的数据大致如下：

....
# HELP pika_compaction 
#The all metrics of compaction_L<N>:
#compaction.L<N>.AvgSec	Average time spent per Compact.
#compaction.L<N>.CompCount	The number of times Compact has been accumulated.
#compaction.L<N>.CompMergeCPU	CPU time used in compression, in seconds.
#compaction.L<N>.CompSec	Compact cumulative time, in seconds.
#compaction.L<N>.CompactedFiles	Number of files that have completed compact.
#compaction.L<N>.KeyDrop	Number of keys deleted in compact.
#compaction.L<N>.KeyIn	Number of records compared during the compaction process.
#compaction.L<N>.MovedGB	During the compaction process, the number of bytes moved to level n+1.
#compaction.L<N>.NumFiles	Total number of sst files.
#compaction.L<N>.RblobGB	The size of data read from the blob file by the compaction, in GB.
#compaction.L<N>.ReadGB	Read size in GB.
#compaction.L<N>.ReadMBps	Read rate in MBps.
#compaction.L<N>.RnGB	When performing compact, read the size of the current layer file in GB.
#compaction.L<N>.Rnp1GB	When performing compact, read the size of the next level file in GB.
#compaction.L<N>.Score	Score, the higher the score, the higher the priority.
#compaction.L<N>.SizeBytes	Total size of SST in bytes.
#compaction.L<N>.WblobGB	The size of the blob file written during compaction, in GB.
#compaction.L<N>.WnewGB	WNP1- Rnp1.
#compaction.L<N>.WriteAmp	Total bytes written to level<N+1>/(total bytes read from level<N>).
#compaction.L<N>.WriteGB	The size of the table file written during the compaction period, in GB.
#compaction.L<N>.WriteMBps	Data write rate in compaction.
# TYPE pika_compaction gauge
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="AvgSec"} 0.048421
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="CompCount"} 2
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="CompMergeCPU"} 0.004067
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="CompSec"} 0.096843
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="CompactedFiles"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="KeyDrop"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="KeyIn"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="MovedGB"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="NumFiles"} 2
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="RblobGB"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="ReadGB"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="ReadMBps"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="RnGB"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="Rnp1GB"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="Score"} 0.5
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="SizeBytes"} 2358
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="WblobGB"} 0
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="WnewGB"} 2e-06
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="WriteAmp"} 1
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="WriteGB"} 2e-06
pika_compaction{addr="127.0.0.1:9221",alias="",compaction_level="0",data_type="strings",info_type="WriteMBps"} 0.02322
# HELP pika_compaction_Sum The all metrics of compaction_Sum.
# TYPE pika_compaction_Sum gauge
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="AvgSec"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="CompCount"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="CompMergeCPU"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="CompSec"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="CompactedFiles"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="KeyDrop"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="KeyIn"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="MovedGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="NumFiles"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="RblobGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="ReadGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="ReadMBps"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="RnGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="Rnp1GB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="Score"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="SizeBytes"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="WblobGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="WnewGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="WriteAmp"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="WriteGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="hashes",info_type="WriteMBps"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="AvgSec"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="CompCount"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="CompMergeCPU"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="CompSec"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="CompactedFiles"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="KeyDrop"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="KeyIn"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="MovedGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="NumFiles"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="RblobGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="ReadGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="ReadMBps"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="RnGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="Rnp1GB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="Score"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="SizeBytes"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="WblobGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="WnewGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="WriteAmp"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="WriteGB"} 0
pika_compaction_Sum{addr="127.0.0.1:9221",alias="",data_type="lists",info_type="WriteMBps"} 0
....

@yaoyinnan
Copy link
Contributor

I think we should discuss which indicators are more important, and expose indicators that are more conducive to helping O&M personnel monitor Pika on grafana. For example, io_stalls can reflect disk I/O performance very well, but there are many compactions stats, so you need to consider whether to display or display part.

@chenbt-hz
Copy link
Collaborator Author

I think we should discuss which indicators are more important, and expose indicators that are more conducive to helping O&M personnel monitor Pika on grafana. For example, io_stalls can reflect disk I/O performance very well, but there are many compactions stats, so you need to consider whether to display or display part.

I discussed your suggestion with my DBA colleagues. Perhaps due to the lack of annotation information and experience in use, it is difficult for everyone to judge the specific practical significance of such indicators. This week, I will try to sort out the specific meaning of each indicator to facilitate discussion.

yaoyinnan
yaoyinnan reviewed Jul 13, 2023
View reviewed changes
Copy link
Contributor

@yaoyinnan yaoyinnan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A modification suggestion is mainly to supplement the description information of the indicator.

I think this PR can be merged after modification. I discussed this with @AlexStocks. After the plan, you can start a new PR, the pika_exporter obtains the method of Metrics into the form of configurable to obtain a part of the information. Now obtained through the info all command. After, then it can be disassembled into multiple info XXX commands to obtain metrics.

What do you think of this plan?

tools/pika_exporter/README.md Outdated Show resolved Hide resolved
tools/pika_exporter/README.md Show resolved Hide resolved
tools/pika_exporter/exporter/metrics/rocksdb.go Outdated Show resolved Hide resolved
cr-gpt[bot]
cr-gpt bot reviewed Jul 22, 2023
View reviewed changes
tools/pika_exporter/Makefile Outdated Show resolved Hide resolved
@yaoyinnan
Copy link
Contributor

@chenbt-hz need to rebase

yaoyinnan
yaoyinnan previously approved these changes Jul 22, 2023
View reviewed changes
@yaoyinnan yaoyinnan self-requested a review July 22, 2023 14:15
yaoyinnan
yaoyinnan approved these changes Jul 23, 2023
View reviewed changes
Copy link
Contributor

@yaoyinnan yaoyinnan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yaoyinnan yaoyinnan requested a review from Mixficsol July 23, 2023 13:42
@Mixficsol Mixficsol merged commit c77a6d3 into OpenAtomFoundation:unstable Jul 24, 2023
11 checks passed
@yaoyinnan yaoyinnan mentioned this pull request Jul 27, 2023
Closed
bigdaronlee163 pushed a commit to bigdaronlee163/pika that referenced this pull request Jun 8, 2024
@chenbt-hz
pika-exporter：Add RocksDB column family metrics(OpenAtomFoundation#1566…
8b4a71c 
…) (OpenAtomFoundation#1696)

* 支持cfstats监控数据采集

* 修改监控指标注释;修复时间解析错误bug;添加mac-amd64编译

* 添加监控指标说明

* bug fix
cheniujh pushed a commit to cheniujh/pika that referenced this pull request Sep 24, 2024
@chenbt-hz
pika-exporter：Add RocksDB column family metrics(OpenAtomFoundation#1566…
a85d46a 
…) (OpenAtomFoundation#1696)

* 支持cfstats监控数据采集

* 修改监控指标注释;修复时间解析错误bug;添加mac-amd64编译

* 添加监控指标说明

* bug fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cr-gpt CR.GPT CR.GPT left review comments

@yaoyinnan yaoyinnan yaoyinnan approved these changes

@Mixficsol Mixficsol Mixficsol approved these changes

@machinly machinly Awaiting requested review from machinly

@luky116 luky116 Awaiting requested review from luky116

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add RocksDB column family metrics
3 participants
@chenbt-hz @yaoyinnan @Mixficsol