Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OPCT-12: Embed on tools image the etcd log parser #40

Conversation

mtulio
Copy link
Contributor

@mtulio mtulio commented Apr 6, 2023

https://issues.redhat.com/browse/OPCT-12

Embed the tool used on the review process to parse etcd logs from must-gather filtering by slowly apply requests. It will aggregate the slow requests into buckets of 100ms, sharing counters for each one and which bucket the slow requests stayed, useful when comparing executions with baseline HW (reference) and new ones on the time frame the e2e tests have.

For reference:

  • expected to have fewer requests reporting higher than 500ms
$ cat ~/opct/tmp/must-gather/must-gather-opct/*/namespaces/openshift-etcd/pods/*/etcd/etcd/logs/current.log  \
 | ./build/ocp-etcd-log-filters 
> Filter Name: ApplyTookTooLong
> Group by: all
>>> Summary <<<
all	 16949	
>500ms	 1485	(8.762 %)
---
>>> Buckets <<<
low-200	 0	(0.000 %)
200-300	 9340	(55.106 %)
300-400	 4169	(24.597 %)
400-500	 1853	(10.933 %)
500-600	 716	(4.224 %)
600-700	 223	(1.316 %)
700-800	 185	(1.092 %)
800-900	 139	(0.820 %)
900-1s	 79	(0.466 %)
1s-inf	 143	(0.844 %)
unkw	 102	(0.602 %)

  • aggregating by hour
$ cat ~/opct/tmp/must-gather/must-gather-opct/*/namespaces/openshift-etcd/pods/*/etcd/etcd/logs/current.log  \
 | ./build/ocp-etcd-log-filters -aggregator hour
> Filter Name: ApplyTookTooLong
> Group by: hour

>> 2023-03-01T17
>>> Summary <<<
all	 558	
>500ms	 54	(9.677 %)
---
>>> Buckets <<<
low-200	 0	(0.000 %)
200-300	 385	(68.996 %)
300-400	 90	(16.129 %)
400-500	 28	(5.018 %)
500-600	 9	(1.613 %)
600-700	 10	(1.792 %)
700-800	 7	(1.254 %)
800-900	 9	(1.613 %)
900-1s	 16	(2.867 %)
1s-inf	 3	(0.538 %)
unkw	 1	(0.179 %)

>> 2023-03-01T15
>>> Summary <<<
all	 7740	
>500ms	 619	(7.997 %)
---
>>> Buckets <<<
low-200	 0	(0.000 %)
200-300	 4122	(53.256 %)
300-400	 2107	(27.222 %)
400-500	 842	(10.879 %)
500-600	 379	(4.897 %)
600-700	 78	(1.008 %)
700-800	 67	(0.866 %)
800-900	 55	(0.711 %)
900-1s	 15	(0.194 %)
1s-inf	 25	(0.323 %)
unkw	 50	(0.646 %)

>> 2023-03-01T16
>>> Summary <<<
all	 8651	
>500ms	 812	(9.386 %)
---
>>> Buckets <<<
low-200	 0	(0.000 %)
200-300	 4833	(55.866 %)
300-400	 1972	(22.795 %)
400-500	 983	(11.363 %)
500-600	 328	(3.791 %)
600-700	 135	(1.561 %)
700-800	 111	(1.283 %)
800-900	 75	(0.867 %)
900-1s	 48	(0.555 %)
1s-inf	 115	(1.329 %)
unkw	 51	(0.590 %)

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 6, 2023
@mtulio mtulio added kind/feature Categorizes issue or PR as related to a new feature. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. labels Apr 6, 2023
@mtulio mtulio marked this pull request as ready for review April 10, 2023 20:06
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 10, 2023
@openshift-ci openshift-ci bot requested review from bostrt and rvanderp3 April 10, 2023 20:06
@mtulio
Copy link
Contributor Author

mtulio commented Apr 10, 2023

Tools image published (it's manual), the CI is failing while the quay.io is processing it. I will re-run the job in a few minutes. PR ready for review.
/assign @rvanderp3 @faermanj

@mtulio mtulio force-pushed the opct-12-embed-etcd-log-parser branch from cf059b8 to 7e8633f Compare April 10, 2023 20:58
@rvanderp3
Copy link
Contributor

/lgtm
/approve

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Apr 10, 2023
@openshift-ci
Copy link

openshift-ci bot commented Apr 10, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rvanderp3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 10, 2023
@openshift-merge-robot openshift-merge-robot merged commit 3baec0e into redhat-openshift-ecosystem:main Apr 10, 2023
3 checks passed
@mtulio mtulio deleted the opct-12-embed-etcd-log-parser branch April 10, 2023 21:56
openshift-merge-robot pushed a commit to redhat-openshift-ecosystem/provider-certification-tool that referenced this pull request Apr 11, 2023
https://issues.redhat.com/browse/OPCT-12

Improving the documentation for 'installation review', providing
references of expected etcd performance when inspecting the etcd logs
for slow requests on must-gather.

Sync with
redhat-openshift-ecosystem/provider-certification-plugins#40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/feature Categorizes issue or PR as related to a new feature. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants