OPCT-226: Added check rules for etcd

[mtulio]

Introducing etcd check rules based in the parsed logs for etcd (etcd request took too long).

The acceptance values are calibrated from the baseline providers:

• AWS (ocp414rc0_AWS_None_202309222127_sonobuoy_47efe9ef-06e4-48f3-a190-4e3523ff1ae0.tar.gz)

• AWS (4.13.9-20230925-HighlyAvailable-aws-None-202309250459_sonobuoy_f4e06587-e7b3-4cbd-bcf4-d1350ec35d9a.tar.gz):

• vSphere (4.13.9-20230821-HighlyAvailable-vsphere-None.tar.gz):

The check rules is implemented in the feature #76

provider-certification-tool/internal/opct/report/checks.go

Lines 321 to 388 in dc21563

	checkSum.Checks = append(checkSum.Checks, &Check{
	ID: "OPCT-010",
	Name: "etcd logs: slow requests: average should be under 500ms",
	Test: func() CheckResult {
	prefix := "Check OPCT-010 Failed"
	if re.Provider.MustGatherInfo == nil {
	log.Debugf("%s: unable to read must-gather information.", prefix)
	return CheckResultFail
	}
	if re.Provider.MustGatherInfo.ErrorEtcdLogs.FilterRequestSlowAll["all"] == nil {
	log.Debugf("%s: unable to read statistics from parsed etcd logs.", prefix)
	return CheckResultFail
	}
	if re.Provider.MustGatherInfo.ErrorEtcdLogs.FilterRequestSlowAll["all"].StatMean == "" {
	log.Debugf("%s: unable to get p50/mean statistics from parsed data: %v", prefix, re.Provider.MustGatherInfo.ErrorEtcdLogs.FilterRequestSlowAll["all"])
	return CheckResultFail
	}
	values := strings.Split(re.Provider.MustGatherInfo.ErrorEtcdLogs.FilterRequestSlowAll["all"].StatMean, " ")
	if values[0] == "" {
	log.Debugf("%s: unable to get parse p50/mean: %v", prefix, values)
	return CheckResultFail
	}
	value, err := strconv.ParseFloat(values[0], 64)
	if err != nil {
	log.Debugf("%s: unable to convert p50/mean to float: %v", prefix, err)
	return CheckResultFail
	}
	if value >= 500 {
	log.Debugf("%s acceptance criteria: want=[%v] got=[%v]", prefix, "<500", value)
	return CheckResultFail
	}
	return CheckResultPass
	},
	})
	checkSum.Checks = append(checkSum.Checks, &Check{
	ID: "OPCT-011",
	Name: "etcd logs: slow requests: maximum should be under 1500ms",
	Test: func() CheckResult {
	prefix := "Check OPCT-011 Failed"
	if re.Provider.MustGatherInfo == nil {
	log.Debugf("%s: unable to read must-gather information.", prefix)
	return CheckResultFail
	}
	if re.Provider.MustGatherInfo.ErrorEtcdLogs.FilterRequestSlowAll["all"] == nil {
	log.Debugf("%s: unable to read statistics from parsed etcd logs.", prefix)
	return CheckResultFail
	}
	if re.Provider.MustGatherInfo.ErrorEtcdLogs.FilterRequestSlowAll["all"].StatMax == "" {
	log.Debugf("%s: unable to get p50/mean statistics from parsed data: %v", prefix, re.Provider.MustGatherInfo.ErrorEtcdLogs.FilterRequestSlowAll["all"])
	return CheckResultFail
	}
	values := strings.Split(re.Provider.MustGatherInfo.ErrorEtcdLogs.FilterRequestSlowAll["all"].StatMax, " ")
	if values[0] == "" {
	log.Debugf("%s: unable to get parse p50/mean: %v", prefix, values)
	return CheckResultFail
	}
	value, err := strconv.ParseFloat(values[0], 64)
	if err != nil {
	log.Debugf("%s: unable to convert p50/mean to float: %v", prefix, err)
	return CheckResultFail
	}
	if value >= 1500 {
	log.Debugf("%s acceptance criteria: want=[%v] got=[%v]", prefix, "<1500", value)
	return CheckResultFail
	}
	return CheckResultPass
	},
	})

[mtulio]

need to be merged after fixing broken master https://github.com/redhat-openshift-ecosystem/provider-certification-tool/commits/main
/hold

[mtulio]

/assign @rvanderp3

[mtulio]

Thanks @rvanderp3 ! Fixed.
Also fixed the main branch doc build (CI):
/hold cancel

[rvanderp3]

/lgtm
/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rvanderp3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rvanderp3]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment