NETOBSERV-1343: generate dashboards from metrics API #609

jotak · 2024-04-03T15:42:01Z

Description

Add dashboard config to metrics API
Use this API internally for predefined dashboards
Allow using SingleStats
New dedicated buckets for latency histograms

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

openshift-ci-robot · 2024-04-03T15:42:04Z

openshift-ci · 2024-04-03T15:42:06Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

jotak · 2024-04-03T15:42:22Z

Pending to do:

~~Clean up API (kubebuilder tags etc.)~~ done
~~Handle new dashboards~~ done
~~Fix and add tests~~ done
~~Handle charts ordering when they come from different FlowMetrics (e.g. new Priority field)~~ done

codecov-commenter · 2024-04-08T11:07:53Z

Codecov Report

Attention: Patch coverage is 93.42561% with 38 lines in your changes are missing coverage. Please review.

Project coverage is 67.03%. Comparing base (10a6846) to head (515c7e5).
Report is 10 commits behind head on main.

❗ Current head 515c7e5 differs from pull request most recent head fe3a9e6. Consider uploading reports for the commit fe3a9e6 to get more accurate results

Files	Patch %	Lines
apis/flowmetrics/v1alpha1/zz_generated.deepcopy.go	44.00%	14 Missing ⚠️
controllers/monitoring/monitoring_controller.go	74.41%	5 Missing and 6 partials ⚠️
controllers/reconcilers/reconcilers.go	71.42%	3 Missing and 1 partial ⚠️
pkg/dashboards/model.go	92.15%	4 Missing ⚠️
controllers/flp/flp_pipeline_builder.go	85.71%	1 Missing and 1 partial ⚠️
controllers/reconcilers/common.go	0.00%	2 Missing ⚠️
pkg/metrics/predefined_charts.go	99.54%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #609      +/-   ##
==========================================
+ Coverage   66.35%   67.03%   +0.68%     
==========================================
  Files          65       66       +1     
  Lines        7321     7557     +236     
==========================================
+ Hits         4858     5066     +208     
- Misses       2104     2133      +29     
+ Partials      359      358       -1

Flag	Coverage Δ
unittests	`67.03% <93.42%> (+0.68%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

jotak · 2024-04-08T14:14:47Z

/test ci-index-noo-bundle

jpinsonneau · 2024-04-15T09:13:14Z

pkg/dashboards/dashboard.go

+	var targets []Target
+	for _, q := range c.Queries {
+		query := strings.ReplaceAll(q.PromQL, "$METRIC", "netobserv_"+c.mptr.Spec.MetricName)
+		query = fmt.Sprintf("topk(7, %s)", query)


Why top 7 here ? Can't it be configurable to fine grain Prom perfs ?

100% subjective yes, but from personal testing it was the best compromise to see relevant info without overloading the charts. Top 10 easily leads to overcrowded charts (especially as it's a top per datapoint, not a "global" top); while too small top leads to showing more partial metrics/chunks. Of course that's very dependent on what we run on the cluster.
So yes I agree it can be made configurable.
We just need to care about finding a good balance between too much and too few settings.
I guess it doesn't hurt to add that one

cool ! thanks @jotak

done in 5b02dd1

github-actions · 2024-04-15T11:52:03Z

New images:

quay.io/netobserv/network-observability-operator:e1458c2
quay.io/netobserv/network-observability-operator-bundle:v0.0.0-e1458c2
quay.io/netobserv/network-observability-operator-catalog:v0.0.0-e1458c2

They will expire after two weeks.

To deploy this build:

# Direct deployment, from operator repo
IMAGE=quay.io/netobserv/network-observability-operator:e1458c2 make deploy

# Or using operator-sdk
operator-sdk run bundle quay.io/netobserv/network-observability-operator-bundle:v0.0.0-e1458c2

Or as a Catalog Source:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: netobserv-dev
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/netobserv/network-observability-operator-catalog:v0.0.0-e1458c2
  displayName: NetObserv development catalog
  publisher: Me
  updateStrategy:
    registryPoll:
      interval: 1m

jpinsonneau

LGTM thanks @jotak !

Do we need to provide more examples ? 🤔
I'm thinking about optionnal features (Drops / DNS / RTT). That can be done in followups

jotak · 2024-04-16T13:24:40Z

Do we need to provide more examples ? 🤔 I'm thinking about optionnal features (Drops / DNS / RTT). That can be done in followups

There's already drops/dns/rtt metrics with their charts as part of the predefined metrics, which can be enabled via the includeList; but we could have more "combined" examples such as external traffic latency; I can add that

- Add dashboard config to metrics API - Use this API internally for predefined dashboards - Allow using SingleStats - New dedicated buckets for latency histograms

- Allow users to reference/create new dashboards (this requires the monitoring controller to fetch all dashboards from the dashboards namespace) All dashboard names are prefixed "NetObserv / " ; so, rename the main dashboard as "NetObserv / Main" - Add/update tests

an Overview section

- generate markdown API reference - document in Metrics.md

memodi · 2024-04-17T14:08:18Z

/ok-to-test

github-actions · 2024-04-17T14:11:45Z

New images:

quay.io/netobserv/network-observability-operator:8487560
quay.io/netobserv/network-observability-operator-bundle:v0.0.0-8487560
quay.io/netobserv/network-observability-operator-catalog:v0.0.0-8487560

They will expire after two weeks.

To deploy this build:

# Direct deployment, from operator repo
IMAGE=quay.io/netobserv/network-observability-operator:8487560 make deploy

# Or using operator-sdk
operator-sdk run bundle quay.io/netobserv/network-observability-operator-bundle:v0.0.0-8487560

Or as a Catalog Source:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: netobserv-dev
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/netobserv/network-observability-operator-catalog:v0.0.0-8487560
  displayName: NetObserv development catalog
  publisher: Me
  updateStrategy:
    registryPoll:
      interval: 1m

memodi · 2024-04-18T19:44:52Z

@jotak - feedback on unit, there's no way to leave it empty as instruction suggest, perhaps it's possible in yaml, is it possible to set custom units like fps (flows per second) ?

config/samples/flowmetrics/service_ports.yaml

Co-authored-by: Mehul Modi <memodi@redhat.com>

jotak · 2024-04-19T08:28:01Z

@jotak - feedback on unit, there's no way to leave it empty as instruction suggest, perhaps it's possible in yaml, is it possible to set custom units like fps (flows per second) ?

No we can't set custom units (that wouldn't work with the console's dashboard engine), but you're right we need to add something as a fallback unit .. perhaps "numeric"

memodi · 2024-04-26T18:47:18Z

/ok-to-test

github-actions · 2024-04-26T18:50:27Z

New images:

quay.io/netobserv/network-observability-operator:e973630
quay.io/netobserv/network-observability-operator-bundle:v0.0.0-e973630
quay.io/netobserv/network-observability-operator-catalog:v0.0.0-e973630

They will expire after two weeks.

To deploy this build:

# Direct deployment, from operator repo
IMAGE=quay.io/netobserv/network-observability-operator:e973630 make deploy

# Or using operator-sdk
operator-sdk run bundle quay.io/netobserv/network-observability-operator-bundle:v0.0.0-e973630

Or as a Catalog Source:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: netobserv-dev
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/netobserv/network-observability-operator-catalog:v0.0.0-e973630
  displayName: NetObserv development catalog
  publisher: Me
  updateStrategy:
    registryPoll:
      interval: 1m

openshift-ci · 2024-04-26T19:03:21Z

@jotak: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-operator	`fe3a9e6`	link	false	`/test e2e-operator`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

memodi · 2024-04-26T20:22:14Z

/label qe-approved

openshift-ci-robot · 2024-04-26T20:22:20Z

jotak · 2024-05-07T08:38:25Z

/approve

openshift-ci · 2024-05-07T08:38:35Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jotak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jotak]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot added the jira/valid-reference label Apr 3, 2024

openshift-ci bot added the do-not-merge/work-in-progress label Apr 3, 2024

jotak force-pushed the custom-metrics-dashboards branch from 7d96d0f to a40f093 Compare April 4, 2024 12:11

jotak force-pushed the custom-metrics-dashboards branch from bcec79e to 044d976 Compare April 8, 2024 11:20

jotak marked this pull request as ready for review April 8, 2024 12:23

openshift-ci bot removed the do-not-merge/work-in-progress label Apr 8, 2024

jotak requested review from msherif1234, OlivierCazade and jpinsonneau April 8, 2024 12:23

jpinsonneau reviewed Apr 15, 2024

View reviewed changes

jotak added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 15, 2024

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 15, 2024

jotak requested a review from jpinsonneau April 15, 2024 13:06

jpinsonneau approved these changes Apr 16, 2024

View reviewed changes

openshift-ci bot assigned jpinsonneau Apr 16, 2024

openshift-ci bot added the lgtm label Apr 16, 2024

openshift-merge-robot added the needs-rebase label Apr 16, 2024

jotak added 6 commits April 16, 2024 17:01

NETOBSERV-1343: generate dashboards from metrics API

01bda0a

- Add dashboard config to metrics API - Use this API internally for predefined dashboards - Allow using SingleStats - New dedicated buckets for latency histograms

Document API, gen bundle, fix tests, update examples

2325b29

update bundle

2ecbff3

enforce consistent charts order

5435224

When too many panels are in top row, put them in

f7ee31f

an Overview section

More examples

ebf5897

jotak force-pushed the custom-metrics-dashboards branch from 5938f74 to ebf5897 Compare April 16, 2024 15:01

openshift-ci bot removed the lgtm label Apr 16, 2024

openshift-merge-robot removed the needs-rebase label Apr 16, 2024

jotak mentioned this pull request Apr 16, 2024

OSDOCS-9480: NetObserv Custom Metrics openshift/openshift-docs#74412

Open

1 task

jotak added 2 commits April 17, 2024 12:10

Remove includeDuplicates, and some minor doc/samples changes

623b58e

Add FlowMetrics upstream doc

88a08ae

- generate markdown API reference - document in Metrics.md

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 17, 2024

memodi reviewed Apr 18, 2024

View reviewed changes

config/samples/flowmetrics/service_ports.yaml Outdated Show resolved Hide resolved

memodi reviewed Apr 18, 2024

View reviewed changes

config/samples/flowmetrics/service_ports.yaml Outdated Show resolved Hide resolved

Apply suggestions from code review

515c7e5

Co-authored-by: Mehul Modi <memodi@redhat.com>

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 19, 2024

Add more info on cardinality

fe3a9e6

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 26, 2024

openshift-ci bot added the qe-approved QE has approved this pull request label Apr 26, 2024

jpinsonneau approved these changes Apr 30, 2024

View reviewed changes

openshift-ci bot added the lgtm label Apr 30, 2024

openshift-ci bot added the approved label May 7, 2024

jotak merged commit 33b5849 into netobserv:main May 7, 2024
9 of 12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NETOBSERV-1343: generate dashboards from metrics API #609

NETOBSERV-1343: generate dashboards from metrics API #609

jotak commented Apr 3, 2024

openshift-ci-robot commented Apr 3, 2024 •

edited by openshift-ci bot

Description

Dependencies

Checklist

openshift-ci bot commented Apr 3, 2024

jotak commented Apr 3, 2024 •

edited

codecov-commenter commented Apr 8, 2024 •

edited

jotak commented Apr 8, 2024

jpinsonneau Apr 15, 2024

jotak Apr 15, 2024 •

edited

jpinsonneau Apr 15, 2024

jotak Apr 15, 2024

github-actions bot commented Apr 15, 2024

jpinsonneau left a comment

jotak commented Apr 16, 2024

memodi commented Apr 17, 2024

github-actions bot commented Apr 17, 2024

memodi commented Apr 18, 2024 •

edited

jotak commented Apr 19, 2024

memodi commented Apr 26, 2024

github-actions bot commented Apr 26, 2024

openshift-ci bot commented Apr 26, 2024

memodi commented Apr 26, 2024

openshift-ci-robot commented Apr 26, 2024 •

edited by openshift-ci bot

Description

Dependencies

Checklist

jotak commented May 7, 2024

openshift-ci bot commented May 7, 2024

NETOBSERV-1343: generate dashboards from metrics API #609

NETOBSERV-1343: generate dashboards from metrics API #609

Conversation

jotak commented Apr 3, 2024

Description

Dependencies

Checklist

openshift-ci-robot commented Apr 3, 2024 • edited by openshift-ci bot

Description

Dependencies

Checklist

openshift-ci bot commented Apr 3, 2024

jotak commented Apr 3, 2024 • edited

codecov-commenter commented Apr 8, 2024 • edited

Codecov Report

jotak commented Apr 8, 2024

jpinsonneau Apr 15, 2024

Choose a reason for hiding this comment

jotak Apr 15, 2024 • edited

Choose a reason for hiding this comment

jpinsonneau Apr 15, 2024

Choose a reason for hiding this comment

jotak Apr 15, 2024

Choose a reason for hiding this comment

github-actions bot commented Apr 15, 2024

jpinsonneau left a comment

Choose a reason for hiding this comment

jotak commented Apr 16, 2024

memodi commented Apr 17, 2024

github-actions bot commented Apr 17, 2024

memodi commented Apr 18, 2024 • edited

jotak commented Apr 19, 2024

memodi commented Apr 26, 2024

github-actions bot commented Apr 26, 2024

openshift-ci bot commented Apr 26, 2024

memodi commented Apr 26, 2024

openshift-ci-robot commented Apr 26, 2024 • edited by openshift-ci bot

Description

Dependencies

Checklist

jotak commented May 7, 2024

openshift-ci bot commented May 7, 2024

openshift-ci-robot commented Apr 3, 2024 •

edited by openshift-ci bot

jotak commented Apr 3, 2024 •

edited

codecov-commenter commented Apr 8, 2024 •

edited

jotak Apr 15, 2024 •

edited

memodi commented Apr 18, 2024 •

edited

openshift-ci-robot commented Apr 26, 2024 •

edited by openshift-ci bot