NETOBSERV-1407 console deduper merge mode #435

jpinsonneau · 2023-11-28T16:29:08Z

Description

Manage Interface and Direction arrays as a single field in the UI according to mark & merge configs
Skip frontend deduplication for both flows & metrics queries
Update filtering engine to query on new fields

To setup deduper merge, simply add the following config in ebpf debug (or advanced) configuration:

      advanced:
        env:
          DEDUPER_JUST_MARK: "false"
          DEDUPER_MERGE: "true"

Dependencies

netobserv/network-observability-operator#501
netobserv/network-observability-operator#529

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

codecov · 2023-11-28T16:31:37Z

Codecov Report

Attention: 55 lines in your changes are missing coverage. Please review.

Comparison is base (8f2df29) 58.61% compared to head (cc28165) 58.34%.
Report is 2 commits behind head on main.

❗ Current head cc28165 differs from pull request most recent head 09a3b91. Consider uploading reports for the commit 09a3b91 to get more accurate results

Files	Patch %	Lines
web/src/components/netflow-record/record-panel.tsx	12.12%	28 Missing and 1 partial ⚠️
web/src/components/netflow-record/record-field.tsx	47.36%	10 Missing ⚠️
pkg/loki/flow_query.go	10.00%	8 Missing and 1 partial ⚠️
pkg/loki/filter.go	33.33%	4 Missing ⚠️
web/src/components/netflow-traffic.tsx	60.00%	1 Missing and 1 partial ⚠️
web/src/api/ipfix.ts	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #435      +/-   ##
==========================================
- Coverage   58.61%   58.34%   -0.28%     
==========================================
  Files         167      167              
  Lines        8312     8385      +73     
  Branches     1061     1074      +13     
==========================================
+ Hits         4872     4892      +20     
- Misses       3132     3182      +50     
- Partials      308      311       +3

Flag	Coverage Δ
uitests	`58.87% <36.36%> (-0.36%)`	⬇️
unittests	`56.79% <58.06%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

msherif1234 · 2024-01-09T21:30:10Z

/ok-to-test

github-actions · 2024-01-22T12:11:23Z

New image:
quay.io/netobserv/network-observability-console-plugin:51c5a2d

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=51c5a2d make set-plugin-image

jpinsonneau · 2024-01-22T12:26:00Z

I sometimes get FlowDirection as string and sometimes as number 🤔
We should be consistent and only use numbers in any case

I'm adapting the plugin to manage both in between

msherif1234 · 2024-01-24T12:18:50Z

config/sample-config.yaml

@@ -812,3 +818,6 @@ frontend:
  alertNamespaces:
    - netobserv
  sampling: 50
+  deduper:
+    mark: false
+    merge: true


isn't better to stay with the current default till we change it ?

msherif1234 · 2024-01-24T12:22:11Z

pkg/handler/config.go

@@ -122,6 +128,10 @@ func ReadConfigFile(version, date, filename string) (*Config, error) {
 			Filters:      []Filter{},
 			QuickFilters: []QuickFilter{},
 			Features:     []string{},
+			Deduper: Deduper{


can u pls add a note here so when the agent default change we need to remember and change this as well ? OR it will be much better to have only one source of truth can u read it from the same cm instead of hard coding it here ?

I've added a TODO for now. We could load by default the sample yaml but that would force us to embed it in the code and parse two yaml, one after the other.

The plugin accept as arguments a file path that is configurable. These defaults are overriden below (line 141)

msherif1234 · 2024-01-24T12:27:37Z

pkg/handler/topology.go

-	return metricType == constants.MetricTypeBytes ||
-		metricType == constants.MetricTypePackets
+func shouldMergeReporters(metricType constants.MetricType, deduper loki.Deduper) bool {
+	return !deduper.Merge && deduper.Mark && (metricType == constants.MetricTypeBytes ||


I don't think it make any sense to have mark true and merge true at the sametime
while we don't enforce this yet but this is something we should consider
IMO valid configs

Mark | Merge

T | F
F | T

since we always need to know about dup so both T or both F is invalid, maybe we can just single config knob at the agent side ?

Both merge and mark could be false 🙃
But I agree both true doesn't make sense on eBPF side. I just wanted to manage every cases here since we have 2 flags for now.

I suggest to keep as is and cleanup the code for enablement in https://issues.redhat.com/browse/NETOBSERV-1459

msherif1234 · 2024-01-24T12:30:07Z

pkg/loki/filter.go

@@ -141,7 +142,7 @@ func (f *lineFilter) asLabelFilters() []labelFilter {
 			valueType: v.valueType,
 			value:     v.value,
 		}
-		if v.valueType == typeRegex || v.valueType == typeRegexContains {
+		if v.valueType == typeRegex || v.valueType == typeRegexContains || v.valueType == typeRegexArrayContains {


maybe worth small helper to check for all regex flavor ?

msherif1234 · 2024-01-24T12:31:10Z

pkg/loki/filter.go

@@ -141,7 +142,7 @@ func (f *lineFilter) asLabelFilters() []labelFilter {
 			valueType: v.valueType,
 			value:     v.value,
 		}
-		if v.valueType == typeRegex || v.valueType == typeRegexContains {
+		if v.valueType == typeRegex || v.valueType == typeRegexContains || v.valueType == typeRegexArrayContains {
 			if f.not {
 				lf.matcher = labelNoMatches
 			} else {


not related to this commit but I think u can avoid if else here just init lf.matcher and check if !f.not

sounds good to me

msherif1234 · 2024-01-24T12:32:11Z

pkg/loki/flow_query.go

+		default:
+			isArray = false
+		}
+	}


I don't understand this its always an array if merge is on ?

Yes !

Currently, when deduper merged is enabled, we have both single and plural fields as:
FlowDirection, Interface, FlowDirections, Interfaces

I'm forcing a `lineMatch`` on plural fields here to avoid any query running on single fields.
In the future, I expect fully removing FlowDirection and Interface fields.

msherif1234 · 2024-01-24T12:36:00Z

web/src/components/netflow-record/record-field.tsx

+        if (
+          flow.fields.Interfaces &&
+          flow.fields.FlowDirections &&
+          flow.fields.Interfaces.length === flow.fields.FlowDirections.length


why u need this check its not possible the two arrays of different len ?

It's not supposed to happen indeed, but just in case, I'm checking the length to avoid crashing in the loop following this test since I'm accessing values by their index.

msherif1234 · 2024-01-24T12:38:04Z

web/src/api/ipfix.ts

+    ? t('Ingress')
+    : value === FlowDirection.Egress
+    ? t('Egress')
+    : value === FlowDirection.Inner


can we ever get Inner flow direction its either ingress or egress right ?

Inner is something calculated from reinterpret direction in FLP.
It's not yet adapted to plural fields

see https://github.com/netobserv/flowlogs-pipeline/blob/40b69485049482aee71b55a5b7b0fa52cd49576e/pkg/pipeline/transform/transform_network_direction.go#L62

msherif1234 · 2024-01-24T18:03:35Z

sounds good to me Thanks @jpinsonneau
/LGTM

openshift-ci · 2024-01-25T11:59:09Z

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot added the do-not-merge/work-in-progress label Nov 28, 2023

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jan 9, 2024

jpinsonneau force-pushed the 1407 branch from 87c7f08 to 8b2773f Compare January 18, 2024 11:48

openshift-merge-robot added the needs-rebase label Jan 18, 2024

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jan 18, 2024

manage deduper merge

4755fba

jpinsonneau force-pushed the 1407 branch from 8b2773f to 4755fba Compare January 18, 2024 11:49

openshift-merge-robot removed the needs-rebase label Jan 18, 2024

jpinsonneau force-pushed the 1407 branch from 0546884 to 1637a6b Compare January 19, 2024 15:37

netobserv deleted a comment from github-actions bot Jan 22, 2024

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jan 22, 2024

jpinsonneau marked this pull request as ready for review January 22, 2024 11:21

openshift-ci bot removed the do-not-merge/work-in-progress label Jan 22, 2024

manage array filtering for deduper

cf8ff66

jpinsonneau force-pushed the 1407 branch from 1637a6b to cf8ff66 Compare January 22, 2024 12:07

netobserv deleted a comment from github-actions bot Jan 22, 2024

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jan 22, 2024

jpinsonneau requested a review from msherif1234 January 22, 2024 12:08

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jan 22, 2024

manage when FlowDirection is number

cc28165

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jan 22, 2024

msherif1234 reviewed Jan 24, 2024

View reviewed changes

addressed feedback

09a3b91

openshift-ci bot assigned msherif1234 Jan 24, 2024

openshift-ci bot added the lgtm label Jan 24, 2024

jpinsonneau added no-qe This PR doesn't necessitate QE approval approved labels Jan 25, 2024

openshift-merge-bot bot merged commit dfe7ac4 into netobserv:main Jan 25, 2024
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NETOBSERV-1407 console deduper merge mode #435

NETOBSERV-1407 console deduper merge mode #435

jpinsonneau commented Nov 28, 2023 •

edited

codecov bot commented Nov 28, 2023 •

edited

msherif1234 commented Jan 9, 2024

github-actions bot commented Jan 22, 2024

jpinsonneau commented Jan 22, 2024 •

edited

msherif1234 Jan 24, 2024

jpinsonneau Jan 24, 2024

msherif1234 Jan 24, 2024

jpinsonneau Jan 24, 2024

msherif1234 Jan 24, 2024 •

edited

jpinsonneau Jan 24, 2024

msherif1234 Jan 24, 2024

jpinsonneau Jan 24, 2024

msherif1234 Jan 24, 2024

jpinsonneau Jan 24, 2024

msherif1234 Jan 24, 2024

jpinsonneau Jan 24, 2024

msherif1234 Jan 24, 2024

jpinsonneau Jan 24, 2024

msherif1234 Jan 24, 2024

jpinsonneau Jan 24, 2024 •

edited

jpinsonneau Jan 24, 2024

msherif1234 commented Jan 24, 2024

openshift-ci bot commented Jan 25, 2024

NETOBSERV-1407 console deduper merge mode #435

NETOBSERV-1407 console deduper merge mode #435

Conversation

jpinsonneau commented Nov 28, 2023 • edited

Description

Dependencies

Checklist

codecov bot commented Nov 28, 2023 • edited

Codecov Report

msherif1234 commented Jan 9, 2024

github-actions bot commented Jan 22, 2024

jpinsonneau commented Jan 22, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msherif1234 Jan 24, 2024 • edited

Choose a reason for hiding this comment

Mark | Merge

T | F F | T

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpinsonneau Jan 24, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msherif1234 commented Jan 24, 2024

openshift-ci bot commented Jan 25, 2024

jpinsonneau commented Nov 28, 2023 •

edited

codecov bot commented Nov 28, 2023 •

edited

jpinsonneau commented Jan 22, 2024 •

edited

msherif1234 Jan 24, 2024 •

edited

T | F
F | T

jpinsonneau Jan 24, 2024 •

edited