NETOBSERV-59 Create oc plugin to display flow table #1

jpinsonneau · 2023-11-17T10:42:17Z

This PR implement first version of oc netobserv cli

Capture flows
Capture packets
Basic filter capabilities

Dependencies

jpinsonneau · 2023-11-17T11:13:16Z

res/flow-capture.yml

+        # image: quay.io/netobserv/netobserv-ebpf-agent:main
+        image: quay.io/jpinsonn/netobserv-ebpf-agent:flow-stream-1


netobserv/netobserv-ebpf-agent#224 needs to be merged first

I wonder if u can reuse the current gRPC exporter from ebpf agent instead of creating new TCP exported ?

Would that work with oc port-forward ?

gRPC uses TCP so I would think so

jpinsonneau · 2023-11-17T13:54:46Z

res/flow-capture.yml

+          # TODO: find why this breaks collection: EOF
+          #- name: ENABLE_PKT_DROPS
+          #  value: "true"


Uncomment this to give a try with pktDrop

After some time running it, I get EOF in the TCP connection read

It seems I was just unlucky ! It's working now

stleerh

It should provide a script or a way to uninstall this on your cluster.

stleerh · 2023-11-17T17:21:15Z

.gitignore

@@ -0,0 +1,2 @@
+build
+output


A number of new files in this commit are missing the linefeed character on the last line. We should clean this up.

stleerh · 2023-11-17T17:28:18Z

Makefile

+OUTPUT := $(DIST_DIR)/$(NAME)
+DOCKER := $(shell which podman)
+ifeq ($(DOCKER),)
+	DOCKER := $(shell which docker)


Perhaps call it OCI or OCI_BIN instead of DOCKER.

Sure, the Docker file is not even provided yet. I'm thinking about being able to run the cli as an extra pod inside the cluster and transfert the generated files locally before deleting it.

That would avoid running all the port-forward and reduce the transfers / latency
I just need to ensure the terminal render properly using oc rsh <pod> / oc exec <pod>

stleerh · 2023-11-17T17:36:22Z

README.md

@@ -3,18 +3,78 @@
 network-observability-cli is a lightweight Flow and Packet visualisation tool.
 It deploy [netobserv eBPF agent](https://github.com/netobserv/netobserv-ebpf-agent) on your k8s cluster to collect flows or packets from nodes network interfaces


Change: "deploys"

stleerh · 2023-11-17T17:38:20Z

README.md

@@ -3,18 +3,78 @@
 network-observability-cli is a lightweight Flow and Packet visualisation tool.
 It deploy [netobserv eBPF agent](https://github.com/netobserv/netobserv-ebpf-agent) on your k8s cluster to collect flows or packets from nodes network interfaces
 and streams data to a local collector for analysis and visualisation.
-Output files are generated under output/flow and output/pcap directories per host name
+Output files are generated under `output/flow` and `output/pcap` directories per host name


Add a section on Prerequisites that describe what you need to have (e.g. need a kind or OpenShift cluster).

stleerh · 2023-11-17T17:39:14Z

cmd/flow-capture.go

+)
+
+const (
+	flowsToShow = 40


Make it configurable?

I did actually, and even more ! I'll make a demo to show all the capabilities

stleerh · 2023-11-17T18:15:39Z

cmd/flow-capture.go

+
+	// lock since we are updating lastFlows concurrently
+	mutex.Lock()
+


Unfortunately, this defeats the benefits of using goroutines.

Correct, the problem is that I'm merging X sources (1 per nodes) to a single display.
Would it be more performant to save flows per node in different slices or map and call mutex.Lock only when reading them for display ?

stleerh · 2023-11-17T18:19:47Z

cmd/flow-capture.go

+		c := exec.Command("clear")
+		c.Stdout = os.Stdout
+		c.Run()
+


I know this tool is not meant to be scalable, but clearing and redrawing the screen for each flow is going to make it very limited. A better alternative is to periodically update the screen at a certain interval.

I've added a mechanism that update only when a delay is over

stleerh · 2023-11-17T18:23:11Z

cmd/flow-capture.go

+	if err != nil {
+		log.Errorf("Create file %s failed: %v", filename, err.Error())
+		log.Fatal(err)
+	}


It would be very useful if you can choose the stdout format, which could just be JSON instead of the UI table. I know technically you can effectively get this with the 'tee' command, but if it didn't output to the UI table, it would be more scalable. This would be simpler to export flows than what we have today, particularly if you just want to test this out and see what data you're getting.

I've added some predefined view capabilities, including raw JSON 👍

stleerh · 2023-11-17T18:35:47Z

cmd/packet-capture.go

+		if err != nil {
+			log.Fatal(err)
+		}
+		go managePcapTable(PcapResult{NodeName: filename, PacketCount: 1, ByteCount: int64(n)})


Not sure how useful this will be by only having node, packet and byte counts, even with a filter applied.

This can become huge in a short time since everything is saved to the output folder
It's more an indicator to avoid filling up your storage more than anything else

jpinsonneau · 2023-11-20T08:24:52Z

It should provide a script or a way to uninstall this on your cluster.

the cleanup is automatic, but I can provide a script that allow user to call it manually 👍

jpinsonneau · 2024-01-30T14:31:20Z

@jotak @msherif1234 @OlivierCazade how do you see the next steps for that ?

I can merge as is if you agree and we can improve the approach if needed. Let me know what works best for you 🐱

OlivierCazade · 2024-01-31T14:44:59Z

I am fine merging and improving from here. This make iterations easier to read.

IMO the first question that we need to answer is, do we want to keep this fully independent or de we want to use a NOO stack already present.

Calling this the network observability cli might be a bit missleading. I would expect from a cli to be the terminal equivalent of the frontend, not an independent project with its own capture stack.

If we keep this independent I would suggest to rename it to avoid confusion.

jpinsonneau · 2024-02-01T09:49:54Z

IMO the first question that we need to answer is, do we want to keep this fully independent or de we want to use a NOO stack already present.

I'm open to any solution that deploy as fast as applying the yaml in this PR. I think that's the key of this tool.

Relying on operator catalog to install netobserv is probably a bit slow and would not work on every cluster
Deploying the controller manager via a YAML is quite fast (same as make deploy from operator repo)

That could bring to less maintenance in the different config when we introduce new features indeed.
We need to investigate on the merge eBPF/FLP components attempt from @jotak first, see if it's worth and if it will be introduced in the operator.

Calling this the network observability cli might be a bit missleading. I would expect from a cli to be the terminal equivalent of the frontend, not an independent project with its own capture stack.
If we keep this independent I would suggest to rename it to avoid confusion.

It will never be the equivalent since we can't handle Loki here. The capture must be different but it can be part of the operator.
Do you have any suggestion in mind ?

jotak · 2024-02-07T09:26:11Z

cmd/flow-capture.go

+func runFlowCapture(cmd *cobra.Command, args []string) {
+	go scanner()
+
+	wg.Add(len(addresses))


wg and goroutines seem wrongly implemented here, we should have a wg.Done() call somewhere... It probably doesn't do what we'd expect here.
I'm trying to change & make sure it doesn't break things

Ok thanks; feel free to update this PR directly if you want to

done - if you want to review my 2 commits

Code looks good ! I'm giving a last try before merging

jotak · 2024-02-07T10:13:41Z

@jpinsonneau I'm ok to merge, there are a couple of points though that I've noted and that we may address in a follow-up :

Ctrl-C doesn't work on packet capture for me (it works in flow capture) - I haven't investigated...
currently, local ports are hardcoded; we should make port-forwarding more robust & able to deal with ports in use
refine packaging; installing yaml files in /usr/bin doesn't sound correct I guess? Perhaps we could embed the resources in scripts at build time?

/lgtm

jpinsonneau · 2024-02-07T10:18:39Z

I have created a followup for code improvments:
https://issues.redhat.com/browse/NETOBSERV-1484

Thanks !

first release

998f1cb

jpinsonneau mentioned this pull request Nov 17, 2023

NETOBSERV-59 Add tcp flow stream exporter netobserv/netobserv-ebpf-agent#224

Closed

11 tasks

jpinsonneau commented Nov 17, 2023

View reviewed changes

pkt drop volume mounts

318c642

jpinsonneau commented Nov 17, 2023

View reviewed changes

enable pkt drop

1de4c88

stleerh reviewed Nov 17, 2023

View reviewed changes

flow table display / filter

b0e429c

feedback + lint

efafe62

jotak reviewed Feb 7, 2024

View reviewed changes

jotak added 2 commits February 7, 2024 11:01

Readme / typo fixes

cc39db6

Fix WaitGroup use (async, defer Done, capture 'i')

72491e2

jpinsonneau added 2 commits February 7, 2024 11:48

fix packet exit

5237a18

make oc-commands

6779632

jpinsonneau merged commit 43fcbf0 into netobserv:main Feb 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NETOBSERV-59 Create oc plugin to display flow table #1

NETOBSERV-59 Create oc plugin to display flow table #1

jpinsonneau commented Nov 17, 2023 •

edited

jpinsonneau Nov 17, 2023

msherif1234 Nov 17, 2023

jpinsonneau Nov 17, 2023

msherif1234 Nov 17, 2023

jpinsonneau Nov 17, 2023

jpinsonneau Nov 17, 2023

stleerh left a comment

stleerh Nov 17, 2023

stleerh Nov 17, 2023

jpinsonneau Nov 20, 2023

stleerh Nov 17, 2023

stleerh Nov 17, 2023

stleerh Nov 17, 2023

jpinsonneau Nov 20, 2023

stleerh Nov 17, 2023

jpinsonneau Nov 20, 2023

stleerh Nov 17, 2023

jpinsonneau Nov 20, 2023

stleerh Nov 17, 2023 •

edited

jpinsonneau Nov 20, 2023

stleerh Nov 17, 2023

jpinsonneau Nov 20, 2023

jpinsonneau commented Nov 20, 2023

jpinsonneau commented Jan 30, 2024

OlivierCazade commented Jan 31, 2024

jpinsonneau commented Feb 1, 2024

jotak Feb 7, 2024

jpinsonneau Feb 7, 2024

jotak Feb 7, 2024

jpinsonneau Feb 7, 2024

jotak commented Feb 7, 2024

jpinsonneau commented Feb 7, 2024

		# image: quay.io/netobserv/netobserv-ebpf-agent:main
		image: quay.io/jpinsonn/netobserv-ebpf-agent:flow-stream-1

		@@ -3,18 +3,78 @@
		network-observability-cli is a lightweight Flow and Packet visualisation tool.
		It deploy [netobserv eBPF agent](https://github.com/netobserv/netobserv-ebpf-agent) on your k8s cluster to collect flows or packets from nodes network interfaces


		// lock since we are updating lastFlows concurrently
		mutex.Lock()

NETOBSERV-59 Create oc plugin to display flow table #1

NETOBSERV-59 Create oc plugin to display flow table #1

Conversation

jpinsonneau commented Nov 17, 2023 • edited

Dependencies

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stleerh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stleerh Nov 17, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpinsonneau commented Nov 20, 2023

jpinsonneau commented Jan 30, 2024

OlivierCazade commented Jan 31, 2024

jpinsonneau commented Feb 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jotak commented Feb 7, 2024

jpinsonneau commented Feb 7, 2024

jpinsonneau commented Nov 17, 2023 •

edited

stleerh Nov 17, 2023 •

edited