|
| 1 | +--- |
| 2 | +layout: :theme/post |
| 3 | +title: "Network Observability On Demand 1.8 Update" |
| 4 | +description: Command line interface improvements in version 1.8 |
| 5 | +tags: CLI,Monitoring,Troubleshooting |
| 6 | +authors: [jpinsonneau] |
| 7 | +--- |
| 8 | + |
| 9 | +# Network Observability On Demand 1.8 Update |
| 10 | + |
| 11 | +_Thanks to: Joël Takvorian, Mehul Modi and Sara Thomas for reviewing_ |
| 12 | + |
| 13 | +``` |
| 14 | +------------------------------------------------------------------------ |
| 15 | + _ _ _ _ ___ _ ___ |
| 16 | + | \| |___| |_ ___| |__ ___ ___ _ ___ __ / __| | |_ _| |
| 17 | + | .' / -_) _/ _ \ '_ (_-</ -_) '_\ V / | (__| |__ | | |
| 18 | + |_|\_\___|\__\___/_.__/__/\___|_| \_/ \___|____|___| |
| 19 | +
|
| 20 | +------------------------------------------------------------------------ |
| 21 | +``` |
| 22 | + |
| 23 | +Since we [introduced the Network Observability CLI](./2024-07-25-cli.md), numerous features have been added. This article covers the improvements in version 1.8 and provide some concrete examples. |
| 24 | + |
| 25 | +## New Options Available |
| 26 | +This update adds several options to the CLI, covering more scenarios and enabling scripting on top of the tool. |
| 27 | + |
| 28 | +### Operate in the Background |
| 29 | +The `--background` option allows you to start a flow or packet capture without connecting your terminal to the **collector pod**. This enables you to let the capture run in the background while you work on something else. |
| 30 | +You can check the capture's progress using the `follow` command. |
| 31 | +The capture is ending automatically when either maximum bytes (50MB) or time (5m) is reached. You can update these accordingly using the `--max-bytes` and `--max-time` options. |
| 32 | +The `stop` command also allows you to end the capture manually. |
| 33 | +In both cases, the collector pod keeps running to let you download the output locally using the `copy` command. |
| 34 | +Once the job is complete, you `cleanup` everything. |
| 35 | + |
| 36 | +```sh |
| 37 | +oc netobserv flows --background # Run a flow capture in the background |
| 38 | + |
| 39 | +oc netobserv follow # Show current capture progress |
| 40 | + |
| 41 | +oc netobserv stop # Stop the capture, keeping the collector pod alive |
| 42 | + |
| 43 | +oc netobserv copy # Copy the output |
| 44 | + |
| 45 | +oc netobserv cleanup # Remove all |
| 46 | +``` |
| 47 | + |
| 48 | +### Customizable Namespace |
| 49 | +You can now customize the capture namespace using the `NETOBSERV_NAMESPACE` environment variable. When the CLI starts, it automatically checks if this namespace exists and stops if it finds any conflict with a pending capture. This is particularly useful if you want to run captures in parallel. |
| 50 | + |
| 51 | +```sh |
| 52 | +NETOBSERV_NAMESPACE=my_ns oc netobserv [flows|packets|metrics|follow|stop|copy|cleanup] |
| 53 | +``` |
| 54 | + |
| 55 | +### Subnets Labelling (for OCP clusters) |
| 56 | +The tool can now read OpenShift configurations from `cluster-config-v1` and `network` to identify **Machine**, **Pods**, and **Services** subnets using the `--get-subnets` option. This automatically adds `SrcSubnetLabel` and `DstSubnetLabel` to your flows. |
| 57 | + |
| 58 | +You can see subnets being configured during the creation of the agents: |
| 59 | +```sh |
| 60 | +creating flow-capture agents: |
| 61 | +opt: get_subnets, value: true |
| 62 | +Found subnets: |
| 63 | + Services: "172.30.0.0/16" |
| 64 | + Pods: "10.128.0.0/14" |
| 65 | + Machines: "10.0.0.0/16" |
| 66 | +``` |
| 67 | + |
| 68 | +}) |
| 69 | + |
| 70 | +## Enhanced Data Filtering |
| 71 | +Filtering is crucial to gather precise network data without involving excessive resources and storage. The CLI focuses on this area, allowing you to deploy agents only where needed and fine-tune what's captured. |
| 72 | + |
| 73 | +### Assigning Agents to Nodes |
| 74 | +It's now possible to define agents `nodeSelector` to capture on a subset of nodes. You can rely on existing labels or create a dedicated one for this usage. For example, you can run the agents on the node with the `kubernetes.io/hostname:my-node` label with the following command: |
| 75 | +```sh |
| 76 | +oc netobserv flows --node-selector=kubernetes.io/hostname:my-node |
| 77 | +### eBPF Filters |
| 78 | +Agents recently introduced [the ability to filter](https://github.com/netobserv/netobserv-ebpf-agent/blob/main/docs/flow_filtering.md) on IPs, Ports, Protocol, Action, TCPFlags, and more simultaneously. You can now apply these filters in the CLI as shown below: |
| 79 | + |
| 80 | +```sh |
| 81 | +oc netobserv flows \ # Capture flows |
| 82 | +--protocol=TCP --port=8080 \ # either on TCP 8080 |
| 83 | +or --protocol=UDP # or UDP |
| 84 | +``` |
| 85 | + |
| 86 | +You can see filters being validated during the creation of the agents: |
| 87 | +```sh |
| 88 | +creating flow-capture agents: |
| 89 | +opt: filter_protocol, value: TCP |
| 90 | +opt: filter_port, value: 8080 |
| 91 | +opt: add_filter |
| 92 | +opt: filter_protocol, value: UDP |
| 93 | +``` |
| 94 | + |
| 95 | +}) |
| 96 | + |
| 97 | +You can add up to 16 sets of filters separated by the `or` operator to create multiple capture scenarios. |
| 98 | + |
| 99 | +### Regular Expressions Usage |
| 100 | +If you need to filter on enriched content beyond the agent-level filters, you can use **regexes** to match any field/value pair. To keep only the traffic from OpenShift namespaces, for example, you can use `--regexes=SrcK8S_Namespace~openshift.*`. |
| 101 | + |
| 102 | +You see regexes being validated during the creation of the agents: |
| 103 | +```sh |
| 104 | +creating flow-capture agents: |
| 105 | +opt: filter_regexes, value: SrcK8S_Namespace~openshift.* |
| 106 | +key: SrcK8S_Namespace value: openshift.* |
| 107 | +``` |
| 108 | + |
| 109 | +}) |
| 110 | + |
| 111 | +Regexes are comma-separated, so you can use multiple at once, such as `--regexes=SrcK8S_Namespace~my-ns,SrcK8S_Name~my-app`. Refer to the [flows format](https://github.com/netobserv/network-observability-operator/blob/main/docs/flows-format.adoc) to see the possible fields. |
| 112 | + |
| 113 | +## Unified Collector User Experience |
| 114 | +All filtering capabilities are now supported for **packets** capture and displays enriched data while collecting. This improvement was made possible by introducing the [flowlogs-pipeline](https://github.com/netobserv/flowlogs-pipeline) component inside [eBPF agents](https://github.com/netobserv/netobserv-ebpf-agent), which parses packets and generates flows from them. |
| 115 | + |
| 116 | +Run a packet capture on a specific port for example: |
| 117 | +```sh |
| 118 | +oc netobserv packets --port=80 |
| 119 | +``` |
| 120 | + |
| 121 | +}) |
| 122 | + |
| 123 | +## Metrics Capture on OpenShift |
| 124 | +Capturing metrics is now possible using the `metrics` command. This creates a `ServiceMonitor` to gather metrics from the agents and store them in [Prometheus](https://prometheus.io/). You can enable all or specific features to gather more information about your network, such in: |
| 125 | +```sh |
| 126 | +oc netobserv metrics --enable_all |
| 127 | +``` |
| 128 | +to capture packet drops, DNS and RTT metrics or |
| 129 | +```sh |
| 130 | +oc netobserv metrics --enable_pktdrop |
| 131 | +``` |
| 132 | +to focus only on drops. |
| 133 | + |
| 134 | +On top of the features, you can use all the filtering capabilities mentioned above to focus on what you're looking for. A `NetObserv / On Demand` dashboard is automatically created, showing the results. |
| 135 | +
|
| 136 | +}) |
| 137 | +
|
| 138 | +In this mode, the maximum bytes or time options are ignored since there is no collector pod involved. Only the eBPF agents are deployed. |
| 139 | +You need to run `stop` or `cleanup` command to remove everything. |
| 140 | +
|
| 141 | +## Help! |
| 142 | +
|
| 143 | +The help has been enhanced to provide examples for each command and option. You can type `oc netobserv help` for the general help message: |
| 144 | +
|
| 145 | +```sh |
| 146 | +$ oc netobserv help |
| 147 | +
|
| 148 | +Netobserv CLI allows you to capture flows, packets and metrics from your cluster. |
| 149 | +Find more information at: https://github.com/netobserv/network-observability-cli/ |
| 150 | +
|
| 151 | +Syntax: netobserv [flows|packets|metrics|follow|stop|copy|cleanup|version] [options] |
| 152 | +
|
| 153 | +main commands: |
| 154 | + flows Capture flows information in JSON format using collector pod. |
| 155 | + metrics Capture metrics information in Prometheus using a ServiceMonitor (OCP cluster only). |
| 156 | + packets Capture packets information in pcap format using collector pod. |
| 157 | +
|
| 158 | +extra commands: |
| 159 | + cleanup Remove netobserv components and configurations. |
| 160 | + copy Copy collector generated files locally. |
| 161 | + follow Follow collector logs when running in background. |
| 162 | + stop Stop collection by removing agent daemonset. |
| 163 | + version Print software version. |
| 164 | +
|
| 165 | +basic examples: |
| 166 | + netobserv flows --drops # Capture dropped flows on all nodes |
| 167 | + netobserv packets --port=8080 # Capture packets on port 8080 |
| 168 | + netobserv metrics --enable_all # Capture all cluster metrics with pktDrop, dns, rtt and network events features |
| 169 | +
|
| 170 | +advanced examples: |
| 171 | + Capture drops in background and copy output locally |
| 172 | + netobserv flows --background \ # Capture flows using background mode |
| 173 | + --max-time=15m \ # for a maximum of 15 minutes |
| 174 | + --protocol=TCP --port=8080 \ # either on TCP 8080 |
| 175 | + or --protocol=UDP # or UDP |
| 176 | + netobserv follow # Display the progression of the background capture |
| 177 | + netobserv stop # Stop the background capture by deleting eBPF agents |
| 178 | + netobserv copy # Copy the background capture output data |
| 179 | + netobserv cleanup # Cleanup netobserv CLI by removing the remaining collector pod |
| 180 | +
|
| 181 | + Capture packets on specific nodes and port |
| 182 | + netobserv packets # Capture packets |
| 183 | + --node-selector=netobserv:true \ # on nodes labelled with netobserv=true |
| 184 | + --port=80 \ # on port 80 only |
| 185 | + --max-bytes=100000000 # for a maximum of 100MB |
| 186 | +``` |
| 187 | +
|
| 188 | +You can also request help on a specific command, such as `oc netobserv metrics help`, to get its options list: |
| 189 | +```sh |
| 190 | +$ oc netobserv metrics help |
| 191 | +
|
| 192 | +Netobserv CLI allows you to capture metrics on your OCP cluster. |
| 193 | +Find more information at: https://github.com/netobserv/network-observability-cli/ |
| 194 | +
|
| 195 | +Syntax: netobserv metrics [options] |
| 196 | +
|
| 197 | +features: |
| 198 | + --enable_all: enable all eBPF features (default: false) |
| 199 | + --enable_dns: enable DNS tracking (default: false) |
| 200 | + --enable_network_events: enable network events monitoring (default: false) |
| 201 | + --enable_pkt_translation: enable packet translation (default: false) |
| 202 | + --enable_pkt_drop: enable packet drop (default: false) |
| 203 | + --enable_rtt: enable RTT tracking (default: false) |
| 204 | + --enable_udn_mapping: enable User Defined Network mapping (default: false) |
| 205 | + --get-subnets: get subnets informations (default: false) |
| 206 | +
|
| 207 | +filters: |
| 208 | + --action: filter action (default: Accept) |
| 209 | + --cidr: filter CIDR (default: 0.0.0.0/0) |
| 210 | + --direction: filter direction (default: n/a) |
| 211 | + --dport: filter destination port (default: n/a) |
| 212 | + --dport_range: filter destination port range (default: n/a) |
| 213 | + --dports: filter on either of two destination ports (default: n/a) |
| 214 | + --drops: filter flows with only dropped packets (default: false) |
| 215 | + --icmp_code: filter ICMP code (default: n/a) |
| 216 | + --icmp_type: filter ICMP type (default: n/a) |
| 217 | + --node-selector: capture on specific nodes (default: n/a) |
| 218 | + --peer_ip: filter peer IP (default: n/a) |
| 219 | + --peer_cidr: filter peer CIDR (default: n/a) |
| 220 | + --port_range: filter port range (default: n/a) |
| 221 | + --port: filter port (default: n/a) |
| 222 | + --ports: filter on either of two ports (default: n/a) |
| 223 | + --protocol: filter protocol (default: n/a) |
| 224 | + --regexes: filter flows using regular expression (default: n/a) |
| 225 | + --sport_range: filter source port range (default: n/a) |
| 226 | + --sport: filter source port (default: n/a) |
| 227 | + --sports: filter on either of two source ports (default: n/a) |
| 228 | + --tcp_flags: filter TCP flags (default: n/a) |
| 229 | + --interfaces: interfaces to monitor (default: n/a) |
| 230 | +``` |
| 231 | +
|
| 232 | +## Feedback |
| 233 | +We hope you enjoyed this article! |
| 234 | +
|
| 235 | +NetObserv is an open source project [available on github](https://github.com/netobserv). |
| 236 | +Feel free to share your [ideas](https://github.com/netobserv/network-observability-operator/discussions/categories/ideas), [use cases](https://github.com/netobserv/network-observability-operator/discussions/categories/show-and-tell) or [ask the community for help](https://github.com/netobserv/network-observability-operator/discussions/categories/q-a). |
0 commit comments