Add subnetCount to pod_ip_metrics.go #205

yiningou · 2023-03-13T23:52:36Z

No description provided.

yiningou · 2023-03-14T22:52:44Z

Add ipv4 & ipv6 subnet Count and Utilization metrics to pod_ip_metrics.go
Please review my codes. Thanks!
/assign jingyuanliang
/assign MrHohn

pkg/metrics/collector/pod_ip_metrics.go

MrHohn

Hey Yining, thanks for the PR! The approach generally looks good to me, leaving a few comments on the details.

MrHohn · 2023-03-21T17:13:05Z

pkg/metrics/collector/pod_ip_metrics.go

@@ -37,6 +39,7 @@ const (
 	ipv6
 	dual

+	gkeCNIConfigDir  = "/etc/cni/net.d/10-gke-ptp.conflist"


I would suggest follow the same pattern as what we did in init container and derive the actual CNI config name from CNI_SPEC_NAME:

netd/scripts/install-cni.sh

Line 251 in a937568

output_file="/host/etc/cni/net.d/${CNI_SPEC_NAME}"

In that way we have a lower chance of hitting any mismatch problem.

This one is tricky. How will we handle the Calico case?

Good catch - with Calico there are a few more things to consider:

CNI file may not be available on netd startup.

Need to pipe through the Calico CNI name.

Parsing is a bit different as Calico puts { "subnet": "usePodCidr" } instead of a plain CIDR.

Thought about parsing the CNI template file instead but that may not work as the template file will be removed by Calico pod promptly.

It may be fine to start without supporting this case though.

CNI file may not be available on netd startup.

Not really. install-cni.sh waits for calico's conflist before moving ahead.

For reference, Calico's regular conf:

{ "name":"k8s-pod-network", "cniVersion":"0.3.1", "plugins":[ { "type":"calico", "mtu":1460, "log_level":"info", "log_file_path":"/var/log/calico/cni/cni.log", "datastore_type":"kubernetes", "nodename":"gke-calico-netd-default-pool-8ce288fc-wb07", "ipam":{ "type":"host-local", "subnet":"usePodCidr" }, "policy":{ "type":"k8s" }, "kubernetes":{ "kubeconfig":"/etc/cni/net.d/calico-kubeconfig" } }, { "type":"portmap", "capabilities":{ "portMappings":true }, "snat":true }, { "type":"bandwidth", "capabilities":{ "bandwidth":true } } ] }

Direct path conf:

{ "name":"gke-pod-network", "cniVersion":"0.3.1", "plugins":[ { "type":"calico", "mtu":1460, "log_level":"warning", "log_file_path":"/var/log/calico/cni/cni.log", "datastore_type":"kubernetes", "nodename":"gke-calico-netd-v6-default-pool-fc507ad4-vmx2", "ipam":{ "type":"host-local", "ranges":[ [ { "subnet":"usePodCidr" } ], [ { "subnet":"2600:2d00:4005:c4e0:a80:3a:0:0/112" } ] ], "routes":[ { "dst":"0.0.0.0/0" }, { "dst":"2001:4860:8040::/42" } ] }, "policy":{ "type":"k8s" }, "kubernetes":{ "kubeconfig":"/etc/cni/net.d/calico-kubeconfig" } }, { "type":"portmap", "capabilities":{ "portMappings":true }, "snat":true }, { "type":"bandwidth", "capabilities":{ "bandwidth":true } } ] }

/etc/cni/net.d/10-calico.conflist

Given the complexity aroundusePodCidr I'm not going to block this PR on calico support - that may come afterward if needed.

pkg/metrics/collector/pod_ip_metrics.go

pkg/metrics/collector/pod_ip_metrics_test.go

jingyuanliang

Didn't review tests because they may be changed based on how other parts change.

jingyuanliang · 2023-03-22T07:00:20Z

pkg/metrics/collector/pod_ip_metrics.go

@@ -37,6 +39,7 @@ const (
 	ipv6
 	dual

+	gkeCNIConfigDir  = "/etc/cni/net.d/10-gke-ptp.conflist"


This one is tricky. How will we handle the Calico case?

pkg/metrics/collector/pod_ip_metrics.go

MrHohn

Thanks Yining, adding second around of comments - this looks good overall.

pkg/metrics/collector/pod_ip_metrics.go

pkg/metrics/collector/pod_ip_metrics_test.go

pkg/metrics/collector/pod_ip_metrics.go

MrHohn

Thanks Yining, two minor comments. Could you fix the commit message to remove the redundant message from the fix-ups?

MrHohn · 2023-04-04T06:59:56Z

pkg/metrics/collector/pod_ip_metrics.go

+	return nil
+}
+
+// hostsPerNetwork returns the number of available hosts in a subnet.


This should match the method name - // countIPsFromRange returns ...

I'm now thinking, why are we doing it by parsing the CNI config file from the beginning?

I previously thought, by doing this we can probably handle more different scenarios where the CNI is configured by some different party or in a different layout, but in the current implementation, what it can handle is pretty limited to the CNI config written by install-cni.sh using the default cni_spec_template (which is even not a part of this repo), and it currently just populates with .spec.podCIDR.

Given this, what about repeating what install-cni.sh does here and read .spec.podCIDR here then export the range size? This will happen to work well with calico's config too. By the way, is the control plane currently populating podCIDRs correctly for IPv6 and what about directpath?

I apologize for not addressing your comment earlier, as I was focusing on fixing the above-mentioned comments. I'd be more than happy to discuss this issue further and explore possible solutions together. How about we schedule a quick meeting or have a discussion session?

MrHohn · 2023-04-04T07:02:10Z

pkg/metrics/collector/pod_ip_metrics.go

+		if cniSpecName == "" {
+			glog.Errorf("CNI_SPEC_NAME environment variable is not set")
+		} else {
+			subnetFile := "/host/etc/cni/net.d/" + cniSpecName


Let's make "/host/etc/cni/net.d/" a constant.

jingyuanliang · 2023-04-04T07:25:28Z

pkg/metrics/collector/pod_ip_metrics.go

@@ -91,18 +94,31 @@ var (
 		"Indicates the IP reuse duration in millisecond for all IPs.",
 		nil, nil,
 	)
-	podIPMetricsWatcherSetup = false
+	assignedIPv4AddrCountDesc = prometheus.NewDesc(
+		"IPv4_assigned_count",


maybe keep this all lowercase for consistency

jingyuanliang · 2023-04-04T07:28:40Z

pkg/metrics/collector/pod_ip_metrics.go

+// Number of hosts is calculated with the formula:
+// IPv4: 2^x – 2, not consider network and broadcast address
+// IPv6: 2^x - 1, not consider network address
+// where x is the number of host bits in the subnet.


do we consider full RangeStart/End support, instead of just the default values?

jingyuanliang · 2023-04-04T07:31:26Z

pkg/metrics/collector/pod_ip_metrics.go

+	}
+	max := uint64(1) << uint(bits-ones)
+	// Don't use the network's ".0" address,
+	if max == 0 {


looks like we'll never run into this case?

jingyuanliang · 2023-04-04T07:31:48Z

pkg/metrics/collector/pod_ip_metrics.go

+		return 0, fmt.Errorf("subnet includes only the network address")
+	}
+	max--
+	if strings.Contains(subnet.IP.String(), ".") {


this condition looks hacky

jingyuanliang · 2023-04-04T07:36:07Z

pkg/metrics/collector/pod_ip_metrics.go

 	return nil
 }

 func (c *podIPMetricsCollector) Update(ch chan<- prometheus.Metric) error {
-	if !podIPMetricsWatcherSetup {
+	if !podIPMetricsWatcherIsInitialized {


can we extract all these into a separate function?

jingyuanliang · 2023-04-04T07:38:49Z

pkg/metrics/collector/pod_ip_metrics.go

+		nil, nil,
+	)
+	podIPMetricsWatcherIsInitialized = false
+	assignedIPExported               = false


we probably want to put these inside struct podIPMetricsCollector.

jingyuanliang · 2023-04-04T08:28:56Z

I'm now thinking, why are we doing it by parsing the CNI config file from the beginning?

I previously thought, by doing this we can probably handle more different scenarios where the CNI is configured by some different party or in a different layout, but in the current implementation, what it can handle is pretty limited to the CNI config written by install-cni.sh using the default cni_spec_template (which is even not a part of this repo), and it currently just populates with .spec.podCIDR.

Given this, what about repeating what install-cni.sh does here and read .spec.podCIDR here then export the range size? This will happen to work well with calico's config too. By the way, is the control plane currently populating podCIDRs correctly for IPv6 and what about directpath?

MrHohn · 2023-04-04T18:53:04Z

Yining, Jingyuan and myself did a quick sync on this and agreed going with .spec.podCIDR and .spec.podCIDRs seems like a cleaner and more consistent way to go with less dependencies to be added.

Regarding directpath - since it doesn't have an angle into IP allocation it should be fine to put it aside (it won't show up in podCIDR).

MrHohn

Thanks Yining! One minor comment and LGTM.
/approve

MrHohn · 2023-04-26T22:18:48Z

pkg/metrics/collector/pod_ip_metrics.go

+			continue
+		}
+		allCIDRs = append(allCIDRs, podCIDR)
+	}


What about something like:

for _, podCIDR := range node.Spec.PodCIDRs { allCIDRs = append(allCIDRs, podCIDR) } if len(allCIDRs) == 0 { if node.Spec.PodCIDR != "" { allCIDRs = append(allCIDRs, node.Spec.PodCIDR) } }

MrHohn · 2023-04-26T22:21:42Z

/hold

MrHohn

Will keep the hold in case Jingyuan wants to take a final look.
/lgtm

google-oss-prow · 2023-04-26T23:59:21Z

New changes are detected. LGTM label has been removed.

google-oss-prow · 2023-04-26T23:59:30Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: MrHohn, yiningou

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-prow bot added the size/L label Mar 13, 2023

yiningou force-pushed the master branch 2 times, most recently from 03e3033 to 164dde2 Compare March 14, 2023 22:36

google-oss-prow bot assigned jingyuanliang and MrHohn Mar 14, 2023

MrHohn reviewed Mar 15, 2023

View reviewed changes

pkg/metrics/collector/pod_ip_metrics.go Outdated Show resolved Hide resolved

pkg/metrics/collector/pod_ip_metrics.go Outdated Show resolved Hide resolved

google-oss-prow bot added size/XXL and removed size/L labels Mar 20, 2023

yiningou force-pushed the master branch from b2eb3fc to 6658f5a Compare March 20, 2023 20:52

yiningou changed the title ~~Add subnetCount and utilization to pod_ip_metrics.go~~ Add subnetCount to pod_ip_metrics.go Mar 20, 2023

yiningou force-pushed the master branch from 6658f5a to a7ed956 Compare March 21, 2023 07:49

MrHohn reviewed Mar 21, 2023

View reviewed changes

jingyuanliang reviewed Mar 22, 2023

View reviewed changes

yiningou force-pushed the master branch 2 times, most recently from ca3b15d to 212cd27 Compare March 24, 2023 18:10

jingyuanliang reviewed Mar 24, 2023

View reviewed changes

pkg/metrics/collector/pod_ip_metrics.go Outdated Show resolved Hide resolved

pkg/metrics/collector/pod_ip_metrics.go Outdated Show resolved Hide resolved

yiningou force-pushed the master branch 2 times, most recently from 4ba3150 to 6380e00 Compare March 28, 2023 18:25

MrHohn reviewed Mar 29, 2023

View reviewed changes

yiningou force-pushed the master branch 3 times, most recently from d2994e7 to f4de66d Compare April 3, 2023 17:58

MrHohn reviewed Apr 4, 2023

View reviewed changes

jingyuanliang reviewed Apr 4, 2023

View reviewed changes

yiningou force-pushed the master branch 3 times, most recently from f44e8b3 to 74f9e80 Compare April 17, 2023 07:39

yiningou force-pushed the master branch 7 times, most recently from 48a9e2c to aca7a91 Compare April 26, 2023 20:23

MrHohn approved these changes Apr 26, 2023

View reviewed changes

google-oss-prow bot added the lgtm label Apr 26, 2023

google-oss-prow bot added approved do-not-merge/hold labels Apr 26, 2023

yiningou force-pushed the master branch from aca7a91 to a0b45d8 Compare April 26, 2023 23:24

google-oss-prow bot removed the lgtm label Apr 26, 2023

MrHohn approved these changes Apr 26, 2023

View reviewed changes

google-oss-prow bot added the lgtm label Apr 26, 2023

yiningou closed this Apr 26, 2023

yiningou force-pushed the master branch from a0b45d8 to 3f7a922 Compare April 26, 2023 23:59

google-oss-prow bot removed the lgtm label Apr 26, 2023

google-oss-prow bot added size/XS and removed approved size/XXL labels Apr 26, 2023

yiningou mentioned this pull request Apr 27, 2023

Add subnetCount to pod_ip_metrics.go #212

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add subnetCount to pod_ip_metrics.go #205

Add subnetCount to pod_ip_metrics.go #205

yiningou commented Mar 13, 2023

yiningou commented Mar 14, 2023

MrHohn left a comment

MrHohn Mar 21, 2023

jingyuanliang Mar 22, 2023

MrHohn Mar 22, 2023

jingyuanliang Mar 22, 2023

jingyuanliang Mar 22, 2023

MrHohn Mar 29, 2023

jingyuanliang left a comment

jingyuanliang Mar 22, 2023

MrHohn left a comment

MrHohn left a comment

MrHohn Apr 4, 2023

yiningou Apr 4, 2023

MrHohn Apr 4, 2023

jingyuanliang Apr 4, 2023

jingyuanliang Apr 4, 2023

jingyuanliang Apr 4, 2023

jingyuanliang Apr 4, 2023

jingyuanliang Apr 4, 2023

jingyuanliang Apr 4, 2023

jingyuanliang commented Apr 4, 2023

MrHohn commented Apr 4, 2023

MrHohn left a comment

MrHohn Apr 26, 2023

MrHohn commented Apr 26, 2023

MrHohn left a comment

google-oss-prow bot commented Apr 26, 2023

google-oss-prow bot commented Apr 26, 2023

Add subnetCount to pod_ip_metrics.go #205

Add subnetCount to pod_ip_metrics.go #205

Conversation

yiningou commented Mar 13, 2023

yiningou commented Mar 14, 2023

MrHohn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jingyuanliang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MrHohn left a comment

Choose a reason for hiding this comment

MrHohn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jingyuanliang commented Apr 4, 2023

MrHohn commented Apr 4, 2023

MrHohn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MrHohn commented Apr 26, 2023

MrHohn left a comment

Choose a reason for hiding this comment

google-oss-prow bot commented Apr 26, 2023

google-oss-prow bot commented Apr 26, 2023