Removing disabled maps causes controllers to fail

[aanm]

Removing disabled maps in

cilium/pkg/datapath/maps/map.go

Lines 210 to 215 in cac5218

	for _, m := range maps {
	p := path.Join(bpf.MapPrefixPath(), m)
	if _, err := os.Stat(p); !os.IsNotExist(err) {
	ms.RemoveMapPath(p)
	}
	}

Causes the controllers to fail:

2021-02-26T14:24:45.460845518Z level=debug msg="Controller func execution time: 15.392µs" name=bpf-map-sync-cilium_snat_v4_external subsys=controller uuid=946f5dda-bd86-44d5-aa8d-7d5ee12daaed
2021-02-26T14:24:45.460882480Z level=debug msg="Controller run failed" consecutiveErrors=1 error="Unable to get object /sys/fs/bpf/tc/globals/cilium_snat_v4_external: no such file or directory" name=bpf-map-sync-cilium_snat_v4_external subsys=controller uuid=946f5dda-bd86-44d5-aa8d-7d5ee12daaed

as seen in our CI https://jenkins.cilium.io/job/Cilium-PR-K8s-1.20-kernel-4.9/799/testReport/junit/Suite-k8s-1/20/K8sServicesTest_Checks_service_across_nodes_Checks_ClusterIP_Connectivity/

This change seems to have been introduced by cac5218

cc @brb @pchaigno @mazzy89

[mazzy89]

Thanks @aanm for reporting this. I will take a look and wire a fix accordingly.

[pchaigno]

@brb If cilium_snat_v4_external is only used when BPF NodePort is enabled, should we create the userspace counterparts only when EnableNodePort is true?

cilium/daemon/cmd/datapath.go

Line 402 in a9374b5

ipv4Nat, ipv6Nat := nat.GlobalMaps(option.Config.EnableIPv4,

and

cilium/pkg/maps/ctmap/ctmap.go

Line 168 in 29c7f21

global4Map, global6Map := nat.GlobalMaps(v4, v6)

[mazzy89]

Thanks, @pchaigno for taking care of this. I was looking at it this morning but you were faster than me 😝

[brb]

If cilium_snat_v4_external is only used when BPF NodePort is enabled, should we create the userspace counterparts only when EnableNodePort is true?

@pchaigno Yep.

[tklauser]

Hit this in #15213 test runs, I think: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.20-kernel-4.9/861/

17:28:14 STEP: Cilium is not ready yet: controllers are failing: cilium-agent 'cilium-jznc4': controller bpf-map-sync-cilium_snat_v4_external is failing: Exitcode: 0 
Stdout:
 	 KVStore:                Ok   Disabled
	 Kubernetes:             Ok   1.20 (v1.20.4) [linux/amd64]
	 Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
	 KubeProxyReplacement:   Disabled   
	 Cilium:                 Ok   1.9.90 (v.1.9.90-r.d55bb42)
	 NodeMonitor:            Listening for events on 3 CPUs with 64x4096 of shared memory
	 IPAM:                   IPv4: 4/255 allocated from 10.0.0.0/24, IPv6: 4/255 allocated from fd02::/120
	 BandwidthManager:       Disabled
	 Host Routing:           Legacy
	 Masquerading:           IPTables [IPv4: Enabled, IPv6: Enabled]
	 Controller Status:      29/30 healthy
	   Name                                          Last success   Last error   Count   Message
	   bpf-map-sync-cilium_lxc                       5s ago         never        0       no error                                                                                         
	   bpf-map-sync-cilium_snat_v4_external          never          3s ago       5       Unable to get object /sys/fs/bpf/tc/globals/cilium_snat_v4_external: no such file or directory 
[...]