OVN watcher daemon exits with erorr "pipe2: too many open files"

[tong101]

OVN kubernetes watcher daemon exits after a few days it started with error complaining about "pipe2: too many files". It seems we didn't close some files in our code.

I0118 18:50:58.058644 29438 reflector.go:286] github.com/openvswitch/ovn-kubernetes/go-controller/pkg/factory/factory.go:81: forcing resync
INFO[197400] Deleting network policy policy1 in namespace ns1
DEBU[197400] deleteAddressSet a15401878422219876457
ERRO[197400] failed to destroy address set a15401878422219876457 (pipe2: too many open files)
DEBU[197400] deleteAddressSet a15401877322708248246
ERRO[197400] failed to destroy address set a15401877322708248246 (pipe2: too many open files)
ERRO[197400] find failed to get the allow rule for namespace=ns1, policy=policy1 (pipe2: too many open files)
INFO[197400] Adding network policy policy1 in namespace ns1
DEBU[197400] Network policy ingress is {Ports:[{Protocol:0xc42472aff0 Port:80} {Protocol:0xc42472b000 Port:1234}] From:[{PodSelector:nil NamespaceSelector:&LabelSelector{MatchLabels:map[string]string{project: myproject1,},MatchExpressions:[],} IPBlock:nil} {PodSelector:&LabelSelector{MatchLabels:map[string]string{role: frontend1,},MatchExpressions:[],} NamespaceSelector:nil IPBlock:nil}]}
DEBU[197400] createAddressSet with ns1.policy1.ingress.0 and []
ERRO[197400] find failed to get address set (pipe2: too many open files) Selector{MatchLabels:map[string]string{proj
DEBU[197400] Network policy ingress is {Ports:[{Protocol:0xc42472b060 Port:80}] From:[{PodSelector:nil NamespaceSelector:&LabelSelector{MatchLabels:map[string]stelector:nil IPBlock:nil}]} ring{project: myproject2,},MatchExpressions:[],} IPBlock:nil} {PodSelector:&LabelSelector{MatchLabels:map[string]string{role: frontend2,},MatchExpressions:[],} NamespaceSelector:nil IPBlock:nil}]}
DEBU[197400] createAddressSet with ns1.policy1.ingress.1 and [] ring{project: myproject2,},MatchExpressions
ERRO[197400] find failed to get address set (pipe2: too many open files)
I0118 18:50:58.059500 29438 reflector.go:202] Starting reflector *v1.Namespace (0s) from github.com/openvswitch/ovn-kubernetes/go-controller/pkg/ovn/policy.go:706
I0118 18:50:58.059514 29438 reflector.go:240] Listing and watching *v1.Namespace from github.com/openvswitch/ovn-kubernetes/go-controller/pkg/ovn/policy.go:706706
I0118 18:50:58.059574 29438 reflector.go:202] Starting reflector *v1.Pod (0s) from github.com/openvswitch/ovn-kubernetes/go-controller/pkg/ovn/policy.go:514
I0118 18:50:58.059585 29438 reflector.go:240] Listing and watching *v1.Pod from github.com/openvswitch/ovn-kubernetes/go-controller/pkg/ovn/policy.go:514
I0118 18:50:58.059611 29438 reflector.go:202] Starting reflector *v1.Namespace (0s) from github.com/openvswitch/ovn-kubernetes/go-controller/pkg/ovn/policy.go:706706
I0118 18:50:58.059624 29438 reflector.go:240] Listing and watching *v1.Namespace from github.com/openvswitch/ovn-kubernetes/go-controller/pkg/ovn/policy.go:706
I0118 18:50:58.059672 29438 reflector.go:202] Starting reflector *v1.Pod (0s) from github.com/openvswitch/ovn-kubernetes/go-controller/pkg/ovn/policy.go:590
I0118 18:50:58.059684 29438 reflector.go:240] Listing and watching *v1.Pod from github.com/openvswitch/ovn-kubernetes/go-controller/pkg/ovn/policy.go:590
I0118 18:50:58.059819 29438 reflector.go:202] Starting reflector *v1.Pod (0s) from github.com/openvswitch/ovn-kubernetes/go-controller/pkg/ovn/policy.go:590
I0118 18:50:58.059848 29438 reflector.go:240] Listing and watching *v1.Pod from github.com/openvswitch/ovn-kubernetes/go-controller/pkg/ovn/policy.go:590
E0118 18:50:58.060158 29438 reflector.go:205] github.com/openvswitch/ovn-kubernetes/go-controller/pkg/ovn/policy.go:514: Failed to list *v1.Pod: Get http://localhost:8080/api/v1/namespaces/ns1/pods?labealhost:8080/api/v1/namespaces/ns1/pods?labelSelector=name%3Dapache&limit=500&resourceVersion=0: dial tcp [::1]:8080: socket: too many open files
E0118 18:50:58.060158 29438 reflector.go:205] github.com/openvswitch/ovn-kubernetes/go-controller/pkg/ovn/policy.go:514: Failed to list *v1.Pod: Get http://localhost:8080/api/v1/namespaces/ns1/pods?labealhost:8080/api/v1/namespaces/ns1/pods?labelSelector=name%3Dapache&limit=500&resourceVersion=0: dial tcp [::1]:8080: socket: too many open files
log: exiting because of error: log: cannot create log: open /tmp/ovnkube.ovn-master.root.log.ERROR.20180118-185058.29438: too many open files

[shettyg]

I think this is likely solved now because we no longer create multiple connections for network policies.