Skip to content

Azure-NPM intermittent issues with namespaceSelector: {} #865

@vakalapa

Description

@vakalapa

What happened:
While using namespaceSelector: {} in network policies, valid pod traffic is getting timed out or blocked.
This problem can happen in two cases:

  1. When NPM is restarted.
  2. When a Pod is created before its namespaces is created.

How to Mitigate:
Instead of using namespaceSelector: {}, namespaces can be labelled and that label of the namespace can be used. For ex:
Use this below command to apply label to all NameSpaces:

kubectl label ns --all all=namespaces

then, replace the "{}" selector with above mentioned label

  egress:
    - to:
        - namespaceSelector: {}
------------------- Replace as below ------------
  egress:
    - to:
        - namespaceSelector: 
             matchLabels:
                all:namespaces

Problem Description
Azure NPM uses iptables and ipsets underneath to decide on packet connectivity. Iptable rules reference ipsets to identify which IPs to be allowed or blocked.
In NPM, an ipset is created for "all-namespaces" and an ipset is created for each NameSpace added to the cluster. "all-namespaces" ipset has other namespace ipsets as members. When a network policy uses namespaceSelector: {} selector, NPM will translate this into using "all-namespaces" ipset in iptable rules, that way all Pod IPs in All-namespaces will be accounted in taking a decision.
Due to a race, when Pod is create event is received before the NS create event is received, NPM is missing to add the Pod's NS in
"all-namespaces". This results in netowkrPolicies with namespaceSelector: {} to not include the pod Ips in this missed NS.

Kubernetes Version:
All

Azure NPM Version:
v1.3.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions