nginx Ingress controller started seeing internal IPs; bad azure-ip-masq-agent update?

[roy-work]

What happened: at 1:48a on Saturday, our cluster appears to have seen some sort of update to azure-ip-masq-agent: both the CM for the agent and the DS were listed as being (re-)created at that time.

Also at that time, nginx-ingress-controller started reporting that the client's IP of incoming HTTP requests as being within the 10.0.0.0/8 subnet / within the pod subnet. (It reported IPs that were node IPs on requests that clearly did not originate on nodes. All of the node IPs we noted in the logs also were from nodes on which nginx-ingress-controller was not running on, which I think might be relevant?)

We looked at some other (unfortunately, different version of k8s) clusters, and they all had this for a azure-ip-masq-agent-config:

nonMasqueradeCIDRs:
  - <stuff>

However, on the problematic cluster, the config was,

data:
  ip-masq-agent: |-
    nonMasqueradeCIDRs:
    masqLinkLocal: true
    resyncInterval: 60s

Lacking any other ideas, we added the pod subnet CIDR to nonMasqueradeCIDRs, & restarted the pods in the azure-ip-masq-agent DS. This appears to have corrected the issue.

Why? AIUI, ip-masq-agent is to re-write the IP addresses of Internet-bound traffic to the node IPs. But traffic from LB to the ingress controller isn't outbound / Internet bound, so I'm at a loss as to why editing that CM had any effect at all. However, much understanding of both kube-proxy & ip-masq-agent is pretty rudimentary.

What you expected to happen: nginx to get the right (external) IP for incoming requests

How to reproduce it (as minimally and precisely as possible): We're not sure.

Anything else we need to know?: We use nginx-ingress in what we think is a bog standard config: there is an Azure LoadBalancer, behind that, nginx-ingress, and behind that, our services.

Environment:

• Kubernetes version (use kubectl version): 1.13.x
• Size of cluster (how many worker nodes are in the cluster?) 7
• General description of workloads in the cluster (e.g. HTTP microservices, Java app, Ruby on Rails, machine learning, etc.)
• Others:

[ghost]

Hi roy-work, AKS bot here 👋
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such:

1. If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster.
2. Please abide by the AKS repo Guidelines and Code of Conduct.
3. If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics?
4. Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS.
5. Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.
6. If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

[ghost]

Triage required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[miwithro]

@roy-work what AKS Version are you currently running, and is the issue still there?

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[ghost]

This issue will now be closed because it hasn't had any activity for 15 days after stale. roy-work feel free to comment again on the next 7 days to reopen or open a new issue after that time if you still have a question/issue or suggestion.