Support for Ingress

[hobti01]

Is this a bug report or a feature request?:
Question

What happened:
I created an Ingress on a cluster with metallb, it was not discoverable as I expected.

What you expected to happen:
I expected/hoped that magic would happen and the loadbalancer hostname would be updated within

apiVersion: extensions/v1beta1
kind: Ingress
...
status:
  loadBalancer:
    ingress:
      - hostname: <looking for a hostname here>

I'm not sure that metallb accommodates Ingresses. If that's the case I'd appreciate any suggestions how to reuse my Ingress configurations (should I simply deploy an ingress controller?)

How to reproduce it (as minimally and precisely as possible):
Create an Ingress?

Anything else we need to know?:
This is my first experience with metallb, please be gentle. However, I've deployed many times on cloud providers.

Environment:

• MetalLB version: {"branch":"HEAD","caller":"main.go:139","commit":"bd678827","msg":"MetalLB controller starting version 0.6.2 (commit bd678827, branch HEAD)","ts":"2018-05-09T08:04:39.34335782Z","version":"0.6.2"}
• Kubernetes version: Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
• BGP router type/version: Sorry, do not know this
• OS (e.g. from /etc/os-release): Not sure of this either, I don't control the hardware
• Kernel (e.g. uname -a):

[eriklindahl]

I too am quite interested in this, as yet another MetalLB-newbie :-)

I just got the plugin itself working well in layer2 mode with kube-router (latest git versions of both), and it works great to expose either internal IPs or external IPs we have on a separate VLAN. Awesome work!

However, the piece-de-resistance would be if I could get name-based virtual hosts working, so traffic for different websites could share a single external IPv4. This far I haven't had any luck with combinations of MetalLB and nginx-ingress, but that's probably more related to PEBKAC.

It seems like something that should be reasonably straightforward to achieve in MetalLB, but again - I'm no expert.

[aledbf]

This far I haven't had any luck with combinations of MetalLB and nginx-ingress, but that's probably more related to PEBKAC.

Please open an issue in https://github.com/kubernetes/ingress-nginx with the details

[danderson]

So, ingress... The basic answer is that no, MetalLB will not support ingress, but instead aims to be a composable pieces that you can use in combination with any self-hosted ingress controller to achieve your goals.

Now for the more detailed answer...

The status.loadBalancer.ingress field in Ingress objects is really designed for cloud platforms. Specifically, platforms where HTTP load-balancers are implemented significantly differently from TCP/UDP load-balancers. In those environments, the platform has to be able to tell you "Okay, I provisioned an HTTP load-balancer for you, and it's available at this IP/hostname."

On bare metal environments, you typically run a "self-hosted" ingress controller like ingress-nginx or Heptio Contour, where the HTTP LB runs inside your cluster. In that scenario, it's up to the ingress controller to decide how it wants to make your balancer available outside the cluster. And, importantly, the way they decide to make the LB available influences what they might (or might not) place in the status.loadBalancer.ingress field of the ingress object.

MetalLB cannot predict the right thing to do for those ingress objects, unless it becomes a full ingress controller that competes with all the other ingress controllers out there. And I don't have the time or desire to do that. Building a good HTTP load-balancer is a lot of work, and better people than me are already doing a great job at it.

In my mind, the answer to this problem is actually the opposite: ingress controllers should leverage Service objects when they need TCP/UDP connectivity with the world. That is the correct Kubernetes object to express "please get TCP/UDP traffic from the World to this set of pods."

If ingress controllers, when they're implementing the user's desires, reach a point where they go "oh, I need to get traffic to myself so I can forward it as the user requested", they should provision a Service with type=LoadBalancer, wait for it to go live, and document in the ingress object's status.loadBalancer.ingress which IP got provisioned.

AFAICT, the answer most ingress controllers have today is more like "getting traffic to me is your problem, but I'll handle it from there." I believe this is mainly because they cannot rely on Service type=LoadBalancer working in bare metal clusters, and in the absence of that functionality, they really don't have any great solution. So they just punt on the problem and say "run it with hostPort, or NodePort, or whatever, your problem."

So, if I could wish for things, I would wish for an option in ingress controllers, something like --cluster-supports-loadbalancers=true, which tells the ingress controller "hey, my cluster has working LoadBalancers. When you're managing ingress objects, please provision LoadBalancer Services as needed, in addition to what you're already doing." That would work universally regardless of the specific LoadBalancer implementation (MetalLB, GCP LB, ...), and would mean that MetalLB and ingress are composable things that people can combine in whatever way makes the most sense to them.

Does that make sense at all? I admit I'm not yet a heavy user of k8s Ingress, so I may be missing a crucial insight that makes everything I've said incorrect. I'd love to be corrected! :)

[aledbf]

@danderson most of the ingress controllers available support Service type=LoadBalancer but by itself that does not work in baremetal. Like you said, "getting traffic to me is your problem, but I'll handle it from there." is a feature provided by k8s, not the ingress controller (even in gce/gke with the gce ingress controller)

MetalLB cannot predict the right thing to do for those ingress objects, unless it becomes a full ingress controller that competes with all the other ingress controllers out there

Please don't do that and continue improving this amazing project

In my mind, the answer to this problem is actually the opposite: ingress controllers should leverage Service objects when they need TCP/UDP connectivity with the world.

Exactly

[hobti01]

Thank you @danderson for the insight to metallb.

On bare metal environments, you typically run a "self-hosted" ingress controller like ingress-nginx or Heptio Contour, where the HTTP LB runs inside your cluster.

This statement presumes a particular network architecture. I have not yet encountered a Kubernetes on-premises infrastructure where an organization has eliminated their traditional, external load balancers. OpenShift recognizes this with the F5 Router/BIG-IP Controller.

ingress controllers should leverage Service objects when they need TCP/UDP connectivity with the world

They do. The nginx-ingress svc is exposed on the nodePorts of the cluster, where an external load balancer routes traffic.

To be more specific about my environment: I'm using external-dns and it uses the status of the ingress to determine the destination for the DNS A record. Is there an appropriate value that could be used from the metallb status or configuration?

Perhaps an external-dns enhancement to inspect the metallb configuration would be a solution.

[danderson]

To be more specific about my environment: I'm using external-dns and it uses the status of the ingress to determine the destination for the DNS A record. Is there an appropriate value that could be used from the metallb status or configuration?

Perhaps an external-dns enhancement to inspect the metallb configuration would be a solution.

So, your architecture is that you have a big iron load-balancer forwarding traffic to an ingress controller listening on NodePort. Is that right?

In that setup, there is nothing MetalLB can do to help you. The external balancer is in full control of the IP layer routing upstream of it, so within the cluster I have no way of knowing what IPs are available/configured.

In this setup, you would need to do one of two things:

• Write an F5/whatever k8s controller that can talk to the balancer and update your Ingress objects based on the balancer's configuration. Effectively, you would be implementing an "F5 LoadBalancer" implementation. In theory, MetalLB could implement that, but that has significant problems of its own (I don't have the funds or time to implement support for proprietary LB boxes, and I have no idea how I would test that integration in the future).
• Somehow tell external-dns manually what IPs it should use for DNS records, based on your own knowledge of the external setup.

I may be missing something about how your external balancers work, in which case maybe there's something we can do to help... But based on what you've described, there's really nothing MetalLB can do since all the IP layer configuration exists outside of the cluster and evolves independently of anything inside the cluster. Do I have that right?

[hobti01]

So, your architecture is that you have a big iron load-balancer forwarding traffic to an ingress controller listening on NodePort. Is that right?

There is a big iron load-balancer forwarding traffic to MetalLB and the ingress controller service is exposed by MetalLB.

Write an F5/whatever k8s controller...
My problem is simpler in that I do not want to interact with the big iron LB at all. I just want to put the MetalLB IP(s) into DNS with a hostname. Perhaps someday the big iron LB will be dynamically configured, but that is not today.

The test environment I'm using has a single node, so I only see one IP address within Service.status.loadBalancer.ingress.ip. Does MetalLB populate the ingress array with entries?

status:
  loadBalancer:
    ingress:
    - ip: 1.2.3.4

...In theory, MetalLB could implement that...

I didn't mean to suggest that this is something MetalLB should do.

I sincerely appreciate the time you're taking to answer my questions while I fit MetalLB into my Kubernetes world view.

[hobti01]

@danderson Thank you for all of your insights to metallb. I've determined that metallb sets enough information to satisfy external-dns for my use case.

[nlamirault]

@hobti01 hi,
i use MetalLB to assign an external IP for my services. Do you use External-DNS to exposes theses IP according to the Ingress ?

[hobti01]

@nlamirault Yes, we use external-dns

To be more specific about my environment: I'm using external-dns and it uses the status of the ingress to determine the destination for the DNS A record.

[dimatha]

Hello @hobti01, would mind sharing how did you manage to integrate external-dns +ingress + metalLB?

[hobti01]

Hi @dimatha,
We use are using nginx-ingress with external-dns on a few clusters where metallb is used but not controlled by us.

We deploy an Ingress with our application (status would be empty for the deployment):

• nginx-ingress sets the Ingress status to the external IP of the Service
• external-dns uses the Ingress status to determine target hostname or IP address for the DNS record it is creating

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-app
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: my-app
          servicePort: http
        path: /
  tls:
  - hosts:
    - example.com
status:
  loadBalancer:
    ingress:
    # Updated by nginx-ingress
    # Used by external-dns
    - ip: 1.2.3.4

nginx-ingress is configured with Service type: LoadBalancer and publishService.enabled: true

• metallb assigns an IP address and sets the Service status
• nginx-ingress uses the Service status to update the Ingress status

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
spec:
  clusterIP: 192.168.131.78
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    nodePort: 32209
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    nodePort: 31948
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app: nginx-ingress
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    # Updated by metallb
    # Used by nginx-ingress
    - ip: 1.2.3.4