document how to run multiple Contours per cluster (e.g. internal and external)

[so0k]

Objective

I need to configure internal and external ingress controllers on AWS with NLB

Internal Ingress is served by an internal NLB (only accessible from within the VPC)
External Ingress is served by an external NLB (publicly accessible)

What steps did you take and what happened:

I'm using deployment/ds-hostnet-split as a guideline.

full final manifest

• I'm using the same rbac / crb / sa for both contour deployments
• I have 2 Contour deployments:
  • name & label:contour-external for deploy / svc (clusterIP) manifest
    the arguments:

    - serve
    - --incluster
    - --xds-address
    - 0.0.0.0
    - --xds-port
    - $(CONTOUR_EXTERNAL_SERVICE_PORT)
    - --envoy-http-port
    - "80"   #<<<<<
    - --envoy-https-port
    - "443" #<<<<<
    - --ingress-class-name=envoy-external  
    

  • name & label: contour-internal for deploy / svc (clusterIP) manifest
    the arguments:

    - serve
    - --incluster
    - --xds-address
    - 0.0.0.0
    - --xds-port
    - $(CONTOUR_INTERNAL_SERVICE_PORT)
    - --envoy-http-port
    - "8080" #<<<<<
    - --envoy-https-port
    - "8443" #<<<<<
    - --ingress-class-name=envoy-internal  
    

• I have 2 Envoy daemonsets with hostNetwork: true:
  • name & label: envoy-external for ds / svc (LoadBalancer:NLB) manifest
    environment:

    - name: NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    

    args:

    - -c
    - /config/contour.yaml
    - --service-cluster
    - envoy-external-cluster
    - --service-node
    - $(NODE_NAME)
    

    bootstrap:

    - --xds-address
    - $(CONTOUR_EXTERNAL_SERVICE_HOST)
    - --xds-port
    - $(CONTOUR_EXTERNAL_SERVICE_PORT)
    - --stats-port
    - "8002"
    - --admin-port
    - "9001"
    

    HealthCheck uses 8002
    preStop uses 9001
  • name & label: envoy-internal for ds / svc (LoadBalancer:NLB) manifest
    environment is the same as above, using fieldRef for spec.NodeName
    args:

    - -c
    - /config/contour.yaml
    - --service-cluster
    - envoy-internal-cluster
    - --service-node
    - $(NODE_NAME)
    

    bootstrap:

    - --xds-address
    - $(CONTOUR_INTERNAL_SERVICE_HOST)
    - --xds-port
    - $(CONTOUR_INTERNAL_SERVICE_PORT)
    - --stats-port
    - "8004"
    - --admin-port
    - "9002"
    

    HealthCheck uses 8004
    preStop uses 9002

What did you expect to happen:

I expect both envoy ds to run side by side, but they exit(1) without any logs, I verified that they do not use the same hostPorts and I reviewed and validated the configuration created by contour bootstrap..
If I delete one ds the other starts without issue and vice versa.

SSH to the node and running a docker container without host network with the same envoy allows both containers to run side by side.

Anything else you would like to add:

here is a gist for full manifests:

https://gist.github.com/so0k/2cb93c405cf05fcb8ecb73fa665efbd0

Environment:

• Contour version: 0.8.1
• Kubernetes version: 1.11.6
• Kubernetes installer & version: kops 1.11
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): CoreOS

[so0k]

without using hostNetwork - this works: new-manifest - but the clientIP is not preserved

[jbeda]

Ok -- here are the ports I see in the original manifest

• service/envoy-external
  • 80->80
  • 443->443
• service/contour-external - 8001->8001
• deployment/contour-external (hostNetwork: false)
  • xds - CONTOUR_EXTERNAL_SERVICE_PORT = 8001
  • envoy-http - 80
  • envoy-https - 443
  • debug - 8000 (only specified in ports section, not on command line)
• daemonset/envoy-external (hostNetwork: true)
  • http: 80
  • https: 443
  • stats: 8002
  • admin: 9001
• service/envoy-internal
  • 80->8080
  • 443->8443
• service/contur-internal
  • 8001->8001
• deployment/contour-internal (hostNetwork: false)
  • xds: CONTOUR_INTERNAL_SERVICE_PORT = 8001
  • envoy http: 8080
  • envoy https: 8443
  • debug - 8000 (only specified in ports section, not on command line)
• daemonset/envoy-internal (hostNetwork: true)
  • http: 8080
  • https: 8443
  • stats: 8004
  • admin: 9002

From first blush, it looks like there are no overlapping ports on the things that are running as host networking and other ports look to be wired up fine.

@davecheney -- do you know if envoy is opening any other ports that aren't specified? Is there a way to make it dump more verbose logging to debug why it is exiting?

@so0k -- might be useful to ssh into the node and use lsof to find what ports the envoy is listening on to see if there is anything unexpected. I don't know the syntax off the top of my head but it looks like some quick googling leads to some sane guides.

[so0k]

running netstat -nltp and playing around the ds to see what ports envoy-external vs envoy-internal each open on the host did not reveal any overlapping ports.

I will try this again tomorrow with a fresh mind as I spent quite a lot of time breaking my head on this already

[stevesloka]

Hey @so0k can you try adding this argument to one of the Envoy daemonset manifests?

- --base-id
- "2"

ref: https://www.envoyproxy.io/docs/envoy/v1.8.0/operations/cli.html#cmdoption-base-id

[so0k]

great find! Adding base-id: new gist - didn't have time to test it yet

[anapsix]

@stevesloka / @so0k, confirmed <3!
Works as advertised (though still having problems with proxy-proto described in #793)

[davecheney]

@anapsix can you please raise a new issue to talk about your problems using proxy-protocol. Thank you.

@so0k what is required to close this issue?

[so0k]

@davecheney - should I do a PR with documentation about adding --base-id flag to envoy if running multiple ds in hostNetwork ?

[davecheney]

Documentation issue, I’m assigning this to beta.1. @so0k feel free to send a docs PR before then.

[davecheney]

@so0k ping

[jpeach]

Here's a working configuration that uses kustomize to deploy internal and external Contours:

https://github.com/jpeach/kustomize/tree/master/knative/configurations/contour

[so0k]

Sorry, I moved company and I'm afraid I won't be able to finish this documentation task until I have a use case to switch ingress over to Contour

[davecheney]

That’s ok, all the best with your new endeavour. On 17 Feb 2020, at 14:32, so0k <notifications@github.com> wrote:  Sorry, I moved company and I'm afraid I won't be able to finish this documentation task — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fprojectcontour%2Fcontour%2Fissues%2F855%3Femail_source%3Dnotifications%26email_token%3DAAABYA3UCMFK4R3EK7RTSJDRDIANTA5CNFSM4GRUCJK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL46PIY%23issuecomment-586803107&data=02%7C01%7Ccheneyd%40vmware.com%7Cb7aed1f560834a08fddc08d7b35a0b80%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637175071633191550&sdata=adKr222vy0TXpR6e7lQA%2FcNtIqkHUMk7b5Ltg9hJA6Q%3D&reserved=0>, or unsubscribe<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAABYAZ3AI774KCZDOAAN5LRDIANTANCNFSM4GRUCJKQ&data=02%7C01%7Ccheneyd%40vmware.com%7Cb7aed1f560834a08fddc08d7b35a0b80%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637175071633191550&sdata=5p%2BKFFkWVeCA0bBezq5FSBWaWh4%2BEIBm6xztrv9gPM8%3D&reserved=0>.

[fernandodeperto]

Hi, guys,

I'm having the a similar issue and I haven't been able to solve it with the suggestions in this thread. Basically, I need to deploy two instances of Contour on the same cluster.

What is happening is that the first one that gets deployed works, Ingresses created with its class get the address assigned. After the second instance is deployed, ingresses deployed with its class do not get the address assigned.

I couldn't find anything weird or wrong on either of the Contour or Envoy's logs. These are the Helm chart version and values I'm using:

NAME               CHART             VERSION
contour-external   bitnami/contour     4.3.5
contour-internal   bitnami/contour     4.3.5

Values for contour-internal

ingressClass: contour-internal
envoy:
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"

Values for contour-external

ingressClass: contour-external
contour:
  manageCRDs: false
envoy:
  extraArgs:
    - --base-id 2
  hostPorts:
    http: 8080
    https: 8443

I would appreciate any suggestions you might have. I didn't go through the sections like when an issue is created since it's just an answer to the ticket, but I can open a new one if that's preferred.

[youngnick]

Hi @fernandodeperto, are both Envoy daemonsets starting up okay? I can't remember if it's this way in the Helm chart or not, but in our example YAMLs, the Envoy daemonset uses a host port, which, if it's being defaulted to something, would mean that the second Envoy deployment wouldn't start, which might produce this outcome?

[fernandodeperto]

Hi, @youngnick, thank you for the response. Indeed that is one of the things that I've checked, and both the Contour Deployment and the Envoy DaemonSet are coming up normally.

I can actually report that I've managed to fix the issue by deploying the two Contour instances on different namespaces. I couldn't find any differences on the logs between the two setups, but somehow in separate namespaces they work, meaning they both manage to assign ELB addresses to Ingress resources.

[youngnick]

Oh, yes, we should really say that you should only do the two instances in separate namespaces. If the two deployments were in the same namespace, there are a few things that could conflict, like the configuration configmap and the leader election.

For other people finding this thread: Contour strongly recommends that, for multiple installations of Contour, they are in separate namespaces.

[EugenMayer]

Just if anybody is interested, using the bitnam helm chart, deploying a internal and external contour would look like this

contour:
  ingressClass:
    name: contour-${name}
    create: true
    default: ${isDefaultIngress}
  manageCRDs: ${manageCRDs}
envoy:
  service:
    externalIPs:
      - ${hostIP}
    ports:
      http: 80
      https: 443
  extraArgs:
    - --base-id ${id}
  hostPorts:
    http: ${hostPortHttp}
    https: ${hostPortHttps}

• name: internal/external
• isDefaultIngress: should be true for only one (notice [bitnami/contour] cannot make ingress non-default bitnami/charts#8450)
• hostIp: should be one of the IPs of the control plain to bind on (if internal and external are routed via different ips)
• be sure to use different ports for hostPortHttp / hostPortHttps, even if you bind those on different externalIPs
• be sure to change the id to a unique value for each deployment, e.g. 1 for internal, 2 for external
• of course, every deployment of the chart must go into a different namespaces

[youngnick]

@OrlinVasilev I think this one is a very good candidate for some updated docs - @EugenMayer has got some great tips here that could cover the Helm usage, but I think a page that covered "how to run multiple Contour instances in a cluster" would help a lot of people.

[OrlinVasilev]

@youngnick - so new Guide or to include it somewhere into the documentation ?
@EugenMayer - do you need help in making this info into blog or guide ? :)

[youngnick]

I think a new Guide is probably best. I'd like to see the Guides section renamed to "cookbook" or something later, but a Guide will get this info in there, we are getting more requests for this.

I think that @EugenMayer was contributing notes, not offering to write the full thing, happy to be proved wrong though.

[EugenMayer]

I would rather handing over to you to put it into a format of your linking - just take it as it were yours and put it where people can use it :)

[skriss]

I'll try to get this documented soon, we're asked fairly regularly about this. Will cover deploying into separate namespaces (easier) and the same namespace (harder, but still possible).