Support several auth (and rate limit) services via single Gateway

[eshepelyuk]

Slack context https://solo-io.slack.com/archives/C9L6VPAUW/p1579627480092100

@christian-posta

Currently only a single custom auth can be declared in Settings object. It would be great to have possibility to declare several custom auth services to use for different routes, virtual services served by single proxy configured by Settings.

For example scenario can be a dev stage k8s cluster with dozens of "env" handling multiple teams \ customers. In this case a single Gloo proxy can serve them all, based on different virtual hosts.
But for this particular scenario several custom auth server maybe required to isolate customer data, hide users between teams etc.

There's no technical proposal from my site, only feature requirement.

[marcogschmidt]

Linking to #2244 as this change would require supporting multiple filter chains for the same Gateway (i.e. Envoy listener).

We should come up with a generic API design that supports both these use cases and lends itself to be easily extended.

@ilackarms

[eshepelyuk]

One suggestion, though.
It would be great if auth service config part could be extracted from Settings to the dedicated CRD.
This would simplify CI \ CD of multiple auth service by eliminating need to patch \ unpatch Settings on adding \ removal of new auth service instances within the cluster.

[ilackarms]

@marcogschmidt thanks for pinging me on this. let me know when you're ready to discuss

[rickducott]

I interpret the original description as wanting multiple auth servers in the same API gateway, but not necessarily in the same Gateway CRD. Instead, as proposed on the duplicate ticket I just closed, we could support this on the Gateway CRD and have multiple Gateway CRDs if that was easier.

[eshepelyuk]

@rickducott well having multiple several specific CRD for customauth, and fhenreferring to them in Virtual Services / Route Tables - would be the best
Idea of this task is to customize auth service at least at Virtual Service level

[rickducott]

Gotcha, thanks.

[rickducott]

Spoke with another user who +1'ed the route- or virtual service-level config (rather than Gateway / listener level).

[derrickburns]

I have a similar need to have different auth at the virtual service level.

I use one Gateway CR for several Virtual Services. For me, a VirtualService represents a user domain.

I could use multiple ELBs, but that adds cost.

[yuval-k]

the problem is that the ext auth filter config is on the listener / gateway level.
doing anything in the virtual service level will require an upstream envoy change

[eshepelyuk]

ok, but what the problem of modyfying envoy config when new auth added for virtual service ? any technical obstacles ? or the matter is only amount of time to develop this ?
currently, one could disable ext auth for virtual service, or for particular route.
so I assume that config is already modified when one disables ext auth for route, for example.

am I wrong on this assumption ?

[yuval-k]

the part in the config that points ext auth to the extauth server is in the listener / gateway level, and is not exposed in the route level config.
if you specify it on the vhost / route allows for disabling or adding context extensions. note that disabling doesnt actually disable the ext auth filter. the filter stills runs, but acts as a no-op.

for reference, here is the envoy config for the ext auth filter:
https://github.com/envoyproxy/envoy/blob/2240490715d8eca2f1d0ad080628b81cc4aae260/api/envoy/config/filter/http/ext_authz/v2/ext_authz.proto#L199-L210

as you can see, only disabled and context_extensions can be modified on the route level.

[eshepelyuk]

@yuval-k does that mean this issue never will be implemented ? Or what are the next steps ?

[bdecoste]

I suggest this is implemented at the gateway vs the settings level. That would enable the use of gateway.virtualServiceSelector to have one control plane manage 2 completely separate data planes.

[derrickburns]

@rickducott Any plans for this implementing @bdecoste 's suggestion above?

[eshepelyuk]

Any updates ?

[rickducott]

@eshepelyuk @derrickburns

Sorry about the delay, we have been discussing internally. We are going to move ahead with @bdecoste's suggestion above, adding support for configuring these upstream services on the Gateway CRD so it can be customized per listener. From the feedback we have been getting, that resolves some of the use cases. I know there is further desire for route-level control, but this is a challenging upstream Envoy change so we can't commit to that at this time.

[derrickburns]

Awesome!

[eshepelyuk]

@rickducott could you provide an example how config will look like ?

[ashleywang1]

Here's an example of what it could look like:

kind: Gateway
metadata:
  labels:
    app: gloo
  name: gateway-proxy
  namespace: gloo-system
spec:
  bindAddress: '::'
  bindPort: 8080
  httpGateway:
    options:
      extauth:
        extauthzServerRef:
          name: auth-server
          namespace: gloo-system
        httpService:
          request:
            allowedHeaders:
            - X-foo
        requestBody:
          maxRequestBytes: 10240
        requestTimeout: 0.5s
  proxyNames:
  - gateway-proxy
  useProxyProto: false

[eshepelyuk]

@ashleywang1 hmmmmm - and where is the multiple custom auth services ??

[ashleywang1]

@ashleywang1 hmmmmm - and where is the multiple custom auth services ??

Gloo supports multiple gateways, so this approach means that one extauth service can be customized per listener. Here's an example of multiple gateways, each with its own extauth config:

kind: Gateway
metadata:
  labels:
    app: gloo
  name: gateway-proxy
  namespace: gloo-system
spec:
  bindAddress: '::'
  bindPort: 8080
  httpGateway:
    options:
      extauth:
        extauthzServerRef:
          name: auth-server
          namespace: gloo-system
        httpService:
          request:
            allowedHeaders:
            - X-foo
        requestBody:
          maxRequestBytes: 10240
        requestTimeout: 0.5s
  proxyNames:
  - gateway-proxy
  useProxyProto: false
---
kind: Gateway
metadata:
  labels:
    app: gloo
  name: gateway-proxy2
  namespace: gloo-system
spec:
  bindAddress: '::'
  bindPort: 8081
  httpGateway:
    options:
      extauth:
        extauthzServerRef:
          name: auth-server2
          namespace: gloo-system
        requestBody:
          maxRequestBytes: 10240
        requestTimeout: 0.5s
  proxyNames:
  - gateway-proxy-ssl
  useProxyProto: false
---
kind: Gateway
metadata:
  labels:
    app: gloo
  name: gateway-proxy-ssl
  namespace: gloo-system
spec:
  bindAddress: '::'
  bindPort: 8443
  httpGateway:
    options:
      extauth:
        extauthzServerRef:
          name: auth-server3
          namespace: gloo-system
        requestBody:
          maxRequestBytes: 10240
        requestTimeout: 0.5s
  proxyNames:
  - gateway-proxy-ssl
  useProxyProto: false

[derrickburns]

More food for thought.

I my company uses jwts as session tokens. I can almost use gloo to authenticate them. However, the tokens can be revoked before they expire. I need a way to check this.

With the gloo design for auth, I don’t see a way to use the gloo built-in auth and add a simple custom auth to check for revocation.

[rickducott]

@derrickburns this use case makes sense - I'm tracking #2601 for similar cases

[eshepelyuk]

@ashleywang1 may you also provide an example how to setup multiple gateways within a single installation of Gloo, preferrably via helm

[ashleywang1]

@ashleywang1 may you also provide an example how to setup multiple gateways within a single installation of Gloo, preferrably via helm

Here's an example of a helm override file:

gatewayProxies:
  gatewayProxy2:
    configMap:
      data: null
    gatewaySettings:
      disableGeneratedGateways: false
    kind:
      deployment:
        replicas: 1
    podTemplate:
      disableNetBind: false
      floatingUserId: false
      httpPort: 8081
      httpsPort: 8444
      image:
        repository: gloo-envoy-wrapper
        tag: 1.4.0-beta10
    readConfig: false
    service:
      httpPort: 80
      httpsFirst: false
      httpsPort: 443
      type: LoadBalancer

This will set up the following 4 Gateways within a single installation of Gloo open source. To get the same result with Gloo Enterprise, the helm override file needs to be modified with all the values going under the gloo subchart:

# Source: gloo/templates/8-default-gateways.yaml
apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
  name: gateway-proxy
  namespace: gloo-system
  labels:
    app: gloo
spec:
  bindAddress: "::"
  bindPort: 8080
  httpGateway: {}
  useProxyProto: false
  ssl: false
  proxyNames:
  - gateway-proxy
---
# Source: gloo/templates/8-default-gateways.yaml
apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
  name: gateway-proxy-ssl
  namespace: gloo-system
  labels:
    app: gloo
spec:
  bindAddress: "::"
  bindPort: 8443
  httpGateway: {}
  useProxyProto: false
  ssl: true
  proxyNames:
  - gateway-proxy
---
# Source: gloo/templates/8-default-gateways.yaml
apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
  name: gateway-proxy-2
  namespace: gloo-system
  labels:
    app: gloo
spec:
  bindAddress: "::"
  bindPort: 8081
  httpGateway: {}
  useProxyProto: 
  ssl: false
  proxyNames:
  - gateway-proxy-2
---
# Source: gloo/templates/8-default-gateways.yaml
apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
  name: gateway-proxy-2-ssl
  namespace: gloo-system
  labels:
    app: gloo
spec:
  bindAddress: "::"
  bindPort: 8444
  httpGateway: {}
  useProxyProto: 
  ssl: true
  proxyNames:
  - gateway-proxy-2

[eshepelyuk]

@ashleywang1 hope last question - how many provider specific load balancers will be created ?
For instance we use AWS, how many NLB\ELB etc will be created if there are several gloo Gateway objects ?

[ashleywang1]

Based on this, there should be one load balancer per gateway-proxy-id (e.g. gateway-proxy or gateway-proxy-2). Each gateway-proxy instance can support a ssl and non-ssl Gateway type.

So in the example above, 2 LoadBalancers would be created, and each would need the helm override annotation:

gatewayProxies:
  gatewayProxy:
    service:
      extraAnnotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb

to integrate with AWS.

[eshepelyuk]

@ashleywang1 so eventually there should be

1 service gloo
1 service discovery
1 service gatewya
2 services of type LoadBalancer

[ashleywang1]

Yep! There will be 2 Kubernetes LoadBalancer services.

[eshepelyuk]

@ashleywang1 Thank you !