TLS Mutual Authentication on the backends

[bsctl]

Do you want to request a feature or report a bug?

feature/bug (?)

What did you do?

I'm trying to use Traefik as APIs gateway in front of remote multiple APIs servers. Remote APIs servers require mutual TLS authentication with certificates (tls.crt/tls.key) while clients on the frontend use simply external authentication.

Once the client is authenticated by Traefik, the client's request is proxied to the backends using mutual TLS authentication between Treafik (acting as client) and the the remote APIs server.

This the desired layout:

                                                                            +---------------+
                                                                            | API Server A  |
                                                              HTTPS         |---------------|
                                                              Mutual TLS    | serverA.crt   |
                                                         +----------------->| serverA.key   |
                                                         |                  |               |
                                 front.crt               |                  |               |
                                 front.key               |                  +---------------+
                                                         |
      +----------+               +----------------------+|tlsA.crt
      |Client    |               |  Traefik             ++tlsA.key
      |----------|   HTTPS       |----------------------|
      |          |+------------> |                      |
      |          |   Ext Auth    |                      |
      |          |               |                      ++
      +----------+               +----------------------+|tlsB.crt
                                             +           |tlsB.key          +---------------+
                                             |           |                  | API Server B  |
                                             |           |                  |---------------|
                                             |           |     Mutual TLS   | serverB.crt   |
                                             v           +----------------->| serverB.key   |
                                      +-------------+          HTTPS        |               |
                                      |    Auth     |                       |               |
                                      |-------------|                       +---------------+
                                      | user: joe   |
                                      | user: alice |
                                      +-------------+

Traefik is running as pod in Kubernetes and I'm using external name k8s services (type: ExternalName) and k8s ingresses to configure Traefik.

Please do not confuse the k8s APIs server with the remote APIs servers in the picture above

This is quite straightforward to achieve with NGINX Ingress controller but struggling to implement with Treafik and the option traefik.ingress.kubernetes.io/pass-tls-cert seems useful only to enable mutual TLS between client and remote APIs server.

With NGINX Ingress Controller we have:

---
apiVersion: v1
kind: Service
metadata:
  labels:
  name: caldera
  namespace: cmp-system
spec:
  externalName: caldera.example.com
  ports:
  - name: https
    port: 443
  type: ExternalName

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: caldera
  namespace: cmp-system
  labels:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/backend-protocol: https
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_ssl_certificate         /etc/apigateway/caldera/tls.crt;
      proxy_ssl_certificate_key     /etc/apigateway/caldera/tls.key;
      proxy_ssl_session_reuse       on;
      proxy_set_header              Impersonate-User $remote_user;
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: caldera
          servicePort: 443
        path: /caldera

What did you expect to see?

The use case above implemented with Traefik

What did you see instead?

It looks to me Traefik only configurable for mutual TLS between clients and backends.

Output of traefik version:

1.7.4

What is your environment & configuration (arguments, toml, provider, platform, ...)?

kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: apigateway
  namespace: cmp-system
  labels:
spec:
  selector:
    matchLabels:
      cmp: apigateway
  template:
    metadata:
      labels:
        cmp: apigateway
    spec:
      serviceAccountName: apigateway
      hostNetwork: true
      nodeSelector:
        kubernetes.io/hostname: cmp
      containers:
      - image: traefik:1.7.4
        name: traefik
        resources:
          requests:
            cpu: 100m
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        volumeMounts:
        - name: config
          mountPath: /etc/traefik
          readOnly: true
        securityContext:
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        args:
        - --kubernetes
      volumes:
      - name: config
        hostPath:
          path: /etc/apigateway

traefik.toml

################################################################
# Global configuration
################################################################

# Enable debug mode
# Optional
# Default: false
debug = true

# Log level
# Optional
# Default: "ERROR"
logLevel = "DEBUG"

# Entrypoints to be used by frontends that do not specify any entrypoint.
# Each frontend can specify its own entrypoints.
# Optional
# Default: ["http"]

defaultEntryPoints = ["http", "https"]

[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.auth.forward]
    address = "https://authserver.com/auth"
    removeHeader = true
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      certFile = "/etc/traefik/pki/front.crt"
      keyFile = "/etc/traefik/pki/front.key"
  [entryPoints.traefik]
  address = ":8082"

insecureSkipVerify = true

If applicable, please paste the log output at DEBUG level (--logLevel=DEBUG switch)

[bsctl]

@jbdoumenjou
Hello, thanks for taking care of this issue. I'm available to provide more info and details if this is the case.

-adriano

[bsctl]

I tried to use the following

---
apiVersion: v1
kind: Service
metadata:
  labels:
  name: caldera
  namespace: cmp-system
spec:
  externalName: caldera.example.com
  ports:
  - name: https
    port: 443
  type: ExternalName

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: caldera
  namespace: cmp-system
  labels:
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.frontend.rule.type: PathPrefixStrip
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: caldera
          servicePort: 443
        path: /caldera

Trying to access with browser the frontend, I get external authentication from traefik (as espected) but I get the following error on the remote APIs server

Dec  4 12:04:53 cmp journal: I1204 11:04:53.929983       1 logs.go:49] http: TLS handshake error from 10.10.10.10:53756: remote error: tls: bad certificate
Dec  4 12:04:54 cmp journal: I1204 11:04:54.156778       1 logs.go:49] http: TLS handshake error from 10.10.10.10:53760: remote error: tls: bad certificate

This make sense because the remote APIs server is especting a mutual TLS authentication but I do not know where to put client certificates /etc/apigateway/caldera/tls.crt; /etc/apigateway/caldera/tls.key; in Traefik.

As additional question, which certificates are Traefik going to use when proxying to the backend with the configuration above?

[bsctl]

Hello any feedback an this?

[jbdoumenjou]

Sorry for the delay. I'll try to take a look before the Xmas Holidays.

[bsctl]

Hello any feedback an this?

[jbdoumenjou]

Hi @Kalise,

To have a mtls between Traefik and your backend, you have to define the ClientCAFiles and set the passTLSCert option to true. The mtls will use the certificate defined on your entrypoint.

But, it will try to do mtls with the EntryPoint too, so you need to specify optional=true on the ClientCA option.

Because it will use the IP of the backend, you need to add the containers IPs to the SANs of your certificate.

A fix (#4438) to apply the global InsecureSkipVerify to the passTLSCert mechanism and avoid the SANs configuration will come in the next version.

Note that all the global options like InsecureSkipVerify must be set at the beginning of your configuration file.

If you have any question, I encourage you to join the #support channel on our slack: https://slack.traefik.io/.

I’ll close the issue for now.