[Bug] Traefik annotations don't work for any paths (except root for old Traefik versions) #30

aleks-fofanov · 2020-06-18T11:54:39Z

Expected Behaviour

All traffic destined to a domain correctly forwarded to a function regardless of the path requested.

Current Behaviour

Depending on installed version of the Traefik, you may observe different behaviors.

If your Traefik version is below v1.7.21, then your function will receive incorrect (duplicated) path, e.g. nodeinfo.example.com/version will be forwarded to gateway/function/nodeinfo/versionversion and your function will get /versionversion as the path. All requests to root path (/) will be forwarded correctly to a function.

If you have Traefik v.1.7.21 or above then you'll get infinite redirect in attempt to request any path (even root)!

The behaviors are different because of this issue that has been addressed in release v1.7.21 of Traefik

Possible Solution

Traefik's rewrite target annotation should be changed from
traefik.ingress.kubernetes.io/rewrite-target: /function/FUNC_NAME/$1
to
traefik.ingress.kubernetes.io/rewrite-target: /function/FUNC_NAME
where FUNC_NAME is the name of the function.
I've created PR which fixes this issue.
(Nice to have) Document (where?) that Traefik v1.7.21 or above is required

Steps to Reproduce (for bugs)

Disclaimer. I tested all of the following on linux, so not really sure that this would work on Mac or Windows.

Create k8s cluster locally

minikube start

Install traefik as ingress controller. For demonstration purposes, I first install v1.7.20

helm upgrade -i traefik \
--version 1.87.1 \
--namespace kube-system \
--set imageTag=1.7.20 \
--set rbac.enabled=true \
--set dashboard.enabled=true \
--set dashboard.serviceType=NodePort \
--set dashboard.domain=traefik.local.dev \
--set dashboard.ingress.annotations."kubernetes\.io\/ingress\.class=traefik" \
--set deployment.hostPort.httpEnabled=true \
--set deployment.hostPort.dashboardEnabled=true \
--set serviceType=NodePort \
--set containerSecurityContext.capabilities.drop[0]=ALL \
--set containerSecurityContext.capabilities.add[0]=NET_BIND_SERVICE \
--set deploymentStrategy.type=Recreate \
stable/traefik

Install openfaas with ingress operator enabled

kubectl create ns dev-openfaas
kubectl create ns dev-openfaas-fn
helm upgrade -i openfaas \
--version 5.8.5 \
--namespace dev-openfaas \
--set functionNamespace=dev-openfaas-fn \
--set basic_auth=false \
--set generateBasicAuth=false \
--set ingressOperator.create=true \
--set ingress.enabled=true \
--set ingress.hosts[0].host=gateway.local.dev \
--set ingress.hosts[0].serviceName=gateway \
--set ingress.hosts[0].servicePort=8080 \
--set ingress.hosts[0].path=/ \
--set ingress.annotations."kubernetes\.io\/ingress\.class=traefik" \
openfaas/openfaas

(Optional) Add Traefik dashboard and OpenFaaS Gateway UI to your hosts if you want to access them via their domain names

echo "$(minikube ip) traefik.local.dev" | sudo tee -a /etc/hosts
echo "$(minikube ip) gateway.local.dev" | sudo tee -a /etc/hosts

or alternatively you can get their services urls to access their UIs

minikube -n kube-system service traefik-dashboard --url
minikube -n dev-openfaas service gateway-external --url

Deploy function

faas-cli deploy \
--image stefanprodan/podinfo:3.1.0 \
--namespace dev-openfaas-fn \
--name podinfo \
--annotation com.openfaas.health.http.initialDelay="2s" \
--annotation com.openfaas.health.http.path="/healthz" \
--env PODINFO_PORT="8080" \
--gateway $(minikube -n dev-openfaas service gateway-external --url) \
--tls-no-verify

Deploy FunctionIngress

cat <<EOF | kubectl apply -f -
apiVersion: openfaas.com/v1alpha2
kind: FunctionIngress
metadata:
  name: podinfo
  namespace: dev-openfaas
spec:
  function: podinfo
  domain: podinfo.local.dev
  ingressType: "traefik"
  bypassGateway: false
  tls:
    enabled: false
EOF

Now the interesting part begins

Let's see what routing rule we have in Traefik for the ingress the operator created

curl -s $(minikube -n kube-system service traefik-dashboard --url)/api/providers/kubernetes/frontends | jq -r '.["podinfo.local.dev/"].routes'

Output:

{
  "/": {
    "rule": "PathPrefix:/;ReplacePathRegex: ^/(.*) /function/podinfo/$1$1"
  },
  "podinfo.local.dev": {
    "rule": "Host:podinfo.local.dev"
  }
}

Note the two $1 at the end of the string with the rule. This is why our function gets duplicated path with this Traefik version.

Add function domain to your hosts and let's see how the routing works

echo "$(minikube ip) podinfo.local.dev" | sudo tee -a /etc/hosts
curl -s http://podinfo.local.dev

Output:

{
  "hostname": "podinfo-5569bb569c-9wvdw",
  "version": "3.1.0",
  "revision": "7b6f11780ab1ce8c7399da32ec6966215b8e43aa",
  "color": "cyan",
  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
  "message": "greetings from podinfo v3.1.0",
  "goos": "linux",
  "goarch": "amd64",
  "runtime": "go1.13.1",
  "num_goroutine": "6",
  "num_cpu": "2"
}

And let's also request the other path

curl -s http://podinfo.local.dev/version

Output:

404 page not found

Function actually gets /versionversion as the path.
Let's also make sure that this path is handled by requesting it through gateway:

curl -s http://gateway.local.dev/function/podinfo/version

or

curl -s $(minikube -n dev-openfaas service gateway-external --url)/function/podinfo/version

Output:

{
  "commit": "7b6f11780ab1ce8c7399da32ec6966215b8e43aa",
  "version": "3.1.0"
}

Now let's change the annotation on the ingress to the proposed and see if it works with this version of Traefik

kubectl -n dev-openfaas annotate --overwrite ing podinfo 'traefik.ingress.kubernetes.io/rewrite-target=/function/podinfo'
curl -s http://podinfo.local.dev/version

Output:

error finding function podinfoenv.dev-openfaas-fn: server returned non-200 status code (404) for function, podinfoenv

Let's rollback changes to the ingress and upgrade traefik to v1.7.23 to see how things work with this version

kubectl -n dev-openfaas annotate --overwrite ing podinfo 'traefik.ingress.kubernetes.io/rewrite-target=/function/podinfo/$1'
helm upgrade traefik \
--namespace kube-system \
--reuse-values \
--set imageTag=1.7.23 \
stable/traefik
kubectl -n kube-system rollout status deploy/traefik

Let's request the same paths and see what we get (spoiler - infinite redirects). I intentionally limited redirects count to keep the output at acceptable level.

curl -sLv --max-redirs 1 http://podinfo.local.dev

Output:

* Rebuilt URL to: http://podinfo.local.dev/                                                                                                                                                                                                                                                                                                                          
*   Trying 192.168.39.39...                                                                                                                                                                                        
* TCP_NODELAY set                                                                                                                                                                                                  
* Connected to podinfo.local.dev (192.168.39.39) port 80 (#0)                                                                                                                                                      
> GET / HTTP/1.1                                                                                                                                                                                                   
> Host: podinfo.local.dev                                                                                                                                                                                          
> User-Agent: curl/7.58.0                                                                                                                                                                                          
> Accept: */*                                                                                                                                                                                                      
>                                                                                                                                                                                                                  
< HTTP/1.1 301 Moved Permanently                                                                                                                                                                                   
< Content-Length: 0
< Date: Thu, 18 Jun 2020 11:18:59 GMT
< Location: /function/podinfo/
< Vary: Accept-Encoding
< 
* Connection #0 to host podinfo.local.dev left intact
* Issue another request to this URL: 'http://podinfo.local.dev/function/podinfo/'
* Found bundle for host podinfo.local.dev: 0x5566c09a19d0 [can pipeline]
* Re-using existing connection! (#0) with host podinfo.local.dev
* Connected to podinfo.local.dev (192.168.39.39) port 80 (#0)
> GET /function/podinfo/ HTTP/1.1
> Host: podinfo.local.dev
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Content-Length: 0
< Date: Thu, 18 Jun 2020 11:18:59 GMT
< Location: /function/podinfo/function/podinfo/function/podinfo/
< Vary: Accept-Encoding
< 
* Connection #0 to host podinfo.local.dev left intact
* Maximum (1) redirects followed

curl -sLv --max-redirs 1 http://podinfo.local.dev/version

Output:

*   Trying 192.168.39.39...                                                                                                                                                                                        
* TCP_NODELAY set                                                                                                                                                                                                  
* Connected to podinfo.local.dev (192.168.39.39) port 80 (#0)                                                                                                                                                      
> GET /version HTTP/1.1                                                                                                                                                                                            
> Host: podinfo.local.dev                                                                                                                                                                                          
> User-Agent: curl/7.58.0                                                                                                                                                                                          
> Accept: */*                                                                                                                                                                                                      
>                                                                                                                                                                                                                  
< HTTP/1.1 301 Moved Permanently
< Date: Thu, 18 Jun 2020 11:22:08 GMT
< Location: /function/podinfo/version/version
< Vary: Accept-Encoding
< 
* Connection #0 to host podinfo.local.dev left intact
* Issue another request to this URL: 'http://podinfo.local.dev/function/podinfo/version/version'
* Found bundle for host podinfo.local.dev: 0x5584618379d0 [can pipeline]
* Re-using existing connection! (#0) with host podinfo.local.dev
* Connected to podinfo.local.dev (192.168.39.39) port 80 (#0)
> GET /function/podinfo/version/version HTTP/1.1
> Host: podinfo.local.dev
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Content-Length: 0
< Date: Thu, 18 Jun 2020 11:22:08 GMT
< Location: /function/podinfo/function/podinfo/version/version/function/podinfo/version/version
< Vary: Accept-Encoding
< 
* Connection #0 to host podinfo.local.dev left intact
* Maximum (1) redirects followed

Now let's change the annotation on the ingress to the proposed and see how it works

kubectl -n dev-openfaas annotate --overwrite ing podinfo 'traefik.ingress.kubernetes.io/rewrite-target=/function/podinfo'

curl -s http://podinfo.local.dev

Output:

{
  "hostname": "podinfo-5569bb569c-9wvdw",
  "version": "3.1.0",
  "revision": "7b6f11780ab1ce8c7399da32ec6966215b8e43aa",
  "color": "cyan",
  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
  "message": "greetings from podinfo v3.1.0",
  "goos": "linux",
  "goarch": "amd64",
  "runtime": "go1.13.1",
  "num_goroutine": "6",
  "num_cpu": "2"
}

curl -s http://podinfo.local.dev/version

Output:

{                                                                                                                                                                                                                  
  "commit": "7b6f11780ab1ce8c7399da32ec6966215b8e43aa",                                                                                                                                                            
  "version": "3.1.0"                                                                                                                                                                                               
}

Those who made this far: Thanks for reading and don't forget to clean your hosts file at /etc/hosts

Context

I'm trying to use ingress operator with traefik to forward all traffic from specific domain to function.

Your Environment

Docker version docker version (e.g. Docker 17.0.05 ):
Docker version 19.03.11, build 42e35e61f3
What version and distriubtion of Kubernetes are you using? kubectl version

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-21T20:58:41Z", GoVersion:"go1.13.11", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Operating System and version (e.g. Linux, Windows, MacOS):
Ubuntu 18.04.4 LTS
Link to your project or a code example to reproduce issue:
n/a
What network driver are you using and what CIDR? i.e. Weave net / Flannel
n/a

The text was updated successfully, but these errors were encountered:

alexellis · 2020-06-18T12:03:43Z

Thanks for looking into this and for providing detailed instructions 👍

alexellis · 2020-06-18T12:15:56Z

I've commented on the PR, where we'd usually have the test outputs too.

aleks-fofanov mentioned this issue Jun 18, 2020

Fix traefik annotations as they are not working properly #31

Merged

11 tasks

alexellis closed this as completed in #31 Jun 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] Traefik annotations don't work for any paths (except root for old Traefik versions) #30

[Bug] Traefik annotations don't work for any paths (except root for old Traefik versions) #30

aleks-fofanov commented Jun 18, 2020

alexellis commented Jun 18, 2020

alexellis commented Jun 18, 2020

[Bug] Traefik annotations don't work for any paths (except root for old Traefik versions) #30

[Bug] Traefik annotations don't work for any paths (except root for old Traefik versions) #30

Comments

aleks-fofanov commented Jun 18, 2020

Expected Behaviour

Current Behaviour

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

alexellis commented Jun 18, 2020

alexellis commented Jun 18, 2020