Origin not allowed error when reverse proxying grafana

[snowbldr]

What is the issue?

When you configure the latest linkerd to use a new grafana instance in the cluster via the grafanaUrl property, it's not possible to log in, and an "origin not allowed" error is displayed.

linkerd 2.11.1
linkerd viz 2.11.1
kube-prometheus-stack 33.2.1
grafana v8.3.6

How can it be reproduced?

Using prometheus-stack-values.yaml:

grafana:
  enabled: true
  adminPassword: password
  grafana.ini:
    server:
      #set to enable integration with linkerd viz
      #https://linkerd.io/2.11/tasks/grafana/index.html#in-cluster-grafana-instances
      root_url: '%(protocol)s://%(domain)s:/grafana/'
      serve_from_sub_path: true

prometheus:
  enabled: true

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

linkerd install | kubectl apply -f -
linkerd check

helm install -f prometheus-stack-values.yaml --version 33.2.1 metrics prometheus-community/kube-prometheus-stack

linkerd viz install --set grafana.enabled=false,grafanaUrl=metrics-grafana.default,prometheus.enabled=false,prometheusUrl=http://prometheus-operated.default:9090 | kubectl apply -f -
linkerd viz check
linkerd viz dashboard &

Logs, error output, etc

Logs from linkerd web

time="2022-03-14T16:14:15Z" level=info msg="running version stable-2.11.1"
time="2022-03-14T16:14:15Z" level=info msg="starting admin server on :9994"
time="2022-03-14T16:14:15Z" level=info msg="starting HTTP server on :8084"

Grafana logs, looks normal for a not logged in user

t=2022-03-14T16:13:28+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=http subUrl=/grafana socket=
t=2022-03-14T16:13:28+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager
t=2022-03-14T16:14:40+0000 lvl=eror msg="Failed to look up user based on cookie" logger=context error="user token not found"
t=2022-03-14T16:14:40+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/dashboard/db/linkerd-namespace status=302 remote_addr=127.0.0.1 time_ms=1 size=37 referer=http://localhost:50750/namespaces
t=2022-03-14T16:14:40+0000 lvl=eror msg="Failed to look up user based on cookie" logger=context error="user token not found"
t=2022-03-14T16:15:14+0000 lvl=eror msg="Failed to look up user based on cookie" logger=context error="user token not found"
t=2022-03-14T16:15:14+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/dashboard/db/linkerd-namespace status=302 remote_addr=127.0.0.1 time_ms=1 size=37 referer=http://localhost:50750/namespaces
t=2022-03-14T16:15:14+0000 lvl=eror msg="Failed to look up user based on cookie" logger=context error="user token not found"

output of linkerd check -o short

linkerd viz not returning valid json?

Linkerd core checks
===================


Status check results are √

Linkerd extensions checks
=========================

linkerd-viz
-----------
× Running: C:\Users\r\.local\bin\linkerd viz check --as-group=[] --linkerd-namespace=linkerd --verbose=false --proxy=false --wait=5m0s --output=json
    invalid extension check output from "C:\Users\r\.local\bin\linkerd viz check --as-group=[] --linkerd-namespace=linkerd --verbose=false --proxy=false --wait=5m0s --output=json" (JSON object expected):

[unexpected end of JSON input]
    see https://linkerd.io/2.11/checks/#extensions for hints

Status check results are ×

here's the output of linkerd viz check (it seems ok...)

linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ tap API service is running
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
√ viz extension proxies are up-to-date
√ viz extension proxies and cli versions match
√ can initialize the client
√ viz extension self-check

Status check results are √

Environment

kind on windows
k8s 1.21.1
linkerd 2.11.1
linkerd viz 2.11.1
kube-prometheus-stack 33.2.1
grafana v8.3.6

Possible solution

Maybe reverse_proxy.go isn't forwarding the host or origin header?

Additional context

Grafana recently released an update that validates the Origin and the Host to prevent CSRF.

https://grafana.com/blog/2022/02/08/grafana-7.5.15-and-8.3.5-released-with-moderate-severity-security-fixes/

https://github.com/grafana/grafana/blob/23956557d8c6a119b7de5be5c42024e29634d002/pkg/middleware/csrf.go#L43

Would you like to work on fixing this bug?

yes

[alpeb]

Thanks for the report @snowbldr , I can confirm the latest Grafana versions can't be linked in the linkerd dashboard. grafana/grafana#45117 also contains some pointers on how to solve this issue; this might need some changes in the reverse proxy in the web component. Will keep you posted.