The oncall plugin does not connect.

[Bolywar]

What went wrong?

What happened:

After the latest changes, the oncall plugin stopped connecting

What did you expect to happen:

I expected it to work normally

How do we reproduce it?

1. Deploy helm
2. Try to connect plugin

Grafana OnCall Version

1.15.2

Product Area

Helm/Kubernetes/Docker

Grafana OnCall Platform?

Kubernetes

User's Browser?

No response

Anything else to add?

No response

[zhou-mo]

Hi @Bolywar,

I met the same issue in grafana oncall v1.15.0 last week. I think it may be a bug in oncall plugin when RBACEnabled is true.
The error occurred when grafana call API api/plugins/grafana-oncall-app/resources/plugin/status for oncall plugin connection. It will trigger an API call /api/access-control/users/1/permissions to grafana but grafana won't process the path any more.

You could check the log of your grafana pod and find such error log:

logger=context userId=xxx orgId=xxx uname=sa-1-sa-autogen-oncall t=xxx level=info msg="Request Completed" method=GET path=/api/access-control/users/1/permissions status=404 remote_addr=xxx time_ms=5 duration=5.295902ms size=24 referer= handler=notfound status_source=server

You could add a config in grafana.ini to work around the issue.

[feature_toggles]
accessControlOnCall = false

[Bolywar]

grafana.ini

grafana:
  enabled: true
  grafana.ini:
    server:
      domain: mydomain.net
      root_url: "%(protocol)s://%(domain)s/grafana/"
      serve_from_sub_path: true
    feature_toggles:
      enable: externalServiceAccounts
      accessControlOnCall: false

[Bolywar]

log

logger=plugin.grafana-oncall-app t=2025-03-10T15:29:36.852244632Z level=error msg="Error validating oncall plugin settings" error="error making request: Get \"http://mydomen.net/grafana/api/org\": dial tcp 192.168.122.53:80: connect: connection refused"
logger=context userId=1 orgId=1 uname=admin t=2025-03-10T15:29:36.852411024Z level=error msg="Request Completed" method=GET path=/api/plugins/grafana-oncall-app/resources/plugin/status status=500 remote_addr=192.168.122.52 time_ms=42 duration=42.966012ms size=126 referer=https://mydomen.net/grafana/plugins/grafana-oncall-app handler=/api/plugins/:pluginId/resources/* status_source=downstream
logger=plugin.grafana-oncall-app t=2025-03-10T15:29:46.337905674Z level=error msg="getting incident plugin settings" error="error making request: Get \"http://mydomen.net/grafana/api/plugins/grafana-incident-app/settings\": dial tcp 192.168.122.53:80: connect: connection refused, http://mydomen.net/grafana/api/plugins/grafana-incident-app/settings"
logger=plugin.grafana-oncall-app t=2025-03-10T15:29:46.341814371Z level=error msg="getting labels plugin settings" error="error making request: Get \"http://mydomen.net/grafana/api/plugins/grafana-labels-app/settings\": dial tcp 192.168.122.53:80: connect: connection refused, http://mydomen.net/grafana/api/plugins/grafana-labels-app/settings"
logger=plugin.grafana-oncall-app t=2025-03-10T15:29:46.347315139Z level=info msg=GetSyncData time=3
logger=plugin.grafana-oncall-app t=2025-03-10T15:29:46.347549698Z level=error msg="Error getting sync data" error="error getting users = error making request: Get \"http://mydomen.net/grafana/api/org/users\": dial tcp 192.168.122.53:80: connect: connection refused"
logger=plugin.grafana-oncall-app t=2025-03-10T15:29:46.368740637Z level=info msg=GetUser user="map[Email:admin@localhost Login:admin Name:admin Role:Admin]"
logger=plugin.grafana-oncall-app t=2025-03-10T15:29:46.372533238Z level=error msg="Error getting user" error="error making request: Get \"http://mydomen.net/grafana/api/org/users\": dial tcp 192.168.122.53:80: connect: connection refused"
logger=plugin.grafana-oncall-app t=2025-03-10T15:29:46.372656127Z level=error msg="Error validating oncall plugin settings" error="error setting up request headers: error making request: Get \"http://mydomen.net/grafana/api/org/users\": dial tcp 192.168.122.53:80: connect: connection refused "
logger=context userId=1 orgId=1 uname=admin t=2025-03-10T15:29:46.373276017Z level=error msg="Request Completed" method=GET path=/api/plugins/grafana-oncall-app/resources/plugin/status status=500 remote_addr=192.168.122.52 time_ms=18 duration=18.741004ms size=167 referer=https://mydomen.net/grafana/plugins/grafana-oncall-app handler=/api/plugins/:pluginId/resources/* status_source=downstream

[Bolywar]

solved by replacing the paths

grafana.ini:
    server:
      domain: oncall-grafana.grafana.svc
      root_url: "%(protocol)s://%(domain)s/grafana/"
      serve_from_sub_path: true
    feature_toggles:
      enable: externalServiceAccounts
      accessControlOnCall: false

grafana-oncall-app-provisioning.yaml:
   apps:
      - type: grafana-oncall-app
        name: grafana-oncall-app
        disabled: false
        jsonData:
          stackId: 5
          orgId: 100
          onCallApiUrl: http://oncall-engine.grafana.svc:8080

[Bolywar]

but if you use the service name, then oauth doesn't work

[Bolywar]

grafana ignore protocols: https

level=error msg="Error getting user" error="error making request: Get \"http://mydomen.net/grafana/api/org/users\": dial tcp 1.1.1.1:80: i/o timeout"