🐛 Bug Report: Authentication doesn't work when kubernetes cluster is added as a resource

[tasiorek27]

📜 Description

When Kubernetes cluster is added as a resource the authentication doesn't work. Instead of showing the related resources at the component level user gets 401 error code.

👍 Expected behavior

User should be able to see its resources when accessing Kubernetes tab at the component.

👎 Actual Behavior with Screenshots

User gets 401 unauthorized.

Request:
GET https://domain/api/kubernetes/clusters

Response:

{
    "error": {
        "name": "ResponseError",
        "message": "Request failed with 500 ResponseError",
        "response": {
            "size": 0,
            "timeout": 0
        },
        "body": {
            "error": {
                "name": "ResponseError",
                "message": "Request failed with 401 AuthenticationError",
                "response": {
                    "size": 0,
                    "timeout": 0
                },
                "body": {
                    "error": {
                        "name": "AuthenticationError",
                        "message": "No token specified"
                    },
                    "request": {
                        "method": "POST",
                        "url": "/authorize"
                    },
                    "response": {
                        "statusCode": 401
                    }
                },
                "cause": {
                    "name": "AuthenticationError",
                    "message": "No token specified"
                }
            },
            "request": {
                "method": "GET",
                "url": "/entities?filter=kind%3DResource%2Cspec.type%3Dkubernetes-cluster%2Cmetadata.annotations.kubernetes.io%2Fapi-server%2Cmetadata.annotations.kubernetes.io%2Fapi-server-certificate-authority%2Cmetadata.annotations.kubernetes.io%2Fauth-provider"
            },
            "response": {
                "statusCode": 500
            }
        },
        "cause": {
            "response": {
                "size": 0,
                "timeout": 0
            },
            "body": {
                "error": {
                    "name": "AuthenticationError",
                    "message": "No token specified"
                },
                "request": {
                    "method": "POST",
                    "url": "/authorize"
                },
                "response": {
                    "statusCode": 401
                }
            },
            "cause": {
                "name": "AuthenticationError",
                "message": "No token specified"
            },
            "name": "ResponseError",
            "message": "Request failed with 401 AuthenticationError"
        }
    },
    "request": {
        "method": "GET",
        "url": "/api/kubernetes/clusters"
    },
    "response": {
        "statusCode": 500
    }
}

👟 Reproduction steps

Below config is added to catalog-info.yaml:

apiVersion: backstage.io/v1alpha1
kind: Resource
metadata:
  name: aks-backstage
  annotations:
    kubernetes.io/api-server: "https://145659-wjalymt8.hcp.westeurope.azmk8s.io:443"
    kubernetes.io/auth-provider: 'aks'
    kubernetes.io/skip-metrics-lookup: "true"
    kubernetes.io/skip-tls-verify: "true"
    kubernetes.io/api-server-certificate-authority: ""

Kubernetes config snippet:

kubernetes:
  serviceLocatorMethod:
    type: 'multiTenant'
  clusterLocatorMethods:
    - type: 'catalog'

The resource is added to backstage but authentication for the cluster doesn't work. At the same time if the cluster is hardcoded within backstage's config as below it works fine.

kubernetes:
  serviceLocatorMethod:
    type: "multiTenant"
  clusterLocatorMethods:
    - type: "config"
      clusters:
        - url: https://145659-wjalymt8.hcp.westeurope.azmk8s.io:443
          name: aks-backstage
          authProvider: aks
          skipTLSVerify: true
          skipMetricsLookup: false

📃 Provide the context for the Bug.

No response

🖥️ Your Environment

OS:   Windows_NT 10.0.22621 - win32/x64
node: v18.16.1
yarn: 1.22.21
cli:  0.25.2 (installed)
backstage:  1.23.3

Dependencies:
  @backstage/app-defaults                                          1.5.0
  @backstage/backend-app-api                                       0.4.5, 0.5.13
  @backstage/backend-common                                        0.15.2, 0.18.5, 0.19.9, 0.20.1, 0.21.2
  @backstage/backend-defaults                                      0.2.12
  @backstage/backend-dev-utils                                     0.1.4
  @backstage/backend-openapi-utils                                 0.1.5
  @backstage/backend-plugin-api                                    0.5.4, 0.6.12
  @backstage/backend-tasks                                         0.5.17
  @backstage/catalog-client                                        1.6.0
  @backstage/catalog-model                                         1.4.4
  @backstage/cli-common                                            0.1.13
  @backstage/cli-node                                              0.2.3
  @backstage/cli                                                   0.25.2
  @backstage/config-loader                                         1.6.2
  @backstage/config                                                1.1.1
  @backstage/core-app-api                                          1.12.0
  @backstage/core-compat-api                                       0.2.0
  @backstage/core-components                                       0.11.2, 0.13.10, 0.14.0
  @backstage/core-plugin-api                                       1.9.0
  @backstage/dev-utils                                             1.0.27
  @backstage/errors                                                1.2.3
  @backstage/eslint-plugin                                         0.1.5
  @backstage/frontend-plugin-api                                   0.6.0
  @backstage/integration-aws-node                                  0.1.9
  @backstage/integration-react                                     1.1.24
  @backstage/integration                                           1.9.0
  @backstage/plugin-api-docs                                       0.11.0
  @backstage/plugin-app-backend                                    0.3.60
  @backstage/plugin-app-node                                       0.1.12
  @backstage/plugin-auth-backend-module-atlassian-provider         0.1.4
  @backstage/plugin-auth-backend-module-aws-alb-provider           0.1.3
  @backstage/plugin-auth-backend-module-gcp-iap-provider           0.2.7
  @backstage/plugin-auth-backend-module-github-provider            0.1.9
  @backstage/plugin-auth-backend-module-gitlab-provider            0.1.9
  @backstage/plugin-auth-backend-module-google-provider            0.1.9
  @backstage/plugin-auth-backend-module-microsoft-provider         0.1.7
  @backstage/plugin-auth-backend-module-oauth2-provider            0.1.9
  @backstage/plugin-auth-backend-module-oauth2-proxy-provider      0.1.5
  @backstage/plugin-auth-backend-module-oidc-provider              0.1.2
  @backstage/plugin-auth-backend-module-okta-provider              0.0.5
  @backstage/plugin-auth-backend                                   0.21.2
  @backstage/plugin-auth-node                                      0.2.19, 0.4.7
  @backstage/plugin-azure-devops-backend                           0.5.4
  @backstage/plugin-azure-devops-common                            0.3.2
  @backstage/plugin-azure-devops                                   0.3.12
  @backstage/plugin-azure-sites-backend                            0.2.2
  @backstage/plugin-azure-sites-common                             0.1.2
  @backstage/plugin-azure-sites                                    0.1.19
  @backstage/plugin-catalog-backend-module-azure                   0.1.31
  @backstage/plugin-catalog-backend-module-github                  0.5.2
  @backstage/plugin-catalog-backend-module-msgraph                 0.5.19
  @backstage/plugin-catalog-backend-module-openapi                 0.1.29
  @backstage/plugin-catalog-backend-module-scaffolder-entity-model 0.1.9
  @backstage/plugin-catalog-backend-module-unprocessed             0.3.9
  @backstage/plugin-catalog-backend                                1.17.2
  @backstage/plugin-catalog-common                                 1.0.21
  @backstage/plugin-catalog-graph                                  0.4.0
  @backstage/plugin-catalog-import                                 0.10.6
  @backstage/plugin-catalog-node                                   1.7.2
  @backstage/plugin-catalog-react                                  1.10.0
  @backstage/plugin-catalog-unprocessed-entities                   0.1.8
  @backstage/plugin-catalog                                        1.17.0
  @backstage/plugin-events-node                                    0.2.21
  @backstage/plugin-git-release-manager                            0.3.42
  @backstage/plugin-github-actions                                 0.6.11
  @backstage/plugin-home-react                                     0.1.8
  @backstage/plugin-home                                           0.6.2
  @backstage/plugin-kubernetes-backend                             0.15.2
  @backstage/plugin-kubernetes-common                              0.7.4
  @backstage/plugin-kubernetes-node                                0.1.6
  @backstage/plugin-kubernetes-react                               0.3.0
  @backstage/plugin-kubernetes                                     0.11.5
  @backstage/plugin-org                                            0.6.20
  @backstage/plugin-permission-backend                             0.5.35
  @backstage/plugin-permission-common                              0.7.12
  @backstage/plugin-permission-node                                0.7.23
  @backstage/plugin-permission-react                               0.4.20
  @backstage/plugin-proxy-backend                                  0.4.10
  @backstage/plugin-scaffolder-backend-module-azure                0.1.4
  @backstage/plugin-scaffolder-backend-module-bitbucket-cloud      0.1.2
  @backstage/plugin-scaffolder-backend-module-bitbucket-server     0.1.2
  @backstage/plugin-scaffolder-backend-module-bitbucket            0.2.2
  @backstage/plugin-scaffolder-backend-module-gerrit               0.1.4
  @backstage/plugin-scaffolder-backend-module-gitea                0.1.2
  @backstage/plugin-scaffolder-backend-module-github               0.2.2
  @backstage/plugin-scaffolder-backend-module-gitlab               0.2.15
  @backstage/plugin-scaffolder-backend                             1.21.2
  @backstage/plugin-scaffolder-common                              1.5.0
  @backstage/plugin-scaffolder-node                                0.1.5, 0.3.2
  @backstage/plugin-scaffolder-react                               1.8.0
  @backstage/plugin-scaffolder                                     1.18.0
  @backstage/plugin-search-backend-module-catalog                  0.1.16
  @backstage/plugin-search-backend-module-pg                       0.5.21
  @backstage/plugin-search-backend-module-techdocs                 0.1.16
  @backstage/plugin-search-backend-node                            1.2.16
  @backstage/plugin-search-backend                                 1.5.2
  @backstage/plugin-search-common                                  1.2.10
  @backstage/plugin-search-react                                   1.7.6
  @backstage/plugin-search                                         1.4.6
  @backstage/plugin-sonarqube-backend                              0.2.14
  @backstage/plugin-sonarqube-react                                0.1.13
  @backstage/plugin-sonarqube                                      0.7.12
  @backstage/plugin-tech-radar                                     0.6.13
  @backstage/plugin-techdocs-backend                               1.9.5
  @backstage/plugin-techdocs-module-addons-contrib                 1.1.5
  @backstage/plugin-techdocs-node                                  1.11.4
  @backstage/plugin-techdocs-react                                 1.1.16
  @backstage/plugin-techdocs                                       1.10.0
  @backstage/plugin-user-settings                                  0.8.1
  @backstage/release-manifests                                     0.0.11
  @backstage/repo-tools                                            0.6.2
  @backstage/test-utils                                            1.5.0
  @backstage/theme                                                 0.2.19, 0.4.4, 0.5.1
  @backstage/types                                                 1.1.1
  @backstage/version-bridge                                        1.0.7
Done in 4.50s.

Will provide the output of yarn backstage-cli info asap.

👀 Have you spent some time to check if this bug has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

Are you willing to submit PR?

None

[benjdlambert]

My guess here is that we need to pass the token through to the CatalogApi call here:

backstage/plugins/kubernetes-backend/src/cluster-locator/CatalogClusterLocator.ts

Lines 59 to 61 in 6009d2a

	const clusters = await this.catalogClient.getEntities({
	filter: [filter],
	});

Using the new auth things that we've been working on recently. Probably going to be a few changes here to the public API through as the request will need drilling down in the interface I think.

@camilaibs do you wanna maybe work on this together?

[tasiorek27]

Hello,
I've verified it on version 1.24.2 and unfortunately it still doesn't work. Please find below the configuration along with the error message.

• kubernetes plugin config:

kubernetes:
  serviceLocatorMethod:
    type: 'multiTenant'
  clusterLocatorMethods:
    - type: 'catalog'

• resource config

apiVersion: backstage.io/v1alpha1
kind: Resource
metadata:
  name: aks-backstage
  annotations:
    kubernetes.io/api-server: "https://145659-wjalymt8.hcp.westeurope.azmk8s.io:443"
    kubernetes.io/auth-provider: 'aks'
    kubernetes.io/skip-metrics-lookup: "true"
    kubernetes.io/skip-tls-verify: "true"
    kubernetes.io/api-server-certificate-authority: "" # base64-encoded CA
spec:
  type: kubernetes-cluster
  owner: "user:default/michal-markiewicz_volvo"

Error message:

Request URL: POST https://<some-url>/api/kubernetes/services/mm-test-secret-value1

Response:

{error: "Cannot read properties of undefined (reading 'credentials')"}
error
: 
"Cannot read properties of undefined (reading 'credentials')"

[benjdlambert]

@tasiorek27 please upgrade to 1.25, it's been fixed in that version.

[tasiorek27]

@benjdlambert alright, will try that. Thanks.

[benjdlambert]

Actually sorry apologies, this actually is going out in todays -next release. If we can confirm it works as expected, I think we can ship 1.25.1 with this fix in it as a patch.

[tasiorek27]

Ok, we will try that and update you tomorrow hopefully.

[benjdlambert]

@tasiorek27 this should be out now for you to test, please let us know if the -next release works as expected, specifically the fix for this issue, although I think that what you're running into now is actually more related to this issue: #23876

[tasiorek27]

@benjdlambert we have following issue when trying to bump the version with command yarn backstage-cli versions:bump --release=next. Could you advise?

Error: Version bump failed for @backstage/plugin-catalog-backend-module-scaffolder-entity-model, @backstage/plugin-catalog-backend, @backstage/plugin-scaffolder-backend-module-azure, @backstage/plugin-scaffolder-backend-module-bitbucket-cloud, @backstage/plugin-scaffolder-backend-module-bitbucket-server, @backstage/plugin-scaffolder-backend-module-bitbucket, @backstage/plugin-scaffolder-backend-module-gerrit, @backstage/plugin-scaffolder-backend-module-gitea, @backstage/plugin-scaffolder-backend-module-github, @backstage/plugin-scaffolder-backend-module-gitlab, @backstage/plugin-scaffolder-backend, @backstage/plugin-scaffolder-backend, @backstage/plugin-search-backend-module-catalog, @backstage/plugin-search
 
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[benjdlambert]

Hmm - that's strange. Do you get any more logs than that?

I wonder if it's just when it's trying to dedupe some dependencies, it's possible that it did actually work? What does the version in backstage.json say?

[tasiorek27]

@benjdlambert yeah, it indeed updated. I can confirm it finally works :)

BTW can I have in single config file multiple clusterLocatorMethods something like this?

kubernetes:
  serviceLocatorMethod:
    type: 'multiTenant'
  clusterLocatorMethods:
    - type: 'catalog'
    - type: 'config'
      clusters:
        - url: https://my-api-cluster.com:6443
          name: xd
          authProvider: serviceAccount
          skipTLSVerify: true
          skipMetricsLookup: false
          serviceAccountToken: ${K8S_XD_TOKEN}