🐛 Bug Report: Unexpected failure for target JWKS check Failed to fetch jwks.json, 403

[matdtr]

📜 Description

Hi,

I'm testing version 1.27.2 before upgrading from 1.23. In my test setup i did the following configurations:

• Setup proxy (same setup in contrib/ folder)
• Setup MicrosoftAD for SSO
• Setup MicrosoftGraph for user Ingestion

When adding the proxy suddenly seems that each time permissions are checked it's logged the error "Unexpected failure for target JWKS check Failed to fetch jwks.json, 403"

Is there any way to fix this in the new backend?

👍 Expected behavior

Backstage should work without any issue

👎 Actual Behavior with Screenshots

backstage error Unexpected failure for target JWKS check Failed to fetch jwks.json, 403 stack=Error: Failed to fetch jwks.json, 403
[1]     at doCheck (/Users/Git/POC/backstage/node_modules/@backstage/backend-app-api/src/services/implementations/auth/plugin/PluginTokenHandler.ts:185:17)
[1]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[1]     at DefaultAuthService.getPluginRequestToken (/Users/Git/POC/backstage/node_modules/@backstage/backend-app-api/src/services/implementations/auth/DefaultAuthService.ts:140:7)
[1]     at _ServerPermissionClient.authorizeConditional (/Users/Git/POC/backstage/node_modules/@backstage/plugin-permission-node/src/ServerPermissionClient.ts:97:9)
[1]     at AuthorizedEntitiesCatalog.entitiesBatch (/Users/Git/POC/backstage/node_modules/@backstage/plugin-catalog-backend/src/service/AuthorizedEntitiesCatalog.ts:87:7)
[1]     at <anonymous> (/Users/Git/POC/backstage/node_modules/@backstage/plugin-catalog-backend/src/service/createRouter.ts:223:26)

backstage error Unexpected failure for target JWKS check Failed to fetch jwks.json, 403 stack=Error: Failed to fetch jwks.json, 403
[1]     at doCheck (/Users/Git/POC/backstage/node_modules/@backstage/backend-app-api/src/services/implementations/auth/plugin/PluginTokenHandler.ts:185:17)
[1]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[1]     at DefaultAuthService.getPluginRequestToken (/Users/Git/POC/backstage/node_modules/@backstage/backend-app-api/src/services/implementations/auth/DefaultAuthService.ts:140:7)
[1]     at handleRequest (/Users/Git/POC/backstage/node_modules/@backstage/plugin-permission-backend/src/service/router.ts:136:23)
[1]     at <anonymous> (/Users/Git/POC/backstage/node_modules/@backstage/plugin-permission-backend/src/service/router.ts:240:16)

backstage error Unexpected failure for target JWKS check Failed to fetch jwks.json, 403 stack=Error: Failed to fetch jwks.json, 403
[1]     at doCheck (/Users/Git/POC/backstage/node_modules/@backstage/backend-app-api/src/services/implementations/auth/plugin/PluginTokenHandler.ts:185:17)
[1]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[1]     at DefaultAuthService.getPluginRequestToken (/Users/Git/POC/backstage/node_modules/@backstage/backend-app-api/src/services/implementations/auth/DefaultAuthService.ts:140:7)
[1]     at CatalogAuthResolverContext.findCatalogUser (/Users/Git/POC/backstage/node_modules/@backstage/plugin-auth-backend/src/lib/resolvers/CatalogAuthResolverContext.ts:109:23)
[1]     at CatalogAuthResolverContext.signInWithCatalogUser (/Users/Git/POC/backstage/node_modules/@backstage/plugin-auth-backend/src/lib/resolvers/CatalogAuthResolverContext.ts:166:24)
[1]     at <anonymous> (/Users/Git/POC/backstage/node_modules/@backstage/plugin-auth-node/src/sign-in/readDeclarativeSignInResolver.ts:60:16)
[1]     at Object.refresh (/Users/Git/POC/backstage/node_modules/@backstage/plugin-auth-node/src/oauth/createOAuthRouteHandlers.ts:331:28)
[1]     at OAuthEnvironmentHandler.refresh (/Users/Git/POC/backstage/node_modules/@backstage/plugin-auth-node/src/oauth/OAuthEnvironmentHandler.ts:60:11)

backstage error Unexpected failure for target JWKS check Failed to fetch jwks.json, 403 stack=Error: Failed to fetch jwks.json, 403
[1]     at doCheck (/Users/Git/POC/backstage/node_modules/@backstage/backend-app-api/src/services/implementations/auth/plugin/PluginTokenHandler.ts:185:17)
[1]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

👟 Reproduction steps

npx @backstage/create-app
yarn add --cwd packages/backend @backstage/plugin-auth-backend-module-microsoft-provider
yarn add --cwd packages/backend @backstage/plugin-catalog-backend-module-msgraph
yarn add --cwd packages/backend undici global-agent  
https://github.com/backstage/backstage/blob/master/contrib/docs/tutorials/help-im-behind-a-corporate-proxy.md

📃 Provide the context for the Bug.

No response

🖥️ Your Environment

yarn run v1.22.21
$ /Users/Git/POC/backstage/node_modules/.bin/backstage-cli info
OS:   Darwin 23.4.0 - darwin/arm64
node: v18.19.0
yarn: 1.22.21
cli:  0.26.5 (installed)
backstage:  1.27.2

Dependencies:
  @backstage/app-defaults                                          1.5.5
  @backstage/backend-app-api                                       0.7.5
  @backstage/backend-common                                        0.22.0
  @backstage/backend-defaults                                      0.2.18
  @backstage/backend-dev-utils                                     0.1.4
  @backstage/backend-openapi-utils                                 0.1.11
  @backstage/backend-plugin-api                                    0.6.18
  @backstage/backend-tasks                                         0.5.23
  @backstage/catalog-client                                        1.6.5
  @backstage/catalog-model                                         1.5.0
  @backstage/cli-common                                            0.1.13
  @backstage/cli-node                                              0.2.5
  @backstage/cli                                                   0.26.5
  @backstage/config-loader                                         1.8.0
  @backstage/config                                                1.2.0
  @backstage/core-app-api                                          1.12.5
  @backstage/core-compat-api                                       0.2.5
  @backstage/core-components                                       0.14.7
  @backstage/core-plugin-api                                       1.9.2
  @backstage/e2e-test-utils                                        0.1.1
  @backstage/errors                                                1.2.4
  @backstage/eslint-plugin                                         0.1.8
  @backstage/frontend-plugin-api                                   0.6.5
  @backstage/integration-aws-node                                  0.1.12
  @backstage/integration-react                                     1.1.27
  @backstage/integration                                           1.11.0
  @backstage/plugin-api-docs                                       0.11.5
  @backstage/plugin-auth-backend-module-atlassian-provider         0.1.10
  @backstage/plugin-auth-backend-module-aws-alb-provider           0.1.10
  @backstage/plugin-auth-backend-module-azure-easyauth-provider    0.1.1
  @backstage/plugin-auth-backend-module-bitbucket-provider         0.1.1
  @backstage/plugin-auth-backend-module-cloudflare-access-provider 0.1.1
  @backstage/plugin-auth-backend-module-gcp-iap-provider           0.2.13
  @backstage/plugin-auth-backend-module-github-provider            0.1.15
  @backstage/plugin-auth-backend-module-gitlab-provider            0.1.15
  @backstage/plugin-auth-backend-module-google-provider            0.1.15
  @backstage/plugin-auth-backend-module-guest-provider             0.1.4
  @backstage/plugin-auth-backend-module-microsoft-provider         0.1.13
  @backstage/plugin-auth-backend-module-oauth2-provider            0.1.15
  @backstage/plugin-auth-backend-module-oauth2-proxy-provider      0.1.11
  @backstage/plugin-auth-backend-module-oidc-provider              0.1.9
  @backstage/plugin-auth-backend-module-okta-provider              0.0.11
  @backstage/plugin-auth-backend                                   0.22.5
  @backstage/plugin-auth-node                                      0.4.13
  @backstage/plugin-auth-react                                     0.1.2
  @backstage/plugin-catalog-backend-module-bitbucket-server        0.1.32
  @backstage/plugin-catalog-backend-module-msgraph                 0.5.26
  @backstage/plugin-catalog-backend-module-scaffolder-entity-model 0.1.16
  @backstage/plugin-catalog-backend                                1.22.0
  @backstage/plugin-catalog-common                                 1.0.23
  @backstage/plugin-catalog-graph                                  0.4.5
  @backstage/plugin-catalog-import                                 0.11.0
  @backstage/plugin-catalog-node                                   1.12.0
  @backstage/plugin-catalog-react                                  1.12.0
  @backstage/plugin-catalog                                        1.20.0
  @backstage/plugin-events-node                                    0.3.4
  @backstage/plugin-org                                            0.6.25
  @backstage/plugin-permission-backend-module-allow-all-policy     0.1.15
  @backstage/plugin-permission-backend                             0.5.42
  @backstage/plugin-permission-common                              0.7.13
  @backstage/plugin-permission-node                                0.7.29
  @backstage/plugin-permission-react                               0.4.22
  @backstage/plugin-proxy-backend                                  0.4.16
  @backstage/plugin-scaffolder-backend-module-azure                0.1.10
  @backstage/plugin-scaffolder-backend-module-bitbucket-cloud      0.1.8
  @backstage/plugin-scaffolder-backend-module-bitbucket-server     0.1.8
  @backstage/plugin-scaffolder-backend-module-bitbucket            0.2.8
  @backstage/plugin-scaffolder-backend-module-gerrit               0.1.10
  @backstage/plugin-scaffolder-backend-module-gitea                0.1.8
  @backstage/plugin-scaffolder-backend-module-github               0.2.8
  @backstage/plugin-scaffolder-backend-module-gitlab               0.4.0
  @backstage/plugin-scaffolder-backend                             1.22.7
  @backstage/plugin-scaffolder-common                              1.5.2
  @backstage/plugin-scaffolder-node                                0.4.4
  @backstage/plugin-scaffolder-react                               1.8.5
  @backstage/plugin-scaffolder                                     1.20.0
  @backstage/plugin-search-backend-module-catalog                  0.1.24
  @backstage/plugin-search-backend-module-techdocs                 0.1.23
  @backstage/plugin-search-backend-node                            1.2.23
  @backstage/plugin-search-backend                                 1.5.9
  @backstage/plugin-search-common                                  1.2.11
  @backstage/plugin-search-react                                   1.7.11
  @backstage/plugin-search                                         1.4.11
  @backstage/plugin-techdocs-backend                               1.10.5
  @backstage/plugin-techdocs-module-addons-contrib                 1.1.10
  @backstage/plugin-techdocs-node                                  1.12.4
  @backstage/plugin-techdocs-react                                 1.2.4
  @backstage/plugin-techdocs                                       1.10.5
  @backstage/plugin-user-settings                                  0.8.6
  @backstage/release-manifests                                     0.0.11
  @backstage/repo-tools                                            0.9.0
  @backstage/test-utils                                            1.5.5
  @backstage/theme                                                 0.5.5
  @backstage/types                                                 1.1.1
  @backstage/version-bridge                                        1.0.8

👀 Have you spent some time to check if this bug has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

Are you willing to submit PR?

None

[stephenglass]

I am also experiencing this issue. New backstage project using guest auth provider and GitHub catalog integration. Followed the tutorial referenced in issue to setup corporate proxy.

[1] 2024-05-21T19:40:21.449Z catalog info Read 4 GitHub repositories (4 matching the pattern) target=github-provider:providerId class=GithubEntityProvider taskId=github-provider:providerId:refresh taskInstanceId=293fb8de-d732-4a37-a4fe-4736b640b4ff
[1] 2024-05-21T19:40:23.410Z search info Collating documents for software-catalog via DefaultCatalogCollatorFactory documentType=software-catalog
[1] 2024-05-21T19:40:23.415Z search info Collating documents for techdocs via DefaultTechDocsCollatorFactory documentType=techdocs
[1] 2024-05-21T19:40:23.510Z backstage error Unexpected failure for target JWKS check fetch failed stack=TypeError: fetch failed
[1]     at node:internal/deps/undici/undici:12502:13
[1]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[1]     at doCheck (/home/stephen/my-backstage-app/node_modules/@backstage/backend-app-api/src/services/implementations/auth/plugin/PluginTokenHandler.ts:175:21)
[1] 2024-05-21T19:40:23.511Z search warn Index for software-catalog was not created: an error was encountered documentType=software-catalog
[1] 2024-05-21T19:40:23.511Z search error Collating documents for software-catalog failed: Error: Unable to generate legacy token for communication with the 'catalog' plugin. You will typically encounter this error when attempting to call a plugin that does not exist, or is deployed with an old version of Backstage; caused by Error: Unable to generate legacy token, no legacy keys are configured in 'backend.auth.keys' or 'backend.auth.externalAccess' documentType=software-catalog
[1] 2024-05-21T19:40:23.512Z search warn Index for techdocs was not created: an error was encountered documentType=techdocs
[1] 2024-05-21T19:40:23.512Z search error Collating documents for techdocs failed: Error: Unable to generate legacy token for communication with the 'catalog' plugin. You will typically encounter this error when attempting to call a plugin that does not exist, or is deployed with an old version of Backstage; caused by Error: Unable to generate legacy token, no legacy keys are configured in 'backend.auth.keys' or 'backend.auth.externalAccess' documentType=techdocs
[1] 2024-05-21T19:40:23.512Z search error Error: Unable to generate legacy token for communication with the 'catalog' plugin. You will typically encounter this error when attempting to call a plugin that does not exist, or is deployed with an old version of Backstage; caused by Error: Unable to generate legacy token, no legacy keys are configured in 'backend.auth.keys' or 'backend.auth.externalAccess' task=search_index_software_catalog
[1] 2024-05-21T19:40:23.512Z search error Error: Unable to generate legacy token for communication with the 'catalog' plugin. You will typically encounter this error when attempting to call a plugin that does not exist, or is deployed with an old version of Backstage; caused by Error: Unable to generate legacy token, no legacy keys are configured in 'backend.auth.keys' or 'backend.auth.externalAccess' task=search_index_techdocs
[0] webpack compiled successfully
[1] 2024-05-21T19:40:38.901Z backstage error Unexpected failure for target JWKS check fetch failed stack=TypeError: fetch failed
[1]     at node:internal/deps/undici/undici:12502:13
[1]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[1]     at doCheck (/home/stephen/my-backstage-app/node_modules/@backstage/backend-app-api/src/services/implementations/auth/plugin/PluginTokenHandler.ts:175:21)
[1]     at DefaultAuthService.getPluginRequestToken (/home/stephen/my-backstage-app/node_modules/@backstage/backend-app-api/src/services/implementations/auth/DefaultAuthService.ts:140:7)
[1]     at CatalogAuthResolverContext.findCatalogUser (/home/stephen/my-backstage-app/node_modules/@backstage/plugin-auth-backend/src/lib/resolvers/CatalogAuthResolverContext.ts:109:23)
[1]     at CatalogAuthResolverContext.signInWithCatalogUser (/home/stephen/my-backstage-app/node_modules/@backstage/plugin-auth-backend/src/lib/resolvers/CatalogAuthResolverContext.ts:166:24)
[1]     at <anonymous> (/home/stephen/my-backstage-app/node_modules/@backstage/plugin-auth-backend-module-guest-provider/src/resolvers.ts:49:14)
[1]     at Object.refresh (/home/stephen/my-backstage-app/node_modules/@backstage/plugin-auth-node/src/proxy/createProxyRouteHandlers.ts:66:24)

[rohanrmallya]

Adding GLOBAL_AGENT_NO_PROXY=localhost in the env should help. I think the requests through localhost are also going through the corporate proxy.

[stephenglass]

This is fixed by #24902 and adding adding localhost to GLOBAL_AGENT_NO_PROXY. You can also setup http interceptors to act as a no proxy handler for the native node fetch too see here: https://github.com/janus-idp/backstage-showcase/blob/main/packages/backend/src/corporate-proxy.ts

[benjdlambert]

@stephenglass thanks for the link there. I'm thinking that we're probably gonna want something like that in place the more the ecosystem grows. Of course we can chose to use node-fetch in our packages, but any transitive deps might upgrade and end up using a polyfill which only fills when global.fetch isn't available, or down right doesn't use a polyfill at all which seems to be the way things are moving.

I guess eventually we will probably follow suit with our packages, so it's good that there's support for both options in that file. Maybe something to bring into the framework at somepoint, or at least add to contrib docs that exist already

[stephenglass]

@stephenglass thanks for the link there. I'm thinking that we're probably gonna want something like that in place the more the ecosystem grows. Of course we can chose to use node-fetch in our packages, but any transitive deps might upgrade and end up using a polyfill which only fills when global.fetch isn't available, or down right doesn't use a polyfill at all which seems to be the way things are moving.

I guess eventually we will probably follow suit with our packages, so it's good that there's support for both options in that file. Maybe something to bring into the framework at somepoint, or at least add to contrib docs that exist already

@benjdlambert I can work on a PR to create a new corporate proxy backstage plugin so users can easily add it to their backend index.ts and optionally override proxy env using app-config.yml file? Of course supporting no proxy conf for both native fetch and node fetch. Does this sound like something that could be accepted into the project at this time?

[darrenyung]

Hi all,

Just out of curiosity, I have similar setup that's been doing well for a while now but until recently. Have you used the permissions framework to create custom policies yet? If so, with default allow-all, did you experience any 401 responses for static content when trying to view your techdocs?

[benjdlambert]

@stephenglass yeah, a new backend module or something which basically sets these things up could be a good start. It might not do much, but it's possible that in the future it could reconfigure a fetchServiceRef that we might end up moving to, similar to the fetchApiRef that we have in the frontend.

Maybe let's open a PR and discuss there?

Might also be worth just updating the current documentation with an example module instead to start though?

[Djiit]

This is fixed by #24902 and adding adding localhost to GLOBAL_AGENT_NO_PROXY. You can also setup http interceptors to act as a no proxy handler for the native node fetch too see here: janus-idp/backstage-showcase@main/packages/backend/src/corporate-proxy.ts

Hey, I just tried with v1.28.0-next-1 and the suggested configuration but without any luck (same error). Any other things we could try ?