Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Bug Report: Github Discovery error #15940

Closed
2 tasks done
darrenyung opened this issue Jan 25, 2023 · 18 comments
Closed
2 tasks done

🐛 Bug Report: Github Discovery error #15940

darrenyung opened this issue Jan 25, 2023 · 18 comments
Labels
bug Something isn't working stale

Comments

@darrenyung
Copy link

darrenyung commented Jan 25, 2023

📜 Description

Problem started when upgrading from 1.7.2 to 1.8.3 and still persists when upgrading to 1.10.1

I've made the changes as advised regarding the naming from GitHub to Github but when the app starts on my local, I get the errors coming from my config for the Github Discovery feature.

I haven't change anything from the config nor the Github App itself as it was working prior to 1.8.3. Did I miss something in the upgrade?

👍 Expected behavior

Github Discovery working and able to pull org data from public Github

👎 Actual Behavior with Screenshots

In the console, error reported the following

1] 2023-01-25T06:39:00.172Z catalog error github-provider:providerId refresh failed fetch failed type=plugin target=github-provider:providerId class=GithubEntityProvider taskId=github-provider:providerId:refresh taskInstanceId=ee0f1bf5-5275-4658-88de-b7f3af8d86bb name=HttpError status=500 request=[object Object] stack=HttpError: fetch failed
[1]     at /Users/yungc/Documents/Projects/backstage/node_modules/@octokit/request/dist-node/index.js:99:11
[1]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[1]     at async hook (/Users/yungc/Documents/Projects/backstage/node_modules/@octokit/auth-app/dist-node/index.js:297:18)
[1]     at async Object.next (/Users/yungc/Documents/Projects/backstage/node_modules/@octokit/plugin-paginate-rest/dist-node/index.js:67:28)
[1]     at async GithubAppManager.getInstallationData (/Users/yungc/Documents/Projects/backstage/node_modules/@backstage/integration/dist/index.cjs.js:1311:30)
[1]     at async /Users/yungc/Documents/Projects/backstage/node_modules/@backstage/integration/dist/index.cjs.js:1278:45
[1]     at async Cache.getOrCreateToken (/Users/yungc/Documents/Projects/backstage/node_modules/@backstage/integration/dist/index.cjs.js:1228:34)
[1]     at async Promise.all (index 0)
[1]     at async GithubAppCredentialsMux.getAppToken (/Users/yungc/Documents/Projects/backstage/node_modules/@backstage/integration/dist/index.cjs.js:1349:21)
[1]     at async _SingleInstanceGithubCredentialsProvider.getCredentials (/Users/yungc/Documents/Projects/backstage/node_modules/@backstage/integration/dist/index.cjs.js:1399:17)

In the UI, Catalog Import returns the following error

{"error":{"name":"InputError","message":"Error: Unable to read url, HttpError: fetch failed","stack":"InputError: Error: Unable to read url, HttpError: fetch failed\n at DefaultLocationService.processEntities (/Users/yungc/Documents/Projects/backstage/node_modules/@backstage/plugin-catalog-backend/dist/index.cjs.js:2188:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async DefaultLocationService.dryRunCreateLocation (/Users/yungc/Documents/Projects/backstage/node_modules/@backstage/plugin-catalog-backend/dist/index.cjs.js:2219:22)\n at async /Users/yungc/Documents/Projects/backstage/node_modules/@backstage/plugin-catalog-backend/dist/index.cjs.js:3512:22"},"request":{"method":"POST","url":"/locations?dryRun=true"},"response":{"statusCode":400}}

👟 Reproduction steps

  1. Run Backstage upgrade as per instructed
  2. Run yarn dev
  3. Error is printed on the console

During runtime,

  1. Go to catalog-import page.
  2. Try to import a catalog
  3. Observe error output on the screen

📃 Provide the context for the Bug.

No response

🖥️ Your Environment

OS: Darwin 22.2.0 - darwin/x64
node: v18.12.1
yarn: 1.22.19
cli: 0.22.1 (installed)
backstage: 1.10.1

Dependencies:
@backstage/app-defaults 1.1.0
@backstage/backend-app-api 0.3.1
@backstage/backend-common 0.18.1
@backstage/backend-plugin-api 0.3.1
@backstage/backend-tasks 0.4.2
@backstage/backend-test-utils 0.1.33
@backstage/catalog-client 1.3.0
@backstage/catalog-model 1.1.5
@backstage/cli-common 0.1.11
@backstage/cli 0.22.1
@backstage/config-loader 1.1.8
@backstage/config 1.0.6
@backstage/core-app-api 1.4.0
@backstage/core-components 0.11.2, 0.12.3
@backstage/core-plugin-api 1.3.0
@backstage/dev-utils 1.0.11
@backstage/errors 1.1.4
@backstage/integration-aws-node 0.1.1
@backstage/integration-react 1.1.9
@backstage/integration 1.4.2
@backstage/plugin-api-docs 0.8.14
@backstage/plugin-app-backend 0.3.41
@backstage/plugin-auth-backend 0.17.4
@backstage/plugin-auth-node 0.2.10
@backstage/plugin-badges-backend 0.1.35
@backstage/plugin-badges 0.2.38
@backstage/plugin-bazaar-backend 0.2.4
@backstage/plugin-bazaar 0.2.3
@backstage/plugin-catalog-backend-module-github 0.2.4
@backstage/plugin-catalog-backend 1.7.1
@backstage/plugin-catalog-common 1.0.10
@backstage/plugin-catalog-graph 0.2.26
@backstage/plugin-catalog-import 0.9.4
@backstage/plugin-catalog-node 1.3.2
@backstage/plugin-catalog-react 1.2.4
@backstage/plugin-catalog 1.7.2
@backstage/plugin-cost-insights-common 0.1.1
@backstage/plugin-cost-insights 0.12.3
@backstage/plugin-events-node 0.2.2
@backstage/plugin-explore-common 0.0.1
@backstage/plugin-explore-react 0.0.25
@backstage/plugin-explore 0.3.45
@backstage/plugin-github-actions 0.5.14
@backstage/plugin-github-issues 0.2.3
@backstage/plugin-github-pull-requests-board 0.1.8
@backstage/plugin-home 0.4.30
@backstage/plugin-kubernetes-backend 0.9.2
@backstage/plugin-kubernetes-common 0.5.1
@backstage/plugin-kubernetes 0.7.7
@backstage/plugin-org 0.6.4
@backstage/plugin-permission-common 0.7.3
@backstage/plugin-permission-node 0.7.4
@backstage/plugin-permission-react 0.4.9
@backstage/plugin-proxy-backend 0.2.35
@backstage/plugin-scaffolder-backend 1.10.1
@backstage/plugin-scaffolder-common 1.2.4
@backstage/plugin-scaffolder-react 1.0.1
@backstage/plugin-scaffolder 1.10.1
@backstage/plugin-search-backend-module-elasticsearch 1.1.2
@backstage/plugin-search-backend-module-pg 0.5.2
@backstage/plugin-search-backend-node 1.1.2
@backstage/plugin-search-backend 1.2.2
@backstage/plugin-search-common 1.2.1
@backstage/plugin-search-react 1.4.0
@backstage/plugin-search 1.0.7
@backstage/plugin-shortcuts 0.3.6
@backstage/plugin-sonarqube-backend 0.1.6
@backstage/plugin-sonarqube-react 0.1.1
@backstage/plugin-sonarqube 0.6.2
@backstage/plugin-stack-overflow 0.1.10
@backstage/plugin-tech-insights-backend-module-jsonfc 0.1.25
@backstage/plugin-tech-insights-backend 0.5.7
@backstage/plugin-tech-insights-common 0.2.9
@backstage/plugin-tech-insights-node 0.3.9
@backstage/plugin-tech-insights 0.3.6
@backstage/plugin-tech-radar 0.6.0
@backstage/plugin-techdocs-addons-test-utils 1.0.9
@backstage/plugin-techdocs-backend 1.5.2
@backstage/plugin-techdocs-module-addons-contrib 1.0.9
@backstage/plugin-techdocs-node 1.4.5
@backstage/plugin-techdocs-react 1.1.2
@backstage/plugin-techdocs 1.4.3
@backstage/plugin-todo-backend 0.1.38
@backstage/plugin-todo 0.2.16
@backstage/plugin-user-settings 0.6.2
@backstage/release-manifests 0.0.8
@backstage/test-utils 1.2.4
@backstage/theme 0.2.16
@backstage/types 1.0.2
@backstage/version-bridge 1.0.3

👀 Have you spent some time to check if this bug has been raised before?

  • I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

Are you willing to submit PR?

None

@darrenyung darrenyung added the bug Something isn't working label Jan 25, 2023
@benjdlambert
Copy link
Member

Hey 👋

Not seen this reported before, but I think we're going to need a little more information on what's going on and what the actual error is. It looks like it's something to do with the Github App credentials or something.

Would it be possible for you to try and debug this a little more to get us some more detailed logging information. Possibly by digging in node_modules a little bit and adding some logs to get some more info on what's going on.

It could be some incompatibility between some of the octokit libraries, or something different it's hard to tell with these error messages.

@darrenyung
Copy link
Author

darrenyung commented Jan 27, 2023

Thanks @benjdlambert . I've managed to narrow it down after an exhaustive check. I hope this is right as what looked like could be a network issue but I'm a little sceptical as why is the newer version producing this.

Log produced in @octokit/request package.

[1]   error: TypeError: fetch failed
[1]       at Object.fetch (node:internal/deps/undici/undici:11118:11)
[1]       at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[1]       at async hook (/Users/yungc/Documents/Projects/backstage/node_modules/@octokit/auth-app/dist-node/index.js:297:18)
[1]       at async Object.next (/Users/yungc/Documents/Projects/backstage/node_modules/@octokit/plugin-paginate-rest/dist-node/index.js:67:28)
[1]       at async GithubAppManager.getInstallationData (/Users/yungc/Documents/Projects/backstage/node_modules/@backstage/integration/dist/index.cjs.js:1311:30)
[1]       at async /Users/yungc/Documents/Projects/backstage/node_modules/@backstage/integration/dist/index.cjs.js:1278:45
[1]       at async Cache.getOrCreateToken (/Users/yungc/Documents/Projects/backstage/node_modules/@backstage/integration/dist/index.cjs.js:1228:34)
[1]       at async Promise.all (index 0)
[1]       at async GithubAppCredentialsMux.getAppToken (/Users/yungc/Documents/Projects/backstage/node_modules/@backstage/integration/dist/index.cjs.js:1349:21)
[1]       at async _SingleInstanceGithubCredentialsProvider.getCredentials (/Users/yungc/Documents/Projects/backstage/node_modules/@backstage/integration/dist/index.cjs.js:1399:17) {
[1]     cause: Error: getaddrinfo ENOTFOUND api.github.com
[1]         at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26) {
[1]       errno: -3008,
[1]       code: 'ENOTFOUND',
[1]       syscall: 'getaddrinfo',
[1]       hostname: 'api.github.com'
[1]     }
[1]   }

I've done a search for @octokit/request package being used in the project, not sure where @lerna is but the request log is coming from the v6.2.3

yarn list v1.22.19
├─ @lerna/github-client@4.0.0
│  ├─ @octokit/request-error@2.1.0
│  └─ @octokit/request@5.6.3
├─ @octokit/request-error@3.0.3
├─ @octokit/request@6.2.3
├─ @roadiehq/backstage-plugin-github-insights@2.3.3
│  ├─ @octokit/request-error@2.1.0
│  └─ @octokit/request@5.6.3
└─ @roadiehq/backstage-plugin-github-pull-requests@2.4.3
   ├─ @octokit/request-error@2.1.0
   └─ @octokit/request@5.6.3
✨  Done in 1.51s.

UPDATE: Just fyi, I'm behind a corporate proxy. Have tested being outside the proxy and it works but just not in and it seems to be pointing to the fetch module.

@benjdlambert
Copy link
Member

benjdlambert commented Jan 27, 2023

I think that the error is here:

[1]     cause: Error: getaddrinfo ENOTFOUND api.github.com

The proxy is getting in the way. Looks like is not respecting the proxies.

Have you looked at these docs? https://github.com/backstage/backstage/blob/master/contrib/docs/tutorials/help-im-behind-a-corporate-proxy.md

@darrenyung
Copy link
Author

Hi @benjdlambert , I have and have set up the package global-agent/bootstrap as per doc in previous version prior to the upgrade and that worked. I'll need to keep digging to see why its not respecting the proxy in this newer version.

@benjdlambert
Copy link
Member

Hmm yeah that's interesting, I wonder if they've changed how they make requests from @octokit/request@5.6.3 -> @octokit/request@6.2.3 and now the global-agent doesn't work anymore. If you can find out some more info or a possible route forward would be much appreciated 🙏

@darrenyung
Copy link
Author

darrenyung commented Jan 30, 2023

Just a quick update, I've downgraded Node 18 to Node 16 (specifically 16.13) and have been presented with a different error. It seems the response for the call to Github API and it returned a parsing error when processing with response.json().

HttpError: Invalid response body while trying to fetch https://api.github.com/app/installations: read ECONNRESET
contentType: 'application/json; charset=utf-8',
[1]   url: 'https://api.github.com/app/installations',
[1]   status: 200,
[1]   response: Response {
[1]     size: 0,
[1]     timeout: 0,
[1]     [Symbol(Body internals)]: { body: [Gunzip], disturbed: false, error: null },
[1]     [Symbol(Response internals)]: {
[1]       url: 'https://api.github.com/app/installations',
[1]       status: 200,
[1]       statusText: 'OK',
[1]       headers: [Headers],
[1]       counter: 0
[1]     }
[1]   }

Wondering if its todo with the socket hang up ECONNRESET since the response size is literally 0 bytes. Additionally, same request done via Postman with the same credentials returned the expected response flawlessly.

@freben
Copy link
Member

freben commented Feb 2, 2023

Just to confirm - are you still behind the corporate proxy here, as the ECONNRESET might hint at?

@darrenyung
Copy link
Author

darrenyung commented Feb 3, 2023

Hi there! Yes, I'm still behind corporate proxy and that is not going anywhere I'm afraid. I'm still digging into some of the issues and testing a different envs and it appears to have mixed results.

The ECONNRESET is bizarre as when I was debugging, the error fails when it tries to perform response.json() . However, when I write my own stream buffer reader, it successfully produced the json payload as expected coming from github but then ends with the ECONNRESET. The error message is terribly misleading.

I have one question, does this change might have any impact for Backstage configured backend auth with Github Apps? We never needed to use PAT tokens since we used Github App instead.

UPDATE: Here's the response I get on the UI side of catalog-import page.

{"error":{"name":"InputError","message":"Error: Unable to read url, HttpError: fetch failed"},"request":{"method":"POST","url":"/locations?dryRun=true"},"response":{"statusCode":400}}

@benjdlambert
Copy link
Member

OK I think i'm starting to understand what's going on here. The bump to @octokit/request now will use the native implementation of fetch if it's available, and it looks like that is the not being wrapped up in the http proxy stuff from global agent. octokit/request.js#481

Then using this information, I had a look at the global-fetch repo and found this: gajus/global-agent#52

Looks like there might be a workaround here at the bottom, but it's not great. I think once we have the backend system we can maybe have the ability to have a fetch service that can be used everywhere that can add these settings so we wont need the global fetch anymore.

I'd look at trying to use that workaround to see if it works for you 🤞 and report back.

@benjdlambert
Copy link
Member

To be clear also, I think you will need both global-agent and the code for the undici workaround for native fetch in node js.

@darrenyung
Copy link
Author

Hi there! Apologies for late response to this. I've given the undici solution a go. It works on my local dev which is fantastic! Unfortunately, when I promote it through my envs, it still fails with the same error. I've pulled the docker image and run as my local container and it doesn't work. 😭

However, I've managed to progress through but only by running on Node16 instead of Node18. At least this buys me further time (a couple of months maybe?) to locate a fix for now and unblocked the upgrade/development of my org's Backstage instance.

OK I think i'm starting to understand what's going on here. The bump to @octokit/request now will use the native implementation of fetch if it's available, and it looks like that is the not being wrapped up in the http proxy stuff from global agent. octokit/request.js#481

Then using this information, I had a look at the global-fetch repo and found this: gajus/global-agent#52

Looks like there might be a workaround here at the bottom, but it's not great. I think once we have the backend system we can maybe have the ability to have a fetch service that can be used everywhere that can add these settings so we wont need the global fetch anymore.

I'd look at trying to use that workaround to see if it works for you 🤞 and report back.

@github-actions
Copy link
Contributor

github-actions bot commented May 5, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label May 5, 2023
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale May 12, 2023
@darrenyung
Copy link
Author

darrenyung commented Sep 13, 2023

For the interest of closure to this issue. I've finally managed to resolve it. Not by using undici package but a setting in Node.

export NODE_OPTIONS="--no-experimental-fetch"

This turns off the experimental fetch api in native node18 and uses the dependency package fetch module instead. This in-turns respects the proxy setting as per documentation. Avoid using experimental feature, I'd say. 😸

Source: node-fetch/node-fetch#1566

@cklewin
Copy link
Contributor

cklewin commented Nov 20, 2023

For the interest of closure to this issue. I've finally managed to resolve it. Not by using undici package but a setting in Node.

export NODE_OPTIONS="--no-experimental-fetch"

This turns off the experimental fetch api in native node18 and uses the dependency package fetch module instead. This in-turns respects the proxy setting as per documentation. Avoid using experimental feature, I'd say. 😸

Source: node-fetch/node-fetch#1566

This also fixed my GitHub Discovery failures - however, the scaffolder was failing to publish:

    ReferenceError: Response is not defined
        at browserDeflate (/app/node_modules/isomorphic-git/index.cjs:4889:35)6
        at deflate (/app/node_modules/isomorphic-git/index.cjs:4882:7)7
        at _writeObject (/app/node_modules/isomorphic-git/index.cjs:4918:32)8
        at async /app/node_modules/isomorphic-git/index.cjs:5044:199
        at async Promise.allSettled (index 0)10
        at async addToIndex (/app/node_modules/isomorphic-git/index.cjs:5049:27)11
        at async Promise.all (index 0)12
        at async /app/node_modules/isomorphic-git/index.cjs:5025:913
        at async Promise.allSettled (index 0)14
        at async addToIndex (/app/node_modules/isomorphic-git/index.cjs:5049:27)15
        at async /app/node_modules/isomorphic-git/index.cjs:993:16

I fixed both the scaffolder and discovery, and also removed the need for the no-experimental-fetch flag with NodeJS 20, by implementing global-agent and undici setGlobalDispatcher:

yarn workspace backend add undici

packages/backend/src/index.ts:

import { bootstrap as globalAgentBootstrap } from "global-agent";
import { ProxyAgent, setGlobalDispatcher } from "undici";
...

async function main() {
  const logger = getRootLogger();
  const config = await loadBackendConfig({ argv: process.argv, logger });

  if (process.env.HTTP_PROXY) {
    logger.info(`Using proxy agent for HTTP requests: ${process.env.HTTP_PROXY}`);
    globalAgentBootstrap();
    setGlobalDispatcher(new ProxyAgent(process.env.HTTP_PROXY));
  } else {
    logger.info("Not using proxy agent for HTTP requests, HTTP_PROXY not set");
  }
...
  const createEnv = makeCreateEnv(config);
...

@jrtaylorJH
Copy link
Contributor

For the interest of closure to this issue. I've finally managed to resolve it. Not by using undici package but a setting in Node.

export NODE_OPTIONS="--no-experimental-fetch"

This turns off the experimental fetch api in native node18 and uses the dependency package fetch module instead. This in-turns respects the proxy setting as per documentation. Avoid using experimental feature, I'd say. 😸

Source: node-fetch/node-fetch#1566

This got me a more informative error, but same overall problem.

HttpError: request to https://api.github.com/app/installations failed, reason: Client network socket disconnected before secure TLS connection was established

@darrenyung
Copy link
Author

For the interest of closure to this issue. I've finally managed to resolve it. Not by using undici package but a setting in Node.

export NODE_OPTIONS="--no-experimental-fetch"

This turns off the experimental fetch api in native node18 and uses the dependency package fetch module instead. This in-turns respects the proxy setting as per documentation. Avoid using experimental feature, I'd say. 😸
Source: node-fetch/node-fetch#1566

This got me a more informative error, but same overall problem.

HttpError: request to https://api.github.com/app/installations failed, reason: Client network socket disconnected before secure TLS connection was established

I noticed Node20 has reached LTS. Would upgrading to that resolve the issue?

@darrenyung
Copy link
Author

For the interest of closure to this issue. I've finally managed to resolve it. Not by using undici package but a setting in Node.

export NODE_OPTIONS="--no-experimental-fetch"

This turns off the experimental fetch api in native node18 and uses the dependency package fetch module instead. This in-turns respects the proxy setting as per documentation. Avoid using experimental feature, I'd say. 😸
Source: node-fetch/node-fetch#1566

This also fixed my GitHub Discovery failures - however, the scaffolder was failing to publish:

    ReferenceError: Response is not defined
        at browserDeflate (/app/node_modules/isomorphic-git/index.cjs:4889:35)6
        at deflate (/app/node_modules/isomorphic-git/index.cjs:4882:7)7
        at _writeObject (/app/node_modules/isomorphic-git/index.cjs:4918:32)8
        at async /app/node_modules/isomorphic-git/index.cjs:5044:199
        at async Promise.allSettled (index 0)10
        at async addToIndex (/app/node_modules/isomorphic-git/index.cjs:5049:27)11
        at async Promise.all (index 0)12
        at async /app/node_modules/isomorphic-git/index.cjs:5025:913
        at async Promise.allSettled (index 0)14
        at async addToIndex (/app/node_modules/isomorphic-git/index.cjs:5049:27)15
        at async /app/node_modules/isomorphic-git/index.cjs:993:16

I fixed both the scaffolder and discovery, and also removed the need for the no-experimental-fetch flag with NodeJS 20, by implementing global-agent and undici setGlobalDispatcher:

yarn workspace backend add undici

packages/backend/src/index.ts:

import { bootstrap as globalAgentBootstrap } from "global-agent";
import { ProxyAgent, setGlobalDispatcher } from "undici";
...

async function main() {
  const logger = getRootLogger();
  const config = await loadBackendConfig({ argv: process.argv, logger });

  if (process.env.HTTP_PROXY) {
    logger.info(`Using proxy agent for HTTP requests: ${process.env.HTTP_PROXY}`);
    globalAgentBootstrap();
    setGlobalDispatcher(new ProxyAgent(process.env.HTTP_PROXY));
  } else {
    logger.info("Not using proxy agent for HTTP requests, HTTP_PROXY not set");
  }
...
  const createEnv = makeCreateEnv(config);
...

Can confirm the issue is also no longer a problem for me after reverting the earlier changes and implementing the suggestion above.

@freben
Copy link
Member

freben commented Feb 2, 2024

@darrenyung Thanks! I made this to follow up #22658

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working stale
Projects
None yet
Development

No branches or pull requests

5 participants