Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internet connection stops working when using Lens #6063

Open
cachila opened this issue Aug 19, 2022 · 76 comments
Open

Internet connection stops working when using Lens #6063

cachila opened this issue Aug 19, 2022 · 76 comments
Labels
bug Something isn't working p1

Comments

@cachila
Copy link

cachila commented Aug 19, 2022

Describe the bug
From time to time when using Lens, the internet connection stops working on my computer. After closing Lens, it starts working again.

To Reproduce
Steps to reproduce the behavior:

  1. Open Lens.
  2. Keep using it and after some hours, Internet will stop working.

Expected behavior
Internet connection keeps active.

Environment (please complete the following information):

  • Lens Version: 6.0.1-latest.20220810.2
  • OS: OSX 12.4
  • Installation method: Installed from .dmg

Additional context
Usually it happens once in a work day (8 - 9 hours).

@cachila cachila added the bug Something isn't working label Aug 19, 2022
@matti
Copy link

matti commented Aug 19, 2022

hmm, I may have experienced this too. weird.

@Nokel81
Copy link
Collaborator

Nokel81 commented Aug 22, 2022

We would like to better understand how you are using Lens to see if we can track down why Lens would be able to do this.

Are you connecting to a lot of clusters at once? Do you use other features such as Lens Spaces? Could you use a network analyzer to display the active connections that Lens is making when this next happens?

@cachila
Copy link
Author

cachila commented Aug 23, 2022

  • Are you connecting to a lot of clusters at once? -> Maybe 5 or 6
  • Do you use other features such as Lens Spaces? -> I'm logged in to Lens Spaces but don't use it. I just use my Local profile
  • Could you use a network analyzer to display the active connections that Lens is making when this next happens? It's been two days without the issue. I'll try to check the network when this happens again.

@BenNF
Copy link

BenNF commented Aug 24, 2022

I've had this happen as well maybe 3-5 times over 2 weeks, but haven't been able to figure out any specific behavior that triggers it.

Lens version: 6.0.1-latest.20220810.2
OS: OSX 12.5.1
Installation: Install from .dmg, arm64 version

  • Are you connecting to a lot of clusters at once? No only 1
  • Do you use other features such as Lens Spaces? No, not enabled

I'm mostly only looking at the logs of pods and will have several tabs open at once for an extended period, so that my best guess for a cause.

@OuFinx
Copy link

OuFinx commented Aug 25, 2022

The same issue.

Two days I tried to understand where is the problem and figured out that when my Lens is open my internet connection after ~10-15 minutes stop working.

Also yesterday there was a situation, Lens was frozen, and I couldn’t close it even through the activity monitor, after which my laptop froze and I had to turn it off while holding the button

Lens version: 6.0.1-latest.20220810.2
OS: OSX 12.5.1
Macbook: Air 2020 M1

I ran ping command and what happened during this weird situations

image

Also, were a lot of situations where ping showed me the same like my wifi didn't work (can't make ping request)

I closed Lens and ping started work again. I opened Lens, worked ~10 mins, and everything repeated

@OuFinx
Copy link

OuFinx commented Aug 25, 2022

In continuation of the previous message, just caught the same thing

CleanShot 2022-08-25 at 12 38 58@2x

@cachila
Copy link
Author

cachila commented Aug 25, 2022

Also yesterday there was a situation, Lens was frozen, and I couldn’t close it even through the activity monitor, after which my laptop froze and I had to turn it off while holding the button

I happened the same to me a few weeks ago. At that time I didn't realice it was Lens.

@nicobistolfi
Copy link

Same issue here, from time to time the internet connection drops, and after closing Lens, everything goes back to normal.

version: Lens: 6.0.1-latest.20220810.2

@Nokel81 I'm connecting to multiple clusters at once, up to 3-4 at the same time and I use Lens Space for 1 cluster.

@msa0311 msa0311 added the p1 label Aug 26, 2022
@jakolehm
Copy link
Contributor

jakolehm commented Aug 26, 2022

I have seen this also once, didn't realize it was Lens fault. Is everyone else also using M1 (arm64)?

@cachila
Copy link
Author

cachila commented Aug 26, 2022

@jakolehm Indeed using M1

@aleksfront aleksfront added this to the 6.0.3 milestone Aug 29, 2022
@Nokel81 Nokel81 modified the milestones: 6.0.3, 6.1.0, 6.2.0 Sep 6, 2022
@Kulagin-G
Copy link

I've also seen this issue twice this week but unfortunately, I don't have specific details at the moment.

MacOS Monterey 12.1 (21C52)
Apple M1 Pro

@monoxane
Copy link

I'm hitting this several times a day, the more clusters I have connected the worse it is, and the more resources in those clusters the worse it is.

Having 1 cluster open that has several hundred pods on 10 nodes will cause a complete network failure in under 40 minutes every time.

It seems to be related to the total amount of resources in currently connected clusters.

M1 MBP with macOS 12.5.1 (21G83)

@monoxane
Copy link

Just had it happen again while monitoring ping and netstat, no unusual amount of sockets opened, once I disconnected the clusters and closed lens everything works again, will add more diagnostics and try catch more data.

@Kulagin-G
Copy link

Kulagin-G commented Sep 12, 2022

I've been connected to EKS cluster but in a one-moment connection session expired, right after that Lens UI got stuck (I think it's electron part) and my VPN connection was dropped. Maybe it's just a coincidence.
In Lens logs I've found only these messages, where xxxxxxxx is a hidden EKS resource. Hope it will be helpful cuz have no time to investigate this issue.

info: [STORE]: SAVING /Users/gkulagin/Library/Application Support/Lens/lens-cluster-store.json
info: [LENS-SPACES-EXTENSION]: 9/12/2022, 1:40:05 PM User has logged out or didn't agree to connect to Lens Cloud, do nothing on resume. .
error: [CLUSTER]: Failed to connect to "xxxxxxxx": StatusCodeError: 500 - "read tcp xxxxxx: read: connection re
set by peer\n"
info: [STORE]: SAVING /Users/gkulagin/Library/Application Support/Lens/lens-cluster-store.json
info: [LENS-SPACES-EXTENSION]: 9/12/2022, 1:40:05 PM User has logged out or didn't agree to connect to Lens Cloud, do nothing on resume. .
info: [LENS-SPACES-EXTENSION]: 9/12/2022, 1:40:05 PM Broadcasted spaces change to SpacesListener on renderer
warn: [LENS-SPACES-EXTENSION]: 9/12/2022, 1:40:05 PM No spaces in SpaceSyncer disposeSpacesReaction, not changing selected space.
info: [LENS-SPACES-EXTENSION]: (from:TokenRefresher) 9/12/2022, 1:40:05 PM Detect token expires in 26597s (26597285.00008583ms), set refresh interval.
info: [LENS-SPACES-EXTENSION]: (from:TokenRefresher) 9/12/2022, 1:40:05 PM Start token refresh iterator... scheduled next refresh 9/12/2022, 8:54:31 PM
info: [CLUSTER]: refresh {"accessible":false,"disconnected":false,"id":"3ad34ed6e316c60c111d99f54ad4134a","name":"xxxxxxxx","online":false,"ready":true}
info: [CLUSTER]: refresh {"accessible":true,"disconnected":false,"id":"3ad34ed6e316c60c111d99f54ad4134a","name":"xxxxxxxx","online":true,"ready":true}
info: [CLUSTER]: refreshMetadata {"accessible":true,"disconnected":false,"id":"3ad34ed6e316c60c111d99f54ad4134a","name":"xxxxxxxx","online":true,"ready":true}
error: [UPDATE-APP/CHECK-FOR-UPDATES] net::ERR_NETWORK_CHANGED {"stack":"Error: net::ERR_NETWORK_CHANGED\n    at SimpleURLLoaderWrapper.<anonymous> (node:electron/js2c/browser_init:105:7068)\n    at SimpleURLLoaderWrapper.emit (node:events:394:28)\n    at SimpleURLLoaderWrapper.emit (node:domain:470:12)"}
info: [CLUSTER-MANAGER]: network is offline
warn: [LENS-SPACES-EXTENSION]: 9/12/2022, 1:55:46 PM isLensCloudStatusOk returns false
info: [LENS-SPACES-EXTENSION]: (from:TokenRefresher) 9/12/2022, 1:55:46 PM Token refresh iterator stopped by ok from false => true
info: [LENS-SPACES-EXTENSION]: 9/12/2022, 1:55:46 PM Offline space Added
info: [LENS-SPACES-EXTENSION]: 9/12/2022, 1:55:46 PM Broadcasted spaces change to SpacesListener on renderer
info: [LENS-SPACES-EXTENSION]: 9/12/2022, 1:55:46 PM Selected the offline space...
info: [LENS-SPACES-EXTENSION]: 9/12/2022, 1:55:46 PM setting resumeState to SUSPENDING
info: [LENS-SPACES-EXTENSION]: 9/12/2022, 1:55:46 PM setting resumeState to undefined
error: [CLUSTER]: Failed to connect to "xxxxxxxx": StatusCodeError: 500 - "dial tcp: lookup xxxxxxxx on [::1]:53: read udp [::1]:57701->[::1]:53: read: connection refused\n"
error: [CLUSTER]: Failed to connect to "xxxxxxxx": StatusCodeError: 500 - "dial tcp: lookup xxxxxxxx on [::1]:53: read udp [::1]:57701->[::1]:53: read: connection refused\n"
info: [CLUSTER-MANAGER]: network is online
error: [CLUSTER]: Failed to connect to "xxxxxxxx": StatusCodeError: 500 - "dial tcp: lookup xxxxxxxx on [::1]:53: read udp [::1]:57715->[::1]:53: read: connection refused\n"
info: [LENS-SPACES-EXTENSION]: 9/12/2022, 1:56:06 PM User has logged out or didn't agree to connect to Lens Cloud, do nothing on resume. .
info: [LENS-SPACES-EXTENSION]: 9/12/2022, 1:56:06 PM Broadcasted spaces change to SpacesListener on renderer
warn: [LENS-SPACES-EXTENSION]: 9/12/2022, 1:56:06 PM No spaces in SpaceSyncer disposeSpacesReaction, not changing selected space.
info: [LENS-SPACES-EXTENSION]: (from:TokenRefresher) 9/12/2022, 1:56:06 PM Detect token expires in 25636s (25636170.000076294ms), set refresh interval.
info: [LENS-SPACES-EXTENSION]: (from:TokenRefresher) 9/12/2022, 1:56:06 PM Start token refresh iterator... scheduled next refresh 9/12/2022, 8:54:50 P

@monoxane
Copy link

Opening lens again after its happened but before a reboot causes it to happen immediately.

@monoxane
Copy link

Seems to be related to the amount of kubernetes updates being processed, potentially something broken in a watch loop.
I've left lens open in the background in the state that usually causes issues for a few hours and it has not done it, started working with the cluster in terminal with kubectl and after applying a bunch of manifests it immediately killed my internet.

@cachila
Copy link
Author

cachila commented Oct 6, 2022

Any news regarding this issue?

@wallacepf
Copy link

Same Thing here. Lost the connection many times during the day.

Latest version of Lens
M1 MBP with OSX 12.6.1

@matti
Copy link

matti commented Oct 28, 2022

@wallacepf and others - are you using Lens with multiple clusters?

@wallacepf
Copy link

In my case, nope. I have a single AKS for demo purposes. Brand new BTW, so I don't believe the number of objects influences the issue.

@gaspo53
Copy link

gaspo53 commented Nov 7, 2022

Hi! Same happening here, it's like the app does something with the host network and overflows something

@cachila
Copy link
Author

cachila commented Nov 8, 2022

Any updates on this issue? It's making the use of Lens a bit frustrating.

@monoxane
Copy link

monoxane commented Nov 9, 2022

I've switched to using OpenLens directly instead of the proprietary binary and have not had a problem since. I think it's something in their telemetry/data harvesting addons that's broken.

@Nokel81 Nokel81 modified the milestones: 6.4.9, 6.4.10, 6.4.11 Mar 22, 2023
@Iku-turso Iku-turso modified the milestones: 6.4.11, 6.4.12, 6.4.13 Mar 27, 2023
@Nokel81 Nokel81 removed this from the 6.4.13 milestone Apr 3, 2023
@smcenlly
Copy link

smcenlly commented Apr 4, 2023

We had a similar issue in one of our products, thought you might be interested in what we found so that it might help you find/fix your problem:

The problem was actually a bug/behavior in one of our dependencies, node-fetch and they way that we were using it. To optimize performance, we were returning content immediately without processing the response body in some cases (the requirement to process the body is not explicitly mentioned in their docs). This is a known bug that can lead to sockets/file descriptors being left open and is apparently an issue on MacOS and Linux (see node-fetch/node-fetch#1673).

@bruno-lopes
Copy link

Any news on this?

@boogi
Copy link

boogi commented Apr 13, 2023

Just found this issue while searching google for "isLensCloudStatusOk". Similar issue here (Ubuntu 20.04, Lens 2023.4.60821-latest) while trying to run LDK (Lens Desktop Kube): after Lens was open for few days, LDK refused to start. Relevant logs provided below.
Just closing and starting afresh Lens helped.

Trying to create LDK (first time, Lens didn't have image locally):

error: [LENS-DESKTOP-KUBE] failed to start vm: RequestError: getaddrinfo EAI_AGAIN downloads.k8slens.dev
    at ClientRequest.<anonymous> (/opt/Lens/resources/app.asar/node_modules/@lensapp/lens-desktop-kube-lens-extension/dist/main.js:2:485643)
    at Object.onceWrapper (node:events:646:26)
    at ClientRequest.emit (node:events:538:35)
    at ClientRequest.emit (node:domain:475:12)
    at e.emit (/opt/Lens/resources/app.asar/node_modules/@lensapp/lens-desktop-kube-lens-extension/dist/main.js:2:263345)
    at TLSSocket.socketErrorListener (node:_http_client:442:9)
    at TLSSocket.emit (node:events:526:28)
    at TLSSocket.emit (node:domain:475:12)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
  code: 'EAI_AGAIN',
  timings: [Object]
}

Unable to configure new cluster because Lens can't fetch list of supported k8s versions (LDK > Settings > Create New Profile):

error: [LENS-DESKTOP-KUBE] unable to fetch supported versions list Unexpected exception [Lens Platform SDK] {"errorCode":null,"rawException":{"code":"EAI_AGAIN","config":{"env":{},"headers":{"Accept":"application/json, text/plain, */*","Authorization":"Bearer: <cut>", "User-Agent":"Lens/2023.4.60821-latest lens-desktop-kube-lens-extension/0.25.0 (Linux)"},"httpsAgent":{"_events":{},"_eventsCount":0,"freeSockets":{},"maxFreeSockets":1,"maxSockets":1,"maxTotalSockets":null,"options":{},"pacProxyAgentOptions":{},"proxyStringCache":{},"requests":{},"sockets":{},"timeout":null},"maxBodyLength":-1,"maxContentLength":-1,"method":"get","timeout":0,"transformRequest":[null],"transformResponse":[null],"transitional":{"clarifyTimeoutError":false,"forcedJSONParsing":true,"silentJSONParsing":true},"url":"https://api.k8slens.dev/lens-desktop-kube/k0s-versions.json","xsrfCookieName":"XSRF-TOKEN","xsrfHeaderName":"X-XSRF-TOKEN"},"message":"getaddrinfo EAI_AGAIN api.k8slens.dev","name":"Error","stack":"Error: getaddrinfo EAI_AGAIN api.k8slens.dev\n    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)","status":null},"stack":"Error: Unexpected exception [Lens Platform SDK]\n    at t.throwExpected (/opt/Lens/resources/app.asar/node_modules/@lensapp/lens-desktop-kube-lens-extension/dist/main.js:2:657585)\n    at process.processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async LensDesktopKubeService.getK0sVersions (/opt/Lens/resources/app.asar/node_modules/@lensapp/lens-desktop-kube-lens-extension/dist/main.js:2:619443)\n    at async /opt/Lens/resources/app.asar/node_modules/@lensapp/lens-desktop-kube-lens-extension/dist/main.js:2:967034\n    at async t.getK0sVersions (/opt/Lens/resources/app.asar/node_modules/@lensapp/lens-desktop-kube-lens-extension/dist/main.js:2:966953)\n    at async /opt/Lens/resources/app.asar/node_modules/@k8slens/core/static/build/library/main.js:8829:32\n    at async node:electron/js2c/browser_init:189:563"}

And from time to time in logs also this line that thankfully brought me here:

warn: [LENS-SPACES-EXTENSION]: 4/13/2023, 4:25:46 PM isLensCloudStatusOk returns false

@jakolehm
Copy link
Contributor

I can see leakage of socket/file descriptors with latest stable version but not with latest alpha versions -> I believe this is fixed in the master branch. Let's close it once we can verify the fix with beta/stable builds.

@matti
Copy link

matti commented Apr 21, 2023

when this happens, please run lsof -p <lens-helper-renderer-pid> | grep TCP | wc -l and post here

@matti
Copy link

matti commented Apr 21, 2023

image

@matti
Copy link

matti commented Apr 24, 2023

image

2023.4.141316-latest

@jwklijnsma
Copy link

so maybe the problem is in Console Ninja if lens using it need to be uppgrade to v1.0.105+.

@gabriel-mirantis
Copy link
Contributor

This has also been described here hashicorp/terraform#31467 and the current workaround seems to be related to disabling IPv6

@jakolehm
Copy link
Contributor

Likely related to golang issue described here: golang/go#52839

@ischenxin
Copy link

Same issue with me. Is there any official person who can reply to this question? This problem has been happening for a year, that's unacceptable.

@matias-cova-skydropx
Copy link

same issue with me

@dshershov
Copy link

one year with this issue
connections are dropped every 15 minutes
looks like we need to add new feature with namespace...

@msa0311
Copy link
Contributor

msa0311 commented Feb 6, 2024

Firstly, thank you all for bringing this issue to our attention and for all your patience as we work to resolve it. We understand how disruptive this can be to your workflow and appreciate the detailed reports you've providing.

We wanted to update the community on this particular issue regarding network connectivity being blocked after using Lens for an extended period. Our team has been actively investigating this matter since quite a while already, and it's been challenging due to the sporadic nature of the problem. Previously we were able to reproduce the issue related to a third-party Golang binary that Lens utilized in lens-proxy for managing network connections. This issue we tired to fix by using the most recent golang version to compile this binary. (see golang/go#52839)

Here are some other tools that have the same issue:

However, as you've experienced, the problem still occurs, albeit rarely. This indicates that there might be more underlying factors contributing to the issue that we've yet to identify.

To move closer to a resolution, we need more data that can only come from occurrences in diverse user environments. If you encounter this issue again, could you please provide us with the following information?

  • Have you been using IPV6 network?
  • Did you configure IPV6 to "link-local" DNS resolve for this network?
  • Are you using MacOS with Apple Silicon Chips (M1, M2)?
  • Are you using the latest version of Lens?
  • Details of the operations being performed within Lens leading up to the issue?

This information will be crucial for us to attempt to replicate the issue under similar conditions. We are committed to resolving this issue and improving Lens for all users.

@dshershov
Copy link

Hello,
Thanks @msa0311 !
So:

  • ipv6 was not use on my PC
  • MacOS with m1Pro chip
  • Latest drop 4 days ago with the next version of Lens - v2024.1.170738-latest
  • Details? hmm, i'm use Lens for default k8s operations such as port-forwarding, logging, checks some manifests, network settings, CRDs and other

@smcenlly
Copy link

smcenlly commented Feb 6, 2024

@msa0311 - did you see my comment above regarding the use of the JavaScript node-fetch package?

I saw Lens is an electron app. I assume you're using node-fetch for network communication and may have the same issue we had. We had exactly the same behavior as you are describing in our own product and it was very hard to track down.

Basically, you will have the problem if you use node-fetch and do not consume the body of the request:

For example:

// This will ultimately leak system resources on the MacOS M1 
// when a certain number of responses have been made. The 
// network will become unavailable at this point for the entire system.

const resourceExists = async (url) => {
  try {
    const response = await fetch(url);

    return response.ok;
  } catch (_) {
    return false;
  }
};

If you're using node-fetch, you'd need to check that all usages of it consume the body (e.g. .json, .text, etc.). The workaround is to dispose of the response with an abort signal.

For example:

// Use of the abort controller to clean up the fetch resources will fix the issue

const controller = new AbortController();

try {
  const response = await fetch(url, {signal: controller.signal});

  return response.ok;
} catch (_) {
  return false;
} finally {
  controller.abort();
}

@jakolehm
Copy link
Contributor

jakolehm commented Feb 7, 2024

@smcenlly we tried to reproduce the node-fetch issue back in April but we could not reproduce it anymore. We should probably verify it has not popped back.

@dshershov
Copy link

still reproduce this issue with the next version:
screen

@dandrade-pedbot
Copy link

same issue here on mac m2 pro

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working p1
Projects
None yet
Development

No branches or pull requests