-
Notifications
You must be signed in to change notification settings - Fork 450
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error while proxying request: context canceled #390
Comments
@jinxin-fu are you able to reproduce the error consistently? If so, does the |
This issue is stale because it has been open for 30 days with no activity. |
@mertyildiran I have the same issue with clean new installation using just tap (tried few times - same result). |
Might be a related issue: kubernetes/kubectl#1004 The error is thrown on line: https://github.com/kubernetes/kubectl/blob/764d523af876fae3eeaf9dc4e44816c79f3e7f47/pkg/proxy/proxy_server.go#L147 There seems like a test case for that: https://github.com/kubernetes/kubectl/blob/67359a916e3f7bb009b5c42b20d965c9ae725786/pkg/drain/drain_test.go#L97 @RamiBerm you might want to take a look. |
Hi @alegmal Thanks for reaching out.
Last option that can help is adding Thanks! |
I found that I got this exact same error in my OpenShift cluster, it seems that for whatever reason it's unable to resolve the api server when the name contains the '.cluster.local'. I'm not sure if this is an issue with my clusters setup or not, but changing the API server URL in the daemonset made it work for me :) |
Hi @monfardineL, Thanks for reaching out, can you please share the log file (/root/.mizu/mizu_cli.log) so I will be able to see if there is any extra information to explain the issue? Also it is possible to run mizu with Thanks |
@gadotroee here it goes. |
Hi @monfardineL , I want to check if this is issue with the proxy that we are starting in mizu, so just to eliminate this I suggest you will do the following:
This will just validate that agent is working and approve that this is proxy issue that we might need to investigate (I know you did something similar but now it will be without mizu proxy so we will validate it is not proxy issue that you have that not related to mizu) |
Got the same error as before:
|
Thanks, @monfardineL I have one last suggestion that might be used as workaround (just to let you experience mizu until we understand the reason for this and solve the case of context canceled) The suggestion is to create (or change) service with type Load Balancer and see if we can access it. while this command is running you can do one of the following:
After some time you should get public ip for this service and see if you can run / use mizu. I'm investigating the issue and try to come with a better solution in the near future. Thanks for the details and I hope it will work for you so you will give more feedback about the usage.. |
@gadotroee I had to do a little change in the command, from deployment to pod, since the mizu app isn't creating a deploy, so: Now, I don't to want mix things, so if you think this is unrelated (as I do), I can open a second issue later, but I noticed this, while accessing Mizu through the Loadbalancer: Confirmation that I did got a few pods at start: And logs from Mizu pod running:
I did a second run without any pod filter (the whole cluster) and the result is the same. |
Ok, this is a good progress. It seems that the issue is not really related but it might tell us some more information, I will investigate a little bit and come back with some more answers as soon as I can. Thanks for the corporation and sharing your status, we will be in touch soon and have a good weekend. |
Hi @monfardineL, unfortunately I don't have any new findings about this issue - the main suspect is the web-socket connection that we have between mizu site that you are viewing and the api server in the cluster. Is it possible to arrange meeting and trying to solve it (maybe when we see the cluster and can "debug live" it can help to understand what is the root cause). Thanks |
Hey @gadotroee I have one question that you can help me clarify so maybe I can do some checks with my network team before we proceed. If my network team doesn't find anything suspect, we can sure schedule a meeting. |
Hi @monfardineL, I will try to answer it and hope it will help them (or you) to understand how mizu works and potentially find a good solution to all issues you currently have. The think is that you are publishing components to you cluster (mizu api server and the tapper daemon set (which starts only on nodes that has "tapped" pods). To be able to see that traffic that mizu captured and actually use mizu - the cli starts proxy (that is very close to the so when you are going to "http://localhost:8899" you are actually accessing something like "http://127.0.0.1:8899/api/v1/namespaces/mizu/services/mizu-api-server:80/proxy" this page is the frontend of mizu and making request to the api-server in 2 ways - HTTP requests and long living web socket connection. (it will be interesting to see if you can give some details about those requests (when you are working with the load balancer which is the only way you actually was able to see the UI. (e.g by going to the chrome dev tools and maybe we can understand from there if there are any failures in those requests and which kind of failures) I hope it is not too much details and I'm here to help if needed in anything else. btw, you can join our community slack channel - here is the invite link if you want https://join.slack.com/t/up9/shared_invite/zt-tfjnduli-QzlR8VV4Z1w3YnPIAJfhlQ, there are some details there and we might answer more quickly. |
Hey @gadotroee just passing by to inform I didn't abandon the case. Just hadn't time (neither network team) to follow on the subject. But I'll join the slack channel as soon as possible. |
We have release pre-release version https://github.com/up9inc/mizu/releases/tag/0.22.13 that should help in this case. |
Hi @jinxin-fu, as you are the original reporter of this issue, I would like to know if this solves your issue. (using version 0.22.13 and above) Thanks. |
Hello guys, thanks for this tool, for the moment I cannot test because I have exactly the same problem that OP described. Waiting for Mizu Agent to start... mizu version kubectl version |
Hi @dalvarezquiroga, can you attach your logs ( Thanks |
Sure @gadotroee if I can help with the debug, you can find here attached using --set dump-logs=true. Thanks. |
Thanks for the logs and for the help and I hope we will find solution so you will be able to run and use mizu.. Ok, from first look over the logs everything looks good. some questions that I have now to be sure I understand correctly:
I hope it make senes I will try to investigate it and come back as soon as I can with more information. |
Thanks!
|
Thanks for the answers. I will validate it in EKS cluster on Amazon, but it is good to hear that you are able t see mizu. Also I'm not sure it is the same problem because the problem that mentioned here before is when you cant get to the mizu app page. If you do see it, it is probably another issue, do you see any information on the top bar of pods that we detect that and should be tapped? When you are "hover" on it you should see list of pods and "v" or "x" to know that we have pod of mizu (tapper pod, which is part of daemon set) that should see them. If you can share screen shot it might help too. |
Thanks @gadotroee , yeah sorry maybe is not the correct issue to discuss about my problem. Sure I attached a screenshot that you can see that all of pods are with X. And the message from console. |
Ok, I think this is what I need for now, might take me some time to reproduce or investigate, hope to come back with answer in a the next few days. Two things that you can do:
The logs from --set dump-logs=true gets the log of those pods so they might help me too (I'm still need to read the zip that you send before because I thought you don't really see the UI and now it is different problem) |
Hello, no problem, take your neccesary time it's only for testing the tool in our kubernetes cluster. Answering your points:
k describe ds mizu-tapper-daemon-set -n mizu I am using Istio like service mesh but I didn't configure yet the following steps: (I don't know if are obligatory or optional ) |
Ok, all information you supplied is very helpful. I will try to explain as many things as possible but first you can also join out slack channel and we can help there (which might be easier than Github issue. also we can schedule f2f meeting (zoom or in slack) to try to solve this interactively. From the last logs you shared I can tell that we created the daemon set successfully and there should be tapper pod created on this node: "ip-10-171-106-191.eu-west-1.compute.internal". if you do have this node and after running you don't see any pods created (daemon set on 0 DESIRED), it will help me to get the yaml of the resource ( about the service mesh option, it can improve the behavior but not required to make it work so you can skip this settings until you are getting working application. btw. I tried to run it in EKS cluster (old one that I have) and it seems to work ok and if all of those actions will not help I think it might be permissions issue (or other tool that changes the resource (Daemon set) before creation and the yaml will help me to ensure it is not changed (if you know about any tool that do those kind of changes you are welcome to share about them and I will try to reproduce it with them) [if you are sharing result from a new run, you can attach the new logs and it will give some context for the information in the message] |
This issue is stale because it has been open for 30 days with no activity. |
This issue was closed because it has been inactive for 14 days since being marked as stale. |
Hi, @dalvarezquiroga just wanted to let you know that we understand the issue that occurred to you and (most likely), and there is a new unstable release that might work for you. if you want to give it another try, please consider using this https://github.com/up9inc/mizu/releases/tag/31.0-dev61, soon we will create stable release but in the meantime you can take this and check if it is working for you. We are here for any questions if you have. |
You are totally right! :-) Thanks for write me! I tested this release and now it works like a charm. During next weeks I will test this tool to be able to see all requests and the memory consuption etc... |
when use " mizu tap ".*"" meet thefollowing error
E1021 21:34:46.539954 3777907 proxy_server.go:147] Error while proxying request: context canceled
logs from mizu-tapper-daemon-set pod:
panic: Error connecting to socket server at ws://mizu-api-server.mizu.svc.cluster.local/wsTapper dial tcp: lookup mizu-api-server.mizu.svc.cluster.local: Try again
The text was updated successfully, but these errors were encountered: