Synchronization timeouts (Dev Spaces) #407
Comments
|
No, we don't know of any issues that would cause synchronization to fail for you. Have any changes in your project caused more or bigger files to be synced? |
|
Hi and thanks for the quick reply Nothing has changed in the project but he project is kinda large (around 20000 files). We keep seeing this performance degregation and it always occurs around 16:30 UTC. We have tried 3 different networks from 3 different carriers with the same result. We have run performance checks on our k8 but the node has plenty or resources left, and the application on the k8 runs smoth and responsive. Does the synchronization happen just between our clients and our k8, or is there some other servers involved aswell? |
|
Hi @Eneuman , it sounds like there may be a pattern of high network traffic around 16:30 in nearby resources in the data center where your cluster and Controller are hosted. Are you in the West Europe region? There are some other Dev Spaces resources involved in the sync operation, but we don't have anything scheduled to run near at that time. |
|
@philon-msft Yes, I am in the West Europe region. And like clockwork, the synchronization just stoped working again :( Running "kubectl get pods" returns the pods in under a second. At the moment, dev spaces is unusable for us after 16:30. |
|
And now its back up and working without any changes on our side. This got to be a resource problem. Do you have any DevSpaces telemetry from the West Europe region you can take a look at? :) |
|
Synchornization stoped working again 6/11 14:45 UTC This is realy frustrating... Please help us figure out what is going on.. |
|
Still not working :( And the command but |
|
Still not working :( |
|
@Eneuman Sorry you're encountering this issue. :( We're having a look into this, and will keep you updated. |
|
Adding @shpathak-msft, who is looking into this. |
|
Thank you! Feel free to access our cluster if you need to troubleshoot. //Per |
|
@Eneuman Could you please reach out to me at shpathak@microsoft.com? I have some logs from our backend service that I would like to share with you. |
|
You got mail :)
…________________________________
Från: shpathak <notifications@github.com>
Skickat: Monday, November 9, 2020 8:00:19 PM
Till: Azure/dev-spaces <dev-spaces@noreply.github.com>
Kopia: Per Bornsjö <per.bornsjo@links.se>; Mention <mention@noreply.github.com>
Ämne: Re: [Azure/dev-spaces] Synchronization timeouts (Dev Spaces) (#407)
@Eneuman<https://github.com/Eneuman> Could you please reach out to me at shpathak@microsoft.com<mailto:shpathak@microsoft.com>? I have some logs from our backend service that I would like to share with you.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#407 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AC4J5TMTFZP6MS26NAH7VKTSPA34HANCNFSM4RYWJA4A>.
|
|
Thank you for reaching out and glad to know code synchronization is now working for you. I suppose you already know this, but I would suggest reaching out to AKS support if you continue to see the "could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1" error. |
|
Thanks you for getting synchronization working again. |
|
@Eneuman Happy that we unblocked you! The issue you encountered is tricky to investigate because it is either transient, or it relies on us restarting the component on our side associated with your Azure Dev Spaces controller. This component didn't show specific errors itself that would help us finding the root cause of this. When this issue happened, this also didn't match to a deployment on our side or to code changes. If I remember correctly, you investigated in the past transitioning to Bridge to Kubernetes but cannot do it presently because you rely on pod identity for your services. Is this correct? We're presently investigating adding support for pod identity and, if it's the only blocker for you, we would be happy to provide you with early bits as soon as we have something to share. |
|
@daniv-msft Yes we are interested in transitioning to Bridge to Kubernetes but two features are missing. Pod Identity (as you said) and the ability to stop all other instances of a particular pod when we start debugging it. We rely heavily on azure service bus and pub/sub, so when debugging a service, we don't want any other running instances, of the pod we are debugging,to receive the events. |
|
@Eneuman Thanks for your quick reply. Regarding pod identity, could you please confirm that your team members are using Windows machines to develop? From previous discussions, I believe that you're using Windows yourself. |
|
@daniv-msft Yes, we are only using Windows machines and Visual Studio 2019 Pro. Really looking forward to trying the new bits out :) |
|
@Eneuman All right, thanks for confirming! |
|
@daniv-msft I think we just got this issue again :( Regards |
|
Hi @Eneuman, I've restarted some of the components on our side. Can you let me know if the issue keeps reproducing? Thanks! |
|
Hi @amsoedal |
|
@amsoedal One is giving me this error. Is this on my side ? pods/klinikportalservice-757c5cc75b-cd6b7: BackOff: Back-off restarting failed container |
|
Hi @Eneuman, sorry you're still having issues! Could you please run |
|
Hmm, for some reason it seems like Azure ADO Cert is no longer valid. Maby a service issue? kubectl logs -n dev pods/klinikportalservice-757c5cc75b-cd6b7 devspaces-build |
|
@Eneuman Thanks for attaching this! Can you try again now? |
|
@amsoedal Yes, now its working again :) Thanks! |
|
Great! Happy to hear it :) |
|
@amsoedal Hi again. It looks like this problem has started to occure again :( Can you please apply your fix again?
Oops... An unexpected error has occurred. A report of the error will be sent to Microsoft. And the log contains this, so it looks like that SSL error again. |
|
Did you just change something? It started working ! |
|
@Eneuman No, we were still digging into the logs we have on our side to try to understand what was happening. We didn't restart anything manually. The only automatic restart script we have (restarting Traefik pods automatically every couple of hours) that we added a couple of weeks ago didn't run in the last minutes. We confirm that we can now see on our side that the build did happen: |
|
Can it be some type of cache that got evicted? |
|
I'm gonna have to call it a night. Been at it for 20 hours straight and I can deploy services to my cluster now so I'm unblocked, even though we don't know why it started working I call it a win :) |
|
@Eneuman Makes sense, it's super late in Sweden right now! Yes, we share your frustration. :( Happy that you're unblocked at least! |
|
@daniv-msft Hi From what I can tell, todays version av Dev-Spaces and any existing versions of Pod-Identity is not compatible so It looks like I can't setup a new cluster and we are stuck. My findings so far: 3) If I install AAD-Pod identity first and then Dev Space controller: The init pods does not get initialized and the build process does not start. I have tried all the existing AAD-Pod identity versions available with the same problem :( You can reproduced this behavior by installing a AKS like this: add a dev namespace add Dev-Spaces and then add AAD-identity and then create a simple .net core service and do a azds up. |
|
@Eneuman thanks for these details. We're still working to get to the bottom of this, but it helps to know that pod identity is the determining difference |
|
@amsoedal Sounds great :) I think one of the problem is that Azure Pod Identity is intercepting the Manage Identity call that the init build container does and that breakes the build. I noticed that if I add a AAD Exception like this: then the webkhook containers starts to work. From NMI logs: The build does not work though |
|
@Eneuman are you able to debug your app after adding this exception? |
|
No, but adding the exception makes the webhookspods work by the look of it. Building the images does not work. The build init container never finishes and the proxy init containers stays in Pending state. |
|
@Eneuman are you still using user assigned managed identity? |
|
Yes |
|
Have you been able to reproduce this behaviour in your cluster? |
|
@Eneuman We're trying this, and will let you know how it goes. |
|
@Eneuman I confirm we do reproduce the issue on multiple clusters. We're still looking into what goes wrong. This issue is challenging. |
|
@Eneuman Something strange for us is that we don't see the webhook issue you were mentioning above. On our side, the webhook seems to work fine (without adding AAD Exception), and we can even see the request to start building the image locally showing in their logs: |
|
Also, we can see the Build service exiting because of an error: Unfortunately, the logs don't show much as of today, but we're working to rebuild an image of this with more logs and see the new info we can get. Do you see something similar on your side? |
|
Yes, I got the exact error as you.
About the webhook, it depended on in witch order I installed Dev spaces and pod identity. See my post higher up. If I installed Dev spaces first and then add, webhooks stoped working for me.
…________________________________
Från: daniv-msft ***@***.***>
Skickat: Friday, April 9, 2021 6:13:45 PM
Till: Azure/dev-spaces ***@***.***>
Kopia: Per Bornsjö ***@***.***>; Mention ***@***.***>
Ämne: Re: [Azure/dev-spaces] Synchronization timeouts (Dev Spaces) (#407)
Also, we can see the Build service exiting because of an error:
λ kubectl logs -n dev webfrontend-76d68d944c-frtpm -c devspaces-build
An error has occurred. Please contact Azure Dev spaces team for troubleshooting.
2021-04-09T15:42:14.7684021Z | BuildService | TRACE | AssemblyLoadContext unloading\nOperation context: <json>{"clientRequestId":"a3e1d09a-a1ef-446a-868b-3b769e2126e7","correlationRequestId":"9423774f-a445-463e-839f-a683ec44dc0e","requestId":null,"userSubscriptionId":null,"startTime":"2021-04-09T15:33:09.3853715Z","userAgent":"BuildSvc/1.0.20210206.1","requestHttpMethod":null,"requestUri":null,"apiVersion":null,"version":"1.0.20210206.1","requestHeaders":{},"loggingProperties":{"applicationName":"BuildService","deviceOperatingSystem":"Linux 5.4.0-1043-azure #45~18.04.1-Ubuntu SMP Sat Mar 20 16:16:05 UTC 2021","framework":".NET Core 4.6.28325.01"}}</json>
2021-04-09T15:43:14.7748874Z | BuildService | TRACE | Process exiting...
2021-04-09T15:43:14.7754218Z | BuildService | TRACE | Process exited
Unfortunately, the logs don't show much as of today, but we're working to rebuild an image of this with more logs and see the new info we can get. Do you see something similar on your side?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#407 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AC4J5TKVCRBCNZY4YTZVX5TTH4RTTANCNFSM4RYWJA4A>.
|
|
@Eneuman, Can you check the nmi and mic pod logs in the default namespace on your cluster? What do they say? |
|
Also, I opened this issue on the pod-identity page. Hopefully they'll be able to shed some light. Azure/aad-pod-identity#1039 |
|
Working on getting you the logs @amsoedal. |
|
My setup: Installed AAD like this: As you can see the webhook gets intercepted by nim and failes. So no init pods gets created. Now the nim loggs look like this: And now the init pods starts, but the BUILD Init does not complete. The state is always Running I think the problem is that the build pod gets intercepted because of this PR Azure/aad-pod-identity#191. It would be very intrested to see what wouold happen if it was possible to exclude init containers from AAD interception... |
|
Hey again,
Could you please run this command on your own AzureIdentityBinding? |
|
@Eneuman Our contact also told us to look out for https://azure.github.io/aad-pod-identity/docs/#v16x-breaking-change. If you follow the AZDS doc I linked above, you have to change the fields to camelcase. |
|
@amsoedal Yes, the casing in my bindings should be correct, I have dbl checked them. Running but I think thats okey since my testpod does not use managed identities yet (I was going to add this after I get it running). |
|
Hi @Eneuman, |
|
@daniv-msft I have not been able to test this yet but Im working on it. I did however found one problem on my side. However, Now I'm only stuck with Pod Identity not working because of a Identity Not Found issue. Tracking it here Azure/aad-pod-identity#1040 I'm not sure if this is related to Dev-Spaces and Pod-Identity interceptions. So we are making progress :) |
|
@Eneuman thanks for the update! Glad you're able to get past the build issue now at least! |
|
We have migrated to Bridge For Kubernetes so we will close this issue. |
About 4 hours ago we started receiving synchronization timeouts.
"Azds up" shows a timeout error after trying for 4 minutes.
Our k8 looks great and we have tried from two different networks with the same result.
Is there a problem with synchronization atm ?
The text was updated successfully, but these errors were encountered: