Jupyterlab pods failing accross 3 independent nebari clusters

[MUvaage]

Hi!

We have three implementations of Nebari running in Microsoft Azure. They are all hosted on the same subscription, but apart from that they are independent. Recently all attempts at spawning jupyterlab instances from the jupyterhub landing page have resulted in timeouts. This includes Holoviz Panel application hosted through Jhub apps as well. The K9s logs have provided no insight.

The error seems to have occurred sometime between 22.05 - 26.05. We have made some modifications to the images used for Jupyterlab in some of our deployments, but not all three.

Since this error has occurred simultaneously across three clusters, I'd figure I would check with the community right away to see if this is something experienced by others, and whether it could be caused by image hosting issues like the previous bitnami-error. I've yet to attempt a redeploy, but this is planned.

We are planning on migrating to NIC at some point, but please let us know if this is a known issue stemming from the deprecation of nebari-classic.

[viniciusdc]

Hey @MUvaage, at first glance this looks like an issue with the auto-scaller (cloud) where it reached the max available instance size for the users group on that subscription, it could've been either an Azure latency issue or unavailability (we've seen this recently nowadays).

If it was related to the images, we usually see some issues like Uanthanticated or PullImagerBackOff... I would try looking over the autoscaler logs on Azure for that time period and also check the subscription quotas in case that's been used in full.

[MUvaage]

Thanks for the reply! Confirmed via the logs that this is a region-specific capacity issue.