ui: DB Console Metrics takes a long time to render graphs and sometimes fails to load at all for 22.1.5 clusters

[thtruo]

Describe the problem

Customers and the field team have been reporting major UI issues affecting CRDB 22.1.4 and 22.1.5 - where metrics/graphs either take a long time to load in DB Console, or at times even fail to load at all with a "Something went wrong" state. Both are regressions

To Reproduce

Links to Loom recordings will be shared on the equivalent Jira ticket.

Environment:

• CockroachDB version 22.1.5 (it's also been reported to happen for 22.1.4)

Additional context

• Affected clusters tend to be running with 35+ or 45+ nodes

Jira issue: CRDB-18353

gz#13551

gz#13591

[daniel-crlabs]

This impacts both cockroach cloud and cockroach on prem customers, as long as they are running 22.1.5. I attempted to obtain screenshots from sql, hardware and overview pages; only the top graph would load, independently of the time frame I chose. Affected nodes are not only large nodes, I tested 5 different cloud customers, all with 9 nodes or less and they all showed this same behavior.
This not only impacts our customers, but also impacts our ability to troubleshoot issues for our customers.

[andreimatei]

@daniel-crlabs it'd be useful if you attached something private pointing to one or two CC clusters where this happens

[daniel-crlabs]

@daniel-crlabs it'd be useful if you attached something private pointing to one or two CC clusters where this happens

@andreimatei there are were quite a few. I'm not sure how to attach a private comment to gh, but we have a view which you can choose and pick the larger clusters (ones with 5 + nodes) that are running 22.1.5 and you'll be able to see this behavior.

I'll add the link to the jira ticket, I think that's internal.

[ddtort]

Some context that may help here... here's a look at a v21.2.x cluster that loads well on the left and a v22.1.5 cluster that takes a long time to load on the right here. The initial request for data goes through for both but all subsequent requests to stream more recent data gets queued and hangs for the 22.1.5 cluster

[daniel-crlabs]

Passing the question along, please see below from one of our CEAs:

What release will this be in: #85636

We can't close the ticket until this has been backported and tested, thanks.

[thtruo]

Thanks @daniel-crlabs - the team is currently on it. If we can merge the fix by ~Aug 16, that will fall in the 22.1.6 release. If we miss that train, then it'll be 22.1.7
We won't have a confirmation yet until a PR is up hopefully soon

[thtruo]

Updated assignees and T- labels since @koorosh is taking the baton from @dhartunian

[daniel-crlabs]

Thank you my friends, I'll pass that info along to the customer, appreciate the update.

[thtruo]

Changes should be merged in time for 22.1.6
Related fixes that should resolve this issue include:

• ui: preserve time window selection on window reload #86157
• release-22.1: ui: fix time window selection with mouse on Metrics charts #85835
• release-22.1: ui: use TimeScaleState reducer to align time window across pages #86066

[ccampbell-crl]

@thtruo Is there currently a workaround for this issue or are customers expected to deal with this issue until they upgrade? Another customer is also currently having this issue on 22.1.5

[thtruo]

@ccampbell-crl the patches to those issues were recently merged, so they are only available in 22.1.6

Unfortunately they will still encounter those regressions if they're on 22.1.5 - we have not found workarounds for that release

[thtruo]

Closing this issue as patches have rolled out into the 22.1.6 release