WaitingForPanels: Change waiting logic for Scenes

[torkelo]

This improves and simplifies the waiting for panel queries and panel rendering logic.

Instead of scanning for how many panels there are using css selectors and comparing this to panel content (which is a bit unreliable / fragile) we just wait for all network requests to be completed and then an additional 100ms for data rendering (this step should be blocking so maybe we can lower this).

• Now the waiting for panels step also works in scenes
• Tested with single panels
• Tested with full dashboard images (using height=-1) (With many slow queries)
• Tested with new and angular panels

[AgnesToulet]

I think this doesn't cover async queries (I discovered them recently and I'd say this is the only reason why we need this complicated code). An example of data source using them is the Athena DS.

As the new PDF feature relying on scenes is working though, maybe it would be possible to use only the full page condition for everything.

[AgnesToulet]

I think this is the same as what we do here:

grafana-image-renderer/src/browser/browser.ts

Line 303 in eae11cd

return page.goto(options.url, { waitUntil: 'networkidle0', timeout: options.timeout * 1000 });

[torkelo]

@AgnesToulet not 100% sure this is the same, that page.goto is just for the initial page load, before we scroll down and trigger more queries. That is why I do a second call to waitForNetworkIdle here as this is after the scroll

[torkelo]

I think this doesn't cover async queries (I discovered them recently and I'd say this is the only reason why we need this complicated code). An example of data source using them is the Athena DS.

@AgnesToulet All queries are async, what is special about the Athena DS?

[AgnesToulet]

@AgnesToulet All queries are async, what is special about the Athena DS?

When you use the TestData data source with the "Slow query" scenario, you can see the query request in a "pending" state for a while and returning after a specific time. This is caught by waitForNetworkIdle because it's waiting for all pending requests to end.

With the Athena DS, a first query call is made, but it returns pretty quickly. Then, when the results are ready, another query call is made to retrieve them. If the time between these two requests is greater than the idleTime option, waitForNetworkIdle will return before this second request is made.

[torkelo]

I see

Then, when the results are ready, another query call is made to retrieve

@AgnesToulet how does know when the results are ready?

[torkelo]

@AgnesToulet is this some closed source athena data source? this ones does nothing special https://github.com/grafana/athena-datasource/blob/main/src/datasource.ts#L110C18-L110C23

[torkelo]

@AgnesToulet oh, I see it's using this https://github.com/grafana/grafana-async-query-data-js/blob/main/src/DatasourceWithAsyncBackend.ts

[torkelo]

@AgnesToulet I see this, https://github.com/grafana/grafana-async-query-data-js/blob/main/src/requestLooper.ts#L79

hm... dammit. why is there always some stupid edge case ruining everything :)

I think I have a way fix it but will require a change in main repo

[torkelo]

@AgnesToulet pushed an update that restores the old waiting logic, except for scenes it will use this window query counter (That is not related to network query state but actual data source requests observable state and responses). So for Athena this will work as they are returning observable loading states (And their observable does not complete until they they are done)

[AgnesToulet]

@torkelo This looks good but doesn't support the TestData slow query scenario. I get this screenshot:

Is it expected? Does it work with slow queries from other DS?

[torkelo]

@AgnesToulet it works for every slow query I have tested for any data source, in both new and old dashboard arch.

What version of Grafana are you using and what is the url? is the feature toggle __feature.dashboardSceneSolo enabled (or in url)?

[AgnesToulet]

@torkelo I'm on latest main, I have the dashboardSceneSolo feature toggle enabled, and it happens when rendering a panel from a Scene dashboard. The URL is this one: http://localhost:3000/render/d-solo/b17337d4-f4b7-40e2-b1a0-d76deee4d6f5?orgId=1&scenes&var-myVar=b&from=2024-02-16T07:33:09.973Z&to=2024-02-16T13:33:09.973Z&panelId=panel-11-clone-0&__feature.dashboardSceneSolo&width=1000&height=500&tz=Europe%2FParis

What's weird is that I see the same behavior with the current version of the image renderer, while all other panels (also using the TestData DS but with the Random Walk scenario) are timing out...

[AgnesToulet]

@torkelo I tried to access the associated /d-solo page, and I'd say the issue is from there. The panel using the TestData DS with the slow query scenario always shows the loading page (the query is supposed to run for 15 seconds, but even after that time, I still see the loading page), and I don't see any query pending.

So, I will approve the PR and will work on releasing it along with the PDF option next week.

[torkelo]

@AgnesToulet strange, d-solo route works well for me no matter the panel or query

[torkelo]

@AgnesToulet you you using latest main?

[AgnesToulet]

@torkelo Actually, I tried again with a simple slow panel, and it's working. The panel that doesn't work is a repeating panel (using a custom variable).

[torkelo]

@torkelo for repeated panels you need use a new syntax for the panelId , example panel-2-clone-2 , it's one of the few breaking changes in scenes

[torkelo]

@AgnesToulet
after grafana/grafana#82441 is merged this should now work for old and scenes dashboards & panels

[AgnesToulet]

LGTM!