Eval errors make jobset and eval pages very expensive to serve and browse

[mweinelt]

Describe the bug
Eval errors make up a large part of what hydra has to ship to users when browsing jobsets or evals.

• The jobset page for nixpkgs/staging-next is 26 MB in size
• 99.9% of that 26 MB page consists of eval errors

To Reproduce
Steps to reproduce the behavior:

1. Go to https://hydra.nixos.org/jobset/nixpkgs/staging-next or https://hydra.nixos.org/eval/1804223
2. Observe the download size
3. Strip the HTML around the eval errors
4. Notice it takes up >99.9% of size

Expected behavior
These pages should be quick to load and not create load on the hydra webserver.

Proposal
Create a separate route for requesting eval errors and embed that endpoint via an iframe that gets lazily loaded, when it comes into the viewport.

The eval errors should also be shipped in plaintext and not be passed through the template engine.

Hydra Server:
hydra.nixos.org

Hydra 0.1.20240126.838648c (using nix-2.19.4).

[delroth]

FWIW when I checked the SQL query associated with loading that page returned 225MB of data. I'm not sure how it ends up getting condensed to 25MB though, but if I did actually get the 225MB measurement right then it's an even more critical performance issue.

Create a separate route for requesting eval errors and embed that endpoint via an iframe that gets lazily loaded, when it comes into the viewport.

Or just make it a link? I don't think we lose much by not having the errors be inline in the page, tbf. If that simplifies the implementation...

[mweinelt]

Given that browsers do the heavy lifting for us, I think iframes can be the drop-in replacement.

https://web.dev/articles/iframe-lazy-loading

[samueldr]

Why even go for an iframe?

Using a discrete page is always the better choice. That is, rather than trying to be "smart" with lazy loading, and/or js, and/or DOM-based tabs.

Though since the expectations vary wildly with tab-based navigation, it would be best if it was a discrete link/action in the blurb above the search input.

[mweinelt]

Why even go for an iframe?

No strong opinion. My impression was that it would be the simplest drop-in replacement.

[delroth]

I've done another pass in 6189ba9 this time actually removing DB fetches when possible. h.n.o loads considerably faster. There are probably some other places where errormsg is being fetched right now, so I'll leave this open for a bit, but I think we might end up just closing the issue if the remaining places don't cause issues.