Automate Benchmarks & Run in CI

[tbillington]

Is it possible to include a shell script to run benchmarks against other terminals that could be automated at CI time to catch speed regressions?

[jwilm]

Yep, would love to have something like that.

[theduke]

OpenGL testing on Travis et al is a bit problematic.

You might be able to set somethig up with Xvfb / OSMesa.

[chrisduerr]

Just as a note, I've tested out automated benchmarking using Xvfb and it was not possible to get any useful results out of the alt-screen-random-write benchmark of vtebench because Xvfb was bottlenecking Alacritty.

I believe the best bet for automatically comparing Alacritty against itself would be to run headless benchmarks which do not exercise any hardware acceleration, which would make it easy to run these on a server. This wouldn't allow comparison against other terminal emulators, but it seems like the best choice to me.

[zacps]

Have you tried using xdummy? I did a bit of research and it seems to be a more recent attempt.

[chrisduerr]

Looking at the performance difference on that page, I don't think this would improve anything.

I don't think anything that doesn't just ignore all render calls will be able to process Alacritty's output without GPU acceleration.

[zacps]

Is there any reason why just ignoring all render calls wouldn't work? The benchmarks shouldn't need the actual rendering to be verified.

[chrisduerr]

In theory there should not be any problem with that. However I'm not certain that there wouldn't be any complications.

[chrisduerr]

See https://github.com/alacritty/termbenchbot.

A follow up for more features can be found here: alacritty/termbenchbot#1.