Wishlist for the speed.yjit.org front page

[maximecb]

Motivation: the speed.yjit.org page is a useful summary of the progress our team is making. It's useful both internally and externally. I look at it all the time to get a quick idea of the progress we're making. I think it's very valuable to optimize the information we display on that page to maximize usefulness, and focus our attention on the metrics we care about the most.

• Bars comparing YJIT 3.3 vs 3.4dev
• Remove the bar for the interpreter, replace it with a horizontal line at 1.0
  • This will make space for more bars or more benchmarks
  • Draw baseline at 1.0 and label it "CRuby <version>" #257
• Baseline currently says "no jit", should be more specificic, eg "CRuby 3.3"
  • Draw baseline at 1.0 and label it "CRuby <version>" #257
• Add bars at the right for the geometric mean (geomean) of CRuby 3.4, YJIT 3.3 and YJIT 3.4
  • This will give us a nice "average speedup" figure (useful shorthand)
  • We may want to write geomean* (with an asterisk to indicate this is not a benchmark) for the column name
  • Add bars to the chart for the geometric mean of all the bars in the chart for each ruby version #266
• Create a normalized memory usage graph with a similar format to the speedup graph, also to be included on the front page
  • This will be very valuable because it's a big blind spot for us right now. We don't pay much attention
    to how memory usage changes over time, and can get blindsided by regressions.
  • Possibly, we should measure this for the 3.4 interpreter as well? It would be good to be able to detect
    regressions there, and know what is or is not due to YJIT.
  • This should also have geomean columns as well
  • Ideally, this should be measured using "max RSS", which yjit-bench is able to track (feel free to copy the code).
  • Add graphs for memory usage comparison #267

Lower priority:

• Tweak colors, styling, current theme not aesthetically pleasing
  • (I realize that style is very subjective)
  • White background, colors that "pop" more for the bars?
  • The legend rectangles could be smaller and better aligned with the text (the text looks too close to the bottom side?)
  • Move "theme" attributes to a root level yaml file and update some styles #264
• We currently show the speedup value when hovering, if there was a way to write this value inside the bars (or at the top at an angle) in a small font, it might be better?
  • The hover setup doesn't work on mobile, it's also not immediately obvious for new visitors.
  • Label bars with their values #269
• Nitpick: we show the time with seconds at the top of the page, but the seconds aren't useful.
  • Just a date and a clickable link to a ruby/ruby commit would be much more useful
  • Reword the header sections, show versions, and link to commits #268

Note:

• if we end up with 3 bars for the memory graph (3.4 interp, 3.3 YJIT, 3.4 YJIT), then maybe we also want to have the same 3 bars for the speedup graph, for consistency. This would also give us some visibility into interpreter performance improvements and regressions.
  • Benchmark against CRuby 3.3 to use as a baseline #258

@rwstauner I ended up writing lots of notes. Hope this is helpful :)