Show the fps history up to 3000 frames in the README

[eregon]

• The first 180 frames tell a partial story: they would last only
  3 seconds at 60 FPS while an actual game will run far longer.

I think many/most people will only see the README and miss the benchmark.md link or not click it.
Also benchmark.md contains a lot more information, which seems not so relevant for most readers.
Since OptCarrot is principally a benchmark I think it makes total sense to show the most relevant charts directly in the README.

[eregon]

@mame Thoughts on this?

[eregon]

@mame Gentle ping.
I think the current state is very misleading, the README shows TruffleRuby at 19.1FPS, but it's actually by far the fastest implementation with enough warmup.
I think adding the second chart addresses that and also shows that warmup is part of the trade-off.

[eregon]

Actually on the current warmup chart, we can also see that the MJIT numbers from the README are misleading (the README says 48.6FPS but it reaches ~60FPS), and that MJIT clearly also needs more than 180 frames to reach peak performance:

So all implementations with a JIT need more than 180 frames of warmup. This is part of the trade-off, and I believe it should be shown right in the README, and not just the arbitrary limit of 180 frames which only shows a (very) small part of the results.

@mame Could you merge this PR?

[mame]

Sorry for leaving this ticket, but honestly, I'm still reluctant to make the top README messy.. I'd like to put a simple chart that just answers "how fast they are, in short". I know that the chart does not reflect the peak performance of Ruby implementations including MJIT, but fast warm-up is actually very important for optcarrot because many games start their opening title in 180 frames from the boot. If 60 fps is not achieved at this time, the game screen is laggy and the music is noisy. I believe "180 frames" is a reasonable criteria for optcarrot, and the current chart shows one certain aspect of the "net" performance of Ruby implementations.

That being said, I admit that optcarrot is now handled as a somewhat influential benchmark program for Ruby, and I'd like to respect Ruby JIT implementors. @k0kubun, do you have any opinion? I'll ask matz's opinion, too.

[mame]

I chatted with matz and akr-san, and matz suggested adding a new histogram at frame 3000 instead of the fps history chart. I think that it is better than showing two different-type charts, so I'll do it.

Akr-san said that the caption of the chart, "Ruby implementation benchmark with Optcarrot (180 frames)", is not easy to understand. He suggested "(three seconds in the emulation time)" or something, but I have no idea about good wording. Do you have any idea?

[eregon]

I chatted with matz and akr-san, and matz suggested adding a new histogram at frame 3000 instead of the fps history chart. I think that it is better than showing two different-type charts, so I'll do it.

Sounds good to me :)

Luckily it seems the performance is mostly stable around frame 3000.
It still only shows 1 data point out of 3000, so I find the warmup chart far more interesting and more telling about what is actually happening. Basically it explains JITs in one picture :) And we could have a line at 60FPS and see when implementations cross it.
But it would probably get messy and not readable if we would add many implementations on that chart.

Akr-san said that the caption of the chart, "Ruby implementation benchmark with Optcarrot (180 frames)", is not easy to understand. He suggested "(three seconds in the emulation time)" or something, but I have no idea about good wording. Do you have any idea?

I think the current title is fairly clear. I think "(three seconds in the emulation time)" would be confusing.
Maybe "Ruby implementations benchmark with Optcarrot for the first 180 frames" would be good and a bit clearer?

If you want something shorter I'd suggest "Optcarrot, first 180 frames".
Although, the actual values are the average of frames 171-180, right?
So then to be precise, I think this would be best:

Optcarrot, average FPS for frames 171-180
Optcarrot, average FPS for frames 2991-3000

[k0kubun]

I'd like to respect Ruby JIT implementors. @k0kubun, do you have any opinion?

I'm interested in the comparison of peek time performance, so I'm +1 for including a graph which shows that. Though I don't have a strong preference on its details. As long as the fps at 3000 frames can show the peak performance of every implementation on @mame's environment, I'm fine with the current proposal #26 (comment).

---

So then to be precise, I think this would be best:

Optcarrot, average FPS for frames 171-180
Optcarrot, average FPS for frames 2991-3000

I like this. TBH I didn't know it's currently showing 171-180, and that makes it clear.

[eregon]

@mame Thanks for the update and addressing this!

Is it intended that f2ad8c4 removes the warmup/history plots though?
I think they were very useful, both for 180FPS and 3000FPS, to get a better idea of variability and how a JIT works.

[mame]

Yes, it is intended. And I also removed the startup-time plot. I agree they were all useful, but they made the tool programs (like run-benchmark.rb and plot.rb) harder to maintain and the benchmark process painful for me. Unlike a paper, I need to update the benchmark at least per year, so I want them to be simpler.