See for example:
where there is a single visible review with a rating of 4 and compare it with
where the summary is a rating of 2.5
The code is in lib/CPANRatings/Control/ShowAll.pm if you feel like making a patch. A related issue is that the system should be including lines that only have unhelpful reviews (but make the rating and count 0) to make sure the data gets overwritten on cpan search.