Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Benchmark Compatibility

jkomoros edited this page · 2 revisions

The primary purpose of RoboHornet is to test for performance, not compatibility. However, every so often there will be tests that implicitly measure compatibility. This page documents how to handle those cases.

Admissible workarounds

A workaround is special code necessary to get a benchmark to work in a given browser, for example due to minor bugs, incompatibilities, or missing functionality. An admissible workaround is one that is a) only required for a minority of browsers for that benchmark and b) is more complex than the base case. The reason for the latter restriction is that we do not want to disincentivize browsers from supporting new capabilities.

Note: An earlier draft of these guidelines allowed some tests to be included in the core set of benchmarks if they could be accomplished with admissible workarounds in browsers that didn't support them directly. There is active discussion around the merits of that approach.

Core vs Extended set of benchmarks

Benchmarks in the "Core" (default) set must work in all of the stable versions of major browsers without workarounds. In some cases there may be a technology whose performance is important to web developers but that is not deployed in all browsers. Those may be included in the "Extended" set of benchmarks--but only if a majority of browsers support the technology without substantive workarounds. As soon as the underlying functionality can be addressed natively in all major browsers the benchmark will automatically be promoted to the "Core" set in the next release of RoboHornet.

Failed Benchmarks

If a browser cannot complete a given benchmark--even including admissible workarounds--due to incompatibilities or unsupported technology, it "fails" the benchmark. In that case it is reported to the user as "Unsupported by your browser", and the score is reported as double the worst score from the set of baseline scores for browsers that can complete the test.

Something went wrong with that request. Please try again.