Do the benchmark on a more realistic schema

[ebdrup]

The current benchmark is run on a subset of the tests in the official JSON-schema test suite.
Although the results seem very much in line with what other stable benchmarks produce (like themis), and exercising as much of the standard as possible is good to catch any slow corners in a validator. It might be a good idea to do a benchmark on data that is more like what you see in production.

Perhaps an performance benchmark on one simple and one advanced schema.

To keep things simple for anyone wanting to chose a validator, the benchmark should still produce one single number for each validator like it does now. Nitty gritty details about performance are for schema validator authors to do in their efforts.

[ebdrup]

I was thinking of machine-generating a bigger schema from the schemas in the official tests, to make sure that the test touches as many parts of the spec as possible. Just to make sure to hit any troublesome parts of a validator.

[gilesbowkett]

ok, so the bad news is I don't have as much time as I'd like, but the JSCK benchmarks use five schemas, and you're more than welcome to use them.

here's a categorization of those 5 schemas:

• draft 3, trivial
• draft 3, medium
• draft 4, trivial
• draft 4, medium
• draft 4, complex

so you can use them to benchmark a range of complexities and draft support.

[JamesNK]

Realistic benchmark idea: Validating the draft 4 schema with the draft 4 schema.

[epoberezkin]

@ebdrup The more I think about it, the more I believe that using test-suite for performance benchmarking is completely wrong.

With the exception of some really old and slow libraries, where slow validation may affect UX in the browser, validation performance only really matters for server-side validation. This is where the performance advantage achieved by compiling validators is becoming important... Server-side validation passes 99+% of the time (excluding the cases of DOS attacks and or broken API consumer - but performance is not going to help too much in these cases).

So I think to be relevant, benchmark should only test passing cases on some data samples of small/average/large size.

Tests can still be used to benchmark compliance, separately from performance.

What do you think?