Fix problem with missing file in benchmarks

[ebdrup]

No description provided.

[gilesbowkett]

I believe the correct solution for this is to just do a git submodule update --init. That used to be in the README, but isn't any more. The README's generated from doc/README.pfm.md, so that'll be the file to change.

[gilesbowkett]

OK, I actually fixed this in the README with 5d3c16d, but thanks for catching this – we want the benchmarks to be easy to run, and we should have documented that dependency.

[ebdrup]

Ok thanks.
BTW I created this: https://github.com/Muscula/json-schema-benchmark
Unfortunately I cannot get numbers for jsck because it fails with the error

jsck could not instantiate with schema for "valid definition". 
This is multiple tests failing. 
This excludes this validator from performance tests 
(Unresolvable $ref values: ["http://json-schema.org/draft-04/schema#"])

Do you know what I need to do to make it be included in the benchmark?

[gilesbowkett]

yes, give git submodule update --init a try. it's looking for a file which is pulled in via a submodule.

[ebdrup]

This is a completely new benchmark, run by simply doing npm install of all the modules - just like any developer using the module would do. I fixed a bug that would not run all the tests in some cases. Now there are these failing tests for jsck:

jsck could not instantiate with schema for "valid definition". This is multiple tests failing. This excludes this validator from performance tests (Unresolvable $ref values: ["http://json-schema.org/draft-04/schema#"])

jsck could not instantiate with schema for "invalid definition". This is multiple tests failing. This excludes this validator from performance tests (Unresolvable $ref values: ["http://json-schema.org/draft-04/schema#"])

jsck failed the test "maxLength validation, two supplementary Unicode code points is long enough". Expected result: true but validator returned: false

jsck failed the test "minLength validation, one supplementary Unicode code point is not long enough". Expected result: false but validator returned: true

jsck failed the test "some languages do not distinguish between different types of numeric value, a float is not an integer even without fractional part". Expected result: false but validator returned: true

jsck could not instantiate with schema for "remote ref, containing refs itself". This is multiple tests failing. This excludes this validator from performance tests (Unresolvable $ref values: ["http://json-schema.org/draft-04/schema#"])

jsck could not instantiate with schema for "remote ref". This is multiple tests failing. This excludes this validator from performance tests (Unresolvable $ref values: ["http://localhost:1234/integer.json"])

jsck could not instantiate with schema for "fragment within remote ref". This is multiple tests failing. This excludes this validator from performance tests (Unresolvable $ref values: ["http://localhost:1234/subSchemas.json#/integer"])

jsck could not instantiate with schema for "ref within remote ref". This is multiple tests failing. This excludes this validator from performance tests (Unresolvable $ref values: ["http://localhost:1234/subSchemas.json#/refToInteger"])

jsck could not instantiate with schema for "change resolution scope". This is multiple tests failing. This excludes this validator from performance tests (Unresolvable $ref values: ["http://localhost:1234/"])

jsck failed the test "uniqueItems validation, non-unique array of integers is invalid". Expected result: false but validator returned: true. This excludes this validator from performance tests

jsck failed the test "uniqueItems validation, numbers are unique if mathematically unequal". Expected result: false but validator returned: true. This excludes this validator from performance tests

jsck failed the test "uniqueItems validation, non-unique array of objects is invalid". Expected result: false but validator returned: true. This excludes this validator from performance tests

jsck failed the test "uniqueItems validation, non-unique array of nested objects is invalid". Expected result: false but validator returned: true. This excludes this validator from performance tests

jsck failed the test "uniqueItems validation, non-unique array of arrays is invalid". Expected result: false but validator returned: true. This excludes this validator from performance tests

jsck failed the test "uniqueItems validation, non-unique heterogeneous types are invalid". Expected result: false but validator returned: true. This excludes this validator from performance tests

All other tests passed.

[automatthew]

run by simply doing npm install of all the modules - just like any developer using the module would do.

Benchmarks are not something I would ever want an npm package to do. Yikes.

[ebdrup]

I have a feeling you might have misunderstood me, but I'm not certain. The benchmark is not an npm package. But the validators tested are simply installed with npm. Just like any user of the validators would do. The benchmark I created is using the benchmark module to provide statistically significant results. Something you might consider for your own benchmark. To get rid of that pesky, long running jayschema measurement.

I relaxed the requirements to be part of the benchmark uniqueItems, just for jsck. So now jsck is part of the benchmark. Right now you place 3rd after is-my-json-validx10 speed and themis x2 speed of jsck

[gilesbowkett]

we'll definitely take a look at integrating the benchmark module. but it seems like this conversation's turning into an Issue that should be filed in https://github.com/Muscula/json-schema-benchmark/

[gilesbowkett]

Right now you place 3rd after is-my-json-valid x10 speed and themis x2 speed of jsck

we've already blogged that imjv is faster than jsck, but a PR re themis is of course welcome.

[ebdrup]

Yes. I tried to add themis to your benchmarks, but something was not working right. I also fiddled with the themis benchmarks, but they take two hours to run.

You might find them interesting. They are extremely detailed, to such a level that they are mostly interesting to schema-validator authors themis benchmarks I tried to add jsck there but also had difficulties.

That's when I decided to do an independent benchmark.

[gilesbowkett]

man, I hope "two hours" is hyperbole, because we will probably will dig into them at some point.

anyway, I opened a ticket in your repo to move this discussion there:

ebdrup/json-schema-benchmark#4

[ebdrup]

If you are interested in the result for themis in your benchmarks, this is what I get when I run my attempt to add it to your benchmarks:

## Benchmarks for Draft 4

Schema: 'Event - Valid document'.  A simple schema, exercising very few attributes
Sample size: 64
Validations per sample: 1024

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  tv4
  Warming up: ................................
  Iterations: ................................................................

  jayschema
  Warming up: ................................
  Iterations: ................................................................

  is-my-json-valid
  Warming up: ................................
  Iterations: ................................................................

  themis
  Warming up: ................................
  Iterations: ................................................................


  JSCK: validations/millisecond
  median: 116.848    max: 135.217    min: 68.582 

  tv4: validations/millisecond
  median: 59.104    max: 67.56    min: 43.675 

  jayschema: validations/millisecond
  median: 0.857    max: 1.068    min: 0.713 

  is-my-json-valid: validations/millisecond
  median: 1208.26    max: 1488.372    min: 594.312 

  themis: validations/millisecond
  median: 488.2    max: 607.715    min: 365.193 

Relative speeds:
is-my-json-valid : 1.000
themis : 2.475
JSCK : 10.340
tv4 : 20.443
jayschema : 1409.692


Schema: 'Configuration'.  A moderately complex schema with some nesting and value constraints
Sample size: 64
Validations per sample: 256

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  tv4
  Warming up: ................................
  Iterations: ................................................................

  jayschema
  Warming up: ................................
  Iterations: ................................................................

  is-my-json-valid
  Warming up: ................................
  Iterations: ................................................................

  themis
  Warming up: ................................
  Iterations: ................................................................


  JSCK: validations/millisecond
  median: 57.612    max: 71.369    min: 38.924 

  tv4: validations/millisecond
  median: 20.741    max: 24.056    min: 15.737 

  jayschema: validations/millisecond
  median: 0.46    max: 0.481    min: 0.423 

  is-my-json-valid: validations/millisecond
  median: 606.635    max: 812.698    min: 179.902 

  themis: validations/millisecond
  median: 72.48    max: 86.341    min: 44.329 

Relative speeds:
is-my-json-valid : 1.000
themis : 8.370
JSCK : 10.530
tv4 : 29.249
jayschema : 1317.964


Schema: 'Transaction'.  
Sample size: 64
Validations per sample: 64

  JSCK
  Warming up: ................................
  Iterations: ................................................................

  tv4
  Warming up: ................................
  Iterations: ................................................................

  jayschema
  Warming up: ................................
  Iterations: ................................................................

  is-my-json-valid
  Warming up: ................................
  Iterations: ................................................................

  themis
  Warming up: TypeError: Object #<Object> has no method '0#transaction'
  at Object.validators.0#/items (<anonymous>:72:43)
  at Object.validators.(anonymous function) [as 0] (<anonymous>:59:48)
  at eval (<anonymous>:3826:37)
  at /Users/allanebdrup/Dropbox/Muscula-jsck/node_modules/themis/src/themis.js:2631:22
  at module.exports.themis.validate (/Users/allanebdrup/Dropbox/Muscula-jsck/benchmarks/draft4/validators.coffee:57:8)
  at Benchmark._measure (/Users/allanebdrup/Dropbox/Muscula-jsck/benchmarks/runner.coffee:31:15)
  at Benchmark.module.exports.Benchmark.run (/Users/allanebdrup/Dropbox/Muscula-jsck/benchmarks/benchmark.coffee:27:10)

[gilesbowkett]

thx @MUSCULA -- I moved that to another Issue also (#81). I kind of just want to end the discussion on this particular PR, so we can stay organized. sorry, I'm a bit anal.

[ebdrup]

Sure thing, sorry for messing up your order.