No JavaScript Benchmarks?

[ohenepee]

Would love to see how the built-in JSON and Protobuf fare against colfer. Its kinda unfair to see C, Go, and Java make it to the benchmark games and JS stays at home. Could it be that the JS version couldn't live up to the Colfer performance promise?

[pascaldekloe]

JavaScript was added to give browsers access to backed data so that no conversion would be needed. NodeJS changes all that and I know little about it's performance characteristics. Don't expect peak results yet it should be easy to make it fast.

So how does one make JavaScript benchmarks work? Would you care to help?

[lewisdiamond]

I just tried running it, I could provide the code but I can't marshal + unmarshal and get the same result. It's also more than 2x slower than JSON in my benchmark.

[pascaldekloe]

Please share @lewisdiamond. I'll be happy to make it fast. No problem. You can make a pull request or mail to pascal@quies.net.

[pascaldekloe]

I have a few days off so the sooner the better @lewisdiamond. No need for a nice delivery. I just need a little help with the benchmark setup.

[lewisdiamond]

@pascaldekloe Well the problem I have right now is that the JS code generated by colf either doesn't work or I don't use it properly. Otherwise you just need the benchmark npm package and use it like this.

const Benchmark = require('benchmark');
const suite = new Benchmark.Suite;
const data = { //Data goes here }
const colfer = new b.bench.Bench(data) // Is this how it's supposed to be used?
const marshaled = colfer.marshal()
const stringified = JSON.stringify(data)

suite.add('Colfer > marshal', function() {
    colfer.marshal()
})
    .add('JSON > stringify', function() {
        JSON.stringify(data)
    })
    .add('Colfer > unmarshal', function() {
        colfer.unmarshal(marshaled)
    })
    .add('JSON > parse', function() {
        JSON.parse(stringified)
    })
    .on('cycle', (event) => {
        console.log(String(event.target))
    })
    .run({async: true})
    

[lewisdiamond]

Also, consider making the generated Javascript code actually just load a C version compiled to WASM.

[pascaldekloe]

Yes WASM sounds really cool yet that's another issue/feature.

I'm not sure where the b. in new b.bench.Bench(data) comes from. If you made a type called foo in a package called bar then it should read new bar.Foo(data).

Also, I just fixed issue #40 with an int64 bug. If you use that datatype please try version 1.7.2 and/or Maven plugin 1.11.2.

[lewisdiamond]

Sorry I omitted it.
b is just const b = require('./MyGeneratedColferFile.js')

[pascaldekloe]

Ah, then yes, your setup seems right.

I downloaded benchmark.js including its dependency lodash.js and I populated test.js with the following.

const Lodash = require('./lodash.js');
const Benchmark = require('./benchmark.js');

console.log(Benchmark.Suite);

For some reason node test.js prints "undefined". Do you understand why @lewisdiamond?

[lewisdiamond]

Did you download benchmark.js from there? https://raw.githubusercontent.com/bestiejs/benchmark.js/master/benchmark.js

 $ node
> const benchmark = require('./benchmark')
undefined
> benchmark.Suite
{ [Function: Suite] options: { name: undefined } }

$ node --version
v9.3.0

[pascaldekloe]

That exact file yes. Now I know this is supposed to work. Thanks for helping out.

% node --version
v8.9.4
% node
> const benchmark = require('./benchmark');
undefined
> benchmark.Suite
undefined
> benchmark
{ [Function: Benchmark]
  runInContext: [Function: runInContext],
  Benchmark: [Circular] }

[lewisdiamond]

It's likely because you don't have the lodash dependency.

const Lodash = require('./lodash.js');

That does not fulfill the dependency requirement.
They get lodash like this:

var _ = context && context._ || require('lodash') || root._;

Try this:

root._ = require('./lodash')
const Benchmark = require('benchmark')

Alternatively, use npm or yarn:

$cd my_directory
$npm init //or yarn init
$npm install lodash //or yarn add lodash
$npm install benchmark //or yarn add benchmark

then you can just do this:

const Benchmark = require('benchmark') //notice no leading ./

[pascaldekloe]

That root import did the trick and the suggested framework plays nicely. I can start tuning now.

[pascaldekloe]

The current (untuned) code compared to the native JSON.

marshal Colfer x 86,798 ops/sec ±0.87% (87 runs sampled)
unmarshal Colfer x 356,519 ops/sec ±0.17% (90 runs sampled)
marshal JSON x 221,298 ops/sec ±0.48% (94 runs sampled)
unmarshal JSON x 214,750 ops/sec ±0.12% (93 runs sampled)

[lewisdiamond]

Check out protobuf.js https://github.com/dcodeIO/ProtoBuf.js/
It's one of the only format faster than the built-in JSON on nodejs. I guess JSON must be written in C/C++. The only way to get close to it may be to use WASM or C++ bindings.

Note that it may still be worth to have a browser compatible version (at least for older browser since new ones support WASM)

[pascaldekloe]

The ProtoBuf.js link was quite useful as a reference @lewisdiamond. I did not expect JavaScript to be 50 to 100 times slower than most languages. 😧 This basically means that if you care about performance at all then JavaScript is a no-go and we need WASM to save the day.

Colfer is a bit faster than Node's native [C++] JSON now but not by much.

marshal Colfer x 197,269 ops/sec ±1.02% (84 runs sampled)
unmarshal Colfer x 354,110 ops/sec ±0.26% (89 runs sampled)
marshal JSON x 220,158 ops/sec ±0.40% (93 runs sampled)
unmarshal JSON x 207,980 ops/sec ±0.69% (91 runs sampled)

For some reason the standard DataView/Uint8Array performs horrible. ProtoBuf.js made it's own libraries for integer and floating point encoding to compensate.

Doubling the speed with more tuning effort is quite possible yet because those numbers still look bad I'll leave things as they are for now and defer to V8 for TypedArray optimisation.

[pascaldekloe]

https://golang.org/doc/go1.11#wasm