New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

benchmark: add microbenchmarks for ES Map #7581

Closed
wants to merge 3 commits into
base: master
from

Conversation

Projects
None yet
10 participants
@rvagg
Member

rvagg commented Jul 7, 2016

Given the discussion in #6102 about Map still being too slow to use I figured I'd see with a benchmark. @mscdex, @jasnell please review and let me know if you think this is valid.

es/map-bench.js method="pojo" millions="10": 1.07989
es/map-bench.js method="fakeMap" millions="10": 0.85223
es/map-bench.js method="map" millions="10": 1.77412

As for why the fakeMap test is faster ... I got nothing ...


Update after @fhinkel noticed a serious flaw:

v6

es/map-bench.js millions=1 method="object": 0.18984850416666324
es/map-bench.js millions=1 method="nullProtoObject": 0.16044478686078825
es/map-bench.js millions=1 method="fakeMap": 0.1311882248350222
es/map-bench.js millions=1 method="map": 0.3121014937928409

master

es/map-bench.js millions=1 method="object": 0.2220851555085364
es/map-bench.js millions=1 method="nullProtoObject": 0.19131381289232038
es/map-bench.js millions=1 method="fakeMap": 0.17931948397493905
es/map-bench.js millions=1 method="map": 0.28127750318241235

Times are ops/sec, so higher is better

@Fishrock123

View changes

Show outdated Hide outdated benchmark/es/map-bench.js Outdated
@addaleax

This comment has been minimized.

Show comment
Hide comment
@addaleax

addaleax Jul 7, 2016

Member

Not sure if it matters, but it might be nice to have one for m = Object.create(null), too.

Member

addaleax commented Jul 7, 2016

Not sure if it matters, but it might be nice to have one for m = Object.create(null), too.

@rvagg

This comment has been minimized.

Show comment
Hide comment
@rvagg

rvagg Jul 7, 2016

Member

ok, s/pojo/object and added one for a null-prototype object

es/map-bench.js method="object" millions="10": 1.07722
es/map-bench.js method="nullProtoObject" millions="10": 1.04962
es/map-bench.js method="fakeMap" millions="10": 0.88726
es/map-bench.js method="map" millions="10": 1.78555
Member

rvagg commented Jul 7, 2016

ok, s/pojo/object and added one for a null-prototype object

es/map-bench.js method="object" millions="10": 1.07722
es/map-bench.js method="nullProtoObject" millions="10": 1.04962
es/map-bench.js method="fakeMap" millions="10": 0.88726
es/map-bench.js method="map" millions="10": 1.78555
@cjihrig

This comment has been minimized.

Show comment
Hide comment
@cjihrig

cjihrig Jul 7, 2016

Contributor

Should this utilize common.v8ForceOptimization()?

Contributor

cjihrig commented Jul 7, 2016

Should this utilize common.v8ForceOptimization()?

@cjihrig

View changes

Show outdated Hide outdated benchmark/es/map-bench.js Outdated
@rvagg

This comment has been minimized.

Show comment
Hide comment
@rvagg

rvagg Jul 7, 2016

Member

s/let/var, thanks for pointing that out.

I don't know if this is a candidate for common.v8ForceOptimization() or not, there's a bit of state to pass around, it'd at least be passing in m to optimized functions from the parent bench functions, would that make a difference?

Member

rvagg commented Jul 7, 2016

s/let/var, thanks for pointing that out.

I don't know if this is a candidate for common.v8ForceOptimization() or not, there's a bit of state to pass around, it'd at least be passing in m to optimized functions from the parent bench functions, would that make a difference?

@cjihrig

This comment has been minimized.

Show comment
Hide comment
@cjihrig

cjihrig Jul 7, 2016

Contributor

I'll defer to someone with more experience working on the benchmarks, but es/restparams-bench.js, for example, forces optimization with arguments.

Contributor

cjihrig commented Jul 7, 2016

I'll defer to someone with more experience working on the benchmarks, but es/restparams-bench.js, for example, forces optimization with arguments.

@mscdex mscdex added the V8 Engine label Jul 7, 2016

@mscdex

This comment has been minimized.

Show comment
Hide comment
@mscdex

mscdex Jul 7, 2016

Contributor

AFAIK that should be (million) ops/sec, so technically Map would be the fastest (largest value). That's quite surprising.

Contributor

mscdex commented Jul 7, 2016

AFAIK that should be (million) ops/sec, so technically Map would be the fastest (largest value). That's quite surprising.

@jasnell

This comment has been minimized.

Show comment
Hide comment
@jasnell

jasnell Jul 7, 2016

Member

Interesting! Map is definitely showing to be faster... even if you take creation of the Map into consideration on each loop.

Unmodified version:

~/node/main/node [check] $ ./node benchmark/es/map-bench.js
es/map-bench.js method="object" millions="10": 0.84279
es/map-bench.js method="nullProtoObject" millions="10": 0.84733
es/map-bench.js method="fakeMap" millions="10": 0.69652
es/map-bench.js method="map" millions="10": 1.71819

Modified to include map creation on each loop:

~/node/main/node [check] $ ./node benchmark/es/map-bench.js
es/map-bench.js method="object" millions="10": 0.77879
es/map-bench.js method="nullProtoObject" millions="10": 0.77240
es/map-bench.js method="fakeMap" millions="10": 0.37927
es/map-bench.js method="map" millions="10": 1.61652
Member

jasnell commented Jul 7, 2016

Interesting! Map is definitely showing to be faster... even if you take creation of the Map into consideration on each loop.

Unmodified version:

~/node/main/node [check] $ ./node benchmark/es/map-bench.js
es/map-bench.js method="object" millions="10": 0.84279
es/map-bench.js method="nullProtoObject" millions="10": 0.84733
es/map-bench.js method="fakeMap" millions="10": 0.69652
es/map-bench.js method="map" millions="10": 1.71819

Modified to include map creation on each loop:

~/node/main/node [check] $ ./node benchmark/es/map-bench.js
es/map-bench.js method="object" millions="10": 0.77879
es/map-bench.js method="nullProtoObject" millions="10": 0.77240
es/map-bench.js method="fakeMap" millions="10": 0.37927
es/map-bench.js method="map" millions="10": 1.61652
@rvagg

This comment has been minimized.

Show comment
Hide comment
@rvagg

rvagg Jul 8, 2016

Member

ok then, that is interesting and explains why fakeMap looked "better" than the pojo, perhaps we go ahead and migrate headers to a Map eh?

Member

rvagg commented Jul 8, 2016

ok then, that is interesting and explains why fakeMap looked "better" than the pojo, perhaps we go ahead and migrate headers to a Map eh?

@mscdex

This comment has been minimized.

Show comment
Hide comment
@mscdex

mscdex Jul 8, 2016

Contributor

@rvagg That would probably have to happen at least in v7 or later since that would be a pretty big change.

Contributor

mscdex commented Jul 8, 2016

@rvagg That would probably have to happen at least in v7 or later since that would be a pretty big change.

@simonkcleung

This comment has been minimized.

Show comment
Hide comment
@simonkcleung

simonkcleung Jul 10, 2016

The number should be the higher the better?

simonkcleung commented Jul 10, 2016

The number should be the higher the better?

@mscdex

This comment has been minimized.

Show comment
Hide comment
@mscdex

mscdex Jul 10, 2016

Contributor

@simonkcleung Yes. Values are always operations per second.

Contributor

mscdex commented Jul 10, 2016

@simonkcleung Yes. Values are always operations per second.

@jasnell

This comment has been minimized.

Show comment
Hide comment
@jasnell

jasnell Jul 20, 2016

Member

Moving the querystring and headers objects to Maps definitely makes sense given the perf results but @Fishrock123 is right about targeting such a change for v7. We can likely go ahead and get that change landed in master now as a semver-major.

Member

jasnell commented Jul 20, 2016

Moving the querystring and headers objects to Maps definitely makes sense given the perf results but @Fishrock123 is right about targeting such a change for v7. We can likely go ahead and get that change landed in master now as a semver-major.

@ofrobots ofrobots referenced this pull request Jul 22, 2016

Closed

http: don't inherit from Object.prototype #6102

3 of 3 tasks complete
@fhinkel

View changes

Show outdated Hide outdated benchmark/es/map-bench.js Outdated
@fhinkel

This comment has been minimized.

Show comment
Hide comment
@fhinkel

fhinkel Jul 29, 2016

Member

@rvagg Thanks for putting this together.

When we're using named keys o[key] on an object, access is usually megamorphic. V8 only has limited space in the megamorphic IC stub cache, around 2500 entries. The same cache is used for all megamorphic ICs, which can be completely unrelated to o.

Every time we add a property to an object, we create a new transition map. Only if we add properties in exactly the same order can we reuse the maps. This is often not the case, for example when we parse request headers into _headers[key]. Because we need to keep track of the transitions, the number of maps quickly increases. All these maps fight for entries in the megamorphic IC stub cache, eventually overwriting maps for other objects that are unrelated to o, which slows down the application overall.

So +1 for moving to maps!

Member

fhinkel commented Jul 29, 2016

@rvagg Thanks for putting this together.

When we're using named keys o[key] on an object, access is usually megamorphic. V8 only has limited space in the megamorphic IC stub cache, around 2500 entries. The same cache is used for all megamorphic ICs, which can be completely unrelated to o.

Every time we add a property to an object, we create a new transition map. Only if we add properties in exactly the same order can we reuse the maps. This is often not the case, for example when we parse request headers into _headers[key]. Because we need to keep track of the transitions, the number of maps quickly increases. All these maps fight for entries in the megamorphic IC stub cache, eventually overwriting maps for other objects that are unrelated to o, which slows down the application overall.

So +1 for moving to maps!

@jasnell

This comment has been minimized.

Show comment
Hide comment
@jasnell

jasnell Jul 29, 2016

Member

btw, LGTM for getting this landed. :-)

Member

jasnell commented Jul 29, 2016

btw, LGTM for getting this landed. :-)

@jasnell

This comment has been minimized.

Show comment
Hide comment
@jasnell

jasnell Aug 4, 2016

Member

@rvagg ... is this ready to land?

Member

jasnell commented Aug 4, 2016

@rvagg ... is this ready to land?

@addaleax

This comment has been minimized.

Show comment
Hide comment
@addaleax

addaleax Aug 4, 2016

Member

I think @fhinkel’s comment should still be addressed?

Member

addaleax commented Aug 4, 2016

I think @fhinkel’s comment should still be addressed?

@Florian-R Florian-R referenced this pull request Sep 20, 2016

Merged

timers: improve setTimeout/Interval performance #8661

2 of 2 tasks complete
@addaleax

This comment has been minimized.

Show comment
Hide comment
@addaleax
Member

addaleax commented Sep 20, 2016

ping @rvagg

@addaleax addaleax referenced this pull request Sep 20, 2016

Closed

module: NativeModule._cache turned to a Map #8576

2 of 2 tasks complete
@rvagg

This comment has been minimized.

Show comment
Hide comment
@rvagg

rvagg Oct 18, 2016

Member

Thanks for catching that @fhinkel, that changes everything!

Updates pushed, pls review folks.

v6

es/map-bench.js thousands=100 method="object": 0.26339184977485885
es/map-bench.js thousands=100 method="nullProtoObject": 0.23501547831916525
es/map-bench.js thousands=100 method="fakeMap": 0.212913507413461
es/map-bench.js thousands=100 method="map": 0.7449254838002358

master

es/map-bench.js thousands=100 method="object": 0.35728933045594424
es/map-bench.js thousands=100 method="nullProtoObject": 0.029615543283847124
es/map-bench.js thousands=100 method="fakeMap": 0.028427891784159526
es/map-bench.js thousands=100 method="map": 0.15582199860352638
Member

rvagg commented Oct 18, 2016

Thanks for catching that @fhinkel, that changes everything!

Updates pushed, pls review folks.

v6

es/map-bench.js thousands=100 method="object": 0.26339184977485885
es/map-bench.js thousands=100 method="nullProtoObject": 0.23501547831916525
es/map-bench.js thousands=100 method="fakeMap": 0.212913507413461
es/map-bench.js thousands=100 method="map": 0.7449254838002358

master

es/map-bench.js thousands=100 method="object": 0.35728933045594424
es/map-bench.js thousands=100 method="nullProtoObject": 0.029615543283847124
es/map-bench.js thousands=100 method="fakeMap": 0.028427891784159526
es/map-bench.js thousands=100 method="map": 0.15582199860352638
@fhinkel

This comment has been minimized.

Show comment
Hide comment
@fhinkel

fhinkel Oct 18, 2016

Member

LGTM. Should we update the first comment with the new times and also point out, that it's ops/sec rather than seconds (so higher is faster).

Member

fhinkel commented Oct 18, 2016

LGTM. Should we update the first comment with the new times and also point out, that it's ops/sec rather than seconds (so higher is faster).

@rvagg

This comment has been minimized.

Show comment
Hide comment
@rvagg

rvagg Oct 18, 2016

Member

Good idea, done! @fhinkel is there a known speed regression for maps in later V8? We're comparing 5.1 to 5.4 here and that's a big slowdown, or could the test not be getting at the real numbers?

Member

rvagg commented Oct 18, 2016

Good idea, done! @fhinkel is there a known speed regression for maps in later V8? We're comparing 5.1 to 5.4 here and that's a big slowdown, or could the test not be getting at the real numbers?

@fhinkel

This comment has been minimized.

Show comment
Hide comment
@fhinkel

fhinkel Oct 18, 2016

Member

I wonder if something else was going on during the benchmarking, because nullProtoObject and fakeMap also have a 10x slowdown. Let me try on my machine ...

Member

fhinkel commented Oct 18, 2016

I wonder if something else was going on during the benchmarking, because nullProtoObject and fakeMap also have a 10x slowdown. Let me try on my machine ...

@rvagg

This comment has been minimized.

Show comment
Hide comment
@rvagg

rvagg Oct 18, 2016

Member

yea, I think you're right, new run on master:

es/map-bench.js thousands=100 method="object": 0.10768888702936505
es/map-bench.js thousands=100 method="nullProtoObject": 0.11350308225508328
es/map-bench.js thousands=100 method="fakeMap": 0.12039689515526422
es/map-bench.js thousands=100 method="map": 0.38445207528844944
Member

rvagg commented Oct 18, 2016

yea, I think you're right, new run on master:

es/map-bench.js thousands=100 method="object": 0.10768888702936505
es/map-bench.js thousands=100 method="nullProtoObject": 0.11350308225508328
es/map-bench.js thousands=100 method="fakeMap": 0.12039689515526422
es/map-bench.js thousands=100 method="map": 0.38445207528844944
@rvagg

This comment has been minimized.

Show comment
Hide comment
@rvagg

rvagg Oct 18, 2016

Member

Too much variability between runs, bumping it back up to the millions and it stabilises a bit more:

v6

es/map-bench.js millions=1 method="object": 0.18984850416666324
es/map-bench.js millions=1 method="nullProtoObject": 0.16044478686078825
es/map-bench.js millions=1 method="fakeMap": 0.1311882248350222
es/map-bench.js millions=1 method="map": 0.3121014937928409

master

es/map-bench.js millions=1 method="object": 0.2220851555085364
es/map-bench.js millions=1 method="nullProtoObject": 0.19131381289232038
es/map-bench.js millions=1 method="fakeMap": 0.17931948397493905
es/map-bench.js millions=1 method="map": 0.28127750318241235
Member

rvagg commented Oct 18, 2016

Too much variability between runs, bumping it back up to the millions and it stabilises a bit more:

v6

es/map-bench.js millions=1 method="object": 0.18984850416666324
es/map-bench.js millions=1 method="nullProtoObject": 0.16044478686078825
es/map-bench.js millions=1 method="fakeMap": 0.1311882248350222
es/map-bench.js millions=1 method="map": 0.3121014937928409

master

es/map-bench.js millions=1 method="object": 0.2220851555085364
es/map-bench.js millions=1 method="nullProtoObject": 0.19131381289232038
es/map-bench.js millions=1 method="fakeMap": 0.17931948397493905
es/map-bench.js millions=1 method="map": 0.28127750318241235
@fhinkel

This comment has been minimized.

Show comment
Hide comment
@fhinkel

fhinkel Oct 18, 2016

Member

I think we can blame the variation on timing issues. I don't get a regression going from v6.0.0 to master.

Node v6.0.0:

~/node/benchmark/es$ node --version
v6.0.0
~/node/benchmark/es$ node -e "console.log(process.versions.v8)"
5.0.71.35
~/node/benchmark/es$ node map-bench.js 
es/map-bench.js millions=1 method="object": 0.3697918357000556
es/map-bench.js millions=1 method="nullProtoObject": 0.41595417356334285
es/map-bench.js millions=1 method="fakeMap": 0.36001723720769746
es/map-bench.js millions=1 method="map":  0.5557669893876821

Node v6.7.0:

~/node/benchmark/es$ node -e "console.log(process.versions.v8)"
5.1.281.83
~/node/benchmark/es$ node map-bench.js 
es/map-bench.js millions=1 method="object": 0.4192028932525165
es/map-bench.js millions=1 method="nullProtoObject": 0.43625871109888
es/map-bench.js millions=1 method="fakeMap": 0.3631661863732719
es/map-bench.js millions=1 method="map": 0.5750817453541195

Node master:

~/node/benchmark/es$ ~/node/node -e "console.log(process.versions.v8)"
5.5.0 (candidate)
~/node/benchmark/es$ ~/node/node map-bench.js 
es/map-bench.js millions=1 method="object": 0.43575744375713865
es/map-bench.js millions=1 method="nullProtoObject": 0.4588149868110948
es/map-bench.js millions=1 method="fakeMap": 0.4051635672142672
es/map-bench.js millions=1 method="map": 0.6342556913214041

Member

fhinkel commented Oct 18, 2016

I think we can blame the variation on timing issues. I don't get a regression going from v6.0.0 to master.

Node v6.0.0:

~/node/benchmark/es$ node --version
v6.0.0
~/node/benchmark/es$ node -e "console.log(process.versions.v8)"
5.0.71.35
~/node/benchmark/es$ node map-bench.js 
es/map-bench.js millions=1 method="object": 0.3697918357000556
es/map-bench.js millions=1 method="nullProtoObject": 0.41595417356334285
es/map-bench.js millions=1 method="fakeMap": 0.36001723720769746
es/map-bench.js millions=1 method="map":  0.5557669893876821

Node v6.7.0:

~/node/benchmark/es$ node -e "console.log(process.versions.v8)"
5.1.281.83
~/node/benchmark/es$ node map-bench.js 
es/map-bench.js millions=1 method="object": 0.4192028932525165
es/map-bench.js millions=1 method="nullProtoObject": 0.43625871109888
es/map-bench.js millions=1 method="fakeMap": 0.3631661863732719
es/map-bench.js millions=1 method="map": 0.5750817453541195

Node master:

~/node/benchmark/es$ ~/node/node -e "console.log(process.versions.v8)"
5.5.0 (candidate)
~/node/benchmark/es$ ~/node/node map-bench.js 
es/map-bench.js millions=1 method="object": 0.43575744375713865
es/map-bench.js millions=1 method="nullProtoObject": 0.4588149868110948
es/map-bench.js millions=1 method="fakeMap": 0.4051635672142672
es/map-bench.js millions=1 method="map": 0.6342556913214041

@rvagg rvagg closed this Oct 29, 2016

@rvagg rvagg deleted the rvagg:es-map-bench branch Oct 29, 2016

@rvagg

This comment has been minimized.

Show comment
Hide comment
@rvagg

rvagg Oct 29, 2016

Member

landed @ 07cc9df

Member

rvagg commented Oct 29, 2016

landed @ 07cc9df

rvagg added a commit that referenced this pull request Oct 29, 2016

benchmark: add microbenchmarks for ES Map
PR-URL: #7581
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Franziska Hinkelmann <ranziska.hinkelmann@gmail.com>

evanlucas added a commit that referenced this pull request Nov 3, 2016

benchmark: add microbenchmarks for ES Map
PR-URL: #7581
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Franziska Hinkelmann <ranziska.hinkelmann@gmail.com>

MylesBorins added a commit that referenced this pull request Nov 22, 2016

benchmark: add microbenchmarks for ES Map
PR-URL: #7581
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Franziska Hinkelmann <ranziska.hinkelmann@gmail.com>

MylesBorins added a commit that referenced this pull request Nov 22, 2016

benchmark: add microbenchmarks for ES Map
PR-URL: #7581
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Franziska Hinkelmann <ranziska.hinkelmann@gmail.com>

This was referenced Nov 22, 2016

@lpinca lpinca referenced this pull request Jan 28, 2017

Closed

timers: use Maps for internal duration pooling #11040

2 of 3 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment