This is a new attempt to fix the add performance

[amorgner]

The new approach uses a classic index which is a better solution for id lookups IMHO.

Details of the issue and the discussion see here

#72

and here

https://groups.google.com/forum/#!msg/neo4j/qgGI60taSmA/POIR3UGks0kJ

[craigtaverner]

I just looked at the new code and I like it since it is much simpler than the old code. But I have two questions about why we are doing this at all?

Making an index of the node-id seems unnecessary, since we can surely just ask neo4j directly if a node with a specific id exists, right and then check if it is a geometry node. That should be faster than using an index, and faster than the old way of iterating a long chain.

[craigtaverner]

Having just read the discussion between Axel and Jonathan on the neo4j forum, I guess perhaps my comments are not competely valid. I made a different comment in the forum.

[peterneubauer]

Very cool! Do we have any test inserting a sizable amount of geometries to test the better performance?

[amorgner]

Yes, there's testAddPerformance() in IndexProviderTest.

When considering to merge this, please look at Jonathan's suggestions regarding index name and deletion in the forum.

I can provide an additional PR tonight for that.

[peterneubauer]

Axel,
I just added you as a committer to Neo4j Spatial for all your great
contribution. Thank you!

On Mon, Jan 7, 2013 at 10:04 AM, amorgner notifications@github.com wrote:

Yes, there's testAddPerformance() in IndexProviderTest.

When considering to merge this, please look at Jonathan's suggestions
regarding index name and deletion in the forum.

I can provide an additional PR tonight for that.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/83#issuecomment-11944764.

[amorgner]

Interesting ... I can't get the testAddPerformance() to pass on my Mac (MPB 8G 2.4GHz SSD). There's a performance drop after abt 14,000 nodes:

testAddPerformance(): 1000 nodes added in 2026ms, total 1000 in 2026ms, speed: 493.58341559723596 adds per second, 2.026 ms per step per node in index
testAddPerformance(): 1000 nodes added in 1241ms, total 2000 in 3267ms, speed: 805.8017727639 adds per second, 0.6205 ms per step per node in index
testAddPerformance(): 1000 nodes added in 1182ms, total 3000 in 4449ms, speed: 846.0236886632827 adds per second, 0.394 ms per step per node in index
testAddPerformance(): 1000 nodes added in 880ms, total 4000 in 5329ms, speed: 1136.3636363636363 adds per second, 0.22 ms per step per node in index
testAddPerformance(): 1000 nodes added in 893ms, total 5000 in 6222ms, speed: 1119.8208286674133 adds per second, 0.1786 ms per step per node in index
testAddPerformance(): 1000 nodes added in 879ms, total 6000 in 7101ms, speed: 1137.6564277588168 adds per second, 0.1465 ms per step per node in index
testAddPerformance(): 1000 nodes added in 870ms, total 7000 in 7971ms, speed: 1149.4252873563219 adds per second, 0.12428571428571429 ms per step per node in index
testAddPerformance(): 1000 nodes added in 1369ms, total 8000 in 9340ms, speed: 730.4601899196493 adds per second, 0.171125 ms per step per node in index
testAddPerformance(): 1000 nodes added in 1294ms, total 9000 in 10634ms, speed: 772.7975270479134 adds per second, 0.14377777777777778 ms per step per node in index
testAddPerformance(): 1000 nodes added in 2173ms, total 10000 in 12807ms, speed: 460.19328117809476 adds per second, 0.2173 ms per step per node in index
testAddPerformance(): 1000 nodes added in 7172ms, total 11000 in 19979ms, speed: 139.43112102621305 adds per second, 0.652 ms per step per node in index
testAddPerformance(): 1000 nodes added in 14154ms, total 12000 in 34133ms, speed: 70.65140596297866 adds per second, 1.1795 ms per step per node in index
testAddPerformance(): 1000 nodes added in 18518ms, total 13000 in 52651ms, speed: 54.00151204233718 adds per second, 1.4244615384615384 ms per step per node in index
testAddPerformance(): 1000 nodes added in 43886ms, total 14000 in 96537ms, speed: 22.786309984961033 adds per second, 3.134714285714286 ms per step per node in index

Same code, tested at my dev box (Linux 3.2.0, Core i7, 8 GB, SSD):

testAddPerformance(): 1000 nodes added in 2115ms, total 1000 in 2115ms, speed: 472.81323877068553 adds per second, 2.115 ms per step per node in index
testAddPerformance(): 1000 nodes added in 1208ms, total 2000 in 3323ms, speed: 827.8145695364238 adds per second, 0.604 ms per step per node in index
testAddPerformance(): 1000 nodes added in 1059ms, total 3000 in 4382ms, speed: 944.2870632672333 adds per second, 0.353 ms per step per node in index
testAddPerformance(): 1000 nodes added in 953ms, total 4000 in 5335ms, speed: 1049.3179433368311 adds per second, 0.23825 ms per step per node in index
testAddPerformance(): 1000 nodes added in 890ms, total 5000 in 6225ms, speed: 1123.5955056179776 adds per second, 0.178 ms per step per node in index
testAddPerformance(): 1000 nodes added in 1240ms, total 6000 in 7465ms, speed: 806.4516129032259 adds per second, 0.20666666666666667 ms per step per node in index
testAddPerformance(): 1000 nodes added in 832ms, total 7000 in 8297ms, speed: 1201.923076923077 adds per second, 0.11885714285714286 ms per step per node in index
testAddPerformance(): 1000 nodes added in 772ms, total 8000 in 9069ms, speed: 1295.3367875647668 adds per second, 0.0965 ms per step per node in index
testAddPerformance(): 1000 nodes added in 864ms, total 9000 in 9933ms, speed: 1157.4074074074074 adds per second, 0.096 ms per step per node in index
testAddPerformance(): 1000 nodes added in 1187ms, total 10000 in 11120ms, speed: 842.4599831508003 adds per second, 0.1187 ms per step per node in index
testAddPerformance(): 1000 nodes added in 789ms, total 11000 in 11909ms, speed: 1267.4271229404308 adds per second, 0.07172727272727272 ms per step per node in index
testAddPerformance(): 1000 nodes added in 728ms, total 12000 in 12637ms, speed: 1373.6263736263736 adds per second, 0.06066666666666667 ms per step per node in index
testAddPerformance(): 1000 nodes added in 601ms, total 13000 in 13238ms, speed: 1663.893510815308 adds per second, 0.04623076923076923 ms per step per node in index
testAddPerformance(): 1000 nodes added in 1306ms, total 14000 in 14544ms, speed: 765.6967840735068 adds per second, 0.09328571428571429 ms per step per node in index
testAddPerformance(): 1000 nodes added in 618ms, total 15000 in 15162ms, speed: 1618.1229773462783 adds per second, 0.0412 ms per step per node in index
testAddPerformance(): 1000 nodes added in 735ms, total 16000 in 15897ms, speed: 1360.544217687075 adds per second, 0.0459375 ms per step per node in index
[...]
testAddPerformance(): 1000 nodes added in 889ms, total 90000 in 82337ms, speed: 1124.859392575928 adds per second, 0.009877777777777777 ms per step per node in index
testAddPerformance(): 1000 nodes added in 736ms, total 91000 in 83073ms, speed: 1358.695652173913 adds per second, 0.008087912087912088 ms per step per node in index
testAddPerformance(): 1000 nodes added in 852ms, total 92000 in 83925ms, speed: 1173.7089201877934 adds per second, 0.009260869565217392 ms per step per node in index
testAddPerformance(): 1000 nodes added in 793ms, total 93000 in 84718ms, speed: 1261.0340479192937 adds per second, 0.008526881720430108 ms per step per node in index
testAddPerformance(): 1000 nodes added in 854ms, total 94000 in 85572ms, speed: 1170.96018735363 adds per second, 0.009085106382978724 ms per step per node in index
testAddPerformance(): 1000 nodes added in 897ms, total 95000 in 86469ms, speed: 1114.8272017837235 adds per second, 0.009442105263157894 ms per step per node in index
testAddPerformance(): 1000 nodes added in 777ms, total 96000 in 87246ms, speed: 1287.001287001287 adds per second, 0.00809375 ms per step per node in index
testAddPerformance(): 1000 nodes added in 815ms, total 97000 in 88061ms, speed: 1226.993865030675 adds per second, 0.008402061855670103 ms per step per node in index
testAddPerformance(): 1000 nodes added in 3555ms, total 98000 in 91616ms, speed: 281.2939521800281 adds per second, 0.03627551020408163 ms per step per node in index
testAddPerformance(): 1000 nodes added in 847ms, total 99000 in 92463ms, speed: 1180.637544273908 adds per second, 0.008555555555555556 ms per step per node in index
testAddPerformance(): 1000 nodes added in 890ms, total 100000 in 93353ms, speed: 1123.5955056179776 adds per second, 0.0089 ms per step per node in index
testAddPerformance(): finishing transaction
Tests run: 8, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 100.23 sec

No time-consuming jobs were running on either system, enough free RAM, no swapping.

The first 7,000 nodes are inserted at nearly equal performance, but after that, the Mac drops off.

Any ideas?

[peterneubauer]

That is very interesting. Any chance to profile that?

On Mon, Jan 7, 2013 at 9:07 PM, amorgner notifications@github.com wrote:

Interesting ... I can't get the testAddPerformance() to pass on my Mac
(MPB 8G 2.4GHz SSD). There's a performance drop after abt 14,000 nodes:

`testAddPerformance(): 1000 nodes added in 2026ms, total 1000 in 2026ms,
speed: 493.58341559723596 adds per second, 2.026 ms per step per node in
index
testAddPerformance(): 1000 nodes added in 1241ms, total 2000 in 3267ms,
speed: 805.8017727639 adds per second, 0.6205 ms per step per node in index
testAddPerformance(): 1000 nodes added in 1182ms, total 3000 in 4449ms,
speed: 846.0236886632827 adds per second, 0.394 ms per step per node in
index
testAddPerformance(): 1000 nodes added in 880ms, total 4000 in 5329ms,
speed: 1136.3636363636363 adds per second, 0.22 ms per step per node in
index
testAddPerformance(): 1000 nodes added in 893ms, total 5000 in 6222ms,
speed: 1119.8208286674133 adds per second, 0.1786 ms per step per node in
index
testAddPerformance(): 1000 nodes added in 879ms, total 6000 in 7101ms,
speed: 1137.6564277588168 adds per second, 0.1465 ms per step per node in
index
testAddPerformance(): 1000 nodes added in 870ms, total 7000 in 7971ms,
speed: 1149.4252873563219 adds per second, 0.12428571428571429 ms per step
per node in index
testAddPerformance(): 1000 nodes added in 1369ms, total 8000 in 9340ms,
speed: 730.4601899196493 adds per second, 0.171125 ms per step per node in
index
testAddPerformance(): 1000 nodes added in 1294ms, total 9000 in 10634ms,
speed: 772.7975270479134 adds per second, 0.14377777777777778 ms per step
per node in index
testAddPerformance(): 1000 nodes added in 2173ms, total 10000 in 12807ms,
speed: 460.19328117809476 adds per second, 0.2173 ms per step per node in
index
testAddPerformance(): 1000 nodes added in 7172ms, total 11000 in 19979ms,
speed: 139.43112102621305 adds per second, 0.652 ms per step per node in
index
testAddPerformance(): 1000 nodes added in 14154ms, total 12000 in 34133ms,
speed: 70.65140596297866 adds per second, 1.1795 ms per step per node in
index
testAddPerformance(): 1000 nodes added in 18518ms, total 13000 in 52651ms,
speed: 54.00151204233718 adds per second, 1.4244615384615384 ms per step
per node in index
testAddPerformance(): 1000 nodes added in 43886ms, total 14000 in 96537ms,
speed: 22.786309984961033 adds per second, 3.134714285714286 ms per step
per node in index

Same code, tested at my dev box (Linux 3.2.9, Core i7, 8 GB, SSD):

testAddPerformance(): 1000 nodes added in 2115ms, total 1000 in 2115ms, speed: 472.81323877068553 adds per second, 2.115 ms per step per node in index
testAddPerformance(): 1000 nodes added in 1208ms, total 2000 in 3323ms, speed: 827.8145695364238 adds per second, 0.604 ms per step per node in index
testAddPerformance(): 1000 nodes added in 1059ms, total 3000 in 4382ms, speed: 944.2870632672333 adds per second, 0.353 ms per step per node in index
testAddPerformance(): 1000 nodes added in 953ms, total 4000 in 5335ms, speed: 1049.3179433368311 adds per second, 0.23825 ms per step per node in index
testAddPerformance(): 1000 nodes added in 890ms, total 5000 in 6225ms, speed: 1123.5955056179776 adds per second, 0.178 ms per step per node in index
testAddPerformance(): 1000 nodes added in 1240ms, total 6000 in 7465ms, speed: 806.4516129032259 adds per second, 0.20666666666666667 ms per step per node in index
testAddPerformance(): 1000 nodes added in 832ms, total 7000 in 8297ms, speed: 1201.923076923077 adds per second, 0.11885714285714286 ms per step per node in index
testAddPerformance(): 1000 nodes added in 772ms, total 8000 in 9069ms, speed: 1295.3367875647668 adds per second, 0.0965 ms per step per node in index
testAddPerformance(): 1000 nodes added in 864ms, total 9000 in 9933ms, speed: 1157.4074074074074 adds per second, 0.096 ms per step per node in index
testAddPerformance(): 1000 nodes added in 1187ms, total 10000 in 11120ms, speed: 842.4599831508003 adds per second, 0.1187 ms per step per node in index
testAddPerformance(): 1000 nodes added in 789ms, total 11000 in 11909ms, speed: 1267.4271229404308 adds per second, 0.07172727272727272 ms per step per node in index
testAddPerformance(): 1000 nodes added in 728ms, total 12000 in 12637ms, speed: 1373.6263736263736 adds per second, 0.06066666666666667 ms per step per node in index
testAddPerformance(): 1000 nodes added in 601ms, total 13000 in 13238ms, speed: 1663.893510815308 adds per second, 0.04623076923076923 ms per step per node in index
testAddPerformance(): 1000 nodes added in 1306ms, total 14000 in 14544ms, speed: 765.6967840735068 adds per second, 0.09328571428571429 ms per step per node in index
testAddPerformance(): 1000 nodes added in 618ms, total 15000 in 15162ms, speed: 1618.1229773462783 adds per second, 0.0412 ms per step per node in index
testAddPerformance(): 1000 nodes added in 735ms, total 16000 in 15897ms, speed: 1360.544217687075 adds per second, 0.0459375 ms per step per node in index
[...]
testAddPerformance(): 1000 nodes added in 889ms, total 90000 in 82337ms, speed: 1124.859392575928 adds per second, 0.009877777777777777 ms per step per node in index
testAddPerformance(): 1000 nodes added in 736ms, total 91000 in 83073ms, speed: 1358.695652173913 adds per second, 0.008087912087912088 ms per step per node in index
testAddPerformance(): 1000 nodes added in 852ms, total 92000 in 83925ms, speed: 1173.7089201877934 adds per second, 0.009260869565217392 ms per step per node in index
testAddPerformance(): 1000 nodes added in 793ms, total 93000 in 84718ms, speed: 1261.0340479192937 adds per second, 0.008526881720430108 ms per step per node in index
testAddPerformance(): 1000 nodes added in 854ms, total 94000 in 85572ms, speed: 1170.96018735363 adds per second, 0.009085106382978724 ms per step per node in index
testAddPerformance(): 1000 nodes added in 897ms, total 95000 in 86469ms, speed: 1114.8272017837235 adds per second, 0.009442105263157894 ms per step per node in index
testAddPerformance(): 1000 nodes added in 777ms, total 96000 in 87246ms, speed: 1287.001287001287 adds per second, 0.00809375 ms per step per node in index
testAddPerformance(): 1000 nodes added in 815ms, total 97000 in 88061ms, speed: 1226.993865030675 adds per second, 0.008402061855670103 ms per step per node in index
testAddPerformance(): 1000 nodes added in 3555ms, total 98000 in 91616ms, speed: 281.2939521800281 adds per second, 0.03627551020408163 ms per step per node in index
testAddPerformance(): 1000 nodes added in 847ms, total 99000 in 92463ms, speed: 1180.637544273908 adds per second, 0.008555555555555556 ms per step per node in index
testAddPerformance(): 1000 nodes added in 890ms, total 100000 in 93353ms, speed: 1123.5955056179776 adds per second, 0.0089 ms per step per node in index
testAddPerformance(): finishing transaction
Tests run: 8, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 100.23 sec

No time-consuming jobs were running on either system, enough free RAM, no
swapping.

The first 7,000 nodes are inserted at nearly equal performance, but after
that, the Mac drops off.

Any ideas?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/83#issuecomment-11968922.

[amorgner]

Got it. Tests failed because of too small heap size. Will add an appropriate section to the pom.xml to make sure tests are running with enough heap space.

[amorgner]

Now the test passes on Travis.

Not sure how to proceed here.

Please up-/downvote if you are pro/con merging this PR.

[peterneubauer]

What is the current performance profile? No degradation anymore? If not, go ahead!

[peterneubauer]

Btw, awesome to get i building on travis, good work! You should announce it on the list or in a blog when merged.

[amorgner]

Performance profile is linear (with some GC pauses depending on tx size and heap size, nothing unusual).

Speed is about 1,250 nodes/s on a Core i7

[craigtaverner]

Seems like a huge performance improvement. I would love to see a comparison of the before and after scenario, so we can really see the difference. I actually tried to do this yesterday, but ran out of time before I could merge your pull requests into my local devenv. I was planning to merge, commit, push, but also think it would be better if you did it.

If all test cases pass (including old ones), then I'm happy that you merge this.