PingPong latency test across /dev/shm/SharedHashMap IPC ...

[Cotton-Ben]

I have a test that involves 2 Java VM PIDs ping-pong'g the put()/get() of a primitive double by K into a /dev/shm/SharedHashMap.

PID 1 represents a ping pong player positoned on the LEFT side of the table.
PID 2 represents a ping pong player positoned on the RIGHT side of the table.

PID 1 only does a shm.put(K, 4.0);
PID 2 only does a shm.put(K,5.0);

After either PID does its .put() it then spins-busy waiting for the player on the other side to "ping-pong" back their put.

PID 1 -- the player on the left -- must always have its JVM started first for the test to work.

I am getting consistently < 10us latency for this ping-pong latency test.

My 2 questions are these -- on line 38 of each .java at

https://github.com/Cotton-Ben/HugeCollections/tree/master/collections/src/test/java/net/openhft/collections/fromdocs/pingpong_latency

(1) why do I have to do a re-assignment

 _bondEntryV = shmLeft.acquireUsing("369604103", bondOffHeap);

instead of just re-using the earlier computed off-Heap reference to '_bondEntryV' in the spin-busy waiting loop?

(2) I see that OpenHFT now has a SharedMapEventListeners API ... is this ready? Are there any existing tests I could inspect to affect my learning how to use this API? This API should provide my relief from having to do these spin-busy waiting loops, correct?

[peter-lawrey]

You don't need to redo the assignment unless the key has been deleted.

This listener will be notified about all changes. This is intended to
allow you to update meta data as required (or any number of other actions)

On 3 June 2014 15:01, Ben Cotton notifications@github.com wrote:

I have a test that involves 2 Java VM PIDs ping-pong'g the put()/get() of
a primitive double by K into a /dev/shm/SharedHashMap.

PID 1 represents a ping pong player positoned on the LEFT side of the
table.
PID 2 represents a ping pong player positoned on the RIGHT side of the
table.

PID 1 only does a shm.put(K, 4.0);
PID 2 only does a shm.put(K,5.0);

After either PID does its .put() it then spins-busy waiting for the player
on the other side to "ping-pong" back their put.

PID 1 -- the player on the left -- must always have its JVM started first
for the test to work.

I am getting consistently < 10us latency for this ping-pong latency test.

My 2 questions are these -- on line 38 of each .java at

https://github.com/Cotton-Ben/HugeCollections/tree/master/collections/src/test/java/net/openhft/collections/fromdocs/pingpong_latency

(1) why do I have to do a re-assignment

_bondEntryV = shmLeft.acquireUsing("369604103", bondOffHeap);

instead of just re-using the earlier computed off-Heap reference to
'_bondEntryV' in the spin-busy waiting loop?

(2) I see that OpenHFT now has a SharedMapEventListeners API ... is this
ready? Are there any existing tests I could inspect to affect my learning
how to use this API? This API should provide my relief from having to do
these spin-busy waiting loops, correct?

—
Reply to this email directly or view it on GitHub
#28.

[Cotton-Ben]

You are 100% correct. I coded the PingPong test wrong. My bad.

[peter-lawrey]

Note: if the key is deleted in another thread, the off heap reference can
become invalid (or point to a value of another key).

On 3 June 2014 16:42, Ben Cotton notifications@github.com wrote:

Closed #28 #28.

—
Reply to this email directly or view it on GitHub
#28 (comment).

[Cotton-Ben]

@peter-lawrey : many thanks for adding the CAS support for primitve double ... by using CAS operators on my existing off-heap reference( in my central spin-busy wait loop) my tests now run with mean ping-pong latency (across PID1 to PID2 via /dev/shm IPC transport) of 80 nanos (instead of 10 micros).

[peter-lawrey]

The ping-pong latency is perhaps a better indication of what you should get
(cf rather than half ping-pong)

BTW I recommend mounting tmpfs onto /tmp.

On 4 June 2014 11:14, Ben Cotton notifications@github.com wrote:

@peter-lawrey https://github.com/peter-lawrey : many thanks for adding
the CAS support for primitve double ... by using CAS operators on my
existing off-heap reference( in my central spin-busy wait loop) my tests
now run with mean ping-pong latency (across PID1 to PID2 via /dev/shm IPC
transport) of 80 nanos (instead of 10 micros).

—
Reply to this email directly or view it on GitHub
#28 (comment)
.

[Cotton-Ben]

BTW I recommend mounting tmpfs onto /tmp.

TBH, I never "got it" wrt to how that can be any better of an off-heap destination than native /dev/shm .... I still don't get it. Can it really be better? Neither involve any kernel context hops, correct?

[peter-lawrey]

It's better in the sense that /tmp is a natural place for temporary files
e.g. java.io.tmpdir and none of the temporary need to be persisted to disk.
i.e. it helps all programs which use temporary files.

For testing /tmp makes more sense IMHO, however for production style use
(which you could be testing) /dev/shm might make more sense as it is kept
separate and has it's own size limit.

On 4 June 2014 11:25, Ben Cotton notifications@github.com wrote:

BTW I recommend mounting tmpfs onto /tmp.

TBH, I never "got it" wrt to how that can be any better of an off-heap
destination than native /dev/shm .... I still don't get it. Can it really
be better? Neither involve any kernel context hops, correct?

—
Reply to this email directly or view it on GitHub
#28 (comment)
.

[RobAustin]

:-), I'm not sure if this is relating to one of the checking's I made, but we try to, when writing unit test point to System.getProperty("java.io.tmpdir") + "/", as this test maybe run on different architecture not just linux. It also is important when run with TeamCity as each agent has its own TMP directory.

[Cotton-Ben]

@peter-lawrey +@BoundedBuffer -- true. and all very reasonable considerations, especially from your "Build for all" vantage ... but do know this: we are unbothered wrt to accommodating non-Linux OS providers. :)

[RobAustin]

:-), sure, but some of our user still use windows ! :-)

[Cotton-Ben]

Ok, we'll behave civilly.

[Frank-Bradley]

This very ineresting to me. Would you guys mind if my asking:

For ping pong latency between PID1s and PID2s processes, I am looking at the link provided, which BTW does not show the CAS usage in the code pushed to the repo URL. Is it true from that the original problem came from there being a need to do sort of an "IPC synchronized(shm) {/* ...*/}" across the 2 PIDs local views with a main view of the SharedHashMap? If so is it also true that to cross some underlying OpenHFT "IPC read/write barrier" across processes that you must use

1. CAS operations (for primiive double -- and all other primitives? Object? can I see sample code?)
2. shmLeft.acquireUsing("369604103", bondOffHeap);
3. SharedMapEventListeners API (can I see sample code?)

but using

1. _coupon = bonddOffHeap.getCoupon();

will not cross not cross a read barrier to the main memory view of SharedHashMap

?

Is what you do with OpenHFT across OS processes and a main view of an off heap SharedHashMap exactly what synchronized/volatile etc. do with acoss JVM ThreadLocal and the Java JMM?

@Cotton-Ben please show us your use of CAS to solve the problem (I don't think you pushed what you are describing to the repo url)

@peter-lawrey please show us use of SharedMapEventListeners to solve the problem.

Sorry for such a long list of an email. but this is very interesting.

[Cotton-Ben]

@Frank-Bradley great questions. you go the idea, but I'll let the experts answer in detail.

I can't push my Tests right at this moment (give me a few hous) for now, check out this link of how to use

bondOffHeap.compareAndSwapCoupon();   //Also see the file BondVOInterface.java at the link below

https://github.com/OpenHFT/HugeCollections/tree/master/collections/src/test/java/net/openhft/collections/fromdocs/pingpong_latency

[peter-lawrey]

Some clients use Windows for development, even if their servers use Linux.

On 4 June 2014 11:36, Rob Austin notifications@github.com wrote:

:-), sure, but some of our user still use windows ! :-)

—
Reply to this email directly or view it on GitHub
#28 (comment)
.

[peter-lawrey]

BTW does not show the CAS usage in the code pushed to the repo URL

You need my version, which Ben has sent.

Is it true from that the original problem came from there being a need
to do sort of an "IPC synchronized(shm) {/* ...*/}" across the 2 PIDs local
views with a main view of the SharedHashMap?

You can't use synchronize across two processes and as such we don't
use/support this. We use other primitives which are safe across processes.

CAS operations (for primiive double -- and all other primitives? Object?
can I see sample code?)

Current CAS is supported for int, long and double. float could be added,
but I discourage the use of float, unless you really, really have a good
use case. You can't have off heap references to on heap Objects. (or at
least you shouldn't as it can be done, just not a good idea)

In this case, CAS is fastest, but if you want a different interaction,
there is busyLockRecord() & unlockRecord() which give exclusive access to
the record for complex operations.

will not cross not cross a read barrier to the main memory view of
SharedHashMap

Correct, which is why it didn't work for Ben, and why I replaced it with a
CAS.

Is what you do with OpenHFT across OS processes and a main view of an off
heap SharedHashMap exactly what synchronized/volatile etc. do with accoss
JVM ThreadLocal and the Java JMM?

Not exactly. It does a minimal subset e.g. there is no blocking lock
operations, no wait/notify, no volatile write (only ordered/lazy write)
There is no requirement for ThreadLocal functionality.

On 4 June 2014 12:15, Frank-Bradley notifications@github.com wrote:

This very ineresting to me. Would you guys mind if my asking:

For ping pong latency between PID1s and PID2s processes, I am looking at
the link provided, which BTW does not show the CAS usage in the code pushed
to the repo URL. Is it true from that the original problem came from there
being a need to do sort of an "IPC synchronized(shm) {/* ...*/}" across the
2 PIDs local views with a main view of the SharedHashMap? If so is it also
true that to cross some underlying OpenHFT "IPC read/write barrier" across
processes that you must use

1. CAS operations (for primiive double -- and all other primitives?
   Object? can I see sample code?)
2. shmLeft.acquireUsing("369604103", bondOffHeap);
3. SharedMapEventListeners API (can I see sample code?)

but using

1. _coupon = bonddOffHeap.getCoupon();

will not cross not cross a read barrier to the main memory view of
SharedHashMap

?

Is what you do with OpenHFT across OS processes and a main view of an off
heap SharedHashMap exactly what synchronized/volatile etc. do with acoss
JVM ThreadLocal and the Java JMM?

@Cotton-Ben https://github.com/Cotton-Ben please show us your use of
CAS to solve the problem (I don't think you pushed what you are describing
to the repo url)

@peter-lawrey https://github.com/peter-lawrey please show us use of
SharedMapEventListeners to solve the problem.

Sorry for such a long list of an email. but this is very interesting.

—
Reply to this email directly or view it on GitHub
#28 (comment)
.

[RobAustin]

BTW earlier today

I’ve ignored these test

net.openhft.collections.fromdocs.pingpong_latency.PingPongPlayerLeft#bondExample
net.openhft.collections.fromdocs.pingpong_latency.PingPongPlayerRight#bondExample

because :

Connected to the target VM, address: '127.0.0.1:51263', transport: 'socket'
/net/openhft/collections/fromdocs/BondVOInterface$$Native.java:41: error: cannot find symbol
return _bytes.compareAndSwapDouble(_offset + COUPON, _1, _2);
^
symbol: method compareAndSwapDouble(long,double,double)
location: variable _bytes of type Bytes
1 error
Disconnected from the target VM, address: '127.0.0.1:51263', transport: 'socket'

java.lang.AssertionError: java.lang.ClassNotFoundException: net.openhft.collections.fromdocs.BondVOInterface$$Native
at net.openhft.lang.model.DataValueGenerator.acquireNativeClass(DataValueGenerator.java:498)
at net.openhft.lang.model.DataValueClassCache.directClassFor(DataValueClassCache.java:48)
at net.openhft.lang.model.DataValueClassCache.newDirectReference(DataValueClassCache.java:37)
at net.openhft.lang.model.DataValueClasses.newDirectReference(DataValueClasses.java:36)
at net.openhft.collections.fromdocs.pingpong_latency.PingPongPlayerLeft.playPingPong(PingPongPlayerLeft.java:48)
at net.openhft.collections.fromdocs.pingpong_latency.PingPongPlayerLeft.bondExample(PingPongPlayerLeft.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
Caused by: java.lang.ClassNotFoundException: net.openhft.collections.fromdocs.BondVOInterface$$Native
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at net.openhft.compiler.CachedCompiler.loadFromJava(CachedCompiler.java:113)
at net.openhft.lang.model.DataValueGenerator.acquireNativeClass(DataValueGenerator.java:496)
... 26 more

[peter-lawrey]

The pom for collections needed to be updated, fixed now though they should
remain disabled as they should only be run manually.

On 4 June 2014 12:46, Rob Austin notifications@github.com wrote:

BTW earlier today

I’ve ignored these test

net.openhft.collections.fromdocs.pingpong_latency.PingPongPlayerLeft#bondExample

net.openhft.collections.fromdocs.pingpong_latency.PingPongPlayerRight#bondExample

because :

Connected to the target VM, address: '127.0.0.1:51263', transport:
'socket'
/net/openhft/collections/fromdocs/BondVOInterface$$Native.java:41: error:
cannot find symbol
return _bytes.compareAndSwapDouble(_offset + COUPON, _1, _2);
^
symbol: method compareAndSwapDouble(long,double,double)
location: variable _bytes of type Bytes
1 error
Disconnected from the target VM, address: '127.0.0.1:51263', transport:
'socket'

java.lang.AssertionError: java.lang.ClassNotFoundException:
net.openhft.collections.fromdocs.BondVOInterface$$Native
at
net.openhft.lang.model.DataValueGenerator.acquireNativeClass(DataValueGenerator.java:498)
at
net.openhft.lang.model.DataValueClassCache.directClassFor(DataValueClassCache.java:48)
at
net.openhft.lang.model.DataValueClassCache.newDirectReference(DataValueClassCache.java:37)
at
net.openhft.lang.model.DataValueClasses.newDirectReference(DataValueClasses.java:36)
at
net.openhft.collections.fromdocs.pingpong_latency.PingPongPlayerLeft.playPingPong(PingPongPlayerLeft.java:48)
at
net.openhft.collections.fromdocs.pingpong_latency.PingPongPlayerLeft.bondExample(PingPongPlayerLeft.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
Caused by: java.lang.ClassNotFoundException:
net.openhft.collections.fromdocs.BondVOInterface$$Native
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at
net.openhft.compiler.CachedCompiler.loadFromJava(CachedCompiler.java:113)
at
net.openhft.lang.model.DataValueGenerator.acquireNativeClass(DataValueGenerator.java:496)
... 26 more

—
Reply to this email directly or view it on GitHub
#28 (comment)
.

[Frank-Bradley]

Thank you. Did you have any code samples or Tests that I could study to learn SharedMapEventListeners API? I've tried looking under test packages and have not seen anything.

[RobAustin]

the net.openhft.collections.SharedMapEventListener's, get called when changes occur to the SHM, for example when shm.put(k,v) is called the sharedMapEventListener.onPut(...) will be called

The SharedMapEventListener is used by the net.openhft.collections.VanillaSharedReplicatedHashMap.ModificationIterator, but that is a rather complicated example, best look at the logger at net.openhft.collections.SharedMapEventListeners#BYTES_LOGGING for a simple example.