off-heap inspn meeting

off-heap/HHM-SHM-expectations-obligations-re-V.msg

@@ -0,0 +1,50 @@
+
+From: Peter Lawrey [mailto:peter.lawrey@higherfrequencytrading.com] 
+Sent: Thursday, April 03, 2014 11:17 AM
+To: Cotton, Ben
+Subject: Re: HHM/SHM expectations/obligations re: V
+
+The minimum requirement is that the key and value are Serializable. Note: Object is not Serializable which some people get confused about. ;)
+
+Serializable is to be avoided of course,  Externalizable or ByteMarshallable is better for keys and values.  Some built in types have their own serializers and you can implement your own for a custom type but I don't know of anyone who uses this feature.
+
+If you want zero copy, you can do this for value (but not keys) and the minimum for this is Byteable.  If you want the library to generate a Byteable for you, you need to followed conventions based on Java Beans. In this case you can have nested types or array like elements via getXxxAt and setXxxAt, provided they are also generated, String or primitive.  If you implement your own Byteable, you can do what you like with it, provided you fit inside the entrySize() you built the collection with.
+
+There could be provision for Map, List, Set, Queue, Deque, but in the short term that would involve serialization/deserialization.
+
+On 3 April 2014 15:44, Cotton, Ben <ben.cotton@jpmorgan.com> wrote:
+Hi Peter,
+
+Sorry for the flurry of terribly worded public questions at https://github.com/OpenHFT/HugeCollections/issues/13  �. I was on the bus, typing it all in on a phone w/o  my glasses, and I get excited (and then worried) to be as clear as possible.  My bad.
+
+Would you mind providing to me, a general statement of  specific expectation/obligation for HHM/SHM <K,V> capability?
+
+I mean based on what I know via our general 3.0e experience and exercising  BondVOInterface (as via the InfoQ samples), is the following completely true:  
+
+1.       As long as we provide an interface graph (i.e. Neseted interfaces supported) that is at all points JBI compliant, We can use such an interface graph as a V in HHM/SHM
+2.      All JBI gettr/settr fields in the Interface graph must be in typeSet = {String, {any primitive}}      (i.e � I can�t have somehing like a sub-Map as part of V in SHM/HHM)
+3.      If both 1 & 2 are completely true, then we can then safely have the expectation that OpenHFT will interoperate soundly/completely as a ConcurrentMap � with both ZC and Externalizable accommodations for V.  
+
+Is the above 3 close to being sound/complete?
+
+If not, could you assist us to provide a sound/complete  statement approx =   �Here is your HHM,SHM V obligation,  if you meet it, here is your valid expectation for HHM,SHM capability wrt to V� ?
+
+i.e. I think we need what I would call the official  OpenHFT SHM/HHM �terms for endearment� wrt to V.
+
+Very grateful, thank you Peter.
+
+
+
+
+
+Ben D. Cotton III
+J.P.Morgan
+Liquidity Risk Technology
+277 Park Ave  Desk 08-GG64
+New York, NY 10172-0003
+212.622.5010
+ben.cotton@jpmorgan.com
+
+
+This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email. 
+

off-heap/OpenHFT_SHM_2_OffHeapRedHatInfinispanCACHE.msg

@@ -0,0 +1,109 @@
+
+
+
+-----Original Message-----
+From: Cotton, Ben 
+Sent: Wednesday, April 02, 2014 2:38 PM
+To: Mircea Markus
+Cc: Wang, Xiaoming; Gordeev, Dmitry; peter.lawrey@higherfrequencytrading.com; Tristan Tarrant; Dildy, Justin P; anistor@redhat.com; ben.cotton@alumni.rutgers.edu
+Subject: RE: build bridge from Off-Heap SHM to ISPN 7 ICE (via DataContainer)
+
+Thanks Mircea. Give us one more day, before we formally elaborate on the road block (we are not perfectly sure ourselves).
+
+What we ultimately want to do is build an ISPN7-valid Test of a new bridge that connects a JCACHE API view to an HPC off-heap capability.
+
+Specifically:
+
+from
+1.  Cache.Entry<K,V> API view     (ISPN7 AdvancedCache CacheImpl) to
+2.  DataContainer<K,V> API view   (JPM OffHeapDefaultDataContainer<K,V> Impl) to
+3.  SharedHashMap<K,V> API view   (OpenHFT VanillaSharedHashMap<K,V> Impl)
+
+We want to model the planned off-heap bridge described above, based off of the symmetry any *exisiting* Red-Hat test that proves an inplace existing on-heap bridge from
+
+1.  Cache.Entry<K,V> API view     (ISPN7 AdvancedCache CacheImpl) to
+2.  DataContainer<K,V> API view   (RedHat DefaultDataContainer Impl) to
+3.  ConcurrentMap<K,V> API view   (RedHat BoundedConcurrentHashMap Impl)
+
+The potential roadblock that has us mildly concerned is that neither of
+
+DataContainer nor DefaultDataContainer are explicilty generic on <K,V> and instead binds as <Object,InternalCacheEntry>. Interestingly, BoundedConcurrentMap is generic on <K,V> (but the current bridge to it may not take adavantage of that). This DataContainer binding to something that is not formally <K,V> may be a problem with our ultimately, generically, building a best bridge to the off-heap capability.  
+
+Today's meeting at JPM is to explore this concern and identify our view of it as an (isA|isNotA) potential roadblock.
+
+Will get back to you tomorrow w update.
+
+-----Original Message-----
+From: Mircea Markus [mailto:mmarkus@redhat.com] 
+Sent: Wednesday, April 02, 2014 1:57 PM
+To: Cotton, Ben
+Cc: Wang, Xiaoming; Gordeev, Dmitry; peter.lawrey@higherfrequencytrading.com; Tristan Tarrant; Dildy, Justin P; anistor@redhat.com
+Subject: Re: build bridge from Off-Heap SHM to ISPN 7 ICE (via DataContainer)
+
+Thanks for sharing this Ben; adding Adrian Nistor to the chain as he's the one that will overview the integration from an Infinispan perspective.
+
+Can you please elaborate on what the roadblock is?  
+
+On Apr 2, 2014, at 18:34, Cotton, Ben <ben.cotton@jpmorgan.com> wrote:
+
+> When: Wednesday, April 02, 2014 4:00 PM-4:30 PM (UTC-05:00) Eastern Time (US & Canada).
+> Where: Ben, Dmitry, Xiao desks
+>  
+> Note: The GMT offset above does not reflect daylight saving time adjustments.
+>  
+> *~*~*~*~*~*~*~*~*~*
+>  
+> Dmitry,Xiao:  Here is the current ROADBLOCK issue.  This meeting�s objective will be to analyze the options for remedy.
+>  
+> (Peter,Mircea,Tristan,Justin: this meeting notice is just FYI)
+>  
+> RedHat has the following @Test of their on-heap DefaultDataContainer.
+>  
+> https://github.com/Cotton-Ben/infinispan/blob/master/core/src/test/java/org/infinispan/container/SimpleDataContainerTest.java
+>  
+> The RedHat DataContainer API is what JPM will use as the basis of connecting OpenHFT�s operand=SharedHashMap.Entry (off-heap) to ISPN 7�s  operand=InternalCacheEntry (on-heap).
+>  
+> To affect this, Dmitry,Xiao, and I will build a new OffHeapDefaultDataContainer (implements DataContainer) that can pass an off-heap (version of the above RedHat) Test at
+>  
+> https://github.com/Cotton-Ben/infinispan/blob/master/off-heap/src/test/java/org/infinispan/offheap/OffHeapDefaultDataContainerTest.java
+>  
+> The current roadblock is that  what ISPN7 currently does do via DefaultDataContainer.java:
+>  
+> ConcurrentMap<Object,InternalCacheEntry> entries = new BoundedConcurrentHashMap<Object,InternalCacheEntry>(/**/);
+>  
+> We cannot do via OffDefaultDataContainer.java:
+>  
+> SharedHashMap<Object, InternalCacheEntry> entries =
+>                    (SharedHashMap<Object, InternalCacheEntry>)  new SharedHashMapBuilder(/**/);
+>  
+>  
+> Musing openly, I think we may have to adapt some notion of a �DataContainer Visitor� capability within any candidate solution.
+>  
+>  
+>  
+> Ben D. Cotton III
+> J.P.Morgan
+> Liquidity Risk Technology
+> 277 Park Ave  Desk 08-GG64
+> New York, NY 10172-0003
+> 212.622.5010
+> ben.cotton@jpmorgan.com
+>  
+>  
+>  
+>  
+>  
+>  
+>  
+> This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available athttp://www.jpmorgan.com/pages/disclosures/email.
+> 
+> <Mail Attachment.ics>
+
+Cheers,
+-- 
+Mircea Markus
+Infinispan lead (www.infinispan.org)
+
+
+
+