Explore possible optimizations

[bzz]

There are some possibly low-hanging fruits in performance optimization that need to be evaluated:

• 4b pointers -XX:+UseCompressedOops if <32Gb RAM per JVM
• enable Kryo serialization --conf spark.serializer='org.apache.spark.serializer.KryoSerializer'
• tune DataStructures to http://fastutil.di.unimi.it
  http://java-performance.info/hashmap-overview-jdk-fastutil-goldman-sachs-hppc-koloboke-trove-january-2015/
  • re-implement vectorization using fastutils.IntArrayList
  • re-implement generateCooccurrence using fastutils.ObjectIntMap
  • test beforehand - how well does it play with Kryo?
• add new CLI to save vectorized or indexed intermediate results: convert to IDs and persist numbers (+measure size/throughput)
• Determine best StorageLevel for caching to coocs: they are used twice, on getting shards and on getting row/col sums - MEMORY_AND_DISK_SER