Repository navigation
1. General Architecture
DVIDSparkServices uses PySpark to orchestrate and perform computation on large image datasets. For many operations involving access of large data, the package uses the DVID API, a distributed, versioned, image-oriented datastore. More details on other parts of DVIDSparkServices are described in subsequent pages:
- sparkdvid for RDD access of DVID (may eventually become a separate package)
- workflow system (see the README for a discussion on how to call a custom workflow)
- evaluation workflow and plugin
- segmentation workflow and plugin
Python was chosen instead of Java or Scala to have the greatest accessibility to potential contributors and to facilitate rapid development. Because the RDDs are often large nD volumes, DVIDSparkServices is not really meant for near-real time computation and is not likely to be dominated by the shuttling around of small data items. We can hopefully therefore address any key deficiencies for large image datasets by careful design.
One potential big performance gap between Python and Java is likely to exist in the serialization and deserialization of large datastructures. By default, PySpark uses cPickle which is very slow. Furthermore, all cached data needs to be serialized when using PySpark, whereas Java does not require serialization. In terms of compression, there are more options in the Java spark version as well.
To alleviate these problems, we implemented a custom serializer for PySpark that runs fast lz4 compression after pickling. The lz4 implementation is very fast and in the case of sparsely labeled 3d volumes, can result in compression factors of >10x. However, the pickler itself is still very slow on a large dataset. Pickling can take over a minute for ~1 GB numpy arrays. To circumvent this problem, sparkdvid implements a CompressedNumpyArray type that quickly serializes/deserializes the numpy array using StringIO and its built-in disk serialize function. The resulting structure can then be lz4 compressed. This, now small, dataset can be quickly run through standard serialization / deserialization in Spark. The core numpy transforms and lz4 for large datasets like this are only on the order of seconds and the compression will greatly improve performance when RDDs are shuffled.