Skip to content
Stuart Berg edited this page Aug 28, 2015 · 2 revisions

sparkdvid is a python module that defines input and output functions over RDDs to access DVID. The module is installed under DVIDSparkServices.sparkdvid.

Currently read/write in DVID is limited by the throughput of the databases behind DVID. If the DVID backend is not clustered, the accesses will be throttled and slow. This is the default behavior currently.

The key parts of the module are:

  • CompressedNumpyArray (custom serialization/deserialization and compression for large numpy arrays)
  • CompressedSerializerLZ4 (custom spark serializer used in the workflow spark context creation)
  • Subvolume (small 3D regions that often define units of work)
  • sparkdvid (core component for reading and writing RDDs to DVID)

sparkdvid mostly works over subvolumes that form a partitioning of the image dataset.

Clone this wiki locally