Add a Thunder format

[hinerm]

Kevin Mader asked about the best way to support Thunder data in applications like KNIME, ImageJ, etc...

One possible solution would be adding a SCIFIO format that could read(/write?) Thunder data

[kmader]

The idea is basically a generic format (@freeman-lab), where data is stored in multiple binary files and has one companion file with all of the metadata and dimensions for the binary files (example: https://github.com/thunder-project/thunder/tree/master/python/thunder/utils/data/fish/series)
with the metadata being currently

{
  "valuetype": "uint8", 
  "nkeys": 3, 
  "keytype": "int16", 
  "dims": [
    76, 
    87, 
    2
  ], 
  "nvalues": 240, 
  "input": "/Users/freemanj11/Desktop/bin"
}

The standard for such 'big data' framworks is a key-value pairs representation and the idea would be to have a key consisting of several numbers (nkeys) of type (keytype) and then a value as an array of type (valuetype) with dimensions (dims) and all of this spread into multiple files so they can be easily written and read in parallel (or on different machines to a shared file system).

The key issues are:

• it needs to be separate files to make saving in parallel or distributed environments easy (imgfile_001.bin, imgfile_002.bin, etc)
• it should be a very simple format to it is also easy to read in Matlab, Python, or whatever other tool one might be using

[ctrueden]

No bandwidth to work on this, no current user demand, and Thunder is not actively developed anymore. Anyone is free to start a scifio-format-thunder repository, though.