Mixed endianness of the ".continuous" data format

[galenlynch]

The current format of continuous recording files is very confusing. Although this isn't in the documentation, after looking at the analysis scripts and the RecordingEngine.cpp files themselves, it seems like ".continuous" recording files have little endian headers, followed by a number of records consisting of blocks of little endian timestamps and sample counts and the a block of big endian ints. In effect this makes the continuous recording files a file with alternating big and little endian data, which seems needlessly complicated.

At first I assumed that the big endian data was just the format of the data collected from the Rhythm API, but after looking at the source code for OriginalRecording object that produces ".continuous" files, I see that the big endian recording blocks are big endian seemingly by design: the floating point data from the Rhythm API is converted to big endian integers with a call to AudioDataConverters::convertFloatToInt16BE. For comparison, the HDF5Recording object converts to little endian integers with AudioDataConverters::convertFloatToInt16LE.

What is the reasoning behind this choice? It makes reading data files much more complicated.

Additionally, what is the status of the HDF5Recording object used by the KWIK recorder? Should it be used to make neural recordings instead of the Original recorder?

[open-ephys]

In short, consistency of endianness wasn't really a factor when we were designing the format. It would be nice (and not too difficult) to have an updated version that's consistent across the board, but we've put most of our recording-related development efforts into the KWIK format, which is robust and working well. I'd recommend using that for your recordings; the convenience of loading HDF5 files is amazing. I've been using it for all my data collection lately, and it's been great.

[galenlynch]

Awesome, I'll start using the KWIK format!

[nikolaskaralis]

Is the open-ephys format going to be supported in the future?
What about the discussions on the klustaviewas mailing list regarding the
format?
It seems they might be moving away from the Kwik format.

On Tue, Jul 28, 2015 at 12:17 AM, open-ephys notifications@github.com
wrote:

In short, consistency of endianness wasn't really a factor when we were
designing the format. It would be nice (and not too difficult) to have an
updated version that's consistent across the board, but we've put most of
our recording-related development efforts into the KWIK format, which is
robust and working well. I'd recommend using that for your recordings; the
convenience of loading HDF5 files is amazing. I've been using it for all my
data collection lately, and it's been great.

—
Reply to this email directly or view it on GitHub
#305 (comment).

Nikolas Karalis - http://www.nikolaskaralis.gr http://nikolaskaralis.gr

Neuroscience Graduate Student
Ludwig-Maximilians Universität München
Centre for Integrative Neuroscience - Universität Tübingen
Neuroscience Graduate
Université Bordeaux 2 - Charité Universitätsmedizin Berlin
Applied Mathematics and Physics Graduate
National Technical University of Athens, Greece

[open-ephys]

Yes, we'll continue supporting the original format.

Once the specs for the updated Kwik format are released, we plan to add the ability to record in that format as well. We'll also keep an HDF5-based recording option, which will likely be the Neurodata Without Borders format that's currently under development (http://neurodatawithoutborders.github.io).