Missing timestamps from continuous recordings, binary format

[leoreh]

Hi,

I encountered the following problem and any help will be very much appreciated.

The first recording in most (but not all) experiments have more data points than timestamps. This is true for continuous but not event streams. The excess data points appear on all channels (including auxiliary) and their value is zero. Thus, it looks as if the data is zero-padded. The number of zeros varies between experiments, from 241 to 139249, and they always occur within the first second of the recording (zeroPaddingData.pdf). To clarify, as far as I could tell this only occurs after I start acquisition (press play) but not between recordings.

I am using GUI v0.4.5 on Windows 10, i5 6-core machine w/ 32 GB RAM and a GTX 1650 GPU.
The acquisition board is connected to a mouse shuttleDrive via the Intan 32ch headstage, and to an I/O box via its digital input (receives TTLs from a camera and stimulator).
Data is saved as flat binary.

[aacuevas]

Hi,

Are the timestamps continuous or is there a gap in the sequence?

[leoreh]

Hi Aarón,

Thank you for your rapid response.
The timestamps are not continuous but rather contain a gap that corresponds to the excess data points (i.e. "zero-padding").

[aacuevas]

That usually indicates a gap in received data. You said it only happens in the first recording. Does that happen when you hit play and then record, when acquisition is stopped and you start it by pressing record or it does not matter?

Have you noticed if the cpu meter gets too high? What kind of hard drive do you have? With your specs it would be rare to have an issue related to this, nonetheless.

[zsong30]

Hi, I am experiencing a smiliar issue. The timestamps are skipping with correspondent voltage data being zero. I plotted the differences between ajacent timestamps and attached the plot here. The x axis is timestamp and the y axis is the difference (that should have been 1). Does anyone know why this is happeninng?

[jsiegle]

Hi @zsong30, this is due to a bug that has been fixed in the latest version of the GUI. Can you try upgrading and see if that fixes the problem?

Edit: just saw your other post, is this using the same signal chain you sent previously?

[jingjie-li]

Hi everyone.
I also encountered this problem with the lastest GUI software (v0.5.5.2), which is jumping timestamp.npy and many zeros in the raw raw recording binary file. Does anyone have any cue to solve this?

[jsiegle]

Can you give some more details about your signal chain? For example, this is a known issue when trying to record Neuropixels data to a hard drive. Switching to an SSD should fix the problem.

[leoreh]

Hi all,

As Josh said, switching to an SSD solved the problem for me.

Still, at the time I wrote a Matlab function to remove the excess zeros from the binary file. See below.

function datInfo = valTstampsOE(varargin)

% fix for bug in open ephys binary format. description of problem: the
% first recording in most (but not all) experiments have more data points
% than timestamps. This is true for continuous but not event streams. The
% excess data points appear on all channels (including auxiliary) and their
% value is zero. Thus, it looks as if the data is zero-padded. The number
% of zeros varies between experiments, from 241 to 139249, and they always
% occur within the first second of the recording (zeroPaddingData.pdf). To
% clarify, as far as I could tell this only occurs after I start
% acquisition (press play) but not between recordings. solution: find
% inconsistencies in timestamps and remove corresponding samples from dat
% file. 
%
% INPUT:
%   basepath    string. path to .dat file (not including dat file itself)
%   fname       string. name of dat file. if empty and more than one dat in
%               path, will be extracted from basepath
%   chunksize   size of data to load at once [samples]{5e6}. 
%               if empty will load entire file (be careful!).
%               for 35 channels in int16, 5e6 samples = 350 MB.
%   bkup        logical. keep original file (true) or not {false}
%   nchans      numeric. original number of channels in dat file {35}.
%   precision   char. sample precision {'int16'} 
%   saveVar     logical. save datInfo {true} or not (false).
%
% OUTPUT
%   datInfo     struct with fields describing original and processed files
%
% CALLS:
%   bz_BasenameFromBasepath
%   class2bytes
%   n2chunks
%
% 22 apr 20 LH      


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% arguments
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

p = inputParser;
addOptional(p, 'basepath', pwd);
addOptional(p, 'fname', '', @ischar);
addOptional(p, 'chunksize', 5e6, @isnumeric);
addOptional(p, 'precision', 'int16', @ischar);
addOptional(p, 'nchans', 35, @isnumeric);
addOptional(p, 'mapch', [], @isnumeric);
addOptional(p, 'rmvch', [], @isnumeric);
addOptional(p, 'pli', 0, @isnumeric);
addOptional(p, 'bkup', false, @islogical);
addOptional(p, 'saveVar', true, @islogical);

parse(p, varargin{:})
basepath = p.Results.basepath;
fname = p.Results.fname;
chunksize = p.Results.chunksize;
precision = p.Results.precision;
nchans = p.Results.nchans;
bkup = p.Results.bkup;
saveVar = p.Results.saveVar;

% size of one data point in bytes
nbytes = class2bytes(precision);

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% preparations 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% handle dat file
cd(basepath)
datFiles = dir([basepath filesep '**' filesep '*dat']);
if isempty(datFiles)
    error('no .dat files found in %s', basepath)
end
if isempty(fname)
    if length(datFiles) == 1
        fname = datFiles.name;
    else
        fname = [bz_BasenameFromBasepath(basepath) '.dat'];
        if ~contains({datFiles.name}, fname)
            error('please specify which dat file to process')
        end
    end
end
[~, basename, ~] = fileparts(fname);
tempname = [basename '.tmp.dat'];
    
% load timestamps
tFiles = dir([basepath filesep '**' filesep 'timestamps.npy']);
for i = 1 : length(datFiles)
    idx = find(strcmp({tFiles.folder}, datFiles(i).folder));
    if length(idx) > 1
        warning(['more than one timestamps.npy found in %s\n',...
            'skipping tstamps concatination'], basepath)
        break
    elseif length(idx) < 1
        warning(['timestamps.npy not found in %s\n',...
            'skipping tstamps concatination'], basepath)
        break
    else
        tstamps = readNPY(fullfile(tFiles(idx).folder, tFiles(idx).name));
    end
end

% find inconsistencies in timestamps and arrange chunks accordingly
idx = find(diff(tstamps) > 1);
if isempty(idx)
    return
end
clip = zeros(length(idx), 2);
for i = 1 : length(idx)
    clip(i, :) = [tstamps(idx(i)), tstamps(idx(i) + 1)];
end
info = dir(fname);
nsamps = info.bytes / nbytes / nchans;
chunks = n2chunks('n', nsamps, 'chunksize', chunksize, 'clip', clip);
nchunks = size(chunks, 1);

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% write dat without clip
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% open temp dat file
fid = fopen(tempname, 'w');
if(fid == -1)
    error('cannot open file');
end

m = memmapfile(fname, 'Format', {'int16' [nchans nsamps] 'mapped'});

% go over chunks
for i = 1 : nchunks
    fprintf('working on chunk %d / %d\n', i, nchunks)
    d = m.data.mapped(:, chunks(i, 1) : chunks(i, 2));
    fwrite(fid, d(:), precision);
end

% close file and map
clear m
rc = fclose(fid);
fclose('all');
if rc == -1
    error('cannot save new file');
end

% rename new and original files, remove if no bkup requested
movefile(fname, [basename '_orig.dat'])
movefile(tempname, fname)
if ~bkup
    delete([basename '_orig.dat'])
end

fprintf('\nfinished processing %s. \nFile size = %.2f MB\n', fname, info.bytes / 1e6);

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% arrange datInfo and save
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
infoname = fullfile(basepath, [basename, '.datInfo.mat']);
if exist(infoname, 'file')
    load(infoname)
end

datInfo.clip = clip;

if saveVar
    save(infoname, 'datInfo');
end


end

% EOF

[jsiegle]

The Binary Format should be more reliable in the latest version of the GUI. Also, as mentioned above, when recording many channels simultaneously, writing to an SSD is critical.

If you encounter any issues with timestamps files in GUI version 0.6.X, please open a new issue.