Some "Binned*.mat" files cannot be opened in Python

[weiglszonja]

Issue Summary

I found issues with opening some of the "Binned*.mat" files using Python (tried packages like h5py, scipy.io, and pymatreader). When attempting to open the files, I received errors indicating that the file signature could not be found (h5py) or that the file format was not recognized properly.

The "Binned*.mat" (e.g. 'Binned405_VGlut-A997-20200201.mat') contains the raw fiber photometry response series data for a given subject and date (multiple sessions for a date).

Workaround:

The workaround for this issue is to load them using MATLAB and resaving them to version 7.3.
The MATLAB snippet that achieves this:

function save_binned(source_filepath, destination_filepath)
    % Load the file
    load(source_filepath);
    % Save to version 7.3
    save(destination_filepath, '-v7.3', 'T');
end

Additional Notes:

When reading these files with scipy.io or pymatreader the data shows up as:
MatlabOpaque object:

[(b'T', b'MCOS', b'table', array([[3707764736],,        [         2],,        [         1],,        [         1],,        [         1],,        [         1]], dtype=uint32))          ]

The appearance of MatlabOpaque objects suggests that the .mat file may contain data structures defined using custom MATLAB classes or data types that are not fully supported or understood by the Python packages being used.

[weiglszonja]

Example MATLAB script to re-save all files and save them version 7.3:

function resave_mat_files(root_folder_path)
    % Get a list of all files and folders in this folder.
    all_files = dir(root_folder_path);
    % Extract the names of all files and folders.
    all_names = {all_files.name};
    % Get a logical vector that tells which is a directory.
    dir_flags = [all_files.isdir];
    % Extract the names of all subfolders.
    subfolders = all_names(dir_flags);
    % Print folder names to command window.
    for k = 1 : length(subfolders)
        if strcmp(subfolders{k}, '.') || strcmp(subfolders{k}, '..')
            continue;
        end
        fprintf('Sub folder #%d = %s\n', k, subfolders{k});
        resave_mat_files(fullfile(root_folder_path, subfolders{k}));
    end
    % Get a list of all mat files in the current directory.
    mat_files = dir(fullfile(root_folder_path, 'Binned405_*.mat'));
    % Loop through each mat file in the current directory.
    for k = 1 : length(mat_files)
        mat_file_path = fullfile(root_folder_path, mat_files(k).name);
        fprintf('Resaving mat file #%d = %s\n', k, mat_file_path);
        % Load the mat file.
        load(mat_file_path);

        % Resave the mat file to version 7.3.
        save(mat_file_path, 'T', '-v7.3');
        % Clear the loaded variables.
        clear('T');
        %clear;
    end
end