Error reading EEGLAB file (buffer too small) #10012

cbrnr · 2021-11-17T11:02:10Z

This issue was reported in our forum using this example file.

raw = mne.io.read_raw_eeglab("EC_CON1.set")
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-90625f7fcaf5> in <module>
----> 1 raw = mne.io.read_raw_eeglab("EC_CON1.set")

/usr/local/lib/python3.9/site-packages/mne/io/eeglab/eeglab.py in read_raw_eeglab(input_fname, eog, preload, uint16_codec, verbose)
    217     .. versionadded:: 0.11.0
    218     """
--> 219     return RawEEGLAB(input_fname=input_fname, preload=preload,
    220                      eog=eog, verbose=verbose, uint16_codec=uint16_codec)
    221 

<decorator-gen-261> in __init__(self, input_fname, eog, preload, uint16_codec, verbose)

/usr/local/lib/python3.9/site-packages/mne/io/eeglab/eeglab.py in __init__(self, input_fname, eog, preload, uint16_codec, verbose)
    316                  preload=False, uint16_codec=None, verbose=None):  # noqa: D102
    317         input_fname = _check_fname(input_fname, 'read', True, 'input_fname')
--> 318         eeg = _check_load_mat(input_fname, uint16_codec)
    319         if eeg.trials != 1:
    320             raise TypeError('The number of trials is %d. It must be 1 for raw'

/usr/local/lib/python3.9/site-packages/mne/io/eeglab/eeglab.py in _check_load_mat(fname, uint16_codec)
     57     """Check if the mat struct contains 'EEG'."""
     58     from ...externals.pymatreader import read_mat
---> 59     eeg = read_mat(fname, uint16_codec=uint16_codec)
     60     if 'ALLEEG' in eeg:
     61         raise NotImplementedError(

/usr/local/lib/python3.9/site-packages/mne/externals/pymatreader/pymatreader.py in read_mat(filename, variable_names, ignore_fields, uint16_codec)
     87                 extra_kwargs['uint16_codec'] = uint16_codec
     88 
---> 89             raw_data = scipy.io.loadmat(fid, struct_as_record=True,
     90                                         squeeze_me=True, mat_dtype=False,
     91                                         variable_names=variable_names,

/usr/local/lib/python3.9/site-packages/scipy/io/matlab/mio.py in loadmat(file_name, mdict, appendmat, **kwargs)
    224     with _open_file_context(file_name, appendmat) as f:
    225         MR, _ = mat_reader_factory(f, **kwargs)
--> 226         matfile_dict = MR.get_variables(variable_names)
    227 
    228     if mdict is not None:

/usr/local/lib/python3.9/site-packages/scipy/io/matlab/mio5.py in get_variables(self, variable_names)
    330                 continue
    331             try:
--> 332                 res = self.read_var_array(hdr, process)
    333             except MatReadError as err:
    334                 warnings.warn(

/usr/local/lib/python3.9/site-packages/scipy/io/matlab/mio5.py in read_var_array(self, header, process)
    290            `process`.
    291         '''
--> 292         return self._matrix_reader.array_from_header(header, process)
    293 
    294     def get_variables(self, variable_names=None):

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.array_from_header()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.array_from_header()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.read_struct()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.read_mi_matrix()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.array_from_header()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.read_char()

TypeError: buffer is too small for requested array

It looks like this is related to reading the .mat file.

The text was updated successfully, but these errors were encountered:

cbrnr · 2021-11-17T11:30:23Z

@thht something you could potentially fix or rather scipy.io.loadmat?

agramfort · 2021-11-17T12:51:23Z

it's due to a change in scipy or in mne?

…

thht · 2021-11-17T14:13:18Z

as it is the scipy function that throws the error, we should ask them. pymatreader only touches what is returned. and i just double-checked: even with the default options, scipy.io.loadmat throws that error....

thht · 2021-11-17T14:16:35Z

the file loads fine in matlab, though.... but i noticed some non-ascii characters in one of the fields:

filepath: 'C:\Users\User\OneDrive - Prince of Songkla University\Desktop\�.����Է\EC\control\RUN_ICA_SELECTED_170S_after remove first 10S '

when i set this field to an empty string, it loads fine with scipy.

cbrnr · 2021-11-17T15:07:30Z

It's always character encodings, that's for sure. Thanks for checking @thht! How do we proceed?

agramfort · 2021-11-17T20:25:45Z

fix scipy but this will take time and before fix the file with matlab to be able to load it as is my 2c

mmagnuski · 2021-11-17T20:58:01Z

I remember we had this problem before (but I couldnt find related issue or pull request now) and this is why we've exposed uint16_codec argument of loadmat. In my case, long time ago, setting it to something like 'latin' (IIRC) helped.

mmagnuski · 2021-11-17T20:59:48Z

Quoting the docs for the uint16_codec parameter of read_raw_eeglab:

uint16_codec : str | None
    If your \*.set file contains non-ascii characters, sometimes reading
    it may fail and give rise to error message stating that "buffer is
    too small". ``uint16_codec`` allows to specify what codec (for example:
    'latin1' or 'utf-8') should be used when reading character arrays and
    can therefore help you solve this problem.

cbrnr · 2021-11-17T21:07:14Z

Nice find @mmagnuski! If anyone can confirm that this works with the problematic file, this issue can be closed.

drammock · 2021-11-17T23:07:58Z

#8226 (comment)

MoonPlatinum · 2022-03-29T07:52:28Z

I also have this problem!

@drammock

cbrnr · 2022-03-29T08:28:24Z

@MoonPlatinum would you be able to share the problematic file with us? Also, it would be great if you posted the errors as text (formatted as code) instead of screenshots here.

mmagnuski · 2022-04-01T08:08:34Z

@MoonPlatinum does the solution work for you?

MoonPlatinum · 2022-04-01T15:18:06Z

Thank you !!!
1.
import mne

raw = mne.io.read_epochs_eeglab('0928H_DFI.set', uint16_codec='latin-1')

/Users/tianbiwang/.conda/envs/pythonProject1/bin/python /Users/tianbiwang/PycharmProjects/pythonProject1/pythonks/demo2.py
Traceback (most recent call last):
File "/Users/tianbiwang/PycharmProjects/pythonProject1/pythonks/demo2.py", line 8, in
raw = mne.io.read_epochs_eeglab('0928H_DFI.set', uint16_codec='latin-1')
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/io/eeglab/eeglab.py", line 304, in read_epochs_eeglab
epochs = EpochsEEGLAB(input_fname=input_fname, events=events, eog=eog,
File "", line 12, in init
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/io/eeglab/eeglab.py", line 488, in init
raise ValueError("The file does not seem to contain epochs "
ValueError: The file does not seem to contain epochs (trials less than 2). You should try using read_raw_eeglab function.
2.
import mne

raw = mne.io.read_raw_eeglab('0928H_DFI.set', uint16_codec='latin-1')
/Users/tianbiwang/.conda/envs/pythonProject1/bin/python /Users/tianbiwang/PycharmProjects/pythonProject1/pythonks/demo2.py
Reading /Users/tianbiwang/PycharmProjects/pythonProject1/pythonks/0928H_DFI.fdt
Traceback (most recent call last):
File "/Users/tianbiwang/PycharmProjects/pythonProject1/pythonks/demo2.py", line 8, in
raw = mne.io.read_raw_eeglab('0928H_DFI.set', uint16_codec='latin-1')
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/io/eeglab/eeglab.py", line 248, in read_raw_eeglab
return RawEEGLAB(input_fname=input_fname, preload=preload,
File "", line 12, in init
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/io/eeglab/eeglab.py", line 383, in init
annot = read_annotations(input_fname)
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/annotations.py", line 1121, in read_annotations
annotations = _read_annotations_eeglab(fname,
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/io/eeglab/eeglab.py", line 630, in _read_annotations_eeglab
eeg = _check_load_mat(eeg, uint16_codec=uint16_codec)
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/io/eeglab/eeglab.py", line 61, in _check_load_mat
eeg = read_mat(fname, uint16_codec=uint16_codec)
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/pymatreader/pymatreader.py", line 92, in read_mat
raw_data = scipy.io.loadmat(fid, struct_as_record=True,
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/scipy/io/matlab/_mio.py", line 226, in loadmat
matfile_dict = MR.get_variables(variable_names)
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/scipy/io/matlab/_mio5.py", line 332, in get_variables
res = self.read_var_array(hdr, process)
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/scipy/io/matlab/_mio5.py", line 292, in read_var_array
return self._matrix_reader.array_from_header(header, process)
File "_mio5_utils.pyx", line 671, in scipy.io.matlab._mio5_utils.VarReader5.array_from_header
File "_mio5_utils.pyx", line 713, in scipy.io.matlab._mio5_utils.VarReader5.array_from_header
File "_mio5_utils.pyx", line 876, in scipy.io.matlab._mio5_utils.VarReader5.read_char
TypeError: buffer is too small for requested array

Process finished with exit code 1

MoonPlatinum · 2022-04-01T15:22:22Z

@MoonPlatinum would you be able to share the problematic file with us? Also, it would be great if you posted the errors as text (formatted as code) instead of screenshots here.
Thank you !!!

1.import mne

raw = mne.io.read_epochs_eeglab('0928H_DFI.set', uint16_codec='latin-1')

/Users/tianbiwang/.conda/envs/pythonProject1/bin/python /Users/tianbiwang/PycharmProjects/pythonProject1/pythonks/demo2.py
Traceback (most recent call last):
File "/Users/tianbiwang/PycharmProjects/pythonProject1/pythonks/demo2.py", line 8, in
raw = mne.io.read_epochs_eeglab('0928H_DFI.set', uint16_codec='latin-1')
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/io/eeglab/eeglab.py", line 304, in read_epochs_eeglab
epochs = EpochsEEGLAB(input_fname=input_fname, events=events, eog=eog,
File "", line 12, in init
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/io/eeglab/eeglab.py", line 488, in init
raise ValueError("The file does not seem to contain epochs "
ValueError: The file does not seem to contain epochs (trials less than 2). You should try using read_raw_eeglab function.

2.import mne

raw = mne.io.read_raw_eeglab('0928H_DFI.set', uint16_codec='latin-1')
/Users/tianbiwang/.conda/envs/pythonProject1/bin/python /Users/tianbiwang/PycharmProjects/pythonProject1/pythonks/demo2.py
Reading /Users/tianbiwang/PycharmProjects/pythonProject1/pythonks/0928H_DFI.fdt
Traceback (most recent call last):
File "/Users/tianbiwang/PycharmProjects/pythonProject1/pythonks/demo2.py", line 8, in
raw = mne.io.read_raw_eeglab('0928H_DFI.set', uint16_codec='latin-1')
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/io/eeglab/eeglab.py", line 248, in read_raw_eeglab
return RawEEGLAB(input_fname=input_fname, preload=preload,
File "", line 12, in init
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/io/eeglab/eeglab.py", line 383, in init
annot = read_annotations(input_fname)
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/annotations.py", line 1121, in read_annotations
annotations = _read_annotations_eeglab(fname,
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/io/eeglab/eeglab.py", line 630, in _read_annotations_eeglab
eeg = _check_load_mat(eeg, uint16_codec=uint16_codec)
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/io/eeglab/eeglab.py", line 61, in _check_load_mat
eeg = read_mat(fname, uint16_codec=uint16_codec)
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/pymatreader/pymatreader.py", line 92, in read_mat
raw_data = scipy.io.loadmat(fid, struct_as_record=True,
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/scipy/io/matlab/_mio.py", line 226, in loadmat
matfile_dict = MR.get_variables(variable_names)
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/scipy/io/matlab/_mio5.py", line 332, in get_variables
res = self.read_var_array(hdr, process)
File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/scipy/io/matlab/_mio5.py", line 292, in read_var_array
return self._matrix_reader.array_from_header(header, process)
File "_mio5_utils.pyx", line 671, in scipy.io.matlab._mio5_utils.VarReader5.array_from_header
File "_mio5_utils.pyx", line 713, in scipy.io.matlab._mio5_utils.VarReader5.array_from_header
File "_mio5_utils.pyx", line 876, in scipy.io.matlab._mio5_utils.VarReader5.read_char
TypeError: buffer is too small for requested array

Process finished with exit code 1
@cbrnr

MoonPlatinum · 2022-04-01T15:47:34Z

@cbrnr
My files is too big. I'm trying to upload it to GitHub。

mmagnuski · 2022-04-01T18:33:31Z

Are you sure the files you are trying to read are epochs? The error you get now suggests you to use read_raw_eeglab.

MoonPlatinum · 2022-04-02T11:02:48Z

Are you sure the files you are trying to read are epochs? The error you get now suggests you to use read_raw_eeglab.

I have tried both methods, but they still can't run successfully.
@mmagnuski

MoonPlatinum · 2022-04-02T11:08:48Z

I asked others to help upload the data file. Can you help me try how to run successfully？
Its link is

https://github.com/ShrimpLG/eeg

@mmagnuski
@cbrnr

mmagnuski · 2022-04-02T12:53:57Z

I have tried both methods, but they still can't run successfully.

Could you post the error you get when using read_raw_eeglab on the file (but still using uint16_codec argument)?

MoonPlatinum · 2022-04-02T12:57:00Z

import mne

raw = mne.io.read_raw_eeglab('0928H_DFI.set', uint16_codec='latin-1') /Users/tianbiwang/.conda/envs/pythonProject1/bin/python /Users/tianbiwang/PycharmProjects/pythonProject1/pythonks/demo2.py Reading /Users/tianbiwang/PycharmProjects/pythonProject1/pythonks/0928H_DFI.fdt Traceback (most recent call last): File "/Users/tianbiwang/PycharmProjects/pythonProject1/pythonks/demo2.py", line 8, in raw = mne.io.read_raw_eeglab('0928H_DFI.set', uint16_codec='latin-1') File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/io/eeglab/eeglab.py", line 248, in read_raw_eeglab return RawEEGLAB(input_fname=input_fname, preload=preload, File "", line 12, in init File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/io/eeglab/eeglab.py", line 383, in init annot = read_annotations(input_fname) File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/annotations.py", line 1121, in read_annotations annotations = _read_annotations_eeglab(fname, File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/io/eeglab/eeglab.py", line 630, in _read_annotations_eeglab eeg = _check_load_mat(eeg, uint16_codec=uint16_codec) File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/mne/io/eeglab/eeglab.py", line 61, in _check_load_mat eeg = read_mat(fname, uint16_codec=uint16_codec) File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/pymatreader/pymatreader.py", line 92, in read_mat raw_data = scipy.io.loadmat(fid, struct_as_record=True, File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/scipy/io/matlab/_mio.py", line 226, in loadmat matfile_dict = MR.get_variables(variable_names) File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/scipy/io/matlab/_mio5.py", line 332, in get_variables res = self.read_var_array(hdr, process) File "/Users/tianbiwang/.conda/envs/pythonProject1/lib/python3.10/site-packages/scipy/io/matlab/_mio5.py", line 292, in read_var_array return self._matrix_reader.array_from_header(header, process) File "_mio5_utils.pyx", line 671, in scipy.io.matlab._mio5_utils.VarReader5.array_from_header File "_mio5_utils.pyx", line 713, in scipy.io.matlab._mio5_utils.VarReader5.array_from_header File "_mio5_utils.pyx", line 876, in scipy.io.matlab._mio5_utils.VarReader5.read_char TypeError: buffer is too small for requested array

Process finished with exit code 1

@mmagnuski

MoonPlatinum · 2022-04-02T12:59:49Z

I have tried both methods, but they still can't run successfully.

Could you post the error you get when using read_raw_eeglab on the file (but still using uint16_codec argument)?

And I also provided the original data in the above answer. Can you help me have a look?

cbrnr · 2022-04-03T17:20:00Z

Not sure why, but scipy.io.loadmat() cannot read the .set file (which is just a .mat file). Opening it with a hex editor, the file starts with MATLAB 5.0 MAT-file. I don't know that format, and this MATLAB page lists only v4, v6, v7, and v7.3. @thht can you provide more insights what's going on? Which .mat format do we have here? The uint16_codec argument does not seem to work, I always get this error. FWIW, scipy.io.loadmat does not even have such a parameter.

thht · 2022-04-04T09:05:23Z

i just tried to load it in matlab:

>> load('0928H_DFI.set');
Error using load
Unable to read file '0928H_DFI.set'. Input must be a MAT-file or an ASCII file containing numeric data with same number of columns
in each row.

so, not even matlab can read the file....

cbrnr · 2022-04-04T09:45:50Z

Thanks @thht! So I guess this explains the issue. @MoonPlatinum can you maybe try to save this in a different format? If this is exported from EEGLAB I think you should ask the EEGLAB developers as this seems like a problem on their end.

MoonPlatinum · 2022-04-05T04:30:09Z

Thanks a lot for your help to me, this file can't be loaded directly with matlab, but it can be read into matlab by eeglab plugin, my problem is that the file can't be read by mne library of python environment, I don't know why this is.
@thht @cbrnr @mmagnuski

cbrnr · 2022-04-05T06:33:11Z

Can you maybe convert the files to a format which is readable in other packages? EEGLAB should let you choose the export format.

cbrnr added the BUG label Nov 17, 2021

cbrnr closed this as completed Nov 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error reading EEGLAB file (buffer too small) #10012

Error reading EEGLAB file (buffer too small) #10012

cbrnr commented Nov 17, 2021 •

edited

cbrnr commented Nov 17, 2021

agramfort commented Nov 17, 2021 via email

thht commented Nov 17, 2021

thht commented Nov 17, 2021

cbrnr commented Nov 17, 2021

agramfort commented Nov 17, 2021 via email

mmagnuski commented Nov 17, 2021

mmagnuski commented Nov 17, 2021

cbrnr commented Nov 17, 2021

drammock commented Nov 17, 2021

MoonPlatinum commented Mar 29, 2022

cbrnr commented Mar 29, 2022

mmagnuski commented Apr 1, 2022

MoonPlatinum commented Apr 1, 2022

MoonPlatinum commented Apr 1, 2022

MoonPlatinum commented Apr 1, 2022

mmagnuski commented Apr 1, 2022

MoonPlatinum commented Apr 2, 2022

MoonPlatinum commented Apr 2, 2022

mmagnuski commented Apr 2, 2022

MoonPlatinum commented Apr 2, 2022

MoonPlatinum commented Apr 2, 2022

cbrnr commented Apr 3, 2022

thht commented Apr 4, 2022

cbrnr commented Apr 4, 2022

MoonPlatinum commented Apr 5, 2022

cbrnr commented Apr 5, 2022

Error reading EEGLAB file (buffer too small) #10012

Error reading EEGLAB file (buffer too small) #10012

Comments

cbrnr commented Nov 17, 2021 • edited

cbrnr commented Nov 17, 2021

agramfort commented Nov 17, 2021 via email

thht commented Nov 17, 2021

thht commented Nov 17, 2021

cbrnr commented Nov 17, 2021

agramfort commented Nov 17, 2021 via email

mmagnuski commented Nov 17, 2021

mmagnuski commented Nov 17, 2021

cbrnr commented Nov 17, 2021

drammock commented Nov 17, 2021

MoonPlatinum commented Mar 29, 2022

cbrnr commented Mar 29, 2022

mmagnuski commented Apr 1, 2022

MoonPlatinum commented Apr 1, 2022

MoonPlatinum commented Apr 1, 2022

MoonPlatinum commented Apr 1, 2022

mmagnuski commented Apr 1, 2022

MoonPlatinum commented Apr 2, 2022

MoonPlatinum commented Apr 2, 2022

mmagnuski commented Apr 2, 2022

MoonPlatinum commented Apr 2, 2022

MoonPlatinum commented Apr 2, 2022

cbrnr commented Apr 3, 2022

thht commented Apr 4, 2022

cbrnr commented Apr 4, 2022

MoonPlatinum commented Apr 5, 2022

cbrnr commented Apr 5, 2022

cbrnr commented Nov 17, 2021 •

edited