Blacrock ValueError: mmap length is greater than file size

[luiztauffer]

An error that happens after I fix #1689 with #1690

This happens for Blackrock spec==2.3

import neo

base_path = ".../H0.2_openField_PPX_062723001"
r6 = neo.rawio.BlackrockRawIO(filename=base_path, nsx_to_load=[6], load_nev=False)
r6.parse_header()
r6.header

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[8], line 2
      1 r6 = neo.rawio.BlackrockRawIO(filename=base_path, nsx_to_load=[6], load_nev=False)
----> 2 r6.parse_header()
      3 r6.header

File /mnt/shared_storage/Github/python-neo/neo/rawio/baserawio.py:211, in BaseRawIO.parse_header(self)
    197 """
    198 Parses the header of the file(s) to allow for faster computations
    199 for all other functions
    200 
    201 """
    202 # this must create
    203 # self.header['nb_block']
    204 # self.header['nb_segment']
   (...)    208 # self.header['spike_channels']
    209 # self.header['event_channels']
--> 211 self._parse_header()
    212 self._check_stream_signal_channel_characteristics()
    213 self.is_header_parsed = True

File /mnt/shared_storage/Github/python-neo/neo/rawio/blackrockrawio.py:394, in BlackrockRawIO._parse_header(self)
    392 else:
    393     _data_reader_fun = self.__nsx_data_reader[spec]
--> 394 self.nsx_datas[nsx_nb] = _data_reader_fun(nsx_nb)
    396 sr = float(self.main_sampling_rate / self.__nsx_basic_header[nsx_nb]["period"])
    397 self.sig_sampling_rates[nsx_nb] = sr

File /mnt/shared_storage/Github/python-neo/neo/rawio/blackrockrawio.py:1087, in BlackrockRawIO.__read_nsx_data_variant_b(self, nsx_nb)
   1084     offset = int(self.__nsx_data_header[nsx_nb][data_bl]["offset_to_data_block"])
   1086     # read data
-> 1087     data[data_bl] = np.memmap(filename, dtype="int16", shape=shape, offset=offset, mode="r")
   1089 return data

File ~/anaconda3/envs/env_nelson/lib/python3.13/site-packages/numpy/_core/memmap.py:289, in memmap.__new__(subtype, filename, dtype, mode, offset, shape, order)
    287     start -= mmap.ALLOCATIONGRANULARITY
    288 array_offset = offset - start
--> 289 mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
    291 self = ndarray.__new__(subtype, shape, dtype=descr, buffer=mm,
    292                        offset=array_offset, order=order)
    293 self._mmap = mm

ValueError: mmap length is greater than file size

Environment:

• OS: Linux
• Python 3.13.2
• Neo 0.15.0.dev0 (master branch)
• NumPy 2.2.4

[zm711]

I think that is due to us still missing other ints. A user had this error and I fixed one of our overflows and that took care of it. But I'm not sure why the overflows I fixed would have helped them.

[luiztauffer]

printing the content of __read_nsx_data_variant_b:

def __read_nsx_data_variant_b(self, nsx_nb):
    filename = ".".join([self._filenames["nsx"], f"ns{nsx_nb}"])
    data = {}
    print(self.__nsx_data_header[nsx_nb].keys())
    for data_bl in self.__nsx_data_header[nsx_nb].keys():
        # get shape and offset of data
        shape = (
            int(self.__nsx_data_header[nsx_nb][data_bl]["nb_data_points"]),
            int(self.__nsx_basic_header[nsx_nb]["channel_count"]),
        )
        offset = int(self.__nsx_data_header[nsx_nb][data_bl]["offset_to_data_block"])

        print("######################################\n")
        print(filename)
        print("File size: ", self.__get_file_size(filename))
        print("Shape: ", shape)
        print("Offset: ", offset)
        print("\n######################################")

I get:

dict_keys([0, 1])
######################################

/media/luiz/seagate_external/storage/catalystneuro/nelson_lab/Raw_BlackRock_withCognitiveEvents/BlackRock_troubleshootingFile_3/H4.3_openField_PPX_091423001.ns6
File size:  38547474810
Shape:  (0, 32)
Offset:  2435

######################################
######################################

/media/luiz/seagate_external/storage/catalystneuro/nelson_lab/Raw_BlackRock_withCognitiveEvents/BlackRock_troubleshootingFile_3/H4.3_openField_PPX_091423001.ns6
File size:  38547474810
Shape:  (3857284608, 32)
Offset:  2444

######################################

and then the error.
It looks like we have some potential sources of error here:

• we're creating blocks with no data
• the shape is really going beyond the file size

[luiztauffer]

also, I realized nsx_to_load is not being respected, and both nsX files are being parsed, even if I use the argument

[zm711]

I think this is all super useful info. Thanks Luiz! I swore off extensively working on this IO because I hate name mangling with a passion. So Sam mostly manages this along with some community contributors. If you want to do some more extensive tweaks we would have to wait for Sam to review which could take a while.

[zm711]

Is this fixed and we should open a new issue for the nsx_to_load thing?

[luiztauffer]

@zm711 I'm still investigating the case of a recording session for which neo code fails, although the system's visualization software successfully reads the data from files. It's not 100% certain yet if there's some corruption in the files or if that is a behavior we should considerate covering.

here's the files, if you'd like to try it out: https://drive.google.com/drive/folders/1qzlbp8SaDiKjv37IVEhiSS1NSHzDbwlV?usp=drive_link

[zm711]

Thanks if I get some time I'll try to take a look too!

[luiztauffer]

This was not a neo error, I’m leaving the solution here for future reference.

The files contained two issues:

• a wrong value of zero for Number of Data Points, in the data packet
• incomplete data at the end of the file, indicated by the number of bytes not being multiple of the number of channels

This leads to the suspicion that the recording system did not finish the writing process successfully.

The data could be opened in Blackrock’s software and replayed with nPlay. I requested the users to export the data from there, excluding the finals seconds of data, and this resulted in correct ns2 and ns6 files.

[zm711]

Thanks @luiztauffer. I think that's what we were thinking about overall. So good to get confirmation. Happy that you're able to open in the company's software to fix!