Recording files not released

[RikkelBob]

I'm trying to do batch processing of my probes / sessions. All my data is located on a network attached storage (NAS). To be able to point to one of my .dat files as a recording object, I first move the file to a temp folder on my C:/ drive, preprocess, save the preprocessed copy to the same folder. Then I perform sorting on that file. After sorting, I move the output back to the NAS.

Then, I'd like to remove all the temporary files from my C:/ drive using shutil.rmtree(temp_path). However, python throws a PermissionError: [WinError 32] The process cannot access the file because it is being used by another process:

Is there a way to release my files so they can be deleted? Or, if not, a way to directly read data from a network drive? It seems si.read.binary cannot deal with windows networks path (i.e. "//NAS/MyName/AcquisitionPC/").

I've tried del() on all the recording, waveformextractor and sorting objects, after which I call gc.collect(). This doesn't solve it.

(I swear this will be my last issue)

[alejoe91]

Unfortunately, that's a well-known issue... https://stackoverflow.com/questions/27215462/permissionerror-winerror-32-the-process-cannot-access-the-file-because-it-is

[alejoe91]

(I swear this will be my last issue)

Open as many issues as you need!!! ;)

[RikkelBob]

Thanks :)

It seems adding f.close() to the read_binary_recording function fixes this. I haven't fully checked if this doesn't break anything, but at least I don't get any errors anymore.

    num_chan = int(num_chan)
    with Path(file).open() as f:
        nsamples = (os.fstat(f.fileno()).st_size - offset) // (num_chan * np.dtype(dtype).itemsize)
    if time_axis == 0:
        samples = np.memmap(file, np.dtype(dtype), mode="r", offset=offset, shape=(nsamples, num_chan))
    else:
        samples = np.memmap(file, np.dtype(dtype), mode="r", offset=offset, shape=(num_chan, nsamples)).T
    f.close()
    return samples

[samuelgarcia]

The with Path(file).open() as f:already close it.

[h-mayorquin]

Thanks :)

It seems adding f.close() to the read_binary_recording function fixes this. I haven't fully checked if this doesn't break anything, but at least I don't get any errors anymore.

    num_chan = int(num_chan)
    with Path(file).open() as f:
        nsamples = (os.fstat(f.fileno()).st_size - offset) // (num_chan * np.dtype(dtype).itemsize)
    if time_axis == 0:
        samples = np.memmap(file, np.dtype(dtype), mode="r", offset=offset, shape=(nsamples, num_chan))
    else:
        samples = np.memmap(file, np.dtype(dtype), mode="r", offset=offset, shape=(num_chan, nsamples)).T
    f.close()
    return samples

How is this function being called? Are you using the latest version? I am wondering if your error disappear in the latest version of spikeinterface as that function is not used anymore.

[zm711]

This is a huge windows issue like Alessio linked. I deal with it relatively commonly with my work computer. Usually if I want to use shutil to remove files I just do it in a separate cleanup script. It's similar to the little windows clean up script I added into the installation_tips folder. On Windows the current python process must completely finish before cleanup can occur, so if the user has spikeinterface and cleanup in the same script it will often fail on windows. It's so annoying I would honestly just recommend always keeping around a "clean-up" only script that you can modify for all your python project when working on windows.

Addressing the NAS, the one issue I run into with Pathlib for windows is that sometimes windows paths (since they are \ instead of / ) can trigger spurious escaping (\b \t \n are big offenders) so sometimes file paths need to be prepended with the "r" prefix. So maybe you need to do r"NAS\PATH" instead. I have definitely got Pathlib to work fine connecting to other drives than C from my computer, but I always just do r"path" now to be safe after running into a bunch of Windows path issues.

[RikkelBob]

@h-mayorquin You're right, updating also fixes it!

@zm711 Thanks for the suggestion, I'll give it a try when I have the time.

[h-mayorquin]

Yei! Issues solved!
We just (a couple of hours) made a new release @RikkelBob be sure to update!