Passing a folder_path that ends with the Record Node does not correctly load timestamps

[bparks13]

When loading data from the Open Ephys GUI (via spikeinterface.extractors.read_openephys), if I give it a folder path that ends with a record node (e.g., /path/to/Record Node 101) then it does not load the timestamps correctly. I noticed this when loading data from the ONIX Source plugin, as the data streams there are hardware timestamped. Instead of relying on a start time and extrapolating based on the assumed sampling frequency, some of the data streams must have their timestamps loaded explicitly to ensure the times are synchronized across streams. For a 32 minute recording, the difference between the two approaches led to a 3.6 second difference between the real timestamps and the assumed timestamps, which caused confusion during post-processing analysis.

The offending line is here:

spikeinterface/src/spikeinterface/extractors/neoextractors/openephys.py

Line 340 in 68d4258

recording_folder = Path(folder_path) / record_node

The current behavior is to create a recording_folder that becomes /path/to/Record Node 101/Record Node 101, which does not contain the requisite sample_numbers.npy file to correctly load the timestamps.

My proposed fix is simple, as seen below:

if not str(folder_path).endswith(record_node):
    recording_folder = Path(folder_path) / record_node
else:
    recording_folder = Path(folder_path)

I wanted to be more comprehensive and add a unit test to confirm that this works, now and in the future. However, as far as I know there are no existing datasets in the datalad project that contain hardware timestamped data, and I am not sure how to contribute a new file to datalad. I would be happy to provide an example recording session that contains the hardware timestamped data, so that a unit test can be added that loads the data and asserts that the end time (or total time) is correct.

[chrishalcrow]

Hey @bparks13 , thanks for finding this! Just to check, my standard output (not ONIX) looks like

Record Node 109/
    setting.xml
    experiment1/
        recording1/
        structure.oebin
            continuous/
                Neuropix-PXI-103.ProbeA/
                    continuous.dat
                    sample_numbers.npy
                    timestamps.npy

So the timestamps and sample_numbers are saved alongside the raw data. Is the ONIX file structure different? Can you share an example file tree?

Thanks!

I'm not sure about adding to datalad. Whatever is added, we try and keep whatever needs to be downloaded small. So if you're able to prepare some data with the right structure and metadata, then modify it so it's a tiny (a few samples) recording, that'd be helpful!

[bparks13]

@chrishalcrow I have the same file structure for the ONIX Source plugin. I do see one difference: in the recording1 folder there is a sync_messages.txt file. Everything else is exactly the same.

I apologize if this wasn't clear in my issue, but the core problem is that the read_openephys method assumes that the folder given is one level above the record node folder. However, the OpenEphysBinaryRecordingExtractor specifies that the folder_path can be "- Specific Record Node folder (e.g., "Record Node 102")", which is inconsistent with its behavior. To maintain consistency, I wanted to add this check in the loading code so that either level is acceptable.

Regarding datalad, I should be able get you a very small recording. I did find the online repo with some additional instructions on how to submit new files, so I will work on that today.

[alejoe91]

Thanks @bparks13, I think what you propose makes sense! Users should be able to load a specific Record Node.

Let me add a patch similar to what you suggested

[alejoe91]

I don't think we need test files, the folder structure is the same :)

[h-mayorquin]

I do think that if you are willing to provide the data @bparks13 that would be great. We might not needed it here but we might need it for something else on the future. Would you still be willing to provide it?

[bparks13]

@h-mayorquin I can definitely provide some data! I will upload that to datalad tomorrow. I will make sure it is a small dataset so that it does not cause issues when downloading it. Are there any other considerations when recording the data? I skimmed the datalad repo, so I have an idea of what to do, but any specifics provided here would be helpful.