unequal number of datapoints in NWB conversion

[ozsat]

I noticed that when I convert DAT to NWB (ipfx.bin.run_x_to_nwb_conversion.convert), the voltage samples and current samples are not equal. In the current samples, 5 of the given points are always missing from the beginning. It seems to be a major problem because the ipfx cant process those NWB files.
I checked the raw data and it contains an equal number of points (I also attached a picture that shows the discrepancy in the NWB file).

When I run run_pipeline on the generated NWB data the following error message appears:

INFO:root:
INFO:root:***** ***** ***** Extract QC features ***** ***** *****
INFO:root:=======================================================
INFO:root:
INFO:root:Ontology is not provided, using default E:\anaconda3\lib\site-packages\ipfx\defaults\stimulus_ontology.json
E:\anaconda3\lib\site-packages\ipfx\qc_feature_extractor.py:28: VisibleDeprecationWarning: Function get_sweep_number is deprecated. call .get_sweep_numbers()[-1] instead
blowout_sweep_number = data_set.get_sweep_number(ontology.blowout_names)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: extpinbath
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: extpbreakn
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Long square
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Rheobase
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Short square
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Capacitance
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Chirp test
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Chirp
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: extpexpend
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\qc_feature_extractor.py:57: VisibleDeprecationWarning: Function get_sweep_number is deprecated. call .get_sweep_numbers()[-1] instead
bath_sweep_number = data_set.get_sweep_number(ontology.bath_names)
E:\anaconda3\lib\site-packages\ipfx\qc_feature_extractor.py:99: VisibleDeprecationWarning: Function get_sweep_number is deprecated. call .get_sweep_numbers()[-1] instead
seal_sweep_number = data_set.get_sweep_number(ontology.seal_names,"VoltageClamp")
E:\anaconda3\lib\site-packages\ipfx\qc_feature_extractor.py:151: VisibleDeprecationWarning: Function get_sweep_number is deprecated. call .get_sweep_numbers()[-1] instead
breakin_sweep_number = data_set.get_sweep_number(ontology.breakin_names,"VoltageClamp")
WARNING:root:Could not compute input/access resistance ratio (sr: None, ir:: None)
WARNING:root:Blowout is not available
WARNING:root:Electrode 0 is not available
WARNING:root:Seal is not available
WARNING:root:Breakin sweep not found
Traceback (most recent call last):
File "e:/ipfx/File_reader.py", line 74, in
output_dir=Output_Full_Path,
File "e:/ipfx/File_reader.py", line 60, in run_pipelinefrom_nwb_file
write_spikes)
File "E:\anaconda3\lib\site-packages\ipfx\bin\run_pipeline.py", line 27, in run_pipeline
stimulus_ontology_file
File "E:\anaconda3\lib\site-packages\ipfx\bin\run_sweep_extraction.py", line 69, in run_sweep_extraction
sweep_features = sweep_qc_features(ds)
File "E:\anaconda3\lib\site-packages\ipfx\qc_feature_extractor.py", line 278, in sweep_qc_features
sweep = data_set.sweep(sweep_num)
File "E:\anaconda3\lib\site-packages\ipfx\dataset\ephys_data_set.py", line 240, in sweep
enforce_equal_length=True,
File "E:\anaconda3\lib\site-packages\ipfx\dataset\ephys_data_set.py", line 431, in _voltage_current
f"found {len(voltage)} voltage samples, "
ValueError: found 302000 voltage samples, but 301995 current samples

The DAT file as an example: https://drive.google.com/file/d/1GgNunvFPqRpjz9A6DAj9jF7uFA_owmtR/view?usp=sharing
The generated NWB: https://drive.google.com/file/d/1eHzVY_2F6fyWNtE0R1OyhkAtZ2a-JMWt/view?usp=sharing

Environment:

• Windows 10
• ipfx 1.0.0
• python 3.7

[tmchartrand]

@ozsat I'm guessing you closed this based on using the fix implemented on my fork? we should keep this issue open to track the work of merging that fix into the primary repository.
If you closed it for some other reason, please let us know!

[tmchartrand]

This is fixed at the new x-to-nwb conversion repo (see above), but should maybe be left for reference until the faulty code is removed here (#492).