improve error reporting when stimulus file cannot be found #81

sgratiy · 2019-07-19T18:28:45Z

I get different behaviour when creating an ABF object using the same version of python (3.7.3) and pyabf (2.0.25) and numpy (1.16.4)

On Windows 10 the code below does not generate an exception, but Centos 7 does.
I would be happy to provide the abf file and protocols files

import numpy as np
import pyabf

def _check(abf):

    for sweep in range(abf.sweepCount):
        for channel in range(abf.channelCount):
            abf.setSweep(sweep, channel=channel)

            if not abf._dacSection.nWaveformEnable[channel]:
                continue

            print(F"sweep {abf.sweepC}")
            if np.isnan(abf.sweepC).any():
                raise ValueError(f"Found at least one 'Not a Number' "
                                 f"entry in stimulus channel {channel} of sweep {sweep} "
                                 f"in file {abf.abfFilePath} using protocol {abf.protocol}.")


protocolStorageDir = "/local1/ephys/abf_convertion/reference_atf"
inFile = "/local1/ephys/abf_convertion/H19_29_150_11_21_01_0011.abf"

pyabf.stimulus.Stimulus.protocolStorageDir = protocolStorageDir
abf = pyabf.ABF(inFile, loadData=False)
_check(abf)

The text was updated successfully, but these errors were encountered:

swharden · 2019-07-19T20:22:11Z

@sgratiy thanks for reporting this! If you send me the ABF I can look further into it this weekend

swharden · 2019-07-20T13:36:39Z

I get different behaviour when creating an ABF object using the same version of python (3.7.3) and pyabf (2.0.25) and numpy (1.16.4)

On Windows 10 the code below does not generate an exception, but Centos 7 does.
I would be happy to provide the abf file and protocols files

Interestingly, I do get an exception on Windows 10. I'm using Python 3.6.8 and Numpy 1.13.3. Without worrying about the Windows/Linux difference for now, I'll try to fix command waveform synthesis for this ABF.

https://github.com/swharden/pyABF/tree/master/data#h19_29_150_11_21_01_0011abf

def assertNanIsNotInSweepC(abf):

    for sweep in range(abf.sweepCount):
        for channel in range(abf.channelCount):
            abf.setSweep(sweep, channel=channel)

            if not abf._dacSection.nWaveformEnable[channel]:
                continue

            print(F"sweep {abf.sweepC}")
            if np.isnan(abf.sweepC).any():
                raise ValueError(f"Found at least one 'Not a Number' "
                                 f"entry in stimulus channel {channel} of sweep {sweep} "
                                 f"in file {abf.abfFilePath} using protocol {abf.protocol}.")


if __name__ == "__main__":
    abf = pyabf.ABF(PATH_DATA+"/H19_29_150_11_21_01_0011.abf")
    assertNanIsNotInSweepC(abf)

PS C:\Users\scott\Documents\GitHub\pyABF\dev\python> python '.\2019-07-20 sweepC test.py'
sweep [ nan  nan  nan ...,  nan  nan  nan]
Traceback (most recent call last):
  File ".\2019-07-20 sweepC test.py", line 34, in <module>
    assertNanIsNotInSweepC(abf)
  File ".\2019-07-20 sweepC test.py", line 27, in assertNanIsNotInSweepC
    raise ValueError(f"Found at least one 'Not a Number' "
ValueError: Found at least one 'Not a Number' entry in stimulus channel 0 of sweep 0 in file C:\Users\scott\Documents\GitHub\pyABF\data\abfs\H19_29_150_11_21_01_0011.abf using protocol C1NSD1SHORT.

swharden · 2019-07-20T14:02:33Z

@sgratiy I think I figured this one out and made a fix that improves it.

My guess is that on your Linux system the stimulus file is not being found in the path you gave. Note that the case sensitivity of Linux files may also contribute to this. According to your file header it's looking for a file with the case C1NSD1SHORT.pro so perhaps double-check and ensure that file exists where you expect it to.

When pyABF can't find the stimulus file it returns a sweepC filled with np.nan, which is what you've been observing. I added a new warning that makes noise in cases like this which may make it easier to recognize when stimulus files cannot be found in the future.

@sgratiy I'm looking forward to hearing if you were able to figure out the issue based on these notes! Also if you think the fix is good (a warning), I'll release it on pypi this weekend.

pyABF/src/pyabf/stimulus.py

Lines 101 to 110 in eb7656a

    
           # try to find the stimulus file 
        
           if os.path.exists(stimFname): 
        
               stimFname = os.path.abspath(stimFname) 
        
           elif os.path.exists(pathSameFolder): 
        
               stimFname = pathSameFolder 
        
           elif pathAlt and os.path.exists(pathAlt): 
        
               stimFname = pathAlt 
        
           else: 
        
               warnings.warn("could not locate stimulus file "+stimBN) 
        
               return np.full(abf.sweepPointCount, np.nan)

swharden · 2019-07-26T15:14:45Z

@sgratiy I haven't heard back in about a week so I'll assume you got it working again. If it turns out this is not the case let me know!

Either way, the new error message that got added when stimulus files cannot be found is an improvement. I'll probably issue a pypi release this weekend. Thanks for posting the initial issue.

sgratiy · 2019-07-26T23:01:58Z

@swharden, sorry I was not able to get back to you sooner. I appreciate your looking into this issue
No, I have not resolved the issue yet. I agree that it must have something to do with the paths. On Linux I get the following error message:

  File "../../sandbox/sandbox3.py", line 17, in _check
    raise ValueError(f"Found at least one 'Not a Number' "
ValueError: Found at least one 'Not a Number' entry in stimulus channel 0 of sweep 0 in file /local1/ephys/abf_convertion/H19_29_150_11_21_01_0011.abf using protocol C:\Allen Institute\Allen Institute protocols with text attachment\IN0\C1NSD1SHORT.

Indeed, I do not have the path C:\Allen Institute\Allen Institute protocols with text attachment\IN0\C1NSD1SHORT. Thanks to you, now I know that it comes from the file header.

The mystery:
On Windows I also do not have the above path, but I get no exception when providing the protocolStorageDir.

I am not familiar with this format. I thought that by providing the
protocolStorageDir = "/local1/ephys/abf_convertion/reference_atf"? as I did my original code snippet, it should read protocol from that path, rather than from some hard coded file (C:\Allen Institute\Allen Institute protocols with text attachment\IN0\C1NSD1SHORT).
I guess you got an exception on Windows because you do not have my protocol directory.
Would you like me to send you the corresponding protocol?
Perhaps if it is worth reopening this issue, or creating a different one. Sorry, again for taking so long to get back to you.

swharden · 2019-07-26T23:26:52Z

Hi @sgratiy, no worries about the delay. I'm happy to help you resolve this issue.

What is the output of this program on your system?

import numpy as np
import pyabf
pyabf.stimulus.Stimulus.protocolStorageDir = "/local1/ephys/abf_convertion/reference_atf"

def checkStorageDirectory(abf):
    assert isinstance(abf, pyabf.ABF)
    protocolDir = abf.stimulusByChannel[0].protocolStorageDir
    protocolFilename = os.path.basename(abf.protocolPath)
    protocolPath = os.path.join(protocolDir, protocolFilename)
    print("Protocol storage directory:", protocolDir)
    print("Protocol filename:", protocolFilename)
    print("Looking for protocol:", protocolPath)
    assert os.path.exists(protocolPath)

abf = pyabf.ABF("/local1/ephys/abf_convertion/H19_29_150_11_21_01_0011.abf", loadData=False)
checkStorageDirectory(abf)

sgratiy · 2019-07-26T23:43:01Z

Thanks, here is the output:

Protocol storage directory: /local1/ephys/abf_convertion/reference_atf
Protocol filename: C:\Allen Institute\Allen Institute protocols with text attachment\IN0\C1NSD1SHORT.pro
Looking for protocol: /local1/ephys/abf_convertion/reference_atf/C:\Allen Institute\Allen Institute protocols with text attachment\IN0\C1NSD1SHORT.pro
Traceback (most recent call last):
  File "../../sandbox/sandbox4.py", line 17, in <module>
    checkStorageDirectory(abf)
  File "../../sandbox/sandbox4.py", line 14, in checkStorageDirectory
    assert os.path.exists(protocolPath)
AssertionError

swharden · 2019-07-26T23:45:16Z

lol @ that third line. That's obviously wrong, but I'm confused about how that can happen...

perhaps os.path.basename(fullPath) fails on Linux if you give it a Windows path with backslashes?

swharden · 2019-07-27T04:11:49Z

This may fix your issue. It's a one-line edit so it should be easy for you to try out. Let me know how it goes!

fd32943

sgratiy · 2019-07-29T17:16:12Z

@swharden, How would be the best way to apply this change? Should I make same change in the source code of the installed package? I tried installing the latest pyABF from github into my environment
pip install git+https://github.com/swharden/pyABF.git
but got an error

swharden · 2019-07-29T17:38:50Z

I just pushed a new release on pypi (2.1.9). You can now

pip install --upgrade pyabf

Let me know if this fixes your problem!

sgratiy · 2019-07-29T17:54:13Z

Unfortunately, the fix did not work. I have updated the package as you instructed to:

pyabf 2.1.9

But running your script I still get the same path issue:

(ice36) [sergeyg@ibs-sergeyg-ux1 bin]$ python ../../sandbox/sandbox4.py 
Protocol storage directory: /local1/ephys/abf_convertion/reference_atf
Protocol filename: C:\Allen Institute\Allen Institute protocols with text attachment\IN0\C1NSD1SHORT.pro
Looking for protocol: /local1/ephys/abf_convertion/reference_atf/C:\Allen Institute\Allen Institute protocols with text attachment\IN0\C1NSD1SHORT.pro
Traceback (most recent call last):
  File "../../sandbox/sandbox4.py", line 17, in <module>
    checkStorageDirectory(abf)
  File "../../sandbox/sandbox4.py", line 14, in checkStorageDirectory
    assert os.path.exists(protocolPath)
AssertionError

swharden · 2019-07-29T18:02:24Z

Actually it may be working even though that test script fails. Can you try your original script from the first message?

import numpy as np
import pyabf

def _check(abf):

    for sweep in range(abf.sweepCount):
        for channel in range(abf.channelCount):
            abf.setSweep(sweep, channel=channel)

            if not abf._dacSection.nWaveformEnable[channel]:
                continue

            print(F"sweep {abf.sweepC}")
            if np.isnan(abf.sweepC).any():
                raise ValueError(f"Found at least one 'Not a Number' "
                                 f"entry in stimulus channel {channel} of sweep {sweep} "
                                 f"in file {abf.abfFilePath} using protocol {abf.protocol}.")


protocolStorageDir = "/local1/ephys/abf_convertion/reference_atf"
inFile = "/local1/ephys/abf_convertion/H19_29_150_11_21_01_0011.abf"

pyabf.stimulus.Stimulus.protocolStorageDir = protocolStorageDir
abf = pyabf.ABF(inFile, loadData=False)
_check(abf)

and keep an eye out for warnings or error messages about the protocol not being found

sgratiy · 2019-07-29T18:12:59Z

Ok, here is the output of the original script. It issues an new warning, but still has the original error:

/allen/aibs/technology/sergeyg/miniconda2/envs/ice36/lib/python3.6/site-packages/pyabf/stimulus.py:109: UserWarning: could not locate stimulus file C1NSD1SHORT.atf
  warnings.warn("could not locate stimulus file "+stimBN)
sweep [nan nan nan ... nan nan nan]
Traceback (most recent call last):
  File "../../sandbox/sandbox3.py", line 28, in <module>
    _check(abf)
  File "../../sandbox/sandbox3.py", line 17, in _check
    raise ValueError(f"Found at least one 'Not a Number' "
ValueError: Found at least one 'Not a Number' entry in stimulus channel 0 of sweep 0 in file /local1/ephys/abf_convertion/H19_29_150_11_21_01_0011.abf using protocol C:\Allen Institute\Allen Institute protocols with text attachment\IN0\C1NSD1SHORT.

As I mentioned earlier, I do not have the path "C:\Allen Institute\Allen Institute protocols with text attachment\IN0\C1NSD1SHORT" either on Windows and Linux, but on Windows it does not seem to be a problem.

swharden · 2019-07-29T18:43:04Z

@sgratiy thanks for your patience! I know the original windows path isn't on your system, but I'm pretty sure this is the path it's currently looking for. This file should exist, in this path, and in this case:

/local1/ephys/abf_convertion/reference_atf/C1NSD1SHORT.pro

I made an updated test script to reflect what pyABF does as of the latest update (fd32943). Does this throw an exception on your system?

import pyabf

def checkStorageDirectory(abf):
    assert isinstance(abf, pyabf.ABF)
    protocolDir = abf.stimulusByChannel[0].protocolStorageDir
    protocolPath = abf.protocolPath.replace("\\", "/") # this is the new line
    protocolFilename = os.path.basename(protocolPath)
    protocolPath = os.path.join(protocolDir, protocolFilename)
    print("Protocol storage directory:", protocolDir)
    print("Protocol filename:", protocolFilename)
    print("Looking for protocol:", protocolPath)
    assert os.path.exists(protocolPath)

if __name__ == "__main__":
    pyabf.stimulus.Stimulus.protocolStorageDir = "/local1/ephys/abf_convertion/reference_atf"
    abf = pyabf.ABF(PATH_DATA+"/H19_29_150_11_21_01_0011.abf")
    checkStorageDirectory(abf)

sgratiy · 2019-07-29T19:05:22Z

@swharden Thank you for help, I really appreciate it. Below is the output of the script. As you can see, I do not have the .pro files but I do have the .atf files (with the same name). Sorry, but I do not understand the difference between the two.Is that the problem?

Protocol storage directory: /local1/ephys/abf_convertion/reference_atf
Protocol filename: C1NSD1SHORT.pro
Looking for protocol: /local1/ephys/abf_convertion/reference_atf/C1NSD1SHORT.pro
Traceback (most recent call last):
  File "../../sandbox/sandbox5.py", line 20, in <module>
    checkStorageDirectory(abf)
  File "../../sandbox/sandbox5.py", line 13, in checkStorageDirectory
    assert os.path.exists(protocolPath)
AssertionError
(ice36) [sergeyg@ibs-sergeyg-ux1 bin]$ ls /local1/ephys/abf_convertion/reference_atf
C1NSD1SHORT.atf  C1NSD2SHORT.atf  capacitance5.atf  chirp.atf  LSCOARSE.atf  LSFINEST.atf  SSFINEST.atf  TRIPPLE.atf

swharden · 2019-07-29T20:11:28Z

I think browser window programming has gotten the best of me! I started mixing up protocol files and stimulus files. Want to zip that folder (or at least the relevant files) and send those to me? (not just the ABF) - I can then test it on a linux machine, fix the issue, and publish the fix without so much back and forth.

sgratiy · 2019-07-29T20:22:11Z

Here is the link to the zipped folder with atf files
https://www.dropbox.com/s/rnkm0frdfoh5a6r/ref_atf.zip?dl=0
Thanks again.

swharden · 2019-07-29T22:42:31Z

I got it working on my raspberry pi.

This doesn't work:

pyabf.stimulus.Stimulus.protocolStorageDir = "/local1/ephys/abf_convertion/reference_atf"
abf = pyabf.ABF(inFile, loadData=False)
print(abf.sweepC)

This worked:

abf = pyabf.ABF(inFile, loadData=False)
abf.stimulusFileFolder = "/local1/ephys/abf_convertion/reference_atf"
print(abf.sweepC)

Does the second method work for you?

sgratiy · 2019-07-30T21:38:12Z

Yes, the second example does work. Thanks. I am not familiar with the format. Is the path to the atf files cannot be written in pyabf.stimulus.Stimulus.protocolStorageDir?

swharden · 2019-07-30T21:49:56Z

I'm glad you got this working! I'll add this example to the documentation page so people are less likely to get confused over this in the future.

I don't remember all the ins and outs of this topic, but I did notice that a warning is thrown when you try to assign to protocolStorageDir telling you to assign to abf.stimulusFileFolder instead:

pyABF/src/pyabf/stimulus.py

Lines 81 to 83 in 0090854

    
           @protocolStorageDir.setter 
        
           def protocolStorageDir(self, val=None): 
        
               warnings.warn("set abf.stimulusFileFolder (not protocolStorageDir)")

I think protocolStorageDir was left in there for legacy support reasons, but in my opinion it's a pretty bad name because protocol files are distinctly different than stimulus files, and they can be stored in separate folders.

Either way I'm glad this is resolved! Thanks again for sharing your issue!

swharden · 2019-07-30T22:11:04Z

This is the new documentation for manually defining stimulus waveform folders:
https://github.com/swharden/pyABF/blob/master/docs/getting-started/advanced.md#stimulus-file-folders-and-caching

sgratiy · 2019-07-31T00:59:40Z

thank you.

sgratiy changed the title ~~Different behaviour on Linux vs Windows~~ Stimulus channel values (None vs 0) - different behaviour on Linux (resulting in None) vs Windows (resulting in 0) Jul 19, 2019

sgratiy mentioned this issue Jul 19, 2019

Test run ephys pipeline with nwb2 files AllenInstitute/ipfx#251

Closed

4 tasks

swharden self-assigned this Jul 19, 2019

swharden closed this as completed Jul 26, 2019

swharden changed the title ~~Stimulus channel values (None vs 0) - different behaviour on Linux (resulting in None) vs Windows (resulting in 0)~~ improve error reporting when stimulus file cannot be found Jul 26, 2019

swharden reopened this Jul 26, 2019

swharden closed this as completed Jul 30, 2019

sgratiy mentioned this issue Sep 9, 2019

Abf to nwb v2 conversion: different behaviour on Windows vs. Linux AllenInstitute/ipfx#282

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve error reporting when stimulus file cannot be found #81

improve error reporting when stimulus file cannot be found #81

sgratiy commented Jul 19, 2019 •

edited

Loading

swharden commented Jul 19, 2019

swharden commented Jul 20, 2019

swharden commented Jul 20, 2019 •

edited

Loading

swharden commented Jul 26, 2019

sgratiy commented Jul 26, 2019

swharden commented Jul 26, 2019

sgratiy commented Jul 26, 2019

swharden commented Jul 26, 2019

swharden commented Jul 27, 2019

sgratiy commented Jul 29, 2019 •

edited

Loading

swharden commented Jul 29, 2019

sgratiy commented Jul 29, 2019

swharden commented Jul 29, 2019

sgratiy commented Jul 29, 2019

swharden commented Jul 29, 2019

sgratiy commented Jul 29, 2019 •

edited

Loading

swharden commented Jul 29, 2019

sgratiy commented Jul 29, 2019

swharden commented Jul 29, 2019

sgratiy commented Jul 30, 2019

swharden commented Jul 30, 2019

swharden commented Jul 30, 2019

sgratiy commented Jul 31, 2019

improve error reporting when stimulus file cannot be found #81

improve error reporting when stimulus file cannot be found #81

Comments

sgratiy commented Jul 19, 2019 • edited Loading

swharden commented Jul 19, 2019

swharden commented Jul 20, 2019

swharden commented Jul 20, 2019 • edited Loading

swharden commented Jul 26, 2019

sgratiy commented Jul 26, 2019

swharden commented Jul 26, 2019

sgratiy commented Jul 26, 2019

swharden commented Jul 26, 2019

swharden commented Jul 27, 2019

sgratiy commented Jul 29, 2019 • edited Loading

swharden commented Jul 29, 2019

sgratiy commented Jul 29, 2019

swharden commented Jul 29, 2019

sgratiy commented Jul 29, 2019

swharden commented Jul 29, 2019

sgratiy commented Jul 29, 2019 • edited Loading

swharden commented Jul 29, 2019

sgratiy commented Jul 29, 2019

swharden commented Jul 29, 2019

sgratiy commented Jul 30, 2019

swharden commented Jul 30, 2019

swharden commented Jul 30, 2019

sgratiy commented Jul 31, 2019

sgratiy commented Jul 19, 2019 •

edited

Loading

swharden commented Jul 20, 2019 •

edited

Loading

sgratiy commented Jul 29, 2019 •

edited

Loading

sgratiy commented Jul 29, 2019 •

edited

Loading