Files no longer added to list after file access problem

[philipstarkey]

Original report (archived issue) by Shaun Johnstone (Bitbucket: shjohnst, GitHub: shjohnst).

---

lyse appears to fail to read a .h5 file that it is adding to its list, after which new incoming files fail to be added to the list.
I can see them coming in in the terminal as Webserver requests, but then the usual "INFO LYSE.FileBox.incoming: adding some files" line never comes.

The traceback from the read fail is:

#!python

Traceback (most recent call last):
  File "C:\pythonlib\labscript_utils\excepthook\__init__.py", line 32, in run
    run_old(*args, **kwargs)
  File "C:\Python27\lib\threading.py", line 483, in run
    self.__target(*self.__args, **self.__kwargs)
  File "main.pyw", line 570, in incoming_buffer_loop
    self.add_files(filepaths, marked=True)
  File "main.pyw", line 690, in add_files
    row = get_dataframe_from_shot(filepath)
  File "C:\pythonlib\lyse\dataframe_utilities.py", line 165, in get_dataframe_from_shot
    nested_dict = get_nested_dict_from_shot(filepath)
  File "C:\pythonlib\lyse\dataframe_utilities.py", line 62, in get_nested_dict_from_shot
    with h5py.File(filepath,'r') as h5_file:
  File "C:\pythonlib\labscript_utils\h5_lock.py", line 42, in __init__
    _orig_init(self, name, mode, driver, libver, **kwds)
  File "C:\Python27\lib\site-packages\h5py\_hl\files.py", line 207, in __init__
    fid = make_fid(name, mode, userblock_size, fapl)
  File "C:\Python27\lib\site-packages\h5py\_hl\files.py", line 79, in make_fid
    fid = h5f.open(name, h5f.ACC_RDONLY, fapl=fapl)
  File "h5f.pyx", line 71, in h5py.h5f.open (h5py\h5f.c:1745)
IOError: unable to open file (File accessability: Unable to open file)

This should fail more gracefully and allow lyse to continue operating rather than requiring a restart. As far as BLACS is concerned, everything is OK and lyse is accepting the files for analysis, so it doesn't even queue them up to send to lyse once it is restarted.

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

OK, yes this is because things are decoupled in lyse - stuff arrives and is put in a queue, and success reported to BLACS. Another thread in lyse then reads the file. This latter thread is obviously crashing upon failure, and then does not read subsequent files.

I can change this around so that the file is read before success is reported to BLACS. Also obviously it needs to be wrapped in a try: except: so that failure doesn't take down the thread in question.

[philipstarkey]

I'm going to close this is it was partially fixed in 0c4fff8. Failure to read the file no longer crashes lyse. BLACS still won't know if a file is handled by lyse correctly, but I don't think it should in order to keep things decoupled. A warning is printed to the lyse terminal if a shot cannot be opened, which I think is sufficient.