Unable to open file (h5py.h5f.open)

[hung-th]

Thank you for developing this super useful tool.

I have installed the instaGRAAL with no-opengl branch because freeglut always showed the display error same as others.

Then I ran the following command:

instagraal hic ph.a60.FALC.fasta

It looks like start running (INFO :: start filtering), but soon encounters an error:

INFO :: start filtering
Traceback (most recent call last):
  File "/home/tin_hang_hung/miniconda3/envs/instagraal/bin/instagraal", line 33, in <module>
    sys.exit(load_entry_point('instagraal==0.1.6', 'console_scripts', 'instagraal')())
  File "/home/tin_hang_hung/miniconda3/envs/instagraal/lib/python3.8/site-packages/instagraal-0.1.6-py3.8.egg/instagraal/instagraal.py", line 1164, in main
    p2 = instagraal_class(
  File "/home/tin_hang_hung/miniconda3/envs/instagraal/lib/python3.8/site-packages/instagraal-0.1.6-py3.8.egg/instagraal/instagraal.py", line 151, in __init__
    self.simulation = simulation(
  File "/home/tin_hang_hung/miniconda3/envs/instagraal/lib/python3.8/site-packages/instagraal-0.1.6-py3.8.egg/instagraal/simu_single.py", line 74, in __init__
    self.select_data_set(name)
  File "/home/tin_hang_hung/miniconda3/envs/instagraal/lib/python3.8/site-packages/instagraal-0.1.6-py3.8.egg/instagraal/simu_single.py", line 739, in select_data_set
    self.hic_pyr = pyr.build_and_filter(
  File "/home/tin_hang_hung/miniconda3/envs/instagraal/lib/python3.8/site-packages/instagraal-0.1.6-py3.8.egg/instagraal/pyramid_sparse.py", line 103, in build_and_filter
    pyramid_0 = h5py.File(init_pyramid_file)
  File "/home/tin_hang_hung/miniconda3/envs/instagraal/lib/python3.8/site-packages/h5py/_hl/files.py", line 424, in __init__
    fid = make_fid(name, mode, userblock_size,
  File "/home/tin_hang_hung/miniconda3/envs/instagraal/lib/python3.8/site-packages/h5py/_hl/files.py", line 190, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 96, in h5py.h5f.open
OSError: Unable to open file (unable to open file: name = './pyramids/pyramid_1_no_thresh/pyramid.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

I will be very grateful if you can help with this problem.

[nadegeguiglielmoni]

Hello,

Could you try deleting the pyramid folder and launching again?

[hung-th]

Hi, unfortunately the same error persisted after deleting that pyramid folder.

[hung-th]

I am providing one more piece of information:

instagraal --version hic ph.a60.FALC.fasta returned 0.1.2

[nadegeguiglielmoni]

Would you mind sharing your hic_folder and your assembly so I can try to reproduce the error and see why there is a missing file?

[hung-th]

No problem at all, thank you so much for your help!

Here they are:
Google Drive

[nadegeguiglielmoni]

Hello,

I was not able to reproduce this error, so I suppose this may be due to some dependency but I cannot figure out yet which one. Have you been able to run instaGRAAL successfully before? Were all the dependencies installed successfully? I see that you are using conda and we had some troubles before to install some dependencies for instaGRAAL when using conda. Could you send me the output of ls in the pyramid folder?

As I did not have any issue to run your data and my GPU was not busy, I let it run completely and I deposited the output here https://we.tl/t-oI1KqffKOJ . You will find the pyramids folder, the instagraal output folder, and the final assembly.

[hung-th]

Basically it creates an empty directory of pyramid_1_no_thresh in the directory pyramid.

ls pyramids
> pyramid_1_no_thresh

ls pyramids/pyramid_1_no_thresh
>

This is my first time trying instaGRAAL - I even started from an empty virtual instance on the Google Cloud Platform. I will try to reinstall the dependencies and let you know if I could pinpoint which dependency has the problem.

I really appreciate for your help and actually the output looks super good. N50 is 57.0 Mb, and 10 chromosomes could be identified as expected. Both SALSA2 and Juicer/3D-DNA found 11 chromosomes and lower rate of mapping on the chromosomes - it seems that instaGRAAL outperformed both of the scaffolders.

[hung-th]

Also, I would like to quickly ask how I could visualise the contact matrix after instaGRAAL?

Thank you a lot for your help!

[cmdoret]

Hi, @hung-th, you can visualise instagraal output directly using the hicstuff view command from hicstuff.
Alternatively, if you prefer to work with the cooler format, you can use hicstuff convert to convert it to cool and then use cooler to visualise it.

[nadegeguiglielmoni]

I'm glad you are happy with the result! I will still try to figure out what may have failed for this file to be missing.

[cmdoret]

Nevermind, you actually need to regenerate the Hi-C map with the new genome after instagraal, can't do it directly on the input file AFAIK, sorry !

[FirePanda007]

From my point of view, the problem is caused by the h5py version conflict.

In instagraal/pyramid_sparse.py, the pyramid is initialized by:

pyramid_0 = h5py.File(init_pyramid_file)

However, if you use h5py > 3.0.0, the default file mode will be 'r':

H5pyDeprecationWarning: The default file mode will change to 'r' (read-only) in h5py 3.0. To suppress this warning, pass the mode you need to h5py.File(), or set the global default h5.get_config().default_file_mode, or set the environment variable H5PY_DEFAULT_READONLY=1. Available modes are: 'r', 'r+', 'w', 'w-'/'x', 'a'. See the docs for details.

The pyramid file won't be created so that the problem is caused.

Go back to h5py 2.X , and the instaGRAAL will be OK!

pip install h5py==2.10.0

[nadegeguiglielmoni]

Hello,

This is indeed a good idea FirePanda007. I have updated the no_opengl accordingly, hung-th could you try again with the latest commit and tell me if this works fine? You will need to delete the pyramid folder before trying to run instaGRAAL.

[hung-th]

Thank you @cmdoret @FirePanda007 @nadegeguiglielmoni for all your useful and helpful inputs. Really much appreciated.

The instaGRAAL is running nicely now and the pyramid files are being generated. Since the h5py problem is resolved, I will close the issue for now.

Have a nice day and take care!