Alleviate HDF5 bottleneck

[clementinboittiaux]

Hi @skydes
I noticed a speed bottleneck when dealing with HDF5 files. When there is a large number of groups, it takes much more time to read / write to the HDF5 file. As a result, if using more than 15,000 images, it takes much more time to write to the file than compute NetVLAD features for example. A solution was suggested in h5py/h5py#1055 and https://stackoverflow.com/questions/45023488/inserting-many-hdf5-datasets-very-slow . You simply need to use the libver='latest' option when opening HDF5 files. I found that it greatly increased writing speed when creating a great number of groups.
Sorry for the little hiccup, I accidentally deleted the branch.

[sarlinpe]

Hi @clementinboittiaux,
Nice catch! Does this break backward compatibility - i.e. are you still able to read files created before this fix? HDF5 has indeed given us some trouble recently. Changing the pair separator to create subgroups (#159) helped for long names.

[clementinboittiaux]

Hey,
I did not stumble across any incompatibility issue so far. I have been using this new branch with features computed before and after the modifications and everything went well !
I just had trouble when reading features from hLoc with PixLoc, but it seems that it is only because the data structure of the h5 files you are using in PixLoc are not the same. (I made a dirty fix on my fork https://github.com/clementinboittiaux/pixloc/blob/0e1c5b1007ea37f1684a5158007ee79bbb4c7b01/pixloc/utils/io.py#L70).

[sarlinpe]

Got it, thanks! I'll do some tests and merge later.

Indeed the structure of the HDF5 file is slightly different, I should mention this somewhere.