Read complex numbers from HDF5

[garrison]

Although there is no standardized way of storing complex numbers in an HDF5 file, a few de facto standards have emerged using compound datatypes. Octave, for instance, uses a compound datatype with fields "real" and "imag", while pytables and h5py both use fields "r" and "i".

Since the goal is to get to the point where we can read any HDF5 file, I think it would be wonderful if HDF5.jl could load arrays from these files into arrays of complex numbers.

The following Python program can be used to make an HDF5 file with two matrices: one real and one complex:

#!/usr/bin/env python

import h5py

f = h5py.File("complex.h5", "w")
f["m1"] = [2, 1]
f["m2"] = [2, 1j]
f.close()

The goal is to get Julia to load both matrices using the following code:

using HDF5

println(h5read("complex.h5", "m1"))
println(h5read("complex.h5", "m2"))

However, currently it fails on the last line while reading the array m2 containing complex numbers.

[timholy]

It doesn't so much fail as need some help:

julia> r = h5read("complex.h5", "m2")
HDF5.HDF5Compound(UInt8[0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x40,0x00,0x00  …  0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0xf0,0x3f],Type[Float64,Float64],ASCIIString["r","i"],UInt64[0x0000000000000000,0x0000000000000008])

julia> reinterpret(Complex{Float64}, r.data)
2-element Array{Complex{Float64},1}:
 2.0+0.0im
 0.0+1.0im

My suspicion is that the best way to handle this would be with a separate module dedicated to reading particular formatting conventions. There's nothing in your h5 file that says "this structure with fields r and i is to be interpreted as a complex number." But an H5PY module could do that. Care to contribute it?

[garrison]

When I filed the bug it really did fail, but I just updated from v0.4.6 to v0.4.7 and now it outputs a compound data type as expected.

Thanks for the tips on how to move forward. So the goal is an H5PY module similar to JLD.jl, but with much less complexity? Like yourself I am quite busy with other things, but I may get around to it at some point.

[timholy]

Yes. I have long-term plans to split JLD out into a separate repository, so probably that would be best for H5PY, too. In other words, it could be your own package.

MAT is another example, and probably closest in spirit to what you're trying to achieve.

[garrison]

On 12/03/2014 11:16 AM, Tim Holy wrote:

MAT is another example, and probably closest in spirit to what you're
trying to achieve.

Thanks, that looks like a much better module to base this on.

[bhawkins]

Sorry to necro an old issue, but I was struggling with this today and this issue was high on the list of search results. The reinterpret trick suggested here and elsewhere no longer seems to work. Using PyCall to use h5py directly works, of course, but I'd prefer to avoid that route.

I'm a pretty green Julia programmer, but I managed to cobble something together in this gist:

https://gist.github.com/bhawkins/6edaab42b9907758eeb58a24399b9426

You can use the low-level HDF5.jl routines to create the h5py complex type given a Julia Complex{T} and vice-versa.

The other difficulty I had was not being able to slice datasets with compound types, so you can't write dset[:,1] = z. Fortunately hyperslab (not exported) seems to work fine in this scenario, and you can use it in conjunction with the low-level h5d_read and h5d_write to read/write slices. I'd love to know if there's a higher-level way to do this.

I also noticed that h5t_get_member_name only works with a pointer, whereas the other H5T methods are happy with HDF5Datatype objects.

[kleinhenz]

I think that complex number support would be nice to have built into HDF5.jl. Requiring a general serialization framework like JLD or writing custom code just to conveniently save/load arrays of complex numbers feels like overkill. The precedent of h5py/octave seem to indicate that transparently adding complex number support to hdf5 wrappers via compound datatypes is a reasonable approach. Would people be open to a pull request implementing this approach?

[musm]

@kleinhenz PRs that improve HDF5.jl are always welcome

[tknopp]

@kleinhenz, @musm: I have started working on this, on this branch:
https://github.com/JuliaIO/HDF5.jl/tree/tk/complex

write support is there, although I am not sure if I added it to the correct layer. Maybe it would be better to let ScalarOrString include the complex type.

Read support is still missing.

[tknopp]

Or more precisely: I know how to read and interpret the data, but we somehow need to special case the compound data code in that case, and I am not sure what would be the cleanest way for that.