biom javascript module

[iimog]

I'm planning to write a generic biom module in javascript to allow easy handling of biom data in web applications. Import of biom v1 data into my module is trivial as it is in JSON format. Import of biom v2 data on the other hand is close to impossible (for javascript running in a browser rather than on node.js, for the latter there is https://github.com/HDF-NI/hdf5.node). As the only way to handle HDF5 data properly seems to be wrapping the existing C library I see no way at all to support biom v2.

Do you have any ideas or suggestions on how to add biom v2 support to a javascript module?
Is a javascript module only capable of handling biom v1 data of any use to the community?
Any suggestions are welcome.

[ebolyen]

There is an LLVM->JavaScript library called emscripten which has been used to port things like the Unreal Engine. Most modern browsers now support Typed Arrays so I don't see any reason transpiling HDF5 wouldn't be possible, then from there you could ostensibly implement the BIOM format. But none of that is going to be easy.

[iimog]

Thanks for the pointer. I will look into that and keep you posted if I make progress.

[iimog]

I was able to compile the hdf5 library with emscripten (with some difficulties). I'm also able to compile and link example hdf5 programs in c to javascript (both for nodejs and browsers). However hdf5 does not play nicely with the fake file system provided by emscripten. It just refuses to read or write any files (even preloaded ones) with error messages like that:

HDF5-DIAG: Error detected in HDF5 (1.8.17) thread 0:
  #000: ../../src/H5T.c line 4508 in H5T_path_find(): unable to initialize conversion function
    major: Datatype
    minor: Unable to initialize object
  #001: ../../src/H5Tconv.c line 7713 in H5T__conv_long_float(): disagreement about datatype size
    major: Datatype
    minor: Unable to initialize object
  #002: ../../src/H5T.c line 2290 in H5T_register(): unable to locate/allocate conversion path
    major: Datatype
    minor: Unable to initialize object
...

Currently I see no way of working around this as the hdf5 library depends heavily on files.
For now I will restrict my javascript module to json data.

My idea to work around the problem of not supporting biom2 format natively is the following:
I will set up a publicly reachable webserver that

• takes hdf5 file content via POST
• calls the official python biom convert --to-json on it
• and sends the json object back

This webservice is called from my javascript module via AJAX if a user provides biom version 2.
Any opinions regarding this workaround?

[ebolyen]

It's a shame HDF5 is so tied the the file-system (and that there is only one reference implementation in C). I think your work-around sounds practical, there isn't much else that can be done without trying to implement HDF5 in JavaScript natively (without transpiling).

[wasade]

I'm going to close this out as it's not surrounding the Python in-memory object, or the format itself. Please reopen if you disagree.