Use MorphIO as the IO layer

[wizmer]

No description provided.

[wizmer]

I had to change the format of sample_v2.h5 as it was using unsupported h5v2 format. The repaired group was labelled under the path /neuron1/2 instead of /neuron1/repaired.

[lidakanari]

I'm afraid we still have data in this "unsupported" format (2015 Cell paper morphology release). Have they been updated or is there a way to update them?

[wizmer]

Lol, ok. It's not in the spec. I'll talk with @mgeplf tomorrow to see if it's best to update the spec and make morphio support it or to update the morphologies.

[lidakanari]

Can we ensure that this change will not have an impact on the loaded data? A "functional" test would be useful in this case. For instance could we test that the 2 actual bio neurons (in test_data/valid, C010398B-P2.CNG.swc, C010398B-P2.h5) are loaded the same way as before?

[lidakanari]

This is actually crucial for supporting "weird" cell types. Removing it will block the analysis of the microglia datasets, BigNeuron data and possible other "unsupported" data. Do you have a plan to include arbitrary tree types that the user can customize?

[wizmer]

Yes, I was just looking at it. I'll re-add the tree_type argument and test it. I'll remove you from the PR, polish it and re-add you.

[lidakanari]

Thanks a lot! That will be useful.

[mgeplf]

I'm afraid we still have data in this "unsupported" format (2015 Cell paper morphology release). Have they been updated or is there a way to update them?

@lidakanari which files in the 2015 release have /neuron1/2 style hdf5 datasets? I'm don't remember ever seeing those.

[lidakanari]

@lidakanari which files in the 2015 release have /neuron1/2 style hdf5 datasets? I'm don't remember ever seeing those.

I think it is from the 2012 release (which is used for the 2015 Cell paper). They are the unraveled cells. Are the new unraveled cells in the updated structure? If that's the case, the issue will only relate to previous releases.

[mgeplf]

From what I can see, the {"repaired", "unraveled", "raw"} h5v2 datasets in the h5 files are supported, which I think covers everything in the 2012 morph release - @wizmer was having to update the sample_v2.h5 because it had /neuron1/2 dataset, which was the 'unsupported' part, I think.

[lidakanari]

This is an example of the unraveled cells I have. Was the format modified after the 2015 Cell paper -for the same cells- or are you referring to a different release? In any case do we have a way to convert them? That will also work and resolve most of the issues.

[mgeplf]

Yeah, the unraveled type is still supported. I believe that @wizmer was referring to (in #18) the handling of neuron1/0, neuron1/1, neuron1/2,
as handled by this:
https://github.com/BlueBrain/TMD/blob/master/tmd/io/h5.py#L140

Which isn't part of the spec, and therefore isn't handled by MorphIO. An example of this h5v2 file would be interesting to see, as I don't believe I have ever seen it before.

[wizmer]

Hi. I'm cleaning my opened PR list so I'm closing this one. And it is probably no longer relevant. Feel free to ping me the day you want it to be re-opened.