Skip to content

NeoMatlabIO: round-trip quantity, array and nested annotations - #1894

Open
adityasingh2400 wants to merge 1 commit into
NeuralEnsemble:masterfrom
adityasingh2400:fix-852-matlab-annotations
Open

NeoMatlabIO: round-trip quantity, array and nested annotations#1894
adityasingh2400 wants to merge 1 commit into
NeuralEnsemble:masterfrom
adityasingh2400:fix-852-matlab-annotations

Conversation

@adityasingh2400

Copy link
Copy Markdown

When NeoMatlabIO writes an object, _get_matlab_value flattens the annotations dict for MATLAB. A quantity is split into a plain magnitude plus a companion <key>_units field, exactly the way quantity attributes such as t_start are stored. A nested mapping becomes a nested struct. None becomes the sentinel string Py_None, because MATLAB has no equivalent and scipy drops a real None.

The read side undid none of that. The elif attrtype == dict branch copied every field of the struct straight into the annotations dict, so a quantity came back as {'yop': array([3., 4., 5.]), 'yop_units': 'ms'} instead of [3, 4, 5] * pq.ms, and a nested mapping came back as a raw scipy.io.matlab.mat_struct object. That is what issue #852 reports.

The same branch has a worse problem that the issue does not mention. It tested the sentinel with if value == PY_NONE, and on an array-valued annotation that comparison returns an array, which is not usable as a condition. So ValueError: The truth value of an array with more than one element is ambiguous came out of read_block for any file holding an array annotation, quantity or not. The file was written without complaint and could then never be read back.

Decoding now mirrors the encoding, in a new create_dict_from_struct alongside the existing create_ob_from_struct. A <key>_units field is folded back into the value it belongs to and is not emitted as a key of its own, but only when the field it names is actually present, so an annotation genuinely called foo_units still round-trips on its own. A nested struct is decoded recursively. The sentinel is tested only on values that are strings. One line changed on the write side too: None inside a nested annotation dict was silently dropped, because the guard that names the annotations attribute did not survive the recursion into the nested mapping, and annotations are the only dict-typed attribute any Neo class declares, which I checked across class_by_name.

Verified with a write then read round trip on a temporary file, which needs no downloaded data. The new parametrized test covers a string, an int, None, a plain array, a quantity array, a quantity scalar, a flat dict and a doubly nested dict holding quantities, and asserts no companion field leaks into the annotations. The annotation the issue asks for, yop=[3, 4, 5] * pq.ms, is also added to test_write_read_single_spike as suggested. Against 35cbce7, neo/test/iotest/test_neomatlabio.py is 6 failed 11 passed before the source change and 17 passed after, and neo/test/coretest stays at 637 passed 0 failed.

One thing I want to flag rather than bury: a quantity and its units are two separate fields in the .mat file, so an annotation dict that genuinely contains both a and a_units is indistinguishable on disk from a single quantity annotation a, and now reads back as the latter. That ambiguity is inherent to the storage format the write side already used, and it is the same convention Neo applies to object attributes, but it is a behaviour change for that one case.

Fixes #852

_get_matlab_value flattens an annotation dict for MATLAB by splitting a
quantity into a magnitude plus a companion <key>_units field, mirroring a
nested mapping as a nested struct and standing None up as a sentinel string.
The read side undid none of that. It copied every field of the struct
straight into the annotations dict, so units came back as a separate key,
nested mappings came back as scipy mat_struct objects, and the comparison
against the sentinel was done with `value == PY_NONE`, which on an array
value yields an array and raises "The truth value of an array with more than
one element is ambiguous". That made any file holding an array-valued
annotation unreadable.

Decoding now mirrors the encoding: a <key>_units field is folded back into
the quantity it belongs to, a nested struct is decoded recursively, and the
sentinel is tested only on values that are actually strings. On the write
side None inside a nested annotation dict was being dropped, because the
guard naming the annotations attribute did not survive the recursion, and
annotations are the only mapping-valued attribute Neo has.

Fixes NeuralEnsemble#852
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support Quantity-like annotations when saving/loading .mat files

1 participant