NeoMatlabIO: round-trip quantity, array and nested annotations - #1894
Open
adityasingh2400 wants to merge 1 commit into
Open
NeoMatlabIO: round-trip quantity, array and nested annotations#1894adityasingh2400 wants to merge 1 commit into
adityasingh2400 wants to merge 1 commit into
Conversation
_get_matlab_value flattens an annotation dict for MATLAB by splitting a quantity into a magnitude plus a companion <key>_units field, mirroring a nested mapping as a nested struct and standing None up as a sentinel string. The read side undid none of that. It copied every field of the struct straight into the annotations dict, so units came back as a separate key, nested mappings came back as scipy mat_struct objects, and the comparison against the sentinel was done with `value == PY_NONE`, which on an array value yields an array and raises "The truth value of an array with more than one element is ambiguous". That made any file holding an array-valued annotation unreadable. Decoding now mirrors the encoding: a <key>_units field is folded back into the quantity it belongs to, a nested struct is decoded recursively, and the sentinel is tested only on values that are actually strings. On the write side None inside a nested annotation dict was being dropped, because the guard naming the annotations attribute did not survive the recursion, and annotations are the only mapping-valued attribute Neo has. Fixes NeuralEnsemble#852
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When
NeoMatlabIOwrites an object,_get_matlab_valueflattens the annotations dict for MATLAB. A quantity is split into a plain magnitude plus a companion<key>_unitsfield, exactly the way quantity attributes such ast_startare stored. A nested mapping becomes a nested struct.Nonebecomes the sentinel stringPy_None, because MATLAB has no equivalent and scipy drops a realNone.The read side undid none of that. The
elif attrtype == dictbranch copied every field of the struct straight into the annotations dict, so a quantity came back as{'yop': array([3., 4., 5.]), 'yop_units': 'ms'}instead of[3, 4, 5] * pq.ms, and a nested mapping came back as a rawscipy.io.matlab.mat_structobject. That is what issue #852 reports.The same branch has a worse problem that the issue does not mention. It tested the sentinel with
if value == PY_NONE, and on an array-valued annotation that comparison returns an array, which is not usable as a condition. SoValueError: The truth value of an array with more than one element is ambiguouscame out ofread_blockfor any file holding an array annotation, quantity or not. The file was written without complaint and could then never be read back.Decoding now mirrors the encoding, in a new
create_dict_from_structalongside the existingcreate_ob_from_struct. A<key>_unitsfield is folded back into the value it belongs to and is not emitted as a key of its own, but only when the field it names is actually present, so an annotation genuinely calledfoo_unitsstill round-trips on its own. A nested struct is decoded recursively. The sentinel is tested only on values that are strings. One line changed on the write side too:Noneinside a nested annotation dict was silently dropped, because the guard that names the annotations attribute did not survive the recursion into the nested mapping, and annotations are the only dict-typed attribute any Neo class declares, which I checked acrossclass_by_name.Verified with a write then read round trip on a temporary file, which needs no downloaded data. The new parametrized test covers a string, an int,
None, a plain array, a quantity array, a quantity scalar, a flat dict and a doubly nested dict holding quantities, and asserts no companion field leaks into the annotations. The annotation the issue asks for,yop=[3, 4, 5] * pq.ms, is also added totest_write_read_single_spikeas suggested. Against 35cbce7,neo/test/iotest/test_neomatlabio.pyis 6 failed 11 passed before the source change and 17 passed after, andneo/test/coreteststays at 637 passed 0 failed.One thing I want to flag rather than bury: a quantity and its units are two separate fields in the
.matfile, so an annotation dict that genuinely contains bothaanda_unitsis indistinguishable on disk from a single quantity annotationa, and now reads back as the latter. That ambiguity is inherent to the storage format the write side already used, and it is the same convention Neo applies to object attributes, but it is a behaviour change for that one case.Fixes #852