JSON metadata requests fail on RAMON Neurons that have malformed segments fields

[j6k4m8]

Token: kasthuri2015_ramon_v1
Channel: neurons

Sample ID: 10016

Though this ID exists, it throws an "Unknown error in annotation."

This json lookup fails for id=10016, though the same url with id=16 works.

Other IDs in the same range (>4 digits) also fail with the same error.

http://openconnecto.me/ocp/ca/kasthuri2015_ramon_v1/neurons/10017/json/

This is not the same error as a not-found, which simply 404s gracefully.

[alexbaden]

I can confirm this exists. When I added a test for big ints in the JSON tests it didn't fail, though, so I'm not sure what's going on. Looking now.

[alexbaden]

@WillGray check this out:

p self.kvpairs['segments'].split(',')
['[17]', '[1]', '[19]', '[58]', '[59]', '[1290]', '[2012]', '[2013]', '[2014]', '[2015]', '[2016]', '[2017]', '[2018]', '[2019]', '[2020]', '[2021]', '[2023]', '[2025]', '[2026]', '[2027]', '[2028]', '[2033]']

The offending code is on line 756 of ocpca/annotation.py:

    if self.kvpairs.get('segments'):
      self.segments = [int(i) for i in self.kvpairs['segments'].split(',')]

Python can't figure out how to interpret the string '[7]' as 7.

Do we think this is an OCP bad or a cajal bad, or something else?

17 works fine because it is a segment, and therefore has no segments.

[j6k4m8]

So it looks to me like this is an error with any RAMON object that has a segment attribute, and coincidentally all of the Kasthuri RAMONNeurons live in the greater-than-four-digit range? I'll update the issue description accordingly

@alexbaden is there an easy way to just switch these over to arrays of int-like strings either programmatically or manually? This looks like a cajal issue where matlab treats scalars as 1D matrices, I think. @WillGray any thoughts on that?

[alexbaden]

Seems reasonable to me. Maybe the two of us and @WillGray should do a
hangout next week to discuss?
On Sun, Jan 3, 2016 at 09:12 Jordan Matelsky notifications@github.com
wrote:

So it looks to me like this is an error with any RAMON object that has a
segment attribute, and coincidentally all of the Kasthuri RAMONNeurons
live in the greater-than-four-digit range? I'll update the issue
description accordingly

—
Reply to this email directly or view it on GitHub
#181 (comment)
.

[wrgr]

Interesting. Do you guys have cajal or ndio or restful snippets?

[j6k4m8]

the easiest way to get this is via RESTful call:

http://openconnecto.me/ocp/ca/kasthuri2015_ramon_v1/neurons/10016/json/

But you could also do the same thing, in ndio, for instance:

import ndio.remote.OCP as OCP
oo = OCP()

oo.get_ramon('kasthuri2015_ramon_v1', 'neurons', 16, resolution=3) # works
oo.get_ramon('kasthuri2015_ramon_v1', 'neurons', 10016, resolution=3) # fails

[jovo]

@alexbaden @j6k4m8 @WillGray this is great progress, and the top priority with regard to getting the ocp paper results completed. so, for example, a hangout tonight or tomorrow would be ideal 👍

[alexbaden]

mysql> select * from neurons_kvpairs where annoid=10016;
+--------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| annoid | kv_key     | kv_value                                                                                                                                                                                                                                                                                                   |
+--------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|  10016 | ann_author | unspecified                                                                                                                                                                                                                                                                                                |
|  10016 | segments   | [16],[1474],[1475],[1476],[1477],[1478],[1483],[1485],[1489],[1553],[1555],[1556],[1557],[1558],[1559],[1560],[1562],[1606],[1607],[1608],[1609],[1610],[1611],[1612],[1661],[1662],[1663],[1665],[1666],[1667],[1668],[1815],[1817],[1818],[1819],[1820],[1821],[1822],[1825],[1830],[3149],[3151],[3529] |
+--------+------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

Somehow the segments have been written in as a bracketed string of numbers, instead of a string of numbers.

@WillGray do you have the code to reproduce this dataset?

[edit] @WillGray better yet, do you have a RAMONified dataset that exhibits the same behavior (has defined segments in kvpairs) but works?

[wrgr]

@alexbaden - per the other thread, check out the same token/channel pair and see if it is well behaved?

[alexbaden]

mysql> select * from neurons_kvpairs where annoid=10016;
+--------+------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| annoid | kv_key     | kv_value                                                                                                                                                                                                             |
+--------+------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|  10016 | ann_author | unspecified                                                                                                                                                                                                          |
|  10016 | segments   | 16,1474,1475,1476,1477,1478,1483,1485,1489,1553,1555,1556,1557,1558,1559,1560,1562,1606,1607,1608,1609,1610,1611,1612,1661,1662,1663,1665,1666,1667,1668,1815,1817,1818,1819,1820,1821,1822,1825,1830,3149,3151,3529 |
+--------+------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

Looks good to me. Really weird....

[j6k4m8]

still borken though?

[alexbaden]

I'll look next week. That error message drives me bonkers.

On Thu, Jan 21, 2016 at 1:02 PM Jordan Matelsky notifications@github.com
wrote:

still borken though
http://openconnecto.me/ocp/ca/kasthuri2015_ramon_v1/neurons/10016/json/?

—
Reply to this email directly or view it on GitHub
#181 (comment)
.

[wrgr]

To be clear, though - this works for the hdf5 interface!

http://openconnecto.me/ocp/ca/kasthuri2015_ramon_v1/neurons/10016/

[jovo]

interesting....ok let's proceed with the hdf5 interface forthwith and anon!
we can fix borken stuff later....

On Fri, Jan 22, 2016 at 4:18 PM, William Gray notifications@github.com
wrote:

To be clear, though - this works for the hdf5 interface!

http://openconnecto.me/ocp/ca/kasthuri2015_ramon_v1/neurons/10016/

—
Reply to this email directly or view it on GitHub
#181 (comment)
.

the glass is all full: half water, half air.
neurodata.io

[j6k4m8]

i switched ndio to use the hdf5 interface, so this is no longer blocking. Closing :)

[wrgr]

We want the json interface to work, right? Reopening (unless people object) - no longer critical path.

[alexbaden]

Yeah, keep it open for now.
On Sun, Jan 24, 2016 at 17:12 William Gray notifications@github.com wrote:

We want the json interface to work, right? Reopening (unless people
object) - no longer critical path.

—
Reply to this email directly or view it on GitHub
#181 (comment)
.

[wrgr]

Guys - this is still open - can we figure out how to fix?

doesn't work:
/ca/kasthuri2015_ramon_v2/neurons_meta/10016/json/ #ramonneuron

does work:
/ca/kasthuri2015_ramon_v2/neurons_meta/3/json/ #ramonsegment

Seems that all neurons are "irretrievable" via json at this point. Perhaps we are writing them poorly? Happy to do differently but don't understand error.

Thanks!

[j6k4m8]

#stoppingscience!

[alexbaden]

It's obviously an 'Unknown exception in annotation' :-)

Looking now...

[alexbaden]

Fixed and pushed to master (a7e0621). There was a typo in the jsonann class.

Also fixed temporarily on dsp061 -- will talk with Kunal today about getting dsp061 to run the latest commit (it's a few commits behind).

The unit test coverage for ramon is poor. But, it's not clear whether we should update the current test coverage or rewrite test coverage for the microns branch. Will add to infrastructure agenda.