Failure to complete significant API calls, e.g. update 3+ neuron ID at same time or flywire.get_synapses()

[masonweinstock]

Background
Hi. I have been at this for weeks now, without resolving the issue. I have tried multiple python versions, IDEs, networks/ethernet, generated new flywire tokens, conda environments... everything I can think of. I am on Mac OS X 14.2.1 (Sonoma) on a MacBook Pro M3 Max, using Pycharm to develop in a Python 3.10 virtual miniconda environment, with fafbseg 3.0.2, CAVE client 5.17.2, When using the newest python/fafbseg, I'd had some quirky issues with installation dependencies due to using a Mac with a Silicon chip, and so I downgraded to the versions my lab colleague uses successfully for flywire. I have an account with flywire & the codex, but do not seem to have production dataset access, but I just set the default database to public for now.

Goal
Anyway, I have been attempting to write a script which gets a synapse list for a specific neuron ID, and then compares it to lists of neurons I have by cell type, in order to produce a basic graph of what cell types this neuron projects to. Eventually I will do a ton of other stuff, but I have been stuck at just getting the synapse list.

What works and what doesn't
I am able to successfully update the ID for 1-4/5 neurons at one time with the update_ids() function. However, if I try to update more than 5 at once using a list, I get the attached timeout error:
ERROR_update_ids.txt

I am also able to use the. get_mesh_neuron() function successfully, though I don't actually need the function.

I am entirely unable to get a response for the get_synapses(), get_connectivity/synapses.fetch_connectivity(), get_transmitter_predictions(), or get_synapse_counts() functions. I have attached the typical error I get when attempting to run the following line (regardless of any combination of parameters):
ERROR_get_synapses.txt

fafbseg.flywire.get_synapses($neuronID, downstream=False, min_score=30)

Since the two errors are extremely similar, I'm thinking the error has something to do with connectivity..? Please let me know if anyone has seen this error before and/or what I could do about it. I have also attached a trimmed down version of my script below. Thanks!

script_for_help.txt

-- Mason Weinstock

[schlegelp]

Hi! The tracebacks suggest that the timeout and the other errors occur in caveclient (or cloud-volume) not in fafbseg itself. In the past, I had issues with CAVE for long running queries which would occasionally timed out (sometimes depending on the time of the day). However, if I understand you correctly your queries always time out?

The folks over at the Allen recently did a major rework of the innards of cave-client which caused connection issues/timeouts and failed queries in my hands. Looking at your version of cave-client that might just be it. Could you update cave-client and fafbseg to the latest versions (5.18.0 and 3.0.5, respectively)? Maybe best to also update cloud-volume while you're at it. Then try running your code again.

If that doesn't work, I would suggest temporarily circumventing fafbseg and calling cave-client directly. E.g. for getting outgoing (pre) synapses of a given neuron:

>>> import caveclient as cv
>>> client = cv.CAVEclient('flywire_fafb_public')
>>> syn = client.materialize.query_view('valid_synapses_nt_v2_view', filter_in_dict=dict(pre_pt_root_id=[720575940603231916]))

I have an account with flywire & the codex, but do not seem to have production dataset access, but I just set the default database to public for now

Yes, my latest info is that access to the production dataset still requires extra admittance

[masonweinstock]

Hi, thanks for getting back to me so quickly! I will try updating the packages to see if that resolves it. When I first tried to set everything up last month, I could not install the then-current fafbseg due to dependency issues… but I’ll absolutely give it another shot! I tried directly using CAVE-client last week, but got the same error unfortunately. Thanks again for your help! I’ll try updating and let you know if that resolves the issue. — Mason Weinstock

…

On Tue, Apr 2, 2024 at 14:32 Philipp Schlegel ***@***.***> wrote: Hi! The tracebacks suggest that the timeout and the other errors occur in caveclient (or cloud-volume) not in fafbseg itself. In the past, I had issues with CAVE for long running queries which would occasionally timed out (sometimes depending on the time of the day). However, if I understand you correctly your queries *always* time out? The folks over at the Allen recently did a major rework of the innards of cave-client which caused connection issues/timeouts and failed queries in my hands. Looking at your version of cave-client that might just be it. Could you update cave-client and fafbseg to the latest versions (5.18.0 and 3.0.5, respectively)? Maybe best to also update cloud-volume while you're at it. Then try running your code again. If that doesn't work, I would suggest temporarily circumventing fafbseg and calling cave-client directly. E.g. for getting outgoing (pre) synapses of a given neuron: >>> import caveclient as cv>>> client = cv.CAVEclient('flywire_fafb_public')>>> syn = client.materialize.query_view('valid_synapses_nt_v2_view', filter_in_dict=dict(pre_pt_root_id=[720575940603231916])) I have an account with flywire & the codex, but do not seem to have production dataset access, but I just set the default database to public for now Yes, my latest info is that access to the production dataset still requires extra admittance — Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A4XMDWOB343UFIIK2BCAOITY3L2VRAVCNFSM6AAAAABFT3DMRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZSG44DAOJQGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[masonweinstock]

And yes— queries *always* timeout above a certain (pretty small) request size!

…

On Tue, Apr 2, 2024 at 3:32 PM MasonWeinstock ***@***.***> wrote: Hi, thanks for getting back to me so quickly! I will try updating the packages to see if that resolves it. When I first tried to set everything up last month, I could not install the then-current fafbseg due to dependency issues… but I’ll absolutely give it another shot! I tried directly using CAVE-client last week, but got the same error unfortunately. Thanks again for your help! I’ll try updating and let you know if that resolves the issue. — Mason Weinstock On Tue, Apr 2, 2024 at 14:32 Philipp Schlegel ***@***.***> wrote: > Hi! The tracebacks suggest that the timeout and the other errors occur in > caveclient (or cloud-volume) not in fafbseg itself. In the past, I had > issues with CAVE for long running queries which would occasionally timed > out (sometimes depending on the time of the day). However, if I understand > you correctly your queries *always* time out? > > The folks over at the Allen recently did a major rework of the innards of > cave-client which caused connection issues/timeouts and failed queries > in my hands. Looking at your version of cave-client that might just be > it. Could you update cave-client and fafbseg to the latest versions ( > 5.18.0 and 3.0.5, respectively)? Maybe best to also update cloud-volume > while you're at it. Then try running your code again. > > If that doesn't work, I would suggest temporarily circumventing fafbseg > and calling cave-client directly. E.g. for getting outgoing (pre) > synapses of a given neuron: > > >>> import caveclient as cv>>> client = cv.CAVEclient('flywire_fafb_public')>>> syn = client.materialize.query_view('valid_synapses_nt_v2_view', filter_in_dict=dict(pre_pt_root_id=[720575940603231916])) > > I have an account with flywire & the codex, but do not seem to have > production dataset access, but I just set the default database to public > for now > > Yes, my latest info is that access to the production dataset still > requires extra admittance > > — > Reply to this email directly, view it on GitHub > <#18 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/A4XMDWOB343UFIIK2BCAOITY3L2VRAVCNFSM6AAAAABFT3DMRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZSG44DAOJQGE> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

[schlegelp]

No problem. Do let me know if you run into dependecy issues!

[masonweinstock]

Hello, Updating fafbseg and cave client worked! Thank you for your help. I am able to access the API. However, while get_synapses() works, get_connectivity() fails with an HTTP500 internal error message. Have you seen this before? Thanks, Mason Weinstock

…

On Tue, Apr 2, 2024 at 16:14 Philipp Schlegel ***@***.***> wrote: No problem. Do let me know if you run into dependecy issues! — Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A4XMDWOGBA2NMXJAPVKYT4TY3MGUFAVCNFSM6AAAAABFT3DMRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZTGAYTKNRXGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[schlegelp]

Can you share a minimal code example to reproduce the HTTP500 error?

[masonweinstock]

For example, flywire.get_connectivity($neuronID) produces the error!

…

On Wed, Apr 3, 2024 at 12:02 Philipp Schlegel ***@***.***> wrote: Can you share a minimal code example to reproduce the HTTP500 error? — Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A4XMDWJ4FDZC56XIRSLUZFLY3QRXXAVCNFSM6AAAAABFT3DMRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZVGAYDMMBSGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[schlegelp]

I just tried this and it worked:

>>> from fafbseg import flywire
>>> flywire.set_default_dataset('public')
>>> flywire.get_connectivity(720575940622771976)
Using materialization version 783.
                     pre                post  weight
0     720575940620610420  720575940622771976      35
1     720575940622771976  720575940626783065      33
2     720575940622771976  720575940622754342      29
...

It would be helpful if you could provide a minimal, self contained example like the above - including the traceback.

[masonweinstock]

Okay, here is the exact code and error:

Code

from fafbseg import flywire as fw
fw.set_default_dataset('public')
navis.patch_cloudvolume()


lplc2 = 720575940616491993
input_synapses = fw.get_synapses(lplc2, pre=False, min_score=30)
input_connections = fw.get_connectivity(lplc2, downstream=False, min_score=30) # this is what throws the error

Output:

INFO : cloud-volume successfully patched! (navis)
Default dataset set to "public".
WARNING : Querying synapse table with filtered=True already removes synaptic connections with cleft_score <= 50. If you want less confident connections set filtered=False. Note that this will also drop the de-duplication (see docstring). (navis)
Using materialization version 783.
Using materialization version 783.
Traceback (most recent call last):
File "/Users/masonweinstock/Documents/School/Spring '24/lab/ausborn/practice scripting/LPLC2 Synapses/masons_version.py", line 51, in
input_connections = fw.get_connectivity(lplc2, downstream=False, min_score=30)
File "/opt/homebrew/Caskroom/miniconda/base/envs/modelling_10/lib/python3.10/site-packages/fafbseg/flywire/annotations.py", line 132, in inner
return func(*args, **kwargs)
File "/opt/homebrew/Caskroom/miniconda/base/envs/modelling_10/lib/python3.10/site-packages/fafbseg/flywire/utils.py", line 167, in inner
return func(*args, **kwargs)
File "/opt/homebrew/Caskroom/miniconda/base/envs/modelling_10/lib/python3.10/site-packages/fafbseg/flywire/synapses.py", line 1188, in get_connectivity
syn.append(func(filter_in_dict=filter_in_dict))
File "/opt/homebrew/Caskroom/miniconda/base/envs/modelling_10/lib/python3.10/site-packages/fafbseg/flywire/utils.py", line 492, in wrapper
return func(*args, **kwargs)
File "/opt/homebrew/Caskroom/miniconda/base/envs/modelling_10/lib/python3.10/site-packages/caveclient/materializationengine.py", line 949, in join_query
self.raise_for_status(response)
File "/opt/homebrew/Caskroom/miniconda/base/envs/modelling_10/lib/python3.10/site-packages/caveclient/base.py", line 233, in raise_for_status
_raise_for_status(r, log_warning=log_warning)
File "/opt/homebrew/Caskroom/miniconda/base/envs/modelling_10/lib/python3.10/site-packages/caveclient/base.py", line 80, in _raise_for_status
raise requests.HTTPError(http_error_msg, response=r)
requests.exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: https://prod.flywire-daf.com/materialize/api/v3/datastack/flywire_fafb_public/version/783/query?return_pyarrow=True&arrow_format=True&split_positions=True content:b'{"schema_errors": {"tables": {"0": {"0": ["Field may not be null."]}}}}\n'

[schlegelp]

Ok, I can reproduce the error. It's a mix of configuration errors on both my side as well as on the cave-client. It should be an easy fix but I'm waiting to hear back.

[schlegelp]

Hi @masonweinstock. Sorry, I lost track of this for a while. Just released fafbseg version 3.0.7 which includes a fix for this error. I'm closing this issue as fixed but feel free to re-open if you encounter further problems..