Error with master / upcoming v9.0: "generic_data_handle{raw=..type=void*} cannot be converted to data_handle<double>"

[pramodk]

(thanks to @arnaudon for creating the issue in the BBP JIRA system with a reproducer! 🙌 )

Overview of the issue

When switching to NEURON 9 module (e.g. neuron-nightly wheel), I encountered a crash using bluepyopt-based evaluation. The error I see is:

terminate called after throwing an instance of 'std::runtime_error' what(): generic_data_handle{raw=0x5483510 type=void*} cannot be converted to data_handle<double>

I have attached the reproducer including the necessary MOD files. The example is just using a pickled class based on bluepyopt, I can provide more info on what it is inside, but it's a step protocol applied to a cell.

This same example runs to completion with NEURON v8.2.4.

Expected result/behavior

If NEURON v8.2.4 works then NEURON 9 should also work. If the model is incompatible, the error should be better/self-explanatory.

NEURON setup

• Version: neuron-nightly/master vs 8.2.4
• Installation method: pip or from source
• Other details don't matter

Minimal working example - MWE

• Unpack the attached tarball
• and do:

python3 -m venv env
. env/bin/activate
pip install bluepyopt emodel-generalisation
nrnivmodl mods
python repro.py

To reproduce the issue, uninstall neuron v8.x from venv and install neuron-nightly.

Logs

$ python repro.py
...
works: {'bpo_holding_current': -0.0015625, 'SearchHoldingCurrent.soma.v': <bluepyopt.ephys.responses.TimeVoltageResponse object at 0x7ffc54adc110>}
terminate called after throwing an instance of 'std::runtime_error'
  what():  generic_data_handle{raw=0x3473190 type=void*} cannot be converted to data_handle<double>
Traceback (most recent call last):
  File "/gpfs/bbp.cscs.ch/home/kumbhar/Downloads/repro_neuron9_bug/repro.py", line 20, in <module>
    response = emodel_class.run(cell_model, params, sim=sim)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/bbp.cscs.ch/home/kumbhar/Downloads/repro_neuron9_bug/env/lib/python3.11/site-packages/emodel_generalisation/model/bpopt.py", line 943, in run
    voltage_min = self.get_voltage_base(
                  ^^^^^^^^^^^^^^^^^^^^^^

Reproducer:: repro_neuron9_bug.tgz

[arnaudon]

Hello, would you have any news on that? I just bumped into it again, thanks!

[1uc]

No, probably the order will be something like:

• Finish Remove CNRN mechanism layout. #2895
• VECTORIZE is misleading and in most places refers to whether a mod file is thread safe #2894
  and then this.

[1uc]

The following is a shorter reproducer, found by checking the backtrace of the reproducer provided above:

import numpy as np

from neuron import h, gui
from neuron.units import ms

mods = [
    "StochKv3",
]

s = h.Section()
for mod in mods:
    print(mod, flush=True)
    s.insert(mod)

# s.deterministic_StochKv3 = 0
h.setdata_StochKv3(0.5, sec=s)
h.setRNG_StochKv3(1, 1)

print(s(0.5).rng_StochKv3)

This ultimately tries to return a data_handle<double>, which must fail because rng isn't a double.

[1uc]

Third version of a reproducer:

import numpy as np

from neuron import h, gui
from neuron.units import ms

mods = [
    "StochKv3",
]

s = h.Section()
for mod in mods:
    print(mod, flush=True)
    s.insert(mod)

# s.deterministic_StochKv3 = 0
h.setdata_StochKv3(0.5, sec=s)
h.setRNG_StochKv3(1, 1)

s.psection()

when calling s.psection we happen to call:

mechname = "rng_StochKv3"
getattr(seg, mechname)

[1uc]

@arnaudon I'd like to test changes related to this issue. However, the morphology files have vanished from GPFS, could you please update the reproducer such that the input files are valid again?

[pramodk]

@arnaudon: Luc saw that the files on GPFS are moved/deleted and hence getting errors from reproducer:

OSError: Morphology not found at '/gpfs/bbp.cscs.ch/project/proj148/scratch/circuits/20240314/bioname/emodels/morphologies/cSTUT.asc'

Maybe it would be easy to provide new pkl files by using available morph directory like below?

/gpfs/bbp.cscs.ch/project/proj148/scratch/circuits/20240531/bioname/emodels/morphologies

We don't know the original program/test and hence the request 🙂

[arnaudon]

Sorry about that, hugo did some cleanup of circuits, and erased this one. I recreated the folder/file, so you should now be able to run the reproducer.

[1uc]

Thank you! When using #2940, I get the same output with 9.0 and 8.2.4:

failed: {'bpo_holding_current': -0.0078125, 'SearchHoldingCurrent.soma.v': <bluepyopt.ephys.responses.TimeVoltageResponse object at 0x7fffb8904850>}

(the only difference is the address.)

[arnaudon]

Great, thanks a lot!