Cannot retrieve ion_channel_models from database, nor examplar_morpholgy.brain_region or examplar_morphology.species

[AurelienJaquier]

I have created a PR for implementing EModel in entitysdk: #44

However, a had to put a few fields as optional, because they would make the entity search fail if they were enforced. I think it is because the data is missing on the database, but I am not sure. The fields that are an issue are: ion_channel_models, examplar_morpholgy.brain_region, examplar_morphology.species

Here is the piece of code I use to get an EModel:

iterator = client.search_entity(
    entity_type=EModel,
    query={"name": "EM__1372346__cADpyr__13"},
    token=access_token,
    limit=1,
)
emodel = next(iterator)

and here is the error I get:

Traceback (most recent call last):
  File "/Users/aurelien/workspace/issues/prod-singlecell-simulation-126/test_entity.py", line 65, in <module>
    emodel = next(iterator)
             ^^^^^^^^^^^^^^
  File "/Users/aurelien/softwares/entitysdk/src/entitysdk/result.py", line 24, in __next__
    return next(self._iterable)
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/aurelien/softwares/entitysdk/src/entitysdk/core.py", line 51, in <genexpr>
    serdes.deserialize_entity(json_data, entity_type) for json_data in iterator
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aurelien/softwares/entitysdk/src/entitysdk/serdes.py", line 17, in deserialize_entity
    return entity_type.model_validate(json_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aurelien/venvs/venv-py312/lib/python3.12/site-packages/pydantic/main.py", line 593, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 3 validation errors for EModel
exemplar_morphology.brain_region
  Field required [type=missing, input_value={'id': 'cfe11a2c-ba66-41b...11-04T15:49:32.616674Z'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
exemplar_morphology.species
  Field required [type=missing, input_value={'id': 'cfe11a2c-ba66-41b...11-04T15:49:32.616674Z'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
ion_channel_models
  Field required [type=missing, input_value={'assets': [{'path': 'EMS...1-04T15:49:32.616674Z'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing

Am I doing something wrong here? Or is there missing data in the database that should be fixed?

[eleftherioszisis]

No, you are not doing anything wrong.

The read_one and read_many emodel endpoints in entitycore deviate from each other and they are not exposing the same behavior. For example, when the emodel is fetched by id the ion channel models are expanded but not when querying/searching:
https://github.com/openbraininstitute/entitycore/blob/145fab5aa9684fb6248f53ee325a9d0aaa68be59/app/service/emodel.py#L58

@g-bar is there a reason the two endpoints do not return the same data? It is quite common for users to search for data and expect to get the full schema, the same one as with fetching a single entity.

There is a similar issue with exemplar_morphology, which is not the actual ReconstructionMorphology schema, but a simplified version of it..
https://github.com/openbraininstitute/entitycore/blob/145fab5aa9684fb6248f53ee325a9d0aaa68be59/app/schemas/emodel.py#L22

Making species and brain_region optional will satisfy the simplified schema that is returned without having to create a new schema just for that field.

[g-bar]

The reason is performance, trying to keep the response small, especially regarding the ion channel models, they're are not needed in the list view but cause the endpoint to run much more slowly.

[eleftherioszisis]

Could we make the two endpoints consistent please?
Otherwise emodels won't not usable from the read_many endpoint.