Add new versions of IBC database

[man-shu]

Adding the newer versions of IBC data.

Currently there is an issue where siibra is only listing out one of the subdirectories in a bucket. Due to this src/ibc_api/data/surface_maps_v2.csv is empty, consequently surface can't be fetched as of now. This wasn't the case earlier. Will need to dig into siibra's codebase what changed since.

[sdascoli]

Hi! When running the following (either on main branch or on this branch)

import ibc_api.utils as ibc
db = ibc.get_info(data_type="volume_maps")

I'm getting the following error

[siibra:INFO] Version: 1.0.1-alpha.11
[siibra:WARNING] This is a development release. Use at your own risk.
[siibra:INFO] Please file bugs and issues at https://github.com/FZJ-INM1-BDA/siibra-python.
[siibra:INFO] Clearing siibra cache at /private/home/sdascoli/.cache/siibra.retrieval
Traceback (most recent call last):
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/site-packages/ibc_api/metadata.py", line 130, in fetch_remote_file
    r.raise_for_status()
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/site-packages/requests/models.py", line 1026, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://raw.githubusercontent.com/individual-brain-charting/api/main/src/ibc_api/data/volume_maps_v3.csv

So I guess there is also an issue with volume maps?

[man-shu]

Ah ok! Then this issue would be solved as soon as we merge this PR. So let's do that now and I will handle the surface maps issue after that.

[man-shu]

@sdascoli could you please try once again?

[sdascoli]

Hi! Now I get

SiibraHttpRequestError: Cannot execute http request.
    Status code: 404
    Url: https://data-proxy.ebrains.eu/api/v1/datasets/131add71-e838-4dab-b953-7b7a69

[man-shu]

That is very weird. Just to make sure, you are on main and you get this error when you run the following?

from ibc_api import utils as ibc
db = ibc.get_info(data_type="volume_maps")

Would it be possible for you to get on call sometime? We could meet here: https://rendez-vous.renater.fr/ibc_api_3f9822-03376e-522344

If not, could you maybe try making the API call here: https://data-proxy.ebrains.eu/api/docs#/datasets/get_dataset_v1_datasets__dataset_id__get. It would help if I could see the response you get.

You'd have to click on "authorize" first and login and then enter the following ID: 131add71-e838-4dab-b953-7b7a69ac5d8f.

[sdascoli]

Hi! Now it looks like I aren't even able to connect to ebrains, I don't receive the email of confirmation after logging in via siibra :(
It would be great to have a call! I'm available tomorrow pretty much all day, would a particular time suit you?
Thanks a lot for your help!

[man-shu]

Sorry about all these issues @sdascoli. But it seems this "email issue" is a known problem on EBRAINS infra side: FZJ-INM1-BDA/siibra-python#687 (comment)

So I am afraid there's not much we can do in terms of the API except for tracking the issue on this link.

@ferponcem and I could still meet you tomorrow to understand better if there is a subset of data that you are interested in and also to discuss alternative options for you to access it. Would 3-4pm tomorrow work for you?

[sdascoli]

Thanks, I will be tracking that!
It would be great to meet tomorrow. I'll join the link you sent at 3pm!

[sdascoli]

Hi, I managed to click the validation link, but looks like the API hangs after that.
Here is a traceback after keyboard interrupt:

[siibra:INFO] Clearing siibra cache at /private/home/sdascoli/.cache/siibra.retrieval
Fetched database for volume_maps, version 3.
Found 24 files to download.
... Fetching token and connecting to EBRAINS ...
[siibra:INFO] Setting EBRAINS Knowledge Graph authentication token: eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJLYU01NTRCM2RmMHBIamZYWi1aRl94bUUwMThPS1R0RkNjMjR3aVVqQmFvIn0.eyJleHAiOjE3Njk2NTE4MDksImlhdCI6MTc2OTYwODYwOSwiYXV0aF90aW1lIjoxNzY5NTIwNzcxLCJqdGkiOiJhOWY0ZjAxMy00MTg4LTQ5NjktOGNmZS1mMmZmNzFiZDk3ZjgiLCJpc3MiOiJodHRwczovL2lhbS5lYnJhaW5zLmV1L2F1dGgvcmVhbG1zL2hicCIsImF1ZCI6WyJqdXB5dGVyaHViIiwianVweXRlcmh1Yi1qc2MiLCJ4d2lraSIsImdyb3VwIl0sInN1YiI6IjQzNzMxN2Y0LWU4MjAtNDhmYi05YWRjLWExNjc0MmYwYWFiYiIsInR5cCI6IkJlYXJlciIsImF6cCI6InNpaWJyYSIsInNpZCI6IjYxMjk0ZGVmLTljZjktNDJhZi04YTJhLTE5YWE2ODE2ZDFmZiIsInNjb3BlIjoicHJvZmlsZSByb2xlcyBlbWFpbCBvcGVuaWQgZ3JvdXAiLCJlbWFpbF92ZXJpZmllZCI6dHJ1ZSwibmFtZSI6Ikphcm9kIExldnkiLCJtaXRyZWlkLXN1YiI6IjY5NTIwNjU1NTgzODAyODQiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJqYXJvZDEyMTIiLCJnaXZlbl9uYW1lIjoiSmFyb2QiLCJmYW1pbHlfbmFtZSI6IkxldnkiLCJlbWFpbCI6Imphcm9kLmxldnlAaW5yaWEuZnIifQ.hqY9i5cEiX_Vx6j4iCobHny8WnZFVLFXqEc_ksvBG6gqAn1D3glRbkFyPx2EzMxycpq49rcuuHCyozmIPwSBvHMwlyJOOrubRvReb3Cvgr50YedB05ch-T_VIIYB6jPqP3W0nKr9vn-mH-T6nhF18Ot8Ju9Jd2WvWFZcIwu28NLuS19KGEAXFcANFN1DuzPLTp1lRI4RnyPbuBQBAg8ZXDur4njjHdyaP8FbX6hmDrbXpQvh7jgwS3omIDXpANM8tuL9R_ZkHs976u1FJVagRkKacqIOh6P2Gn-XbUXMurPxHgpnY2JZ_f7Bh0CGRWllmtLJBx3sUdOOvvOu1fgR_A




^CTraceback (most recent call last):
  File "/private/home/sdascoli/brainai/scratch/sdascoli/projects/tribe/analyses/localizer/download_ibc.py", line 5, in <module>
    downloaded_db = ibc.download_data(filtered_db)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/site-packages/ibc_api/utils.py", line 481, in download_data
    connector = _connect_ebrains(data_type)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/site-packages/ibc_api/utils.py", line 94, in _connect_ebrains
    return EbrainsHdgConnector(dataset_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/site-packages/siibra/retrieval/repositories.py", line 508, in __init__
    result = EbrainsRequest(url, DECODERS[".json"]).get()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/site-packages/siibra/retrieval/requests.py", line 544, in get
    return super().get()
           ^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/site-packages/siibra/retrieval/requests.py", line 253, in get
    self._retrieve()
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/site-packages/siibra/retrieval/requests.py", line 213, in _retrieve
    r = http_method(
        ^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/site-packages/urllib3/connection.py", line 516, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/http/client.py", line 1430, in getresponse
    response.begin()
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/http/client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/http/client.py", line 292, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/socket.py", line 720, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/ssl.py", line 1251, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/home/sdascoli/micromamba/envs/brainai/lib/python3.12/ssl.py", line 1103, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

[ferponcem]

Hi @sdascoli thanks for your patience.
There are two parts to this problem:

1. The API hangs after that: actually, it is not really hanging. The connection to Ebrains is very slow because it's done through the EbrainsHdgConnector, and it needs to try multiple times when there's some trouble at siibra side. It is slow but it eventually falls into the "correct" bucket, so if you hold a bit longer you'll see the "starting download" message on your terminal.
2. But then, right after the "starting download" message, it breaks. This is a problem of siibra logging, where it fails to get the right ID of the data, and acts as if it didn't exist at all. I've reported it here.

So unfortunally I haven't heard back but if you have a specific task you need, we can check other ways to share the maps with you in the meantime.

[bthirion]

Hi @sdascoli thanks for your patience. There are two parts to this problem:

1. The API hangs after that: actually, it is not really hanging. The connection to Ebrains is very slow because it's done through the [EbrainsHdgConnector](https://github.com/individual-brain-charting/api/blob/db8ae9eeae29f48186e6bfca5b0493428a2bdd6f/src/ibc_api/utils.py#L65), and it needs to try multiple times when there's some trouble at siibra side. It is slow but it eventually falls into the "correct" bucket, so if you hold a bit longer you'll see the "starting download" message on your terminal.

2. But then, right after the "starting download" message, it breaks. This is a problem of siibra logging, where it fails to get the right ID of the data, and acts as if it didn't exist at all. I've reported it [here](https://github.com/FZJ-INM1-BDA/siibra-python/issues/692#issuecomment-3816372712).

So unfortunally I haven't heard back but if you have a specific task you need, we can check other ways to share the maps with you in the meantime.

Thx for taking care of that.