Moving away from EbrainsHdgConnector, to ebrains-drive

[ferponcem]

This PR is for the transition from siibra's EbrainsHdgConnector to ebrains-drive to access IBC files from the command line.
This first commit updates list of files for volume maps repo, version 3.
There was a problem with the listing all the files available in Ebrains for each collection, we were experiencing a data size limit.
The script modified now uses ebrains_drive's BucketApiClient to get available data in a given collection.

[ferponcem]

Commit a66ca2b has the necessary changes to use the BucketApiClient from ebrains-drive to retrieve the data !

For testing, first make sure to install
pip install ebrains_drive

Then try:

import ibc_api.utils as ibc
# Test with version 2, as version 3 is the csv file that is incomplete, addressed also in this PR
db = ibc.get_info(data_type="volume_maps", save_to="/test_ebrains_drive", version=2)

filtered_db = ibc.filter_data(db, subject_list=["08"], task_list=["Lec1"])

# Manually adjust to the new naming convention (also to be solved by the new csv file on this PR)
filtered_db["path"] = filtered_db["path"].str.replace(
    "space-MNI152NLin2009cAsym_desc-preproc", "space-MNI305", regex=False
)

downloaded_db = ibc.download_data(filtered_db,  save_to="/test_ebrains_drive")

If you encounter issues with the request access because it does not load on time, simply run
downloaded_db = ibc.download_data(filtered_db, save_to="/test_ebrains_drive")

And make sure to be in editable mode

git checkout moving_to_ebrains_drive
pip install -e .

[bthirion]

Thx !
However, when i run the example, I stillget an issue:

Fetched database for volume_maps, version 2.
Found 36 files for subjects ['08'] and tasks ['Lec1'].
Found 36 files to download.
... Fetching token and connecting to EBRAINS ...
***
To continue, please go to https://iam.ebrains.eu/auth/realms/hbp/device?user_code=KUSU-LJOR
***
ebrains token successfully set.
---------------------------------------------------------------------------
ClientHttpError                           Traceback (most recent call last)
~/.local/lib/python3.10/site-packages/ebrains_drive/buckets.py in get_dataset(self, dataset_id, public, request_access)
     24             try:
---> 25                 resp = self.client.get(f"/v1/datasets/{dataset_id}/stat")
     26                 return Bucket.from_json(

~/.local/lib/python3.10/site-packages/ebrains_drive/client.py in get(self, *args, **kwargs)
     66     def get(self, *args, **kwargs):
---> 67         return self.send_request("GET", *args, **kwargs)
     68 

~/.local/lib/python3.10/site-packages/ebrains_drive/client.py in send_request(self, method, url, *args, **kwargs)
    277 
--> 278         return super().send_request(method, url, *args, **kwargs)
    279 

~/.local/lib/python3.10/site-packages/ebrains_drive/client.py in send_request(self, method, url, *args, **kwargs)
    102             msg = f"Expected {expected}, but got {resp.status_code}"
--> 103             raise ClientHttpError(resp.status_code, msg)
    104 

ClientHttpError: ClientHttpError[401: Expected (200,), but got 401]

During handling of the above exception, another exception occurred:

Unauthorized                              Traceback (most recent call last)
<ipython-input-6-b90c41e0683a> in <module>
     10 )
     11 
---> 12 downloaded_db = ibc.download_data(filtered_db,  save_to="test_ebrains_drive")

~/mygit/ibc/api/src/ibc_api/utils.py in download_data(db, save_to)
    532     # connect to ebrains dataset
    533     print("... Fetching token and connecting to EBRAINS ...")
--> 534     connector = _connect_ebrains(data_type)
    535 
    536     # set the save directory

~/mygit/ibc/api/src/ibc_api/utils.py in _connect_ebrains(data_type, metadata, version)
     96     # use the token to create a client and get a data bucket
     97     client = BucketApiClient(token=EbrainsRequest._KG_API_TOKEN)
---> 98     bucket = client.buckets.get_dataset(dataset_id)
     99 
    100     return bucket

~/.local/lib/python3.10/site-packages/ebrains_drive/buckets.py in get_dataset(self, dataset_id, public, request_access)
     32 
     33                 if not request_access:
---> 34                     raise Unauthorized(
     35                         f"You do not have access to this dataset. If this is a private dataset, try to set request_access flag to true. We can start the procedure of requesting access for you."
     36                     )

Unauthorized: Unauthorized. This could be a result of either incorrect path or insufficient privilege. You do not have access to this dataset. If this is a private dataset, try to set request_access flag to true. We can start the procedure of requesting access for you.

[ferponcem]

Oh ! I am assuming this happens while you're clicking on the link for authentication. If that's the case, could you try running this line again? downloaded_db = ibc.download_data(filtered_db, save_to="/test_ebrains_drive")
Sometimes while clicking it times out but when retrying it starts from where it was.

In the meantime I am also running it again to check !

[bthirion]

Unfortunately, I cannot make it work...

[ferponcem]

Found the issue (hopefully): IBC is a private repo. Setting request_access=True solves the problem.

Could you try running this again ? Now when you run
downloaded_db = ibc.download_data(filtered_db, save_to="/test_ebrains_drive")
it will ask you to click on the link first:

***
To continue, please go to https://iam.ebrains.eu/auth/realms/hbp/device?user_code=APCA-YCSI
***

and after you click you'll see a message like:

Request sent. Please check the mail box associated with the token.

Then you should go to your email and click on the link, you'll see something like this:

By clicking this *link*, you confirm that you have... .

Connection might time out in your terminal while you're waiting for the email and clicking, it's okay, just run from downloaded_db = ibc.download_data(filtered_db, save_to="/test_ebrains_drive") again.

Let me know how it works, and thanks for your help !

[bthirion]

Hm. Doesn't work.
To make sure I do things right, ca we take a small slot tomorrow to look at that tomorrow. e.g. 10:30 ?
I'll be at NSP, but we can do it remotely if need be.