Parallel download by ferponcem · Pull Request #11 · individual-brain-charting/api

ferponcem · 2024-03-15T19:19:18Z

Make the IBC data fetching parallel, this is not ready yet!

ferponcem · 2024-07-18T15:10:28Z

So far I have:

added some joblib to handle single files download
adjusted tqdm to get a better estimation of progress
modified some functions to work accordingly
added joblib to dependecies
added to ebrains-token to the gitignore list

Some issues:

it is still very slow !

expand the ibc.filter_data options to filter according to data extension to, with this testing will be faster
test options for optimization
test in a different environment to see if some issues are related to my working env or if new issues arise

bthirion

Thx ! LMK when it's ready.

bthirion · 2024-07-18T20:57:23Z

+#%%
+import pdb
+import sys
+sys.path.append('/home/fer/HBP_IBC/api/src/ibc_api')


You should not have to do that ?

@ferponcem, you should first install the package via pip as mentioned in the README and then run the example.py.

You wouldn't have to do this then.

bthirion · 2024-07-18T20:57:55Z

    return remote_file_names, local_file_names


+# def _update_local_db(db_file, files_data):


remove deprected lines

bthirion · 2024-07-18T20:58:06Z

        updated local database
    """

+    # file_names = [file_data[0] for file_data in files_data]


Co-authored-by: Himanshu Aggarwal <himanshuaggarwal1997@gmail.com>

man-shu · 2024-07-31T11:59:11Z

@ferponcem, I just realized I wasn't watching the repo, so I did not get notified about this.
Let me know if you need a review or some help with this :)

…by_download fix: curl -> request for metadata

bthirion · 2024-08-07T09:40:43Z

Hello, what is the status for this PR ?

…ing/api into parallel_download

ferponcem · 2024-08-08T15:47:24Z

It is good for a review now, I'm preparing a small example

man-shu

I couldn't test it yet, maybe update the example.py appropriately?

man-shu · 2024-08-09T11:53:03Z

+#%%
+import pdb
+import sys
+sys.path.append('/home/fer/HBP_IBC/api/src/ibc_api')


@ferponcem, you should first install the package via pip as mentioned in the README and then run the example.py.

You wouldn't have to do this then.

Co-authored-by: Himanshu Aggarwal <himanshuaggarwal1997@gmail.com>

ferponcem · 2024-08-23T14:44:48Z

Thank you both for the feedback! I've corrected the example in examples/example.py 😸
For testing just make sure you checkout to this branch and run the install command wherever you've cloned the repo:

cd ~/api
git checkout parallel_download
pip install -e .
python examples/example.py

bthirion · 2024-08-23T19:52:48Z

Thx !
When running it, I hit an issue;

Found 36 files to download.
... Fetching token and connecting to EBRAINS ...
***
To continue, please go to https://iam.ebrains.eu/auth/realms/hbp/device?user_code=GNTL-FWKT
***
[siibra:ERROR] exceeded max attempts: 12, aborting...
Traceback (most recent call last):
  File "/home/bertrandthirion/mygit/ibc/api/examples/example.py", line 13, in <module>
    downloaded_db = ibc.download_data(filtered_db, n_jobs=2)
  File "/home/bertrandthirion/mygit/ibc/api/src/ibc_api/utils.py", line 444, in download_data
    connector = _connect_ebrains(data_type)
  File "/home/bertrandthirion/mygit/ibc/api/src/ibc_api/utils.py", line 92, in _connect_ebrains
    token_file = _authenticate()
  File "/home/bertrandthirion/mygit/ibc/api/src/ibc_api/utils.py", line 55, in _authenticate
    siibra.fetch_ebrains_token()
  File "/home/bertrandthirion/.local/lib/python3.10/site-packages/siibra/retrieval/requests.py", line 281, in fetch_token
    cls.device_flow(**kwargs)
  File "/home/bertrandthirion/.local/lib/python3.10/site-packages/siibra/retrieval/requests.py", line 348, in device_flow
    raise EbrainsAuthenticationError(message)
siibra.retrieval.exceptions.EbrainsAuthenticationError: exceeded max attempts: 12, aborting...

man-shu · 2024-08-26T09:32:27Z

Thx ! When running it, I hit an issue;
[siibra:ERROR] exceeded max attempts: 12, aborting...

This happens when it takes you a while to log in and verify your EBRAINS account (it could be due to a forgotten password etc.). You can simply log into your EBRAINS account beforehand and run the example.py again, click on the link, and verify quickly.

The example runs fine for me! Thanks @ferponcem!

man-shu

LGTM!

bthirion · 2024-08-27T15:06:17Z

I can't get it to work. I would need to do it with you.
Can be merged meanwhile I think.

bthirion

Works on my side.

ferponcem · 2024-09-03T11:44:29Z

Thank you both for the feedback ! Merging now 🚀

ferponcem and others added 6 commits March 15, 2024 20:16

first changes to parallelize download

cc5acf1

going paralleeeeeeel

4bc3a13

minor changes

251c313

going paralleeeeeeeeel (better I hope)

3557bf9

untrack the token

1362c12

adding joblib to dependencies

855a49f

bthirion reviewed Jul 18, 2024

View reviewed changes

ferponcem and others added 3 commits July 30, 2024 17:27

curl -> request for metadata

a6359d6

formatting of request function

35d6b7d

raising error instead of just printing

5c15a44

Co-authored-by: Himanshu Aggarwal <himanshuaggarwal1997@gmail.com>

Merge pull request #13 from individual-brain-charting/replacing_curl_…

124ff49

…by_download fix: curl -> request for metadata

ferponcem and others added 11 commits August 7, 2024 18:03

removing tqdm dependency and changes after testing

dae7251

first changes to parallelize download

871efa8

going paralleeeeeeel

a0f46fc

minor changes

c6073b6

going paralleeeeeeeeel (better I hope)

5e8a6b8

untrack the token

dfe61eb

adding joblib to dependencies

b9784d0

removing tqdm dependency and changes after testing

29836c5

Merge branch 'parallel_download' of github.com:individual-brain-chart…

ed07ef6

…ing/api into parallel_download

managing download dataframe differently

030c22b

small format

6336a63

ferponcem marked this pull request as ready for review August 8, 2024 15:47

man-shu reviewed Aug 9, 2024

View reviewed changes

ferponcem added 2 commits August 23, 2024 16:14

adjustments to example and cmd feedback

a09073c

removing unused imports

b5aecee

ferponcem and others added 6 commits August 23, 2024 16:25

raising error instead of just printing

f64bd29

Co-authored-by: Himanshu Aggarwal <himanshuaggarwal1997@gmail.com>

num_jobs -> n_jobs

5f270f7

Co-authored-by: Himanshu Aggarwal <himanshuaggarwal1997@gmail.com>

num_jobs -> n_jobs 2

f1cdf1d

Co-authored-by: Himanshu Aggarwal <himanshuaggarwal1997@gmail.com>

num_jobs -> n_jobs 3

1406ea0

Co-authored-by: Himanshu Aggarwal <himanshuaggarwal1997@gmail.com>

expand n_jobs description

4266486

Co-authored-by: Himanshu Aggarwal <himanshuaggarwal1997@gmail.com>

num_jobs -> n_jobs in example

2303012

man-shu approved these changes Aug 26, 2024

View reviewed changes

bthirion approved these changes Sep 3, 2024

View reviewed changes

ferponcem merged commit 6e5cc23 into main Sep 3, 2024

		return remote_file_names, local_file_names


		# def _update_local_db(db_file, files_data):

Conversation

ferponcem commented Mar 15, 2024

Uh oh!

ferponcem commented Jul 18, 2024

Uh oh!

bthirion left a comment

Choose a reason for hiding this comment

Uh oh!

bthirion Jul 18, 2024

Choose a reason for hiding this comment

Uh oh!

man-shu Aug 9, 2024

Choose a reason for hiding this comment

Uh oh!

bthirion Jul 18, 2024

Choose a reason for hiding this comment

Uh oh!

bthirion Jul 18, 2024

Choose a reason for hiding this comment

Uh oh!

man-shu commented Jul 31, 2024

Uh oh!

bthirion commented Aug 7, 2024

Uh oh!

ferponcem commented Aug 8, 2024

Uh oh!

man-shu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

man-shu Aug 9, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ferponcem commented Aug 23, 2024

Uh oh!

bthirion commented Aug 23, 2024

Uh oh!

man-shu commented Aug 26, 2024

Uh oh!

man-shu left a comment

Choose a reason for hiding this comment

Uh oh!

bthirion commented Aug 27, 2024

Uh oh!

bthirion left a comment

Choose a reason for hiding this comment

Uh oh!

ferponcem commented Sep 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants