Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Alternative to pyega3 ? #200

Closed
PFRoux opened this issue Jul 24, 2023 · 4 comments
Closed

Alternative to pyega3 ? #200

PFRoux opened this issue Jul 24, 2023 · 4 comments
Labels
bug Something isn't working

Comments

@PFRoux
Copy link

PFRoux commented Jul 24, 2023

Dear all,

I've been trying to use pyega3 on 2 different academic computational clusters and my own laptop (macOS) for more than a week and I've been through all possible issues so far : pyega3 incompatible with the cluster architecture, connection problem, connection broken, md5 check inconsistencies ... and haven't been able to download one single file properly. I am losing my mind !

So far, I am still waiting for the support to answer my inquiries. But the thing is that I need to download the data I am supposed to have access to quickly, since one of the master student I am supervising needs it to finish is thesis (ending at the end of the month ...).

Is there an alternative to this download client ?

Thanks a lot for your help.

Pierre-François
@PFRoux PFRoux added the bug Something isn't working label Jul 24, 2023
@CsabaHalmagyi
Copy link
Contributor

Dear Pierre-Francois,
Thank you for contacting us. I would like to confirm that the EGA development team is investigating the downloading issues. For further assistance, please email the EGA Helpdesk at helpdesk@ega-archive.org, they can provide alternative ways to download the datasets.
Regards,
Csaba

@JianGuoZhou3
Copy link

[2023-07-30 05:06:47 -0400]
[2023-07-30 05:06:47 -0400] pyEGA3 - EGA python client version 5.0.2 (https://github.com/EGA-archive/ega-download-client)
[2023-07-30 05:06:47 -0400] Parts of this software are derived from pyEGA (https://github.com/blachlylab/pyega) by James Blachly
[2023-07-30 05:06:47 -0400] Python version : 3.11.4
[2023-07-30 05:06:47 -0400] OS version : Linux #1 SMP Debian 5.10.179-3 (2023-07-27)
[2023-07-30 05:06:47 -0400] Server URL: https://ega.ebi.ac.uk:8443/v2
[2023-07-30 05:06:47 -0400] Session-Id: 2861361300
[2023-07-30 05:06:48 -0400]
[2023-07-30 05:06:48 -0400] Authentication success for user 'jianguo.zhou@zmu.edu.cn'
[2023-07-30 05:06:50 -0400] File Id: 'EGAF00005292867'(25222818 bytes).
[2023-07-30 05:06:50 -0400] Total space : 1692.88 GiB
[2023-07-30 05:06:50 -0400] Used space : 3.13 GiB
[2023-07-30 05:06:50 -0400] Free space : 1620.93 GiB
[2023-07-30 05:06:50 -0400] Download starting [using 1 connection(s), file size 25222802 and chunk length 104857600]...
[2023-07-30 05:18:56 -0400] HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))
urllib3.exceptions.ResponseError: too many 504 error responses

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
[Previous line repeated 7 more times]
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 931, in urlopen
retries = retries.increment(method, url, response=response, _pool=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 323, in download_file_retry
self.download_file(output_file, num_connections, max_slice_size)
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 159, in download_file
for part_file_name in executor.map(self.download_file_slice_, params):
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self.exception
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 189, in download_file_slice
return self.download_file_slice(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 221, in download_file_slice
with self.data_client.get_stream(path, extra_headers) as r:
File "/root/miniconda3/envs/pyega3/lib/python3.11/contextlib.py", line 137, in enter
return next(self.gen)
^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_client.py", line 76, in get_stream
with self.session.get(url, headers=headers, stream=True, timeout=request_timeout_in_sec) as r:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/adapters.py", line 510, in send
raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))
[2023-07-30 05:19:56 -0400] retry attempt 1
[2023-07-30 05:19:56 -0400] Download starting [using 1 connection(s), file size 25222802 and chunk length 104857600]...
[2023-07-30 05:32:01 -0400] HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))
urllib3.exceptions.ResponseError: too many 504 error responses

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
[Previous line repeated 7 more times]
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 931, in urlopen
retries = retries.increment(method, url, response=response, _pool=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 323, in download_file_retry
self.download_file(output_file, num_connections, max_slice_size)
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 159, in download_file
for part_file_name in executor.map(self.download_file_slice_, params):
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self.exception
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 189, in download_file_slice
return self.download_file_slice(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 221, in download_file_slice
with self.data_client.get_stream(path, extra_headers) as r:
File "/root/miniconda3/envs/pyega3/lib/python3.11/contextlib.py", line 137, in enter
return next(self.gen)
^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_client.py", line 76, in get_stream
with self.session.get(url, headers=headers, stream=True, timeout=request_timeout_in_sec) as r:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/adapters.py", line 510, in send
raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))
[2023-07-30 05:33:01 -0400] retry attempt 2
[2023-07-30 05:33:01 -0400] Download starting [using 1 connection(s), file size 25222802 and chunk length 104857600]...
[2023-07-30 05:45:07 -0400] HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))
urllib3.exceptions.ResponseError: too many 504 error responses

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
[Previous line repeated 7 more times]
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 931, in urlopen
retries = retries.increment(method, url, response=response, _pool=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 323, in download_file_retry
self.download_file(output_file, num_connections, max_slice_size)
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 159, in download_file
for part_file_name in executor.map(self.download_file_slice_, params):
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self.exception
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 189, in download_file_slice
return self.download_file_slice(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 221, in download_file_slice
with self.data_client.get_stream(path, extra_headers) as r:
File "/root/miniconda3/envs/pyega3/lib/python3.11/contextlib.py", line 137, in enter
return next(self.gen)
^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_client.py", line 76, in get_stream
with self.session.get(url, headers=headers, stream=True, timeout=request_timeout_in_sec) as r:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/adapters.py", line 510, in send
raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))
[2023-07-30 05:46:07 -0400] retry attempt 3
[2023-07-30 05:46:07 -0400] Download starting [using 1 connection(s), file size 25222802 and chunk length 104857600]...
[2023-07-30 05:58:12 -0400] HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))
urllib3.exceptions.ResponseError: too many 504 error responses

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 941, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
[Previous line repeated 7 more times]
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/connectionpool.py", line 931, in urlopen
retries = retries.increment(method, url, response=response, _pool=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 323, in download_file_retry
self.download_file(output_file, num_connections, max_slice_size)
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 159, in download_file
for part_file_name in executor.map(self.download_file_slice_, params):
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self.exception
File "/root/miniconda3/envs/pyega3/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 189, in download_file_slice
return self.download_file_slice(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_file.py", line 221, in download_file_slice
with self.data_client.get_stream(path, extra_headers) as r:
File "/root/miniconda3/envs/pyega3/lib/python3.11/contextlib.py", line 137, in enter
return next(self.gen)
^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/pyega3/libs/data_client.py", line 76, in get_stream
with self.session.get(url, headers=headers, stream=True, timeout=request_timeout_in_sec) as r:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/pyega3/lib/python3.11/site-packages/requests/adapters.py", line 510, in send
raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))
[2023-07-30 05:59:12 -0400] retry attempt 4
[2023-07-30 05:59:12 -0400] Download starting [using 1 connection(s), file size 25222802 and chunk length 104857600]...
[2023-07-30 06:06:37 -0400]
[2023-07-30 06:06:37 -0400] pyEGA3 - EGA python client version 5.0.2 (https://github.com/EGA-archive/ega-download-client)
[2023-07-30 06:06:37 -0400] Parts of this software are derived from pyEGA (https://github.com/blachlylab/pyega) by James Blachly
[2023-07-30 06:06:37 -0400] Python version : 3.11.4
[2023-07-30 06:06:37 -0400] OS version : Linux #1 SMP Debian 5.10.179-3 (2023-07-27)
[2023-07-30 06:06:37 -0400] Server URL: https://ega.ebi.ac.uk:8443/v2
[2023-07-30 06:06:37 -0400] Session-Id: 1480591647
[2023-07-30 06:06:39 -0400]
[2023-07-30 06:06:39 -0400] Authentication success for user 'jianguo.zhou@zmu.edu.cn'
[2023-07-30 06:06:41 -0400] File Id: 'EGAF00005292867'(25222818 bytes).
[2023-07-30 06:06:41 -0400] Total space : 1692.88 GiB
[2023-07-30 06:06:41 -0400] Used space : 3.13 GiB
[2023-07-30 06:06:41 -0400] Free space : 1620.93 GiB
[2023-07-30 06:06:41 -0400] Download starting [using 1 connection(s), file size 25222802 and chunk length 104857600]...
[2023-07-30 06:11:18 -0400] HTTPSConnectionPool(host='ega.ebi.ac.uk', port=8443): Max retries exceeded with url: /v2/files/EGAF00005292867?destinationFormat=plain (Caused by ResponseError('too many 504 error responses'))
urllib3.exceptions.ResponseError: too many 504 error responses

The above exception was the direct cause of the following exception:

@JianGuoZhou3
Copy link

JianGuoZhou3 commented Jul 31, 2023 •
edited
Loading

I still can't download the EGA datasets...
I can confirm my internet is fine.
Jian-Guo
image

@CsabaHalmagyi

@CsabaHalmagyi
Copy link
Contributor

Dear @jianguozhouzunyimedicaluniversity
Thank you for submitting the error log. We looked into the report and identified the root cause issue. Could you please retry downloading the dataset(s)? Should you encounter any issues please contact our helpdesk team at helpdesk@ega-archive.org, they can provide alternative ways to download the datasets.
Regards,
Csaba

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@CsabaHalmagyi @PFRoux @JianGuoZhou3