Retry download if closed connection without response #134
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If server close connection, you dont have a "response code" and fail into a Exception.
Proof's:
`/usr/local/lib/python3.6/dist-packages/librosa/util/decorators.py:9: NumbaDeprecationWarning: An import was requested from a module that has moved location.
Import of 'jit' requested from: 'numba.decorators', please update to use 'numba.core.decorators' or pin to Numba version 0.48.0. This alias will not be present in Numba version 0.50.0.
from numba.decorators import jit as optional_jit
Downloading voxforge-es ...
INFO:audiomate:Found 2244 available archives to download
Download Files: 0%| | 0/2244 [00:00<?, ?it/s]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/ARI-20100216-hds.tgz" with size: 1361655 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AbdielNavarro-20130409-xvp.tgz" with size: 2625640 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AGOs-20100831-jdm.tgz" with size: 1915692 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AGOs-20100831-tmi.tgz" with size: 2009016 B
INFO:audiomate:Finished download
INFO:audiomate:Finished download
INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Adrian-20120203-lit.tgz" with size: 2607824 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AdrianRamirezEscalante-20111202-pgl.tgz" with size: 2594505 B
INFO:audiomate:Finished download
INFO:audiomate:Finished download
Download Files: 0%| | 1/2244 [00:12<7:30:05, 12.04s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Adrin-20120203-uup.tgz" with size: 2230035 B
INFO:audiomate:Finished download
Download Files: 0%|▎ | 5/2244 [00:18<5:31:54, 8.89s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Aguatacuperche-20100823-zvv.tgz" with size: 2132406 B
INFO:audiomate:Finished download
Download Files: 0%|▍ | 7/2244 [00:21<4:11:57, 6.76s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Agus-20120528-sza.tgz" with size: 1533718 B
INFO:audiomate:Finished download
Download Files: 0%|▌ | 8/2244 [00:28<4:06:45, 6.62s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AlexVixgeck-20090827-qms.tgz" with size: 1105585 B
INFO:audiomate:Finished download
Download Files: 0%|▌ | 9/2244 [00:29<3:05:11, 4.97s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AlanNaiman-20121014-tke.tgz" with size: 2065020 B
INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alejandra-20111109-oaf.tgz" with size: 1311472 B
INFO:audiomate:Finished download
Download Files: 0%|▋ | 10/2244 [00:37<3:37:16, 5.84s/it]INFO:audiomate:Finished download
Download Files: 0%|▋ | 11/2244 [00:38<2:45:46, 4.45s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091127-dls.tgz" with size: 2144399 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091127-wrq.tgz" with size: 2084240 B
INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AlexanderRubioCaceres-20090902-set.tgz" with size: 1260802 B
INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091128-fan.tgz" with size: 2444151 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091127-dis.tgz" with size: 2013014 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091128-ozc.tgz" with size: 2397126 B
INFO:audiomate:Finished download
Download Files: 1%|▊ | 13/2244 [00:46<2:39:44, 4.30s/it]INFO:audiomate:Finished download
INFO:audiomate:Finished download
INFO:audiomate:Finished download
Download Files: 1%|▉ | 14/2244 [00:53<3:14:10, 5.22s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20100520-svz.tgz" with size: 2348385 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091128-ygd.tgz" with size: 2095451 B
INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091128-zoj.tgz" with size: 2226773 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20100520-wvo.tgz" with size: 2189445 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20091128-suc.tgz" with size: 2266510 B
INFO:audiomate:Finished download
INFO:audiomate:Finished download
INFO:audiomate:Finished download
Download Files: 1%|█▏ | 19/2244 [01:06<2:43:43, 4.42s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20110915-mnf.tgz" with size: 2442328 B
INFO:audiomate:Finished download
Download Files: 1%|█▍ | 23/2244 [01:08<2:00:56, 3.27s/it]INFO:audiomate:Finished download
Download Files: 1%|█▌ | 24/2244 [01:09<1:38:17, 2.66s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20110915-sff.tgz" with size: 2128985 B
INFO:audiomate:Finished download
Download Files: 1%|█▋ | 25/2244 [01:12<1:44:04, 2.81s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alf_-20110915-wfh.tgz" with size: 2021008 B
INFO:audiomate:Finished download
Download Files: 1%|█▋ | 26/2244 [01:20<2:35:21, 4.20s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alfredo-20090314-you.tgz" with size: 2117498 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alfredo-20090314-bwm.tgz" with size: 1780407 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Alfredo-20090314-nui.tgz" with size: 2000040 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreaBurgos-20120424-pef.tgz" with size: 1929018 B
INFO:audiomate:Finished download
INFO:audiomate:Finished download
Download Files: 1%|█▊ | 27/2244 [01:32<4:01:22, 6.53s/it]INFO:audiomate:Finished download
Download Files: 1%|█▊ | 28/2244 [01:33<2:58:04, 4.82s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130708-inn.tgz" with size: 1721089 B
INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-dtw.tgz" with size: 1736300 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130708-eaz.tgz" with size: 1909391 B
INFO:audiomate:Finished download
Download Files: 1%|█▉ | 30/2244 [01:38<2:35:58, 4.23s/it]INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-bhv.tgz" with size: 1776404 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-iqt.tgz" with size: 1986977 B
INFO:audiomate:Finished download
INFO:audiomate:Finished download
Download Files: 1%|██ | 31/2244 [01:45<3:04:53, 5.01s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-ttq.tgz" with size: 1749826 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-stu.tgz" with size: 1753877 B
INFO:audiomate:Finished download
Download Files: 1%|██▏ | 33/2244 [01:49<2:27:32, 4.00s/it]INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-rat.tgz" with size: 1553412 B
INFO:audiomate:Finished download
INFO:audiomate:Finished download
Download Files: 2%|██▎ | 36/2244 [01:55<2:07:09, 3.46s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130710-twg.tgz" with size: 1853932 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130720-mkx.tgz" with size: 1973814 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130720-gix.tgz" with size: 1687520 B
INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130711-jje.tgz" with size: 1589503 B
INFO:audiomate:Finished download
Download Files: 2%|██▌ | 39/2244 [02:02<1:55:15, 3.14s/it]INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AntonioH-20160531-gwl.tgz" with size: 849178 B
INFO:audiomate:Finished download
INFO:audiomate:Finished download
Download Files: 2%|██▌ | 40/2244 [02:08<2:22:21, 3.88s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Andros-20120408-age.tgz" with size: 2205975 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Argorn-20141126-igm.tgz" with size: 2165540 B
INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Ases-20120213-esa.tgz" with size: 2552878 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AndreyBarrientosCostaRica-20130720-slz.tgz" with size: 1946756 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/AntonioMedina-20121208-kzq.tgz" with size: 2329390 B
INFO:audiomate:Finished download
INFO:audiomate:Finished download
Download Files: 2%|██▊ | 43/2244 [02:22<2:33:13, 4.18s/it]INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Ashrey-20100714-yhd.tgz" with size: 1937711 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/BLs-20151219-kwe.tgz" with size: 2128918 B
INFO:audiomate:Finished download
Download Files: 2%|██▉ | 46/2244 [02:26<2:01:53, 3.33s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/BLs-20151219-tsj.tgz" with size: 1742771 B
INFO:audiomate:Finished download
Download Files: 2%|███▏ | 49/2244 [02:32<1:44:45, 2.86s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/BLs-20151219-vvx.tgz" with size: 2222457 B
INFO:audiomate:Finished download
INFO:audiomate:Finished download
INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Azul4-20131019-ssk.tgz" with size: 2093735 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Campotrigo-20130607-bnz.tgz" with size: 3093233 B
INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/CarlosEspinoAngulo-20140525-vip.tgz" with size: 2302281 B
INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/BLs-20151219-xzd.tgz" with size: 1994095 B
INFO:audiomate:Finished download
INFO:audiomate:Finished download
Download Files: 2%|███▎ | 50/2244 [02:49<4:22:27, 7.18s/it]INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/CarlosMedina-20130524-wqx.tgz" with size: 2357299 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/CarlosFiebigCosmelli-20121128-aud.tgz" with size: 1756883 B
INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Carlos_Daniel-20120325-khj.tgz" with size: 2414791 B
INFO:audiomate:Finished download
INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Condor-20120214-prz.tgz" with size: 1716730 B
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Bernardo-20101210-tiq.tgz" with size: 1849584 B
INFO:audiomate:Finished download
INFO:audiomate:Download file from "http://www.repository.voxforge1.org/downloads/es/Trunk/Audio/Main/16kHz_16bit/Cixert-20150207-ywe.tgz" with size: 1980326 B
INFO:audiomate:Finished download
Download Files: 3%|███▊ | 58/2244 [03:05<1:56:45, 3.20s/it]
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.6/http/client.py", line 1356, in getresponse
response.begin()
File "/usr/lib/python3.6/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.6/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 403, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.6/dist-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.6/http/client.py", line 1356, in getresponse
response.begin()
File "/usr/lib/python3.6/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.6/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/DeepSpeech/deepspeech-polyglot/preprocessing/download_data.py", line 108, in
main()
File "/DeepSpeech/deepspeech-polyglot/preprocessing/download_data.py", line 97, in main
dl.download(os.path.join(args.target_path, "voxforge"))
File "/usr/local/lib/python3.6/dist-packages/audiomate/corpus/io/base.py", line 40, in download
return self._download(target_path)
File "/usr/local/lib/python3.6/dist-packages/audiomate/corpus/io/voxforge.py", line 73, in _download
downloaded = self.download_files(available, temp_folder)
File "/usr/local/lib/python3.6/dist-packages/audiomate/corpus/io/voxforge.py", line 110, in download_files
dl_result = download.download_files(url_to_target, num_threads=self.num_workers)
File "/usr/local/lib/python3.6/dist-packages/audiomate/utils/download.py", line 35, in download_files
description='Download Files'
File "/usr/local/lib/python3.6/dist-packages/audiomate/logutil.py", line 21, in progress
for x in tqdm.tqdm(iterable, desc=description, total=total):
File "/usr/local/lib/python3.6/dist-packages/tqdm/std.py", line 1165, in iter
for obj in iterable:
File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next
raise value
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/local/lib/python3.6/dist-packages/audiomate/utils/download.py", line 43, in _download_file
return download_file(item[0], item[1])
File "/usr/local/lib/python3.6/dist-packages/audiomate/utils/download.py", line 62, in download_file
r = requests.get(url, stream=True)
File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))`