New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Video downloads break when connection is unstable #5528

Closed
jiangjianshan opened this Issue Oct 20, 2017 · 31 comments

Comments

Projects
None yet
3 participants
@jiangjianshan

jiangjianshan commented Oct 20, 2017

Hello,

My system's information is as below:

  • Operating system: Windows 7 Professional SP1 x64

  • Version: 0.17.3

  • Browser: Chrome

  • Python: 2.7.14

    Recently I have my laptop can download the video via ka-lite. Every the progress bar will be block at the position very close to the end. And then the progress bar disappear.

image

The video can't download anymore.
@mrpau-eugene

This comment has been minimized.

Show comment
Hide comment
@mrpau-eugene

mrpau-eugene Oct 21, 2017

Member

Hi @jiangjianshan can you post the logs here? You can do this by clicking the Show KA-Lite Logs from the system tray.

Member

mrpau-eugene commented Oct 21, 2017

Hi @jiangjianshan can you post the logs here? You can do this by clicking the Show KA-Lite Logs from the system tray.

@jiangjianshan

This comment has been minimized.

Show comment
Hide comment
@jiangjianshan

jiangjianshan Oct 21, 2017

Hello @mrpau-eugene , I have found the log file inside the folder $HOME.kalite\logs. In order to upload the log files to here, the file extension have been changed from .log to .txt.

django-2017-10-20.txt
django-2017-10-19.txt
django-2017-10-18.txt
django-2017-10-18.txt

jiangjianshan commented Oct 21, 2017

Hello @mrpau-eugene , I have found the log file inside the folder $HOME.kalite\logs. In order to upload the log files to here, the file extension have been changed from .log to .txt.

django-2017-10-20.txt
django-2017-10-19.txt
django-2017-10-18.txt
django-2017-10-18.txt

@mrpau-eugene

This comment has been minimized.

Show comment
Hide comment
@mrpau-eugene

mrpau-eugene Oct 22, 2017

Member

@jiangjianshan Great! Thanks a lot for the logs. However, I wasn't able to reproduce this issue..

May I also know what kind of connection are you using? Also, do you have another PC/Laptop which you can try to test it again?

Member

mrpau-eugene commented Oct 22, 2017

@jiangjianshan Great! Thanks a lot for the logs. However, I wasn't able to reproduce this issue..

May I also know what kind of connection are you using? Also, do you have another PC/Laptop which you can try to test it again?

@benjaoming

This comment has been minimized.

Show comment
Hide comment
@benjaoming

benjaoming Oct 23, 2017

Member

There are a lot of unhandled connection errors it seems:

Traceback (most recent call last):
  File "D:\Python27\lib\site-packages\kalite\packages\bundled\fle_utils\videos.py", line 48, in download_video
    response = download_file(thumb_url, thumb_filepath, callback_percent_proxy(callback, start_percent=95, end_percent=100))
  File "D:\Python27\lib\site-packages\kalite\packages\bundled\fle_utils\internet\download.py", line 64, in download_file
    headers={"user-agent": user_agent()}
  File "D:\Python27\lib\site-packages\kalite\packages\dist\requests\api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "D:\Python27\lib\site-packages\kalite\packages\dist\requests\api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "D:\Python27\lib\site-packages\kalite\packages\dist\requests\sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "D:\Python27\lib\site-packages\kalite\packages\dist\requests\sessions.py", line 617, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "D:\Python27\lib\site-packages\kalite\packages\dist\requests\sessions.py", line 177, in resolve_redirects
    **adapter_kwargs
  File "D:\Python27\lib\site-packages\kalite\packages\dist\requests\sessions.py", line 596, in send
    r = adapter.send(request, **kwargs)
  File "D:\Python27\lib\site-packages\kalite\packages\dist\requests\adapters.py", line 487, in send
    raise ConnectionError(e, request=request)
Member

benjaoming commented Oct 23, 2017

There are a lot of unhandled connection errors it seems:

Traceback (most recent call last):
  File "D:\Python27\lib\site-packages\kalite\packages\bundled\fle_utils\videos.py", line 48, in download_video
    response = download_file(thumb_url, thumb_filepath, callback_percent_proxy(callback, start_percent=95, end_percent=100))
  File "D:\Python27\lib\site-packages\kalite\packages\bundled\fle_utils\internet\download.py", line 64, in download_file
    headers={"user-agent": user_agent()}
  File "D:\Python27\lib\site-packages\kalite\packages\dist\requests\api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "D:\Python27\lib\site-packages\kalite\packages\dist\requests\api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "D:\Python27\lib\site-packages\kalite\packages\dist\requests\sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "D:\Python27\lib\site-packages\kalite\packages\dist\requests\sessions.py", line 617, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "D:\Python27\lib\site-packages\kalite\packages\dist\requests\sessions.py", line 177, in resolve_redirects
    **adapter_kwargs
  File "D:\Python27\lib\site-packages\kalite\packages\dist\requests\sessions.py", line 596, in send
    r = adapter.send(request, **kwargs)
  File "D:\Python27\lib\site-packages\kalite\packages\dist\requests\adapters.py", line 487, in send
    raise ConnectionError(e, request=request)

@benjaoming benjaoming changed the title from Video can't be download at the version 0.17.3 to Unhandled connection errors consistently breaking video download Oct 23, 2017

@benjaoming benjaoming added this to the 0.17.4 milestone Oct 23, 2017

@benjaoming benjaoming added the bug label Oct 23, 2017

@benjaoming benjaoming referenced this issue Oct 23, 2017

Merged

0.5 into 0.6 #2494

3 of 3 tasks complete
@jiangjianshan

This comment has been minimized.

Show comment
Hide comment
@jiangjianshan

jiangjianshan Oct 24, 2017

@mrpau-eugene , My laptop was using wireless or LAN at home which it band is 20Mbit/s. Also, I have to change my hosts file in order to access google at China.
hosts.zip

@benjaoming , I have meet this issue when install and upgrade to 0.17.2, but even I latterly update to 0.17.3 but still have the same issue. So I have to wait the bug fix on the 0.17.4 release?

jiangjianshan commented Oct 24, 2017

@mrpau-eugene , My laptop was using wireless or LAN at home which it band is 20Mbit/s. Also, I have to change my hosts file in order to access google at China.
hosts.zip

@benjaoming , I have meet this issue when install and upgrade to 0.17.2, but even I latterly update to 0.17.3 but still have the same issue. So I have to wait the bug fix on the 0.17.4 release?

@mrpau-eugene

This comment has been minimized.

Show comment
Hide comment
@mrpau-eugene

mrpau-eugene Oct 24, 2017

Member

@jiangjianshan I don't think it's a bug but rather a problem with China's ISP blocking some websites. Are you able to watch YouTube videos without any problems using the hosts file?

I'm not quite sure if we can do anything about this. Maybe @benjaoming has some ideas?

Member

mrpau-eugene commented Oct 24, 2017

@jiangjianshan I don't think it's a bug but rather a problem with China's ISP blocking some websites. Are you able to watch YouTube videos without any problems using the hosts file?

I'm not quite sure if we can do anything about this. Maybe @benjaoming has some ideas?

@jiangjianshan

This comment has been minimized.

Show comment
Hide comment
@jiangjianshan

jiangjianshan Oct 25, 2017

@mrpau-eugene ,I can't watch YouTube videos even changed the hosts file. But I remember for version 0.17.1, even I can't watch the YouTube video but still can download the video by ka-lite.

jiangjianshan commented Oct 25, 2017

@mrpau-eugene ,I can't watch YouTube videos even changed the hosts file. But I remember for version 0.17.1, even I can't watch the YouTube video but still can download the video by ka-lite.

@mrpau-eugene

This comment has been minimized.

Show comment
Hide comment
@mrpau-eugene

mrpau-eugene Oct 25, 2017

Member

@jiangjianshan I can't seem to find any significant changes that affects downloading videos on the 0.17.1 based on the Release Notes.

But just to make sure, can you try installing 0.17.1 again to see if you still get the same errors?

You can download the 0.17.1 installer here.

Member

mrpau-eugene commented Oct 25, 2017

@jiangjianshan I can't seem to find any significant changes that affects downloading videos on the 0.17.1 based on the Release Notes.

But just to make sure, can you try installing 0.17.1 again to see if you still get the same errors?

You can download the 0.17.1 installer here.

@benjaoming

This comment has been minimized.

Show comment
Hide comment
@benjaoming

benjaoming Oct 25, 2017

Member

I looked through 2 of the log files and all errors are related to download_file(thumb_url, ...), so they are about downloading thumbnails. What is weird is:

  1. Thumbnail downloads happen after downloading videos, but in the same function. So if the video download would fail, it should not proceed to thumbnails. This indicates that video downloads work, but it's the thumbnail part that's broken.
  2. Both videos and thumbnails are hosted at http://s3.amazonaws.com/KA-youtube-converted/

This supports the description by @jiangjianshan :

Recently I have my laptop can download the video via ka-lite. Every the progress bar will be block at the position very close to the end. And then the progress bar disappear.

@mrpau-eugene could you supply us with a URL and a video that @jiangjianshan can try out?

Member

benjaoming commented Oct 25, 2017

I looked through 2 of the log files and all errors are related to download_file(thumb_url, ...), so they are about downloading thumbnails. What is weird is:

  1. Thumbnail downloads happen after downloading videos, but in the same function. So if the video download would fail, it should not proceed to thumbnails. This indicates that video downloads work, but it's the thumbnail part that's broken.
  2. Both videos and thumbnails are hosted at http://s3.amazonaws.com/KA-youtube-converted/

This supports the description by @jiangjianshan :

Recently I have my laptop can download the video via ka-lite. Every the progress bar will be block at the position very close to the end. And then the progress bar disappear.

@mrpau-eugene could you supply us with a URL and a video that @jiangjianshan can try out?

@jiangjianshan

This comment has been minimized.

Show comment
Hide comment
@jiangjianshan

jiangjianshan Oct 26, 2017

@mrpau-eugene , maybe you can send the URL link of PXlvKtpvUEk.png, which one I have successfully download it by KA Lite at latest time and I can't search its URL link by www.baidu.com. Currently I can't use google because of the hosts file not work for it now.

jiangjianshan commented Oct 26, 2017

@mrpau-eugene , maybe you can send the URL link of PXlvKtpvUEk.png, which one I have successfully download it by KA Lite at latest time and I can't search its URL link by www.baidu.com. Currently I can't use google because of the hosts file not work for it now.

@mrpau-eugene

This comment has been minimized.

Show comment
Hide comment
@mrpau-eugene

mrpau-eugene Oct 26, 2017

Member

@jiangjianshan Here is the link for it http://s3.amazonaws.com/KA-youtube-converted/PXlvKtpvUEk.mp4/PXlvKtpvUEk.mp4

Can you try it out and see if you can watch the video?

Member

mrpau-eugene commented Oct 26, 2017

@jiangjianshan Here is the link for it http://s3.amazonaws.com/KA-youtube-converted/PXlvKtpvUEk.mp4/PXlvKtpvUEk.mp4

Can you try it out and see if you can watch the video?

@jiangjianshan

This comment has been minimized.

Show comment
Hide comment
@jiangjianshan

jiangjianshan Oct 26, 2017

@mrpau-eugene , after I copy the link to URL address bar in Chrome, the video can be play.
But I think the issue I meet is I can't download the .png file but not .mp4 file via KA Lite. Could you send me the link of PXlvKtpvUEk.png but not PXlvKtpvUEk.mp4?

jiangjianshan commented Oct 26, 2017

@mrpau-eugene , after I copy the link to URL address bar in Chrome, the video can be play.
But I think the issue I meet is I can't download the .png file but not .mp4 file via KA Lite. Could you send me the link of PXlvKtpvUEk.png but not PXlvKtpvUEk.mp4?

@mrpau-eugene

This comment has been minimized.

Show comment
Hide comment
@mrpau-eugene

mrpau-eugene Oct 26, 2017

Member

@jiangjianshan sorry about that.. here is the link to the thumbnail http://s3.amazonaws.com/KA-youtube-converted/PXlvKtpvUEk.mp4/PXlvKtpvUEk.png

It seems like the videos you downloaded such as this one: 1tSrRYU6LKM and PZM6acXvsoQ does not have a video thumbnail on s3.

And it seems like it is trying to download the thumbnail from YouTube instead. I think that's the reason why it's displaying errors? Am I right @benjaoming ?

Max retries exceeded with url: /vi/PZM6acXvsoQ/mqdefault.jpg (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x000000000B06BDD8>: Failed to establish a new connection: [Errno 10060] ',))

Try to access https://img.youtube.com/vi/PZM6acXvsoQ/mqdefault.jpg and I think you will have problems while accessing it..

Member

mrpau-eugene commented Oct 26, 2017

@jiangjianshan sorry about that.. here is the link to the thumbnail http://s3.amazonaws.com/KA-youtube-converted/PXlvKtpvUEk.mp4/PXlvKtpvUEk.png

It seems like the videos you downloaded such as this one: 1tSrRYU6LKM and PZM6acXvsoQ does not have a video thumbnail on s3.

And it seems like it is trying to download the thumbnail from YouTube instead. I think that's the reason why it's displaying errors? Am I right @benjaoming ?

Max retries exceeded with url: /vi/PZM6acXvsoQ/mqdefault.jpg (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x000000000B06BDD8>: Failed to establish a new connection: [Errno 10060] ',))

Try to access https://img.youtube.com/vi/PZM6acXvsoQ/mqdefault.jpg and I think you will have problems while accessing it..

@jiangjianshan

This comment has been minimized.

Show comment
Hide comment
@jiangjianshan

jiangjianshan Oct 26, 2017

@mrpau-eugene , I can download PXlvKtpvUEk.png for the link you given.
image

But for the one from the URL link https://img.youtube.com/vi/PZM6acXvsoQ/mqdefault.jpg, as you said, I can't download it.

jiangjianshan commented Oct 26, 2017

@mrpau-eugene , I can download PXlvKtpvUEk.png for the link you given.
image

But for the one from the URL link https://img.youtube.com/vi/PZM6acXvsoQ/mqdefault.jpg, as you said, I can't download it.

@mrpau-eugene

This comment has been minimized.

Show comment
Hide comment
@mrpau-eugene

mrpau-eugene Oct 26, 2017

Member

@jiangjianshan Thanks for the info! I would just like to confirm if you are able to download these 2 videos:

screen shot 2017-10-26 at 2 11 52 pm

They are both available on Amazon S3. If you are able to download it, then I think the case has been resolved (?) and someone just need to update the server to include the missing videos.

Member

mrpau-eugene commented Oct 26, 2017

@jiangjianshan Thanks for the info! I would just like to confirm if you are able to download these 2 videos:

screen shot 2017-10-26 at 2 11 52 pm

They are both available on Amazon S3. If you are able to download it, then I think the case has been resolved (?) and someone just need to update the server to include the missing videos.

@benjaoming benjaoming changed the title from Unhandled connection errors consistently breaking video download to Video downloads break: Handle new set of locations for thumbnail URLs Oct 26, 2017

@benjaoming

This comment has been minimized.

Show comment
Hide comment
@benjaoming

benjaoming Oct 26, 2017

Member

Received another report by email..

Am writing a patch now.

Member

benjaoming commented Oct 26, 2017

Received another report by email..

Am writing a patch now.

@jiangjianshan

This comment has been minimized.

Show comment
Hide comment
@jiangjianshan

jiangjianshan Oct 26, 2017

@mrpau-eugene , I have just tried your suggestion but it failed to download those two videos. What I have done is firstly delete these two videos by KA Lite, and then log out and log in again. After I select them again to download. I have found they can't be download. But actually I was download it before at several months ago.
image

The log file inside $HOME\.kalite\logs as attachment here.

django-2017-10-26.txt

  Also, the hosts file has been changed to the original one for my laptop.

hosts.zip

jiangjianshan commented Oct 26, 2017

@mrpau-eugene , I have just tried your suggestion but it failed to download those two videos. What I have done is firstly delete these two videos by KA Lite, and then log out and log in again. After I select them again to download. I have found they can't be download. But actually I was download it before at several months ago.
image

The log file inside $HOME\.kalite\logs as attachment here.

django-2017-10-26.txt

  Also, the hosts file has been changed to the original one for my laptop.

hosts.zip

@jiangjianshan

This comment has been minimized.

Show comment
Hide comment
@jiangjianshan

jiangjianshan Oct 26, 2017

@benjaoming , thanks for your hard working for the patch.

jiangjianshan commented Oct 26, 2017

@benjaoming , thanks for your hard working for the patch.

@benjaoming

This comment has been minimized.

Show comment
Hide comment
@benjaoming

benjaoming Oct 26, 2017

Member

@jiangjianshan thanks for your further investigation -- at the moment, we have a pretty good idea that it's because thumb nail URLs have changed. Would you be able to try out a pre-release and confirm if it is fixed?

Member

benjaoming commented Oct 26, 2017

@jiangjianshan thanks for your further investigation -- at the moment, we have a pretty good idea that it's because thumb nail URLs have changed. Would you be able to try out a pre-release and confirm if it is fixed?

@jiangjianshan

This comment has been minimized.

Show comment
Hide comment
@jiangjianshan

jiangjianshan Oct 26, 2017

@benjaoming , I'm uninstall the current version 0.17.3 and going to install the version 0.17.1. It may take several hours to install the new version and run the command "kalite manage setup" after each installation. I will report the result to you soon if I finish them.

jiangjianshan commented Oct 26, 2017

@benjaoming , I'm uninstall the current version 0.17.3 and going to install the version 0.17.1. It may take several hours to install the new version and run the command "kalite manage setup" after each installation. I will report the result to you soon if I finish them.

@benjaoming

This comment has been minimized.

Show comment
Hide comment
@benjaoming

benjaoming Oct 26, 2017

Member

Sorry to hear that it takes so long to install on Windows for you. I'm not sure if this is a general issue.

Anyways, I will get it reproduced and fixed -- then I will let you know when a Windows installer with the final fix is released, so you don't have to spend several hours installing to test it :)

Because it will normally take two or three hours to complete the install and do the command 'kalite manage setup'.

@mrpau-richard this is a separate issue, do you have some guidance that you can refer to? Or maybe open a new installers issue that we can refer to with these sporadic reports?

Member

benjaoming commented Oct 26, 2017

Sorry to hear that it takes so long to install on Windows for you. I'm not sure if this is a general issue.

Anyways, I will get it reproduced and fixed -- then I will let you know when a Windows installer with the final fix is released, so you don't have to spend several hours installing to test it :)

Because it will normally take two or three hours to complete the install and do the command 'kalite manage setup'.

@mrpau-richard this is a separate issue, do you have some guidance that you can refer to? Or maybe open a new installers issue that we can refer to with these sporadic reports?

@jiangjianshan

This comment has been minimized.

Show comment
Hide comment
@jiangjianshan

jiangjianshan Oct 26, 2017

@benjaoming , it doesn't matter, I run the command "kalite manage setup" and then go to sleep. This morning I try the version 0.17.1, but got the same issue. Here is the log file.
django-log-2017-10-27.txt

jiangjianshan commented Oct 26, 2017

@benjaoming , it doesn't matter, I run the command "kalite manage setup" and then go to sleep. This morning I try the version 0.17.1, but got the same issue. Here is the log file.
django-log-2017-10-27.txt

@benjaoming

This comment has been minimized.

Show comment
Hide comment
@benjaoming

benjaoming Oct 27, 2017

Member

Sorry about the delay -- we have a slight complexity: The video download routes through LE servers, so perhaps we can fix this without a new release, but simply by patching the central server so it works for any release.

Take this example:

http://kalite.learningequality.org/download/videos/9GQdh2eGP-Y.mp4/9GQdh2eGP-Y.mp4

Redirects to:

http://s3.amazonaws.com/KA-youtube-converted/9GQdh2eGP-Y.mp4/9GQdh2eGP-Y.mp4

Member

benjaoming commented Oct 27, 2017

Sorry about the delay -- we have a slight complexity: The video download routes through LE servers, so perhaps we can fix this without a new release, but simply by patching the central server so it works for any release.

Take this example:

http://kalite.learningequality.org/download/videos/9GQdh2eGP-Y.mp4/9GQdh2eGP-Y.mp4

Redirects to:

http://s3.amazonaws.com/KA-youtube-converted/9GQdh2eGP-Y.mp4/9GQdh2eGP-Y.mp4

@jiangjianshan

This comment has been minimized.

Show comment
Hide comment
@jiangjianshan

jiangjianshan Oct 28, 2017

@benjaoming , attachment is the current log file of my ka-lite installed on my laptop.
django-2017-10-31.txt
So I have to wait for you after apply all the patching inside central server and then I can start the ka-lite to download the video?

jiangjianshan commented Oct 28, 2017

@benjaoming , attachment is the current log file of my ka-lite installed on my laptop.
django-2017-10-31.txt
So I have to wait for you after apply all the patching inside central server and then I can start the ka-lite to download the video?

@benjaoming

This comment has been minimized.

Show comment
Hide comment
@benjaoming

benjaoming Nov 6, 2017

Member

Sorry about the delay in solving this, am now back to the initial plan: We need to create a new release of KA Lite, because we currently don't have a proxy (the assumption in my previous comment doesn't hold).

Example:

~ kalite manage shell
>>> from kalite.topic_tools import content_models
>>> items = content_models.get_content_items()
>>> from fle_utils import videos
>>> for i in items:
...     if not i['youtube_id']:
...         continue
...     videos.get_outside_video_urls(i['youtube_id'])
...     break
...     
... 
(u'http://s3.amazonaws.com/KA-youtube-converted/y2-uaPiyoxc.mp4/y2-uaPiyoxc.mp4', u'http://s3.amazonaws.com/KA-youtube-converted/y2-uaPiyoxc.mp4/y2-uaP
iyoxc.png')
Member

benjaoming commented Nov 6, 2017

Sorry about the delay in solving this, am now back to the initial plan: We need to create a new release of KA Lite, because we currently don't have a proxy (the assumption in my previous comment doesn't hold).

Example:

~ kalite manage shell
>>> from kalite.topic_tools import content_models
>>> items = content_models.get_content_items()
>>> from fle_utils import videos
>>> for i in items:
...     if not i['youtube_id']:
...         continue
...     videos.get_outside_video_urls(i['youtube_id'])
...     break
...     
... 
(u'http://s3.amazonaws.com/KA-youtube-converted/y2-uaPiyoxc.mp4/y2-uaPiyoxc.mp4', u'http://s3.amazonaws.com/KA-youtube-converted/y2-uaPiyoxc.mp4/y2-uaP
iyoxc.png')
@jiangjianshan

This comment has been minimized.

Show comment
Hide comment
@jiangjianshan

jiangjianshan Nov 7, 2017

@benjaoming , thanks for your reply, so I have to wait for the new release 0.17.4 and then the issue should be solved.

jiangjianshan commented Nov 7, 2017

@benjaoming , thanks for your reply, so I have to wait for the new release 0.17.4 and then the issue should be solved.

@benjaoming

This comment has been minimized.

Show comment
Hide comment
@benjaoming

benjaoming Nov 7, 2017

Member

@jiangjianshan yes, seems so, but I will get back directly as soon as we know when it happens!

Member

benjaoming commented Nov 7, 2017

@jiangjianshan yes, seems so, but I will get back directly as soon as we know when it happens!

@benjaoming

This comment has been minimized.

Show comment
Hide comment
@benjaoming

benjaoming Nov 12, 2017

Member

@jiangjianshan interesting - I ran a script that downloads all thumbnails in the English database, and I had the same connection errors happen 197 times.

We should fix this issue by:

  1. Allowing connection retries https://stackoverflow.com/questions/15431044/can-i-set-max-retries-for-requests-request
  2. Failing gracefully and logging on CRITICAL level if thumbnails are missing during the download

I'll try to fix 1) and check if I can download all the thumbnails in one go. If successful, I will release 0.17.4.

Member

benjaoming commented Nov 12, 2017

@jiangjianshan interesting - I ran a script that downloads all thumbnails in the English database, and I had the same connection errors happen 197 times.

We should fix this issue by:

  1. Allowing connection retries https://stackoverflow.com/questions/15431044/can-i-set-max-retries-for-requests-request
  2. Failing gracefully and logging on CRITICAL level if thumbnails are missing during the download

I'll try to fix 1) and check if I can download all the thumbnails in one go. If successful, I will release 0.17.4.

@jiangjianshan

This comment has been minimized.

Show comment
Hide comment
@jiangjianshan

jiangjianshan Nov 12, 2017

@benjaoming , thanks for your hard working.

jiangjianshan commented Nov 12, 2017

@benjaoming , thanks for your hard working.

@benjaoming benjaoming referenced this issue Nov 13, 2017

Merged

Make video/thumbnail download more robust #5536

1 of 2 tasks complete

@benjaoming benjaoming changed the title from Video downloads break: Handle new set of locations for thumbnail URLs to Video downloads break: Robustness missing when connection is unstable Nov 13, 2017

@benjaoming benjaoming changed the title from Video downloads break: Robustness missing when connection is unstable to Video downloads break when connection is unstable Nov 13, 2017

@benjaoming

This comment has been minimized.

Show comment
Hide comment
@benjaoming

benjaoming Nov 14, 2017

Member

This is still not complete! There are more issues in the video download that weren't resolved in #5536 :/

Member

benjaoming commented Nov 14, 2017

This is still not complete! There are more issues in the video download that weren't resolved in #5536 :/

@benjaoming

This comment has been minimized.

Show comment
Hide comment
@benjaoming

benjaoming Jan 19, 2018

Member

Further robustness added in #5545 -- closing this, but in case there are further problems, we can target those in more specific issue reports.

Thanks for reporting @jiangjianshan - this problem in specifically addressed in 0.17.4, which is released on PyPi, and installers will be ready in a few days.

Member

benjaoming commented Jan 19, 2018

Further robustness added in #5545 -- closing this, but in case there are further problems, we can target those in more specific issue reports.

Thanks for reporting @jiangjianshan - this problem in specifically addressed in 0.17.4, which is released on PyPi, and installers will be ready in a few days.

@benjaoming benjaoming closed this Jan 19, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment