Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The unfinished video(part file) is marked as done and recorded to the archive file. #21543

Closed
zfdev opened this issue Jun 27, 2019 · 4 comments
Closed
Labels

Comments

@zfdev
Copy link

@zfdev zfdev commented Jun 27, 2019

Checklist

  • I'm reporting a broken site support issue
  • I've verified that I'm running youtube-dl version 2019.06.21
  • I've checked that all provided URLs are alive and playable in a browser
  • I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • I've searched the bugtracker for similar bug reports including closed ones
  • I've read bugs section in FAQ

Verbose log

D:\workspace\youtube_dl>youtube-dl --verbose --yes-playlist --proxy socks5://127.0.0.1:10808 --continue --ignore-errors --sleep-interval 10 --retries 20 --download-archive archive.txt https://www.youtube.com/watch?v=E9QYZrRzI6U&list=PLKWcPfZiScgBACqEFo5TuhJWAReRNNTc8
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '--yes-playlist', '--proxy', 'socks5://127.0.0.1:10808', '--continue', '--ignore-errors', '--sleep-interval', '10', '--retries', '20', '--download-archive', 'archive.txt', 'https://www.youtube.com/watch?v=E9QYZrRzI6U']
[debug] Encodings: locale cp936, fs utf-8, out utf-8, pref cp936
[debug] youtube-dl version 2019.06.21
[debug] Python version 3.6.5 (CPython) - Windows-10-10.0.17763-SP0
[debug] exe versions: ffmpeg N-93016-gb8ebce4f84, ffprobe N-93016-gb8ebce4f84
[debug] Proxy map: {'http': 'socks5://127.0.0.1:10808', 'https': 'socks5://127.0.0.1:10808'}
[youtube] E9QYZrRzI6U: Downloading webpage
[youtube] E9QYZrRzI6U: Downloading video info webpage
WARNING: unable to download video info webpage: <urlopen error EOF occurred in violation of protocol (_ssl.c:833)>
WARNING: unable to download video info webpage: <urlopen error EOF occurred in violation of protocol (_ssl.c:833)>
[debug] Default format spec: bestvideo+bestaudio/best
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on 'https://r2---sn-a5mlrn7s.googlevideo.com/videoplayback?expire=1561667572&ei=lNMUXa-qApaLkwaypKOQBA&ip=66.42.102.6&id=o-AHT24rJnaM9gUrupMYO3utP4bkKwzww4OyC9KLASvQGm&itag=247&aitags=133%2C134%2C135%2C136%2C160%2C242%2C243%2C244%2C247%2C278&source=youtube&requiressl=yes&mm=31%2C29&mn=sn-a5mlrn7s%2Csn-a5meknsd&ms=au%2Crdu&mv=m&pl=23&initcwndbps=18785000&mime=video%2Fwebm&gir=yes&clen=138771555&dur=1510.200&lmt=1543631328042175&mt=1561645857&fvip=2&keepalive=yes&beids=9466588&c=WEB&txp=5535432&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ALgxI2wwRgIhAPr_2JBFI2hxO5vV9KW7Udg-DtJmPHzsjRbPMMSXsiqgAiEAgGLJiHRr1bemmaw3V-QEbRBT4w6pxiosLk0lrYRackM%3D&lsparams=mm%2Cmn%2Cms%2Cmv%2Cpl%2Cinitcwndbps&lsig=AHylml4wRQIhAMAhKUap_AzKxMdTHymq-uQvuVHKGG3EcOl81pAOwrGRAiBJyUuwf7Xr2KYvvWFqvQYGDKd9u8-MI27qZ_lQX9gR9A%3D%3D&ratebypass=yes'
[download] Sleeping 10 seconds...
[download] Destination: English Conversation Lesson - Learn 3 advanced expressions-E9QYZrRzI6U.f247.webm
[download]  52.8% of 132.34MiB at  1.29MiB/s ETA 00:48         ERROR: unable to download video data: <urlopen error EOF occurred in violation of protocol (_ssl.c:833)>
Traceback (most recent call last):
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\urllib\request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\http\client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\http\client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\http\client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\http\client.py", line 1026, in _send_output
    self.send(msg)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\http\client.py", line 964, in send
    self.connect()
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\youtube_dl\utils.py", line 1111, in connect
    self.sock, server_hostname=self.host)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\ssl.py", line 814, in __init__
    self.do_handshake()
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:833)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\youtube_dl\YoutubeDL.py", line 1909, in process_info
    partial_success = dl(fname, new_info)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\youtube_dl\YoutubeDL.py", line 1854, in dl
    return fd.download(name, info)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\youtube_dl\downloader\common.py", line 366, in download
    return self.real_download(filename, info_dict)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\youtube_dl\downloader\http.py", line 341, in real_download
    establish_connection()
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\youtube_dl\downloader\http.py", line 109, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\youtube_dl\YoutubeDL.py", line 2227, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\urllib\request.py", line 526, in open
    response = self._open(req, data)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\urllib\request.py", line 544, in _open
    '_open', req)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\site-packages\youtube_dl\utils.py", line 1140, in https_open
    req, **kwargs)
  File "c:\users\administrator\appdata\local\programs\python\python36\lib\urllib\request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error EOF occurred in violation of protocol (_ssl.c:833)>

Description

When I try to download a playlist, some videos can't download completely. It will generate a part file in the download directory. I know that the youtube downloader will ignore the error and download the next video when there is some network problem by using the --ignore-errors.

After the playlist is done, I try to download the whole playlist again to resume the unfinished video. But some unfinished videos can't be resume. The log shows that XXX has already been recorded in archive. But the video is a part file. Why the unfinished video will be recorded to the archive file?

Is there something wrong with my parameters? I need your help.

Below is the unfinished video screenshot.
2019-06-27 22_26_50-Phrasal Verb Lessons

@zfdev zfdev changed the title The unfinished task(part file) is marked as done and recorded to the archive file. The unfinished video(part file) is marked as done and recorded to the archive file. Jun 27, 2019
@dstftw
Copy link
Collaborator

@dstftw dstftw commented Jun 27, 2019

Unfinished file will not be recorded to archive file.
It anything is reported to be already in archive then it's either been downloaded previously and added to archive by hand.

@dstftw dstftw closed this Jun 27, 2019
@dstftw dstftw added the invalid label Jun 27, 2019
@zfdev
Copy link
Author

@zfdev zfdev commented Jun 27, 2019

Unfinished file will not be recorded to archive file.
It anything is reported to be already in archive then it's either been downloaded previously and added to archive by hand.

Thanks a lot.

But in my case, there are a lot of part files in my download directory. I must find out which video is unfinished. Sometimes I have to re-download the playlist by deleting all the files and clearing the archive file content to make sure that all the videos are download correctly.

If I want to download the whole playlist without part file. I don't want to check which video is unfinished. I need the downloader to finish all download tasks one by one without any error even the network is disconnected sometimes. What can I do?

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Jun 27, 2019

You don't need to check anything. Rerunning the same command will finish all unfinished downloads.

@zfdev
Copy link
Author

@zfdev zfdev commented Jun 30, 2019

You don't need to check anything. Rerunning the same command will finish all unfinished downloads.

I can make sure this bug can be reproduced many times. @dstftw Please help me to change the status of the issue to open. Please check the debug information below. I have tried to run the same command more than 5 times. But the unfinished video is still a part file, and it had been recorded to the archive file mistakenly.

And I have many same situations for different download playlist. This problem often occurs when there is a problem with the network. The download queue will be interrupted because the proxy network is unstable.

youtube-dl --verbose --yes-playlist -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best' -o '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' --proxy socks5://192.168.1.12:10808 --continue --ignore-errors --sleep-interval 10 --retries 20 --download-archive archive.txt https://www.youtube.com/playlist?list=PLMR3lYBIrUzg0K26GFQDfB7VchN94wpho
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '--yes-playlist', '-f', 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best', '-o', '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s', '--proxy', 'socks5://192.168.1.12:10808', '--continue', '--ignore-errors', '--sleep-interval', '10', '--retries', '20', '--download-archive', 'archive.txt', 'https://www.youtube.com/playlist?list=PLMR3lYBIrUzg0K26GFQDfB7VchN94wpho']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.06.21
[debug] Python version 3.6.5 (CPython) - Linux-3.10.35-x86_64-with-debian-8.10
[debug] exe versions: ffmpeg 4.0-static, ffprobe 4.0-static
[debug] Proxy map: {'http': 'socks5://192.168.1.12:10808', 'https': 'socks5://192.168.1.12:10808'}
[youtube:playlist] PLMR3lYBIrUzg0K26GFQDfB7VchN94wpho: Downloading webpage
[download] Downloading playlist: English Writing
[youtube:playlist] playlist English Writing: Downloading 71 videos
...
[download] Downloading video 21 of 71
[download] 99 Powerful Ways to Say "GOOD JOB" in English | Good Job Synonyms has already been recorded in archive
[download] Downloading video 22 of 71
[download] How to Introduce Yourself !!! How to Give Self Introduction in English with Examples has already been recorded in archive
...
[download] Downloading video 68 of 71
[download] List of Prepositions in English from A to Z | Preposition List has already been recorded in archive
[download] Downloading video 69 of 71
[download] Abstract Nouns: 100+ Common English Abstract Nouns from A to Z has already been recorded in archive
...
[download] Finished downloading playlist: English Writing

The unfinished video 21 is still a part file:
2019-07-01 02_14_14-English Writing

[download] Downloading video 21 of 71
[download] 99 Powerful Ways to Say "GOOD JOB" in English | Good Job Synonyms has already been recorded in archive

The unfinished video 68 is still a part file:
2019-07-01 02_16_28-English Writing

[download] Downloading video 68 of 71
[download] List of Prepositions in English from A to Z | Preposition List has already been recorded in archive

Other same situation:
2019-07-01 02_57_37-IELTS

2019-07-01 02_59_12-youtube_dl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.