Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"OSError: Failed to write string" // Bilibili // Windows 7 x64 // Python 3.6.1 #16019

Open
Kadigan opened this issue Mar 28, 2018 · 8 comments
Open

Comments

@Kadigan
Copy link

@Kadigan Kadigan commented Mar 28, 2018

  • I've verified and I assure that I'm running youtube-dl 2018.03.26.1
  • I have at least skimmed through the README, most notably the FAQ and BUGS sections
  • I have Searched the bugtracker for similar issues including closed ones
  • I have checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

  • I'm filing a bug report (encountered problems with youtube-dl)

Log

G:\[T] == TORRENTS COMPLETE ==>youtube-dl --verbose https://www.bilibili.com/video/av19673092/
[debug] System config: []
[debug] User config: ['-o', 'G:/DOWNLOADED/%(title)s.%(ext)s', '-f', 'bestvideo[height<=?1080][vcodec!=?vp9][ext=mp4]+bestaudio[ext=m4a]/best[height<=?1080][ext
=mp4]/best', '--merge-output-format', 'mkv']
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.bilibili.com/video/av19673092/']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dl version 2018.03.26.1
[debug] Python version 3.6.1 (CPython) - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg N-65476-ge18d9d9, ffprobe N-65476-ge18d9d9, phantomjs 2.1.1
[debug] Proxy map: {}
[BiliBili] 19673092: Downloading webpage
[BiliBili] 19673092: Downloading video info page
Traceback (most recent call last):
  File "e:\python3\3.6.1x64\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "e:\python3\3.6.1x64\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "E:\Python3\3.6.1x64\Scripts\youtube-dl.exe\__main__.py", line 9, in <module>
  File "e:\python3\3.6.1x64\lib\site-packages\youtube_dl\__init__.py", line 471, in main
    _real_main(argv)
  File "e:\python3\3.6.1x64\lib\site-packages\youtube_dl\__init__.py", line 461, in _real_main
    retcode = ydl.download(all_urls)
  File "e:\python3\3.6.1x64\lib\site-packages\youtube_dl\YoutubeDL.py", line 1989, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "e:\python3\3.6.1x64\lib\site-packages\youtube_dl\YoutubeDL.py", line 796, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "e:\python3\3.6.1x64\lib\site-packages\youtube_dl\YoutubeDL.py", line 892, in process_ie_result
    self.to_screen('[download] Downloading playlist: %s' % playlist)
  File "e:\python3\3.6.1x64\lib\site-packages\youtube_dl\YoutubeDL.py", line 495, in to_screen
    return self.to_stdout(message, skip_eol, check_quiet=True)
  File "e:\python3\3.6.1x64\lib\site-packages\youtube_dl\YoutubeDL.py", line 509, in to_stdout
    self._write_string(output, self._screen_file)
  File "e:\python3\3.6.1x64\lib\site-packages\youtube_dl\YoutubeDL.py", line 498, in _write_string
    write_string(s, out=out, encoding=self.params.get('encoding'))
  File "e:\python3\3.6.1x64\lib\site-packages\youtube_dl\utils.py", line 1408, in write_string
    if _windows_write_string(s, out):
  File "e:\python3\3.6.1x64\lib\site-packages\youtube_dl\utils.py", line 1392, in _windows_write_string
    raise OSError('Failed to write string')
OSError: Failed to write string

I feel that no further description is necessary. I can otherwise download from other sources just fine -- only Bilibili seems to have this issue. I was also unable to find "OSError" or "failed to write string" in conjunction w/ Bilibili anywhere in the issues.

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Mar 28, 2018

Post the verbose log from the original .exe.

@Kadigan
Copy link
Author

@Kadigan Kadigan commented Mar 31, 2018

I did, as you can see. If you need me to post anything else, please specify what command line I should use.

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Mar 31, 2018

No, it's a custom build. Original build does not use this python version.

@dstftw dstftw closed this Mar 31, 2018
@Kadigan
Copy link
Author

@Kadigan Kadigan commented Mar 31, 2018

It is not. I installed it via pip install -U youtube-dl.

I have this Python installed, alongside ffmpeg and such similar. Should it not use this Python version?

I assure you that other than that command, I did not perform any custom actions or custom builds.


The executable file is located in e:\Python3\3.6.1x64\Scripts\ and its file hashes are:

MD5: 8EABA272515EE869FA0E97642B50C3A5
SHA1: 4DE8AB450EA6FF704424C73426125FFAF618194F

@dstftw dstftw reopened this Mar 31, 2018
@Kadigan
Copy link
Author

@Kadigan Kadigan commented Apr 1, 2018

I'm open to helping debug this, if you need to run commands on my system.

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Apr 1, 2018

For any work on this issue you must provide access to reproducible environment.

@Kadigan
Copy link
Author

@Kadigan Kadigan commented May 2, 2018

Hoboy... I'm not exactly sure how to do this. Can I zip some files up for you? I obviously don't want to allow you full access to my machine, however I would be open to running some test code that could help pinpoint the issue. I've since run into the issue again, on a different video - yufR7Ch2TFM.

If I drop my output naming, everything works just fine:

C:\Users\User>youtube-dl -v --id --ignore-config --merge-output-format mkv -f "bestvideo[height<=?1080][vcodec!=?vp9][ext=mp4]+bestaudio[ext=m4a]/best[height<=?1080][ext=mp4]/best" yufR7Ch2TFM
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--id', '--ignore-config', '--merge-output-format', 'mkv', '-f', 'bestvideo[height<=?1080][vcodec!=?vp9][ext=mp4]+bestaudio[ext=m4a]/best[height<=?1080][ext=mp4]/best', 'yufR7Ch2TFM']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dl version 2018.04.25
[debug] Python version 3.6.1 (CPython) - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg N-65476-ge18d9d9, ffprobe N-65476-ge18d9d9, phantomjs 2.1.1
[debug] Proxy map: {}
[youtube] yufR7Ch2TFM: Downloading webpage
[youtube] yufR7Ch2TFM: Downloading video info webpage
[youtube] yufR7Ch2TFM: Extracting video information
[debug] Invoking downloader on 'https://r1---sn-pj2-2v1l.googlevideo.com/videoplayback?gir=yes&clen=102922858&keepalive=yes&source=youtube&mime=video%2Fmp4&itag=299&key=yt6&signature=2F08C26F546B79660D67EA2559F8AF0A6C55F42F.A634EEB7E81138EF0F45961AEBE9EBD2985D6AA2&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C298%2C299%2C302%2C303&c=WEB&ei=2yrqWtGnAe3J7ASal56YDA&fvip=1&mm=31%2C29&mn=sn-pj2-2v1l%2Csn-f5f7lne6&pl=17&id=o-ADNSrxYxu0hHAPqMYmTVmpW_S_XmAn7jluhoVsU6izZh&requiressl=yes&dur=209.016&mt=1525295755&mv=m&fexp=23724337&ms=au%2Crdu&ip=62.21.49.42&initcwndbps=863750&lmt=1509241383811619&ipbits=0&expire=1525317435&sparams=aitags%2Cclen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cexpire&ratebypass=yes'
[download] Resuming download at byte 85956286
[download] Destination: yufR7Ch2TFM.f299.mp4
[download] 100% of 98.15MiB in 00:02
[debug] Invoking downloader on 'https://r1---sn-pj2-2v1l.googlevideo.com/videoplayback?gir=yes&clen=3321207&keepalive=yes&source=youtube&mime=audio%2Fmp4&signature=4EACAF8EF1051271747DAC6812EFB827E9F5A73B.6898B8CF35FC1C706E7538C340EBE6C6A12B4942&key=yt6&itag=140&c=WEB&ei=2yrqWtGnAe3J7ASal56YDA&fvip=1&mm=31%2C29&mn=sn-pj2-2v1l%2Csn-f5f7lne6&pl=17&id=o-ADNSrxYxu0hHAPqMYmTVmpW_S_XmAn7jluhoVsU6izZh&requiressl=yes&dur=209.072&mt=1525295755&mv=m&fexp=23724337&ms=au%2Crdu&ip=62.21.49.42&initcwndbps=863750&lmt=1509241419629859&ipbits=0&expire=1525317435&sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cexpire&ratebypass=yes'
[download] Destination: yufR7Ch2TFM.f140.m4a
[download] 100% of 3.17MiB in 00:00
[ffmpeg] Merging formats into "yufR7Ch2TFM.mkv"
[debug] ffmpeg command line: ffmpeg -y -i "file:yufR7Ch2TFM.f299.mp4" -i "file:yufR7Ch2TFM.f140.m4a" -c copy -map "0:v:0" -map "1:a:0" "file:yufR7Ch2TFM.temp.mkv"
Deleting original file yufR7Ch2TFM.f299.mp4 (pass -k to keep)
Deleting original file yufR7Ch2TFM.f140.m4a (pass -k to keep)

For now, I can confirm that, at least for the YouTube one, the problem is in the "…" character (literally, three periods as one character) - I have uploaded some random video to my private YT account, changed its name to reflect the one in the video above and tried downloading it -- this got me a "Failed to write string" error.

The same happened if I simply renamed the video to "…" -- same error.

As soon as I replaced the "…" with "..." (separate periods), the download worked w/o a hitch. Meaning, the function doing the Windows filename writing is still having problems w/ certain characters, at least on some systems.

Since I don't know Python, can you write me a bit of code that attempts to create empty files w/ all Unicode characters? I could then report which ones don't work/raise an error on my system.


I did one more test. I created a tmp directory, and inside I created two files:

  • ....test, and
  • ….test.

Here are the results of dir:

G:\Downloaded\tmp>dir
 Volume in drive G is DOWNLOADS
 Volume Serial Number is ZX12-1901

 Directory of G:\Downloaded\tmp

2018:05:02  23:28    <DIR>          .
2018:05:02  23:28    <DIR>          ..
2018:05:02  23:28                 2 ....test
The system cannot write to the specified device.

Since I once-upon-a-time reinvented the wheel and wrote ls for myself (don't ask...), I also got to compare against it, and I got:

screenshot

I have added a screenshot, because a copy & paste of that properly rendered as in virtually anything that understands UTF-8.

The code tests for readability by attempting to fopen() the file for reading... it was written ages ago. The only reason it counted it as "writeable" is because of an even deeper issue (the underlying implementation splintered, created a number of other files instead, and reported writeability on one of them -- I assume).

If it matters, the output of chcp is

C:\Users\User>chcp
Active code page: 932

Of course, if I replace that OSError() with a debug log (to the effect of logging.debug()) followed by a return True, the download finishes correctly -- except I get a few more "Failed to write string" errors in the process:

DEBUG:root:Failed to write filename string to the console.
[download] 100% of 98.15MiB in 00:17
[debug] Invoking downloader on (...)
DEBUG:root:Failed to write filename string to the console.
[download] 100% of 3.17MiB in 00:00
DEBUG:root:Failed to write filename string to the console.
DEBUG:root:Failed to write filename string to the console.
DEBUG:root:Failed to write filename string to the console.
DEBUG:root:Failed to write filename string to the console.

Does this need to be a show-stopper? For now, I'm going with

        if ret == 0:
            logging.debug(" Failed to write string to console; ret = %r, written = %r, count = %r", ret, written.value, count);
            return True

because despite having issues writing UTF-8 to the Windows console (since it doesn't understand UTF-8, and probably never will), the file still manages to get downloaded just fine.


I should note that the change doesn't help the bilibili video I came in with originally -- it still claims to download, but only one part is ever saved:

C:\Users\User>youtube-dl https://www.bilibili.com/video/av19673092/
[BiliBili] 19673092: Downloading webpage
[BiliBili] 19673092: Downloading video info page
DEBUG:root: Failed to write string to console; ret = 0, written = 0, count = 48
DEBUG:root: Failed to write string to console; ret = 0, written = 0, count = 82
[download] Downloading video 1 of 7
DEBUG:root: Failed to write string to console; ret = 0, written = 0, count = 57
[download] 100% of 67.81MiB in 02:33
[download] Downloading video 2 of 7
DEBUG:root: Failed to write string to console; ret = 0, written = 0, count = 72
[download] 100% of 67.81MiB
[download] Downloading video 3 of 7
DEBUG:root: Failed to write string to console; ret = 0, written = 0, count = 72
[download] 100% of 67.81MiB
[download] Downloading video 4 of 7
DEBUG:root: Failed to write string to console; ret = 0, written = 0, count = 72
[download] 100% of 67.81MiB
[download] Downloading video 5 of 7
DEBUG:root: Failed to write string to console; ret = 0, written = 0, count = 72
[download] 100% of 67.81MiB
[download] Downloading video 6 of 7
DEBUG:root: Failed to write string to console; ret = 0, written = 0, count = 72
[download] 100% of 67.81MiB
[download] Downloading video 7 of 7
DEBUG:root: Failed to write string to console; ret = 0, written = 0, count = 72
[download] 100% of 67.81MiB
DEBUG:root: Failed to write string to console; ret = 0, written = 0, count = 57

Still, only one .flv file as 67.81 MiB is ever retained. But, I suppose that's a different issue.

@Kadigan
Copy link
Author

@Kadigan Kadigan commented Aug 5, 2018

A gentle reminder that this is still an issue in 2018.8.4 and could use some love. The issue is about writing to the console, not to disk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.