Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Non-existent paths passed to ffmpeg/ffprobe/etc due to encodeFilename on Windows python2 #4787

Closed
dstftw opened this issue Jan 26, 2015 · 7 comments
Labels
bug

Comments

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Jan 26, 2015

b0e87c3 did not fix the conceptual problem. When filepath is encoded with encodeFilename(path, True) on python 2 @ Windows it may end up with non existent path (e.g. Trentemøller - Miss You (Mahmut Orhan Remix)-Y9o7ILCPSFU.m4a becomes Trentemoller - Miss You (Mahmut Orhan Remix)-Y9o7ILCPSFU.m4a, ø becomes o). And this non-existent path is passed to ffmpeg/ffprobe/whatever.
Possible workarounds are:

  • Encode filepaths with encodeFilename(path, True) before writing to disk.
  • Perform all the subprocessing on temporary file with plain ASCII name and rollback to original name right after it's done.
  • Provide compat subprocess implementation that works with unicode, as we do for lots of broken stuff in python2 (this would probably require copypasting and adapting the whole subprocess.py from cpython 3 or so).
>python -m youtube_dl -f 160+140 -v Y9o7ILCPSFU
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-f', '160+140', '-v', 'Y9o7ILCPSFU']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2015.01.23.4
[debug] Git HEAD: 80a49d3
[debug] Python version 2.7.6 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: ffmpeg N-68756-g627f565, ffprobe N-68694-g7c210c4, rtmpdump 2.3
[debug] Proxy map: {}
[youtube] Y9o7ILCPSFU: Downloading webpage
[youtube] Y9o7ILCPSFU: Extracting video information
[youtube] Y9o7ILCPSFU: Downloading DASH manifest
[debug] Invoking downloader on 'https://r19---sn-oguesne6.googlevideo.com/videoplayback?id=63da3b20b08f4855&itag=160&source=youtube&requiressl=yes&pl=22&mv=m&ms=au&mm=31&nh=IgpwcjAyLm5ydDE5KgkxMjcuMC4wLjE&ratebypass=yes&mime=video/mp4&gir=yes&clen=1646278&lmt=1418493117455845&dur=272.105&mt=1422286131&upn=ST6SA4yKetw&signature=5982C88337984594B47DD0AE038E9E079D694B4F.09BBB44FE6C75594F04EF70F97E252516F31940D&key=dg_yt0&sver=3&fexp=900718,907263,922243,924638,927622,930016,9406
082,9406326,941004,943917,947225,948124,952302,952605,952901,955301,957103,95710
5,957201,959701,962715&ip=PRIVATE&ipbits=0&expire=1422307758&sparams=ip,ip
bits,expire,id,itag,source,requiressl,pl,mv,ms,mm,nh,ratebypass,mime,gir,clen,lmt,dur'
[download] Destination: Trentemøller - Miss You (Mahmut Orhan Remix)-Y9o7ILCPSFU.f160.mp4
[download] 100% of 1.57MiB in 00:33
[debug] Invoking downloader on 'https://r19---sn-oguesne6.googlevideo.com/videoplayback?id=63da3b20b08f4855&itag=140&source=youtube&requiressl=yes&pl=22&mv=m&ms=au&mm=31&nh=IgpwcjAyLm5ydDE5KgkxMjcuMC4wLjE&ratebypass=yes&mime=audio/mp4&gir=yes&clen=4371063&lmt=1418493113768940&dur=272.230&mt=1422286131&upn=ST6SA4yKetw&signature=23BBC9979391613C6CBE8C5FC1E54B3E01BEF5D7.8686958B23B344D9E7E24C1AF687379F4D291A47&key=dg_yt0&sver=3&fexp=900718,907263,922243,924638,927622,930016,9406082,9406326,941004,943917,947225,948124,952302,952605,952901,955301,957103,957105,957201,959701,962715&ip=PRIVATE&ipbits=0&expire=1422307758&sparams=ip,ipbits,expire,id,itag,source,requiressl,pl,mv,ms,mm,nh,ratebypass,mime,gir,clen,lm
t,dur'
[download] Destination: Trentemøller - Miss You (Mahmut Orhan Remix)-Y9o7ILCPSFU.f140.m4a
[download] 100% of 4.17MiB in 01:46
[ffmpeg] Merging formats into "Trentemøller - Miss You (Mahmut Orhan Remix)-Y9o7ILCPSFU.mp4"
[debug] ffmpeg command line: ffmpeg -y -i 'Trentemller - Miss You (Mahmut OrhanRemix)-Y9o7ILCPSFU.f160.mp4' -i 'Trentemller - Miss You (Mahmut Orhan Remix)-Y9o7ILCPSFU.f140.m4a' -c copy -map 0:v:0 -map 1:a:0 'Trentemller - Miss You (Mahmut Orhan Remix)-Y9o7ILCPSFU.mp4'
ERROR: Trentemller - Miss You (Mahmut Orhan Remix)-Y9o7ILCPSFU.f160.mp4: No such file or directory
Traceback (most recent call last):
  File "youtube_dl\YoutubeDL.py", line 1384, in post_process
    keep_video_wish, info = pp.run(info)
  File "youtube_dl\postprocessor\ffmpeg.py", line 544, in run
    self.run_ffmpeg_multiple_files(info['__files_to_merge'], filename, args)
  File "youtube_dl\postprocessor\ffmpeg.py", line 103, in run_ffmpeg_multiple_files
    raise FFmpegPostProcessorError(msg)
FFmpegPostProcessorError
@dstftw dstftw added the bug label Jan 26, 2015
@dstftw dstftw changed the title Non-existent paths passed to ffmpeg/ffprobe/etc due to `encodeFilename` Non-existent paths passed to ffmpeg/ffprobe/etc due to encodeFilename on Windows python2 Jan 26, 2015
@Croccydile
Copy link

@Croccydile Croccydile commented Jan 26, 2015

Hello, I have run into a similar problem when trying to use the merge feature to download 60 fps DASH videos from YouTube. It seems to effect video names with non-latin characters in them.

I noticed the line [debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252 as using codepage 437, which I don't believe supports such text. Using Windows 7 with the US English language setting.

I have included the original title from YouTube for each video before the block as reference.

Notably in the first it can be seen that "Kαzυσ" is converted to "Kz" which causes it to fail to find the intermediate files.

Mario Circuit - 1:44.414 - Kαzυσ (Mario Kart 8 World Record) 60FPS WITH CHROME

youtube-dl.exe -cikwv -f "bestvideo[height<=720]+bestaudio/best" --console-title --no-mtime --no-playlist --no-resize-buffer --buffer-size 4096 --write-description --all-subs -o "D:\Downloads\youtube-dl\%(extractor)s - %(uploader)s - %(upload_date)s - %(title)s - %(format)s - %(id)s.%(ext)s" https://www.youtube.com/watch?v=_zPm3SSj6W8
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-cikwv', '-f', 'bestvideo[height<=720]+bestaudio/best', '--console-title', '--no-mtime', '--no-playlist
', '--no-resize-buffer', '--buffer-size', '4096', '--write-description', '--all-subs', '-o', 'D:\\Downloads\\youtube-dl\\%(extractor
)s - %(uploader)s - %(upload_date)s - %(title)s - %(format)s - %(id)s.%(ext)s', 'https://www.youtube.com/watch?v=_zPm3SSj6W8']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2015.01.25
[debug] Python version 2.7.8 - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg N-69258-gf994000
[debug] Proxy map: {}
[youtube] _zPm3SSj6W8: Downloading webpage
[youtube] _zPm3SSj6W8: Extracting video information
WARNING: video doesn't have subtitles
[youtube] _zPm3SSj6W8: Downloading DASH manifest
[info] Writing video description to: D:\Downloads\youtube-dl\youtube - MK8 Records - 20141029 - Mario Circuit - 1_44.414 - Kαz?σ (Ma
rio Kart 8 World Record) 60FPS WITH CHROME - bestvideo[height_=720]+bestaudio - _zPm3SSj6W8.mp4.description
[debug] Invoking downloader on 'https://r18---sn-p5qlsm7r.googlevideo.com/videoplayback?id=ff33e6dd24a3e96f&itag=298&source=youtube&
requiressl=yes&mm=31&nh=EAE&ms=au&pl=24&mv=m&ratebypass=yes&mime=video/mp4&gir=yes&clen=47872235&lmt=1415070154052485&dur=117.884&fe
xp=900718,907263,919149,927622,9406415,941004,943917,947225,948124,952302,952605,952710,952901,955301,957103,957105,957201,959701,96
1201&key=dg_yt0&signature=96D0B18FF3768AF829E1F0AE5A07EB12EA41B4A6.4AC798CFD7065F665E8043D23FDC6BFCAF9FDE26&mt=1422286654&upn=A07NmB
KNF-4&sver=3&ip=PRIVATE&ipbits=0&expire=1422308363&sparams=ip,ipbits,expire,id,itag,source,requiressl,mm,nh,ms,pl,mv,ratebypass
,mime,gir,clen,lmt,dur'
[download] Destination: D:\Downloads\youtube-dl\youtube - MK8 Records - 20141029 - Mario Circuit - 1_44.414 - Kαz?σ (Mario Kart 8 Wo
rld Record) 60FPS WITH CHROME - 298 - 1280x720 (DASH video) - _zPm3SSj6W8.f298.mp4
[download] 100% of 45.65MiB in 00:23
[debug] Invoking downloader on 'https://r18---sn-p5qlsm7r.googlevideo.com/videoplayback?id=ff33e6dd24a3e96f&itag=141&source=youtube&
requiressl=yes&mm=31&nh=EAE&ms=au&pl=24&mv=m&ratebypass=yes&mime=audio/mp4&gir=yes&clen=3761363&lmt=1415070218185735&dur=117.957&fex
p=900718,907263,919149,927622,9406415,941004,943917,947225,948124,952302,952605,952710,952901,955301,957103,957105,957201,959701,961
201&key=dg_yt0&signature=4984C3401576D31E73B7C3004FE98145FD921215.29F6188691162E5369080D28ED212E625EC59D9B&mt=1422286654&upn=A07NmBK
NF-4&sver=3&ip=PRIVATE&ipbits=0&expire=1422308363&sparams=ip,ipbits,expire,id,itag,source,requiressl,mm,nh,ms,pl,mv,ratebypass,
mime,gir,clen,lmt,dur'
[download] Destination: D:\Downloads\youtube-dl\youtube - MK8 Records - 20141029 - Mario Circuit - 1_44.414 - Kαz?σ (Mario Kart 8 Wo
rld Record) 60FPS WITH CHROME - 141 - audio only (DASH audio) - _zPm3SSj6W8.f141.m4a
[download] 100% of 3.59MiB in 00:02
[ffmpeg] Merging formats into "D:\Downloads\youtube-dl\youtube - MK8 Records - 20141029 - Mario Circuit - 1_44.414 - Kαz?σ (Mario Ka
rt 8 World Record) 60FPS WITH CHROME - bestvideo[height_=720]+bestaudio - _zPm3SSj6W8.mp4"
[debug] ffmpeg command line: ffmpeg -y -i 'D:\Downloads\youtube-dl\youtube - MK8 Records - 20141029 - Mario Circuit - 1_44.414 - Kz
(Mario Kart 8 World Record) 60FPS WITH CHROME - 298 - 1280x720 (DASH video) - _zPm3SSj6W8.f298.mp4' -i 'D:\Downloads\youtube-dl\yout
ube - MK8 Records - 20141029 - Mario Circuit - 1_44.414 - Kz (Mario Kart 8 World Record) 60FPS WITH CHROME - 141 - audio only (DASH
audio) - _zPm3SSj6W8.f141.m4a' -c copy -map 0:v:0 -map 1:a:0 'D:\Downloads\youtube-dl\youtube - MK8 Records - 20141029 - Mario Circu
it - 1_44.414 - Kz (Mario Kart 8 World Record) 60FPS WITH CHROME - bestvideo[height_=720]+bestaudio - _zPm3SSj6W8.mp4'
ERROR: D:\Downloads\youtube-dl\youtube - MK8 Records - 20141029 - Mario Circuit - 1_44.414 - Kz (Mario Kart 8 World Record) 60FPS WI
TH CHROME - 298 - 1280x720 (DASH video) - _zPm3SSj6W8.f298.mp4: No such file or directory
Traceback (most recent call last):
  File "youtube_dl\YoutubeDL.pyo", line 1384, in post_process
  File "youtube_dl\postprocessor\ffmpeg.pyo", line 544, in run
  File "youtube_dl\postprocessor\ffmpeg.pyo", line 103, in run_ffmpeg_multiple_files
FFmpegPostProcessorError

[60fps Luka→Rin→Miku] Tricolore Airline トリコロール・エア・ライン - 巡音ルカ 鏡音リン 初音ミク DIVA English lyrics

youtube-dl.exe -cikwv -f "bestvideo[height<=720]+bestaudio/best" --console-title --no-mtime --no-playlist --no-resize-buffer --buffer-size 4096 --write-description --all-subs -o "D:\Downloads\youtube-dl\%(extractor)s - %(uploader)s - %(upload_date)s - %(title)s - %(format)s - %(id)s.%(ext)s" https://www.youtube.com/watch?v=Ab4MZPP2b_k
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-cikwv', '-f', 'bestvideo[height<=720]+bestaudio/best', '--console-title', '--no-mtime', '--no-playlist
', '--no-resize-buffer', '--buffer-size', '4096', '--write-description', '--all-subs', '-o', 'D:\\Downloads\\youtube-dl\\%(extractor
)s - %(uploader)s - %(upload_date)s - %(title)s - %(format)s - %(id)s.%(ext)s', 'https://www.youtube.com/watch?v=Ab4MZPP2b_k']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2015.01.25
[debug] Python version 2.7.8 - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg N-69258-gf994000
[debug] Proxy map: {}
[youtube] Ab4MZPP2b_k: Downloading webpage
[youtube] Ab4MZPP2b_k: Extracting video information
WARNING: video doesn't have subtitles
[youtube] Ab4MZPP2b_k: Downloading DASH manifest
[info] Writing video description to: D:\Downloads\youtube-dl\youtube - googoo888 - 20141226 - [60fps Luka→Rin→Miku] Tricolore Airlin
e ??????·??·??? - ???? ???? ???? DIVA English lyrics - bestvideo[height_=720]+bestaudio - Ab4MZPP2b_k.mp4.description
[debug] Invoking downloader on 'https://r14---sn-p5qlsm7s.googlevideo.com/videoplayback?id=01be0c64f3f66ff9&itag=298&source=youtube&
requiressl=yes&ms=au&nh=EAM&mv=m&pl=24&mm=31&ratebypass=yes&mime=video/mp4&gir=yes&clen=83084845&lmt=1420799941038109&dur=206.506&sv
er=3&signature=4A1DA16B5BC9033BAFDF1DD40DE829BB3A25C26D.1E01EDBA6C55642CAC2E1CFEAE085872B6351054&mt=1422286844&upn=FefNLMfBFMs&fexp=
900718,904843,907263,924638,927622,931373,937241,9406279,941004,943605,943917,946008,947225,948124,952302,952605,952901,955301,95710
3,957105,957201,959701&key=dg_yt0&ip=PRIVATE&ipbits=0&expire=1422308526&sparams=ip,ipbits,expire,id,itag,source,requiressl,ms,n
h,mv,pl,mm,ratebypass,mime,gir,clen,lmt,dur'
[download] Destination: D:\Downloads\youtube-dl\youtube - googoo888 - 20141226 - [60fps Luka→Rin→Miku] Tricolore Airline ??????·??·?
?? - ???? ???? ???? DIVA English lyrics - 298 - 1280x720 (DASH video) - Ab4MZPP2b_k.f298.mp4
[download] 100% of 79.24MiB in 03:26
[debug] Invoking downloader on 'https://r14---sn-p5qlsm7s.googlevideo.com/videoplayback?id=01be0c64f3f66ff9&itag=141&source=youtube&
requiressl=yes&ms=au&nh=EAM&mv=m&pl=24&mm=31&ratebypass=yes&mime=audio/mp4&gir=yes&clen=6589319&lmt=1420799329139179&dur=206.657&sve
r=3&signature=6748484F67735E95D66413251A4B2691242E22B0.1EED222716C004E2D0FD1580AF224FE1D7257864&mt=1422286844&upn=FefNLMfBFMs&fexp=9
00718,904843,907263,924638,927622,931373,937241,9406279,941004,943605,943917,946008,947225,948124,952302,952605,952901,955301,957103
,957105,957201,959701&key=dg_yt0&ip=PRIVATE&ipbits=0&expire=1422308526&sparams=ip,ipbits,expire,id,itag,source,requiressl,ms,nh
,mv,pl,mm,ratebypass,mime,gir,clen,lmt,dur'
[download] Destination: D:\Downloads\youtube-dl\youtube - googoo888 - 20141226 - [60fps Luka→Rin→Miku] Tricolore Airline ??????·??·?
?? - ???? ???? ???? DIVA English lyrics - 141 - audio only (DASH audio) - Ab4MZPP2b_k.f141.m4a
[download] 100% of 6.28MiB in 00:07
[ffmpeg] Merging formats into "D:\Downloads\youtube-dl\youtube - googoo888 - 20141226 - [60fps Luka→Rin→Miku] Tricolore Airline ????
??·??·??? - ???? ???? ???? DIVA English lyrics - bestvideo[height_=720]+bestaudio - Ab4MZPP2b_k.mp4"
[debug] ffmpeg command line: ffmpeg -y -i 'D:\Downloads\youtube-dl\youtube - googoo888 - 20141226 - [60fps LukaRinMiku] Tricolore Ai
rline  -    DIVA English lyrics - 298 - 1280x720 (DASH video) - Ab4MZPP2b_k.f298.mp4' -i 'D:\Downloads\youtube-dl\youtube - googoo88
8 - 20141226 - [60fps LukaRinMiku] Tricolore Airline  -    DIVA English lyrics - 141 - audio only (DASH audio) - Ab4MZPP2b_k.f141.m4
a' -c copy -map 0:v:0 -map 1:a:0 'D:\Downloads\youtube-dl\youtube - googoo888 - 20141226 - [60fps LukaRinMiku] Tricolore Airline  -
   DIVA English lyrics - bestvideo[height_=720]+bestaudio - Ab4MZPP2b_k.mp4'
ERROR: D:\Downloads\youtube-dl\youtube - googoo888 - 20141226 - [60fps LukaRinMiku] Tricolore Airline  -    DIVA English lyrics - 29
8 - 1280x720 (DASH video) - Ab4MZPP2b_k.f298.mp4: No such file or directory
Traceback (most recent call last):
  File "youtube_dl\YoutubeDL.pyo", line 1384, in post_process
  File "youtube_dl\postprocessor\ffmpeg.pyo", line 544, in run
  File "youtube_dl\postprocessor\ffmpeg.pyo", line 103, in run_ffmpeg_multiple_files
FFmpegPostProcessorError
@MrS0m30n3
Copy link

@MrS0m30n3 MrS0m30n3 commented Apr 25, 2015

Hi, @dstftw

I still have the same issue on WIndows using youtube-dl.exe (version '2015.04.17').
The error is raised because ffmpeg can't locate the file duo to the filename encoding:
https://github.com/rg3/youtube-dl/blob/master/youtube_dl/postprocessor/ffmpeg.py#L178

@MrS0m30n3
Copy link

@MrS0m30n3 MrS0m30n3 commented Apr 25, 2015

@dstftw

Maybe we could change your solution to keep all the characters by keeping the original filename (with all the characters) in a variable and write the file to disk using the encoded filename. Then you could directly pass the encoded filename to the subprocess for the ffmpeg and after post processing the file you could rename it using the os.rename(encoded filename, original filename)

Basic Logic

original_fname = 'τεστ-test'
encoded_fname = encodeFilename(original_fname, True) # Use this filename to write to disk & for the ffmpeg subprocess.

# Rename both video & audio file after PostProcess
os.rename(encoded_fname + video_extension, original_fname + video_extension)
os.rename(encoded_fname + audio_extension, original_fname + audio_extension)
@dstftw
Copy link
Collaborator Author

@dstftw dstftw commented Apr 25, 2015

This is the solution _#_2 I suggested in the first post, but I remember @phihag was not so happy about it.

@dstftw
Copy link
Collaborator Author

@dstftw dstftw commented Apr 25, 2015

In any case, open a new issue with full verbose output.

@MrS0m30n3
Copy link

@MrS0m30n3 MrS0m30n3 commented Apr 25, 2015

@dstftw

Sorry it was my mistake i was using my own custom py2exe build and i probably had some changes in the code.
Anyway now it works fine but I still think that your _#_2 is better. Should i open a new issue to discuss about using the _#_2 solution?

@dstftw
Copy link
Collaborator Author

@dstftw dstftw commented Apr 25, 2015

Well, I also think _#_2 is more reasonable (apart from _#_3) than current tempfix. At least it will allow to workaround subprocess-related encoding mess on python2 under windows and besides will keep the original filenames. So let's open a new issue to discuss.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.