Youtube-dl unnecessarily peeking at stdin

[hdclark]

Youtube-dl appears to be peeking at stdin for no reason.

Steps to recreate:

Step 1. Create a file with one url per line.

echo 'https://www.youtube.com/watch?v=x4WFTpU8yHk'  > /tmp/playlist
echo 'https://www.youtube.com/watch?v=HsoICLXZpao' >> /tmp/playlist
echo 'https://www.youtube.com/watch?v=rYmeUDnwIAo' >> /tmp/playlist
echo 'https://www.youtube.com/watch?v=zGZKui5SRyo' >> /tmp/playlist
echo 'https://www.youtube.com/watch?v=TVBRG4EN2Bs' >> /tmp/playlist
echo 'https://www.youtube.com/watch?v=C-Dmu5AofZg' >> /tmp/playlist
echo 'https://www.youtube.com/watch?v=kIGmrBObUxQ' >> /tmp/playlist
echo 'https://www.youtube.com/watch?v=kazwE_4Vv84' >> /tmp/playlist
echo 'https://www.youtube.com/watch?v=UQLyLr3cxLw' >> /tmp/playlist
echo 'https://www.youtube.com/watch?v=dC2I-PDXbKc' >> /tmp/playlist

Step 2. Loop over the file with a shell loop.

while read aurl ; do
    youtube-dl --extract-audio "${aurl}"
done < /tmp/playlist

Output on my machine

[hal@MACHINE temporarydir]$ while read aurl ; do youtube-dl --extract-audio "${aurl}"; done < /tmp/playlist
[youtube] x4WFTpU8yHk: Downloading webpage
[youtube] x4WFTpU8yHk: Extracting video information
[youtube] x4WFTpU8yHk: Downloading js player en_US-vflZvrDTQ
[youtube] x4WFTpU8yHk: Downloading DASH manifest
[download] Destination: Iron Maiden - Gates of Tomorrow (lyrics)-x4WFTpU8yHk.m4a
[download] 100% of 9.49MiB in 00:24
[ffmpeg] Correcting container in "Iron Maiden - Gates of Tomorrow (lyrics)-x4WFTpU8yHk.m4a"
[youtube] Post-process file Iron Maiden - Gates of Tomorrow (lyrics)-x4WFTpU8yHk.m4a exists, skipping
WARNING: The url doesn't specify the protocol, trying with http
[generic] watch?v=HsoICLXZpao: Requesting header
WARNING: Could not send HEAD request to http://://www.youtube.com/watch?v=HsoICLXZpao: <urlopen error [Errno -2] Name or service not known>
[generic] watch?v=HsoICLXZpao: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [Errno -2] Name or service not known> (caused by URLError(gaierror(-2, 'Name or service not known'),))
[youtube] rYmeUDnwIAo: Downloading webpage
[youtube] rYmeUDnwIAo: Extracting video information
[youtube] rYmeUDnwIAo: Downloading DASH manifest
[download] Destination: Iron Maiden - Face in the Sand (lyrics)-rYmeUDnwIAo.m4a
[download] 100% of 11.89MiB in 00:16
[ffmpeg] Correcting container in "Iron Maiden - Face in the Sand (lyrics)-rYmeUDnwIAo.m4a"
[youtube] Post-process file Iron Maiden - Face in the Sand (lyrics)-rYmeUDnwIAo.m4a exists, skipping
[generic] watch?v=zGZKui5SRyo: Requesting header
WARNING: Could not send HEAD request to s://www.youtube.com/watch?v=zGZKui5SRyo: <urlopen error unknown url type: s>
[generic] watch?v=zGZKui5SRyo: Downloading webpage
ERROR: Unable to download webpage: <urlopen error unknown url type: s> (caused by URLError('unknown url type: s',))
[youtube] TVBRG4EN2Bs: Downloading webpage
[youtube] TVBRG4EN2Bs: Extracting video information
[youtube] TVBRG4EN2Bs: Downloading js player en_US-vflZvrDTQ
[youtube] TVBRG4EN2Bs: Downloading DASH manifest
[download] Destination: Iron Maiden - Dance of Death (lyrics)-TVBRG4EN2Bs.m4a
[download] 100% of 15.71MiB in 00:17
[ffmpeg] Correcting container in "Iron Maiden - Dance of Death (lyrics)-TVBRG4EN2Bs.m4a"
[youtube] Post-process file Iron Maiden - Dance of Death (lyrics)-TVBRG4EN2Bs.m4a exists, skipping
WARNING: The url doesn't specify the protocol, trying with http
[generic] watch?v=C-Dmu5AofZg: Requesting header
WARNING: Could not send HEAD request to http://://www.youtube.com/watch?v=C-Dmu5AofZg: <urlopen error [Errno -2] Name or service not known>
[generic] watch?v=C-Dmu5AofZg: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [Errno -2] Name or service not known> (caused by URLError(gaierror(-2, 'Name or service not known'),))
[youtube] kIGmrBObUxQ: Downloading webpage
[youtube] kIGmrBObUxQ: Extracting video information
[youtube] kIGmrBObUxQ: Downloading DASH manifest
[download] Destination: Iron Maiden - Rainmaker (lyrics)-kIGmrBObUxQ.m4a
[download] 100% of 6.96MiB in 00:12
[ffmpeg] Correcting container in "Iron Maiden - Rainmaker (lyrics)-kIGmrBObUxQ.m4a"
[youtube] Post-process file Iron Maiden - Rainmaker (lyrics)-kIGmrBObUxQ.m4a exists, skipping
[generic] watch?v=kazwE_4Vv84: Requesting header
WARNING: Could not send HEAD request to s://www.youtube.com/watch?v=kazwE_4Vv84: <urlopen error unknown url type: s>
[generic] watch?v=kazwE_4Vv84: Downloading webpage
ERROR: Unable to download webpage: <urlopen error unknown url type: s> (caused by URLError('unknown url type: s',))
[youtube] UQLyLr3cxLw: Downloading webpage
[youtube] UQLyLr3cxLw: Extracting video information
[youtube] UQLyLr3cxLw: Downloading DASH manifest
[download] Destination: Iron Maiden - Journeyman (lyrics)-UQLyLr3cxLw.m4a
[download] 100% of 12.99MiB in 00:12
[ffmpeg] Correcting container in "Iron Maiden - Journeyman (lyrics)-UQLyLr3cxLw.m4a"
[youtube] Post-process file Iron Maiden - Journeyman (lyrics)-UQLyLr3cxLw.m4a exists, skipping
WARNING: The url doesn't specify the protocol, trying with http
[youtube] dC2I-PDXbKc: Downloading webpage
[youtube] dC2I-PDXbKc: Extracting video information
[youtube] dC2I-PDXbKc: Downloading DASH manifest
[download] Destination: Iron Maiden - Wildest Dreams (lyrics)-dC2I-PDXbKc.m4a
[download] 100% of 7.08MiB in 00:04
[ffmpeg] Correcting container in "Iron Maiden - Wildest Dreams (lyrics)-dC2I-PDXbKc.m4a"
[youtube] Post-process file Iron Maiden - Wildest Dreams (lyrics)-dC2I-PDXbKc.m4a exists, skipping

Issue

stdin is being read for no reason, consuming input before shell's read can read it.

Workarounds

• Use another file descriptor. Avoid using stdin.

while read -u 9 aurl ; do
    youtube-dl --extract-audio "${aurl}"
done 9< /tmp/playlist

• Pass all urls to youtube-dl at once. This is not always possible.

Possibly-relevant information

• Happens when using urxvt and xterm. Identical results happen with each.
• Issue occurs also when dumping json.
• System info:

[hal@MACHINE ~]$ uname -a
Linux luderro 3.18.4-1-ARCH #1 SMP PREEMPT Tue Jan 27 20:45:02 CET 2015 x86_64 GNU/Linux

[hal@MACHINE ~]$ fribidi --version
fribidi (GNU FriBidi) 0.19.6
interface version 3,
Unicode Character Database version 6.2.0,
Configure options --enable-debug --enable-malloc --with-glib.

Copyright (C) 2004  Sharif FarsiWeb, Inc.
Copyright (C) 2001, 2002, 2004, 2005  Behdad Esfahbod
Copyright (C) 1999, 2000  Dov Grobgeld
GNU FriBidi comes with NO WARRANTY, to the extent permitted by law.
You may redistribute copies of GNU FriBidi under
the terms of the GNU Lesser General Public License.
For more information about these matters, see the file named COPYING.

Written by Behdad Esfahbod and Dov Grobgeld.

Suspected cause

urxvt is not bi-directional text safe. xterm is. Using --bidi-workaround makes dumping json fail. Is Fribidi or some other library being used peeking at stdin?

Wishlist fix

Separate the youtube-dl into a core library and a command line wrapper. Having a proper api would obviate issues resulting from driving youtube-dl through a terminal emulator. :)

[jaimeMF]

Thanks for the report! It should work in the next version, the problem was when running ffmpeg for extracting the audio.
Note that for this usercase, you could just use --batch-file <filename>.

Separate the youtube-dl into a core library and a command line wrapper. Having a proper api would obviate issues resulting from driving youtube-dl through a terminal emulator. :)

Maybe I misunderstood this part, but you can just import the youtube_dl module from a python script. If that's not what you wanted, you'll have to open a new issue.

[hdclark]

Great, thank you.

My final comment was tongue-in-cheek wishing for a C api. However, I was unaware the internal api was stable enough to call from Python. Thanks for the tip.

[jaimeMF]

You can always call python from C.

Most of the changes are for adding/improving websites supports, so there aren't many breaking changes.