Launching Youtube-dl as a background command

[vanilla38]

Hi, I just noticed when i try to launch $ youtube-dl http://www.ZZzzzZZ.com &
It fails, program cant terminate.

Any clue ? Thanks.

[phihag]

Can you elaborate what you mean by "cant terminate"? Also, what do you mean by "fails"? Do you get an error message?

It works fine for me:

$ youtube-dl http://www.ZZzzzZZ.com &
[1] 9302
$ Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)                                                                                                                                                                             
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code                                                                                                                                                          
    exec code in run_globals                                                                                                                                                                                         
  File "/home/phihag/projects/youtube-dl/youtube_dl/__main__.py", line 18, in <module>                                                                                                                               
    youtube_dl.main()                                                                                                                                                                                                
  File "/home/phihag/projects/youtube-dl/youtube_dl/__init__.py", line 613, in main
    _real_main(argv)
  File "/home/phihag/projects/youtube-dl/youtube_dl/__init__.py", line 597, in _real_main
    retcode = ydl.download(all_urls)
  File "/home/phihag/projects/youtube-dl/youtube_dl/YoutubeDL.py", line 567, in download
    videos = self.extract_info(url)
  File "/home/phihag/projects/youtube-dl/youtube_dl/YoutubeDL.py", line 305, in extract_info
    ie_result = ie.extract(url)
  File "/home/phihag/projects/youtube-dl/youtube_dl/extractor/common.py", line 91, in extract
    return self._real_extract(url)
  File "/home/phihag/projects/youtube-dl/youtube_dl/extractor/generic.py", line 95, in _real_extract
    new_url = self._test_redirect(url)
  File "/home/phihag/projects/youtube-dl/youtube_dl/extractor/generic.py", line 83, in _test_redirect
    response = opener.open(HeadRequest(url))
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 502: Bad Gateway

[1]+  Exit 1                  youtube-dl http://www.ZZzzzZZ.com

[yasoob]

@vanilla38 Can you give us some more info so that we can investigate the issue ? It is working for phihag.

[canavan]

I've got (probably) the same problem, using python 2.7 on IRIX.

$ ./youtube-dl http://vimeo.com/79695097&
[2] 2859
$

[2]+ Stopped ./youtube-dl http://vimeo.com/79695097
$

However, it continues as normal if I let it run in the foreground again (e.g. with 'fg'), which would be consistent with youtube-dl trying to read stdin. Backgrounding youtube-dl works as well when stdin is redirected from somewhere else:

$ ./youtube-dl http://player.vimeo.com/video/79140726/ < /dev/null &
[2] 3098
$ [vimeo] 79140726: Downloading webpage
[vimeo] 79140726: Extracting information
[download] Verruckt - KC 2014-79140726.mp4 has already been downloaded

[2]+ Done ./youtube-dl http://player.vimeo.com/video/79140726/ < /dev/null

The last bits I can see from python -m trace -t
[...]
init.py(91): def parseOpts(overrideArguments=None):
init.py(446): def _real_main(argv=None):
init.py(742): def _setup_opener(jar=None, opts=None, timeout=300):
init.py(772): def main(argv=None):
main.py(17): if name == 'main':
main.py(18): youtube_dl.main()
--- modulename: init, funcname: main
init.py(773): try:
init.py(774): _real_main(argv)
--- modulename: init, funcname: _real_main
init.py(448): if sys.platform == 'win32':
init.py(452): parser, opts, args = parseOpts(argv)
--- modulename: init, funcname: parseOpts
init.py(92): def _readOptions(filename_bytes):
init.py(105): def _format_option_string(option):
init.py(121): def _comma_separated_values_options_callback(option, opt_str, value, parser):
init.py(124): def _find_term_columns():
init.py(137): def _hide_login_info(opts):
init.py(147): max_width = 80
init.py(148): max_help_position = 80
init.py(151): columns = _find_term_columns()
--- modulename: init, funcname: _find_term_columns
init.py(125): columns = os.environ.get('COLUMNS', None)
--- modulename: UserDict, funcname: get
UserDict.py(59): if key not in self:
--- modulename: UserDict, funcname: contains
UserDict.py(71): return key in self.data
UserDict.py(60): return failobj
init.py(126): if columns:
init.py(129): try:
init.py(130): sp = subprocess.Popen(['stty', 'size'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
--- modulename: subprocess, funcname: init
subprocess.py(625): _cleanup()
--- modulename: subprocess, funcname: _cleanup
subprocess.py(461): for inst in _active[:]:
subprocess.py(627): self._child_created = False
subprocess.py(628): if not isinstance(bufsize, (int, long)):
subprocess.py(631): if mswindows:
subprocess.py(641): if startupinfo is not None:
subprocess.py(644): if creationflags != 0:
subprocess.py(648): self.stdin = None
subprocess.py(649): self.stdout = None
subprocess.py(650): self.stderr = None
subprocess.py(651): self.pid = None
subprocess.py(652): self.returncode = None
subprocess.py(653): self.universal_newlines = universal_newlines
subprocess.py(672): errread, errwrite) = self._get_handles(stdin, stdout, stderr)
--- modulename: subprocess, funcname: _get_handles
subprocess.py(1031): p2cread, p2cwrite = None, None
subprocess.py(1032): c2pread, c2pwrite = None, None
subprocess.py(1033): errread, errwrite = None, None
subprocess.py(1035): if stdin is None:
subprocess.py(1036): pass
subprocess.py(1045): if stdout is None:
subprocess.py(1047): elif stdout == PIPE:
subprocess.py(1048): c2pread, c2pwrite = self.pipe_cloexec()
--- modulename: subprocess, funcname: pipe_cloexec
subprocess.py(1091): r, w = os.pipe()
subprocess.py(1092): self._set_cloexec_flag(r)
--- modulename: subprocess, funcname: _set_cloexec_flag
subprocess.py(1073): try:
subprocess.py(1074): cloexec_flag = fcntl.FD_CLOEXEC
subprocess.py(1078): old = fcntl.fcntl(fd, fcntl.F_GETFD)
subprocess.py(1079): if cloexec:
subprocess.py(1080): fcntl.fcntl(fd, fcntl.F_SETFD, old | cloexec_flag)
subprocess.py(1093): self._set_cloexec_flag(w)
--- modulename: subprocess, funcname: _set_cloexec_flag
subprocess.py(1073): try:
subprocess.py(1074): cloexec_flag = fcntl.FD_CLOEXEC
subprocess.py(1078): old = fcntl.fcntl(fd, fcntl.F_GETFD)
subprocess.py(1079): if cloexec:
subprocess.py(1080): fcntl.fcntl(fd, fcntl.F_SETFD, old | cloexec_flag)
subprocess.py(1094): return r, w
subprocess.py(1055): if stderr is None:
subprocess.py(1057): elif stderr == PIPE:
subprocess.py(1058): errread, errwrite = self.pipe_cloexec()
--- modulename: subprocess, funcname: pipe_cloexec
subprocess.py(1091): r, w = os.pipe()
subprocess.py(1092): self._set_cloexec_flag(r)
--- modulename: subprocess, funcname: _set_cloexec_flag
subprocess.py(1073): try:
subprocess.py(1074): cloexec_flag = fcntl.FD_CLOEXEC
subprocess.py(1078): old = fcntl.fcntl(fd, fcntl.F_GETFD)
subprocess.py(1079): if cloexec:
subprocess.py(1080): fcntl.fcntl(fd, fcntl.F_SETFD, old | cloexec_flag)
subprocess.py(1093): self._set_cloexec_flag(w)
--- modulename: subprocess, funcname: _set_cloexec_flag
subprocess.py(1073): try:
subprocess.py(1074): cloexec_flag = fcntl.FD_CLOEXEC
subprocess.py(1078): old = fcntl.fcntl(fd, fcntl.F_GETFD)
subprocess.py(1079): if cloexec:
subprocess.py(1080): fcntl.fcntl(fd, fcntl.F_SETFD, old | cloexec_flag)
subprocess.py(1094): return r, w
subprocess.py(1067): return (p2cread, p2cwrite,
subprocess.py(1068): c2pread, c2pwrite,
subprocess.py(1069): errread, errwrite)
subprocess.py(674): self._execute_child(args, executable, preexec_fn, close_fds,
subprocess.py(675): cwd, env, universal_newlines,
subprocess.py(676): startupinfo, creationflags, shell,
subprocess.py(677): p2cread, p2cwrite,
subprocess.py(678): c2pread, c2pwrite,
subprocess.py(679): errread, errwrite)
--- modulename: subprocess, funcname: _execute_child
subprocess.py(1119): if isinstance(args, types.StringTypes):
subprocess.py(1122): args = list(args)
subprocess.py(1124): if shell:
subprocess.py(1129): if executable is None:
subprocess.py(1130): executable = args[0]
subprocess.py(1135): errpipe_read, errpipe_write = self.pipe_cloexec()
--- modulename: subprocess, funcname: pipe_cloexec
subprocess.py(1091): r, w = os.pipe()
subprocess.py(1092): self._set_cloexec_flag(r)
--- modulename: subprocess, funcname: _set_cloexec_flag
subprocess.py(1073): try:
subprocess.py(1074): cloexec_flag = fcntl.FD_CLOEXEC
subprocess.py(1078): old = fcntl.fcntl(fd, fcntl.F_GETFD)
subprocess.py(1079): if cloexec:
subprocess.py(1080): fcntl.fcntl(fd, fcntl.F_SETFD, old | cloexec_flag)
subprocess.py(1093): self._set_cloexec_flag(w)
--- modulename: subprocess, funcname: _set_cloexec_flag
subprocess.py(1073): try:
subprocess.py(1074): cloexec_flag = fcntl.FD_CLOEXEC
subprocess.py(1078): old = fcntl.fcntl(fd, fcntl.F_GETFD)
subprocess.py(1079): if cloexec:
subprocess.py(1080): fcntl.fcntl(fd, fcntl.F_SETFD, old | cloexec_flag)
subprocess.py(1094): return r, w
subprocess.py(1136): try:
subprocess.py(1137): try:
subprocess.py(1138): gc_was_enabled = gc.isenabled()
subprocess.py(1141): gc.disable()
subprocess.py(1142): try:
subprocess.py(1143): self.pid = os.fork()
subprocess.py(1148): self._child_created = True
subprocess.py(1149): if self.pid == 0:
subprocess.py(1151): try:
subprocess.py(1153): if p2cwrite is not None:
subprocess.py(1155): if c2pread is not None:
subprocess.py(1156): os.close(c2pread)
subprocess.py(1157): if errread is not None:
subprocess.py(1158): os.close(errread)
subprocess.py(1159): os.close(errpipe_read)
subprocess.py(1164): if c2pwrite == 0:
subprocess.py(1166): if errwrite == 0 or errwrite == 1:
subprocess.py(1170): def _dup2(a, b):
subprocess.py(1178): _dup2(p2cread, 0)
--- modulename: subprocess, funcname: _dup2
subprocess.py(1174): if a == b:
subprocess.py(1176): elif a is not None:
subprocess.py(1179): _dup2(c2pwrite, 1)
--- modulename: subprocess, funcname: _dup2
subprocess.py(1174): if a == b:
subprocess.py(1176): elif a is not None:
subprocess.py(1177): os.dup2(a, b)
subprocess.py(1148): self._child_created = True
subprocess.py(1149): if self.pid == 0:
subprocess.py(1219): if gc_was_enabled:
subprocess.py(1220): gc.enable()
subprocess.py(1223): os.close(errpipe_write)
subprocess.py(1225): if p2cread is not None and p2cwrite is not None:
subprocess.py(1227): if c2pwrite is not None and c2pread is not None:
subprocess.py(1228): os.close(c2pwrite)
subprocess.py(1229): if errwrite is not None and errread is not None:
subprocess.py(1230): os.close(errwrite)
subprocess.py(1234): data = _eintr_retry_call(os.read, errpipe_read, 1048576)
--- modulename: subprocess, funcname: _eintr_retry_call
subprocess.py(476): while True:
subprocess.py(477): try:
subprocess.py(478): return func(*args)
subprocess.py(1237): os.close(errpipe_read)
subprocess.py(1239): if data != "":
subprocess.py(681): if mswindows:
subprocess.py(689): if p2cwrite is not None:
subprocess.py(691): if c2pread is not None:
subprocess.py(692): if universal_newlines:
subprocess.py(695): self.stdout = os.fdopen(c2pread, 'rb', bufsize)
subprocess.py(696): if errread is not None:
subprocess.py(697): if universal_newlines:
subprocess.py(700): self.stderr = os.fdopen(errread, 'rb', bufsize)
init.py(131): out,err = sp.communicate()
--- modulename: subprocess, funcname: communicate
subprocess.py(734): if [self.stdin, self.stdout, self.stderr].count(None) >= 2:
subprocess.py(754): return self._communicate(input)
--- modulename: subprocess, funcname: _communicate
subprocess.py(1304): if self.stdin:
subprocess.py(1311): if _has_poll:
subprocess.py(1312): stdout, stderr = self._communicate_with_poll(input)
--- modulename: subprocess, funcname: _communicate_with_poll
--- modulename: subprocess, funcname: communicate
subprocess.py(734): if [self.stdin, self.stdout, self.stderr].count(None) >= 2:
subprocess.py(754): return self._communicate(input)
--- modulename: subprocess, funcname: _communicate
subprocess.py(1304): if self.stdin:
subprocess.py(1311): if _has_poll:
subprocess.py(1312): stdout, stderr = self._communicate_with_poll(input)
--- modulename: subprocess, funcname: _communicate_with_poll
subprocess.py(1337): stdout = None # Return
subprocess.py(1338): stderr = None # Return
subprocess.py(1339): fd2file = {}
subprocess.py(1340): fd2output = {}

par (like strace) continues with the stuff below just after 'fg':

0mS[  1]                ( 2908): was sent signal SIGCLD by the kernel
0mS[  1]                ( 2908): was sent signal SIGCONT by pid 2534
0mS[  0]                ( 2908): was sent signal SIGCLD by the kernel
0mS[  2]                 : END-poll() = 2
0mS[  2]                 : close(3) OK
0mS[  2]                 : read(5, "unknown mode: size\n", 4096) = 19
1mS[  2]                 : poll(0x105546f8, 1, -1)
1mS[  2]                 : END-poll() = 1
1mS[  2]                 : close(5) OK

[ayushgoel]

+1
Running youtube-dl in background stops the download. Bringing it back with fg starts the download.

[yan12125]

The problematic lines lays in ffmpeg:

        if (tcgetattr (0, &tty) == 0) {
            oldtty = tty;
            restore_tty = 1;

            tty.c_iflag &= ~(IGNBRK|BRKINT|PARMRK|ISTRIP
                             |INLCR|IGNCR|ICRNL|IXON);
            tty.c_oflag |= OPOST;
            tty.c_lflag &= ~(ECHO|ECHONL|ICANON|IEXTEN);
            tty.c_cflag &= ~(CSIZE|PARENB);
            tty.c_cflag |= CS8;
            tty.c_cc[VMIN] = 1;
            tty.c_cc[VTIME] = 0;

            tcsetattr (0, TCSANOW, &tty);
        }

(See: https://github.com/FFmpeg/FFmpeg/blob/6f69f7a8bf6a0d013985578df2ef42ee6b1c7994/ffmpeg.c#L371-L385)
The code segment above tries to ioctl() on STDIN. Kernel then sends SIGTTOU to the calling process, youtube-dl. A workaround is avoiding ffmpeg/avconv calls by tricking youtube-dl that ffmpeg does not exist:

$ youtube-dl -v http://vimeo.com/79695097 --ffmpeg-location /path/not/exist &
[1] 22603

$ [debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', 'http://vimeo.com/79695097', '--ffmpeg-location', '/path/not/exist']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.04.13
[debug] Git HEAD: d1c4e4b
[debug] Python version 3.5.1 - Linux-4.5.0-1-ARCH-x86_64-with-arch-Arch-Linux
WARNING: ffmpeg-location /path/not/exist does not exist! Continuing without avconv/ffmpeg.
[debug] exe versions: rtmpdump 2.4
[debug] Proxy map: {}
[vimeo] 79695097: Downloading webpage
[vimeo] 79695097: Extracting information
[vimeo] 79695097: Downloading webpage
[vimeo] 79695097: Downloading JSON metadata
[vimeo] 79695097: Checking Original video URL
[vimeo] 79695097: Original video URL is invalid, skipping
[vimeo] 79695097: Downloading m3u8 information
WARNING: ffmpeg-location /path/not/exist does not exist! Continuing without avconv/ffmpeg.
[debug] Invoking downloader on 'https://10-lvl3-pdl.vimeocdn.com/01/939/3/79695097/206208039.mp4?expires=1460620853&token=06d817e44ae6452e6b2f6'
[download] Destination: IDIOTS-79695097.mp4
[download] 100% of 67.08MiB in 00:19

[1]  + 22603 done       youtube-dl -v http://vimeo.com/79695097 --ffmpeg-location /path/not/exist

This is the case on Linux. Don't know the case in IRIX.

I guess there's nothing youtube-dl can do unless we have a patched ffmpeg or a patched Linux kernel, so closing this ticket here. Feel free to propose your solution and I or others will reopen this ticket.

[canavan]

I don't think ffmpeg is involved. youtube-dl stops before even trying to download the video:

$ youtube-dl  https://youtu.be/RE9mswQbz6M &
[1] 1864
$ 

[1]+  Stopped                 youtube-dl https://youtu.be/RE9mswQbz6M
$ fg
youtube-dl https://youtu.be/RE9mswQbz6M
[youtube] RE9mswQbz6M: Downloading webpage
[youtube] RE9mswQbz6M: Downloading video info webpage
[youtube] RE9mswQbz6M: Extracting video information
[youtube] RE9mswQbz6M: Downloading MPD manifest
[...]

[yan12125]

It is. Before the video starts downloading, youtube-dl checks whether ffmpeg/avconv exists or not, and prepend bestvideo+bestaudio to requested formats. (https://github.com/rg3/youtube-dl/blob/e0986e3/youtube_dl/YoutubeDL.py#L1359-L1361)

[canavan]

In IRIX, the problem persists, even if ffmpeg is not in the path.

[yan12125]

Could you run the command stty -tostop and try again?

[canavan]

stty -tostop doesn't change the behaviour. Python version is 2.7.11. youtube-dl even stops if the video has previously been downloaded completely.

 $ stty -tostop
 $ /usr/nekoware/bin/python /usr/nekoware/bin/youtube-dl $YOUTUBEDLFLAGS  https://youtu.be/RE9mswQbz6M  &
[1] 3183
 $

[1]+  Stopped                 /usr/nekoware/bin/python /usr/nekoware/bin/youtube-dl $YOUTUBEDLFLAGS https://youtu.be/RE9mswQbz6M

[yan12125]

Well seems the cause is quite different. From your 2013 logs youtube-dl is stopped when calling stty size. This call occurs very early, even before input URL parsing. I guess when you redirect /dev/null to STDIN, stty size just fails as no terminal exists, so youtube-dl can go down the way. This approach can be considered as a workaround for IRIX.

[yan12125]

Thanks to @johnhawkinson (#10996), this should be fixed in the next version. No more workarounds are necessary. Feel free to open a new issue if there's still something wrong.