youtube-dl output messages mix cr lf with cr confusing string processing utilities

[earl-ducaine]

Checklist

• I'm reporting a broken site support issue
• I've verified that I'm running youtube-dl version 2019.05.11
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar bug reports including closed ones
• I've read bugs section in FAQ

Verbose log

Note, I'm specifically omitting a verbose log since I've only managed to recreate the bug when youtube-dl runs in non-verbose mode.

Description

I automate youtube-dl by executing it in a separate process. Recently my automation routine began to fail. On investigation I noticed that youtube-dl was mixing MS (CRLF) and Mac (CR) newline styles. Ideally youtube-dl would use the newline style of the platform it's being run on, for a GNU/Linux system that would be LF, when outputting messages to the shell. But it is definitely a bug to mix styles. Below is example output for running

youtube-dl "https://www.youtube.com/watch?v=qFhdJjjFe5k" --audio-format "flac" > output.txt

from a bash shell. I've listed the first part of output.txt in hexedecimal so the problematic character can be seen. Note that the problem is intermittent and dependent on whether the

"Unknown speed ETA Unknown ETA"

message (which contains the CRLF) is generated.

For example:

00000020: 6e67 2077 6562 7061 6765 *0a* 5b 796f 7574 ng webpage.[yout

but,

000000c0: 34*0a 0d*1b 5b4b 5b64 6f77 6e6c 6f61 645d  4...[K[download]

The full, first part of the listing in Hex:

00000000: 5b79 6f75 7475 6265 5d20 7146 6864 4a6a  [youtube] qFhdJj
00000010: 6a46 6535 6b3a 2044 6f77 6e6c 6f61 6469  jFe5k: Downloadi
00000020: 6e67 2077 6562 7061 6765 0a5b 796f 7574  ng webpage.[yout
00000030: 7562 655d 2071 4668 644a 6a6a 4665 356b  ube] qFhdJjjFe5k
00000040: 3a20 446f 776e 6c6f 6164 696e 6720 7669  : Downloading vi
00000050: 6465 6f20 696e 666f 2077 6562 7061 6765  deo info webpage
00000060: 0a5b 646f 776e 6c6f 6164 5d20 4465 7374  .[download] Dest
00000070: 696e 6174 696f 6e3a 2055 6e6c 696d 6974  ination: Unlimit
00000080: 6564 2054 6f75 6368 202d 2053 6561 7263  ed Touch - Searc
00000090: 6869 6e67 2054 6f20 4669 6e64 2054 6865  hing To Find The
000000a0: 204f 6e65 2028 5265 6d69 7829 2d71 4668   One (Remix)-qFh
000000b0: 644a 6a6a 4665 356b 2e66 3133 362e 6d70  dJjjFe5k.f136.mp
000000c0: 340a 0d1b 5b4b 5b64 6f77 6e6c 6f61 645d  4...[K[download]
000000d0: 2020 2030 2e30 2520 6f66 2032 322e 3732     0.0% of 22.72
000000e0: 4d69 4220 6174 2055 6e6b 6e6f 776e 2073  MiB at Unknown s
000000f0: 7065 6564 2045 5441 2055 6e6b 6e6f 776e  peed ETA Unknown
00000100: 2045 5441 0d1b 5b4b 5b64 6f77 6e6c 6f61   ETA..[K[downloa
00000110: 645d 2020 2030 2e30 2520 6f66 2032 322e  d]   0.0% of 22.
00000120: 3732 4d69 4220 6174 2055 6e6b 6e6f 776e  72MiB at Unknown
00000130: 2073 7065 6564 2045 5441 2055 6e6b 6e6f   speed ETA Unkno
00000140: 776e 2045 5441 0d1b 5b4b 5b64 6f77 6e6c  wn ETA..[K[downl
00000150: 6f61 645d 2020 2030 2e30 2520 6f66 2032  oad]   0.0% of 2
00000160: 322e 3732 4d69 4220 6174 2055 6e6b 6e6f  2.72MiB at Unkno
00000170: 776e 2073 7065 6564 2045 5441 2055 6e6b  wn speed ETA Unk
00000180: 6e6f 776e 2045 5441 0d1b 5b4b 5b64 6f77  nown ETA..[K[dow
00000190: 6e6c 6f61 645d 2020 2030 2e31 2520 6f66  nload]   0.1% of
000001a0: 2032 322e 3732 4d69 4220 6174 2055 6e6b   22.72MiB at Unk
000001b0: 6e6f 776e 2073 7065 6564 2045 5441 2055  nown speed ETA U
000001c0: 6e6b 6e6f 776e 2045 5441 0d1b 5b4b 5b64  nknown ETA..[K[d
000001d0: 6f77 6e6c 6f61 645d 2020 2030 2e31 2520  ownload]   0.1% 

[dstftw]

This is expected behavior for progress bar string.

    --newline                        Output progress bar as new lines

[earl-ducaine]

Thanks for the quick response. --newline was exactly what I should have been using. I didn't notice that there was an extra, non-printing left bracket after the x0a, i.e. a terminal control sequence.