Possible encoding conflict when saving output and errors to file. Missing comma in title.

[jprokos]

Checklist

• I'm asking a question
• I've looked through the README and FAQ for similar questions
• I've searched the bugtracker for similar questions including closed ones

Question

WRITE QUESTION HERE
This would be the best place to ask regarding an issue I am encountering with youtube-dl. I am writing errors and output to a text file called vidstatus and referencing those lines to give feedback to AppleScript on the progress of the youtube-dl request.

The title of the video is altered when writing errors and output to that temp file. When I reference the title in the tmp file, to use as a path to the downloaded file, the script fails since the title has been altered. It's apparent that there's an encoding conflict when writing output and errors to the file, but I am not sure what process or arguments are causing this or how to correct it.

After I run this in AppleScript a comma is removed from the title in the /tmp/vidstatus file.

do shell script "cd /tmp/; /usr/local/bin/youtube-dl --newline --ffmpeg-location /usr/local/bin/ffmpeg " & currentURL & " > /tmp/vidstatus 2>&1 & echo $!"

I get a report like this:

[youtube] wvYJ_j92bjU: Downloading webpage
[youtube] wvYJ_j92bjU: Downloading video info webpage
[download] /Volumes/Godzilla/Video/YouTube Video/Nathan Adrian vs Caeleb Dressel _ Mens 100m Free B Final _ 2020 TYR Pro Swim Series - Des Moines-wvYJ_j92bjU.mp4 has already been downloaded and merged

In this instance the issue is in the word Mens. The comma has been removed when writing the output to the file. I am not sure what process is causing this or how to correct it.

This line in my configuration file:

-o '/Volumes/Godzilla/Video/YouTube Video/%(title)s-%(id)s.%(ext)s'

When the file is downloaded the path looks like this:

/Volumes/Godzilla/Video/YouTube Video/Nathan Adrian vs Caeleb Dressel _ Men’s 100m Free B Final _ 2020 TYR Pro Swim Series - Des Moines-wvYJ_j92bjU.mp4

The difference between Men's and Mens is a stumbling block for my AppleScript.

When running youtube-dl in terminal the comma is not removed from errors and output.

I don't see a request in the reporting template to include debug info but I will include it here as requested:

[debug] System config: []
[debug] User config: [u'-f', u'bestvideo[ext=mp4]+bestaudio[ext=m4a]', u'-o', u'/Volumes/Godzilla/Video/YouTube Video/%(title)s-%(id)s.%(ext)s', u'--no-part']
[debug] Custom config: []
[debug] Command-line args: [u'--newline', u'--verbose', u'--ffmpeg-location', u'/usr/local/bin/ffmpeg', u'https://www.youtube.com/watch?v=HlB3noj2xsI']
[debug] Encodings: locale US-ASCII, fs utf-8, out None, pref US-ASCII
[debug] youtube-dl version 2020.03.06
[debug] Python version 2.7.16 (CPython) - Darwin-18.7.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.2.2, ffprobe 4.2.2
[debug] Proxy map: {}
[youtube] HlB3noj2xsI: Downloading webpage
[youtube] HlB3noj2xsI: Downloading video info webpage
[debug] Invoking downloader on u'https://r6---sn-vgqsrn76.googlevideo.com/videoplayback?expire=1584058327&ei=d3tqXufRL6eAir4PgqKCmAc&ip=73.246.88.217&id=o-AAwhxXZHB4Fu4tAhuuIw0ZPcYz9fvLJ7xErMTQM7Uy_T&itag=137&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&source=youtube&requiressl=yes&mh=lg&mm=31%2C26&mn=sn-vgqsrn76%2Csn-ab5sznld&ms=au%2Conr&mv=m&mvi=5&pl=15&initcwndbps=1990000&vprv=1&mime=video%2Fmp4&gir=yes&clen=53294375&dur=98.231&lmt=1583634788823819&mt=1584036643&fvip=5&keepalive=yes&fexp=23842630&c=WEB&txp=2316222&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ADKhkGMwRAIgC3GkgHwrxJbJ0AyyqccjsDHZQUVtgnpbafRRG9wyWMUCIBRRZXRGo6aqzeycFX22VfLPhmO5TU62qJ2-9ajduYxb&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=ABSNjpQwRQIhAPHN2n9BGJfaw5fBMMmObucwwEf4I9kHrN1DS8MvgTmUAiBT-3Vt5Mwhl-X-Cgwv9N_D8B2ZpbHIZwEu2pBPMQgcDA%3D%3D&ratebypass=yes'
[download] Destination: /Volumes/Godzilla/Video/YouTube Video/Ryan Murphy posts his fastest time since 2016 _ Mens 100m Back A Final _ 2020 TYR Pro Swim Series-HlB3noj2xsI.f137.mp4
[download]   0.0% of 50.83MiB at 282.58KiB/s ETA 03:04
[download]   0.0% of 50.83MiB at 821.71KiB/s ETA 01:03
[download]   0.0% of 50.83MiB at  1.84MiB/s ETA 00:27
[download]   0.0% of 50.83MiB at  3.87MiB/s ETA 00:13
[download]   0.1% of 50.83MiB at  1.87MiB/s ETA 00:27
[download]   0.1% of 50.83MiB at  2.20MiB/s ETA 00:23
[download]   0.2% of 50.83MiB at  2.94MiB/s ETA 00:17
[download]   0.5% of 50.83MiB at  4.23MiB/s ETA 00:11
[download]   1.0% of 50.83MiB at  5.83MiB/s ETA 00:08
[download]   2.0% of 50.83MiB at  8.83MiB/s ETA 00:05
[download]   3.9% of 50.83MiB at 11.35MiB/s ETA 00:04
[download]   7.9% of 50.83MiB at 13.04MiB/s ETA 00:03
[download]  15.7% of 50.83MiB at 16.82MiB/s ETA 00:02
[download]  19.2% of 50.83MiB at 18.00MiB/s ETA 00:02
[download]  19.2% of 50.83MiB at 233.04KiB/s ETA 03:00
[download]  19.2% of 50.83MiB at 666.50KiB/s ETA 01:03
[download]  19.3% of 50.83MiB at  1.47MiB/s ETA 00:28
[download]  19.3% of 50.83MiB at  3.08MiB/s ETA 00:13
[download]  19.3% of 50.83MiB at  2.17MiB/s ETA 00:18
[download]  19.4% of 50.83MiB at  2.11MiB/s ETA 00:19
[download]  19.5% of 50.83MiB at  3.13MiB/s ETA 00:13
[download]  19.7% of 50.83MiB at  4.47MiB/s ETA 00:09
[download]  20.2% of 50.83MiB at  6.96MiB/s ETA 00:05
[download]  21.2% of 50.83MiB at 10.00MiB/s ETA 00:04
[download]  23.2% of 50.83MiB at 14.51MiB/s ETA 00:02
[download]  27.1% of 50.83MiB at 19.11MiB/s ETA 00:01
[download]  35.0% of 50.83MiB at 22.69MiB/s ETA 00:01
[download]  38.7% of 50.83MiB at 23.63MiB/s ETA 00:01
[download]  38.8% of 50.83MiB at 321.65KiB/s ETA 01:39
[download]  38.8% of 50.83MiB at 926.24KiB/s ETA 00:34
[download]  38.8% of 50.83MiB at  2.06MiB/s ETA 00:15
[download]  38.8% of 50.83MiB at  4.32MiB/s ETA 00:07
[download]  38.8% of 50.83MiB at  1.57MiB/s ETA 00:19
[download]  38.9% of 50.83MiB at  2.10MiB/s ETA 00:14
[download]  39.0% of 50.83MiB at  2.61MiB/s ETA 00:11
[download]  39.2% of 50.83MiB at  3.93MiB/s ETA 00:07
[download]  39.7% of 50.83MiB at  5.75MiB/s ETA 00:05
[download]  40.7% of 50.83MiB at  7.76MiB/s ETA 00:03
[download]  42.7% of 50.83MiB at  9.94MiB/s ETA 00:02
[download]  46.6% of 50.83MiB at 12.69MiB/s ETA 00:02
[download]  54.5% of 50.83MiB at 11.58MiB/s ETA 00:01
[download]  58.0% of 50.83MiB at 12.97MiB/s ETA 00:01
[download]  58.0% of 50.83MiB at 656.18KiB/s ETA 00:33
[download]  58.0% of 50.83MiB at  1.79MiB/s ETA 00:11
[download]  58.0% of 50.83MiB at  4.00MiB/s ETA 00:05
[download]  58.0% of 50.83MiB at  8.22MiB/s ETA 00:02
[download]  58.0% of 50.83MiB at  1.89MiB/s ETA 00:11
[download]  58.1% of 50.83MiB at  2.08MiB/s ETA 00:10
[download]  58.2% of 50.83MiB at  2.89MiB/s ETA 00:07
[download]  58.4% of 50.83MiB at  4.06MiB/s ETA 00:05
[download]  58.9% of 50.83MiB at  6.39MiB/s ETA 00:03
[download]  59.9% of 50.83MiB at  7.49MiB/s ETA 00:02
[download]  61.9% of 50.83MiB at 11.92MiB/s ETA 00:01
[download]  65.8% of 50.83MiB at 16.71MiB/s ETA 00:01
[download]  73.7% of 50.83MiB at 20.83MiB/s ETA 00:00
[download]  77.5% of 50.83MiB at 22.06MiB/s ETA 00:00
[download]  77.5% of 50.83MiB at 196.69KiB/s ETA 00:59
[download]  77.5% of 50.83MiB at 576.35KiB/s ETA 00:20
[download]  77.5% of 50.83MiB at  1.30MiB/s ETA 00:08
[download]  77.5% of 50.83MiB at  2.74MiB/s ETA 00:04
[download]  77.6% of 50.83MiB at  1.78MiB/s ETA 00:06
[download]  77.6% of 50.83MiB at  2.18MiB/s ETA 00:05
[download]  77.8% of 50.83MiB at  3.08MiB/s ETA 00:03
[download]  78.0% of 50.83MiB at  4.65MiB/s ETA 00:02
[download]  78.5% of 50.83MiB at  6.34MiB/s ETA 00:01
[download]  79.5% of 50.83MiB at  8.72MiB/s ETA 00:01
[download]  81.4% of 50.83MiB at 13.17MiB/s ETA 00:00
[download]  85.4% of 50.83MiB at 17.96MiB/s ETA 00:00
[download]  93.2% of 50.83MiB at 17.80MiB/s ETA 00:00
[download]  96.4% of 50.83MiB at 19.68MiB/s ETA 00:00
[download]  96.4% of 50.83MiB at 151.77KiB/s ETA 00:12
[download]  96.4% of 50.83MiB at 443.33KiB/s ETA 00:04
[download]  96.4% of 50.83MiB at 1020.55KiB/s ETA 00:01
[download]  96.5% of 50.83MiB at  2.11MiB/s ETA 00:00
[download]  96.5% of 50.83MiB at  1.64MiB/s ETA 00:01
[download]  96.5% of 50.83MiB at  2.02MiB/s ETA 00:00
[download]  96.7% of 50.83MiB at  2.86MiB/s ETA 00:00
[download]  96.9% of 50.83MiB at  4.22MiB/s ETA 00:00
[download]  97.4% of 50.83MiB at  6.36MiB/s ETA 00:00
[download]  98.4% of 50.83MiB at  9.04MiB/s ETA 00:00
[download] 100.0% of 50.83MiB at 11.49MiB/s ETA 00:00
[download] 100% of 50.83MiB in 00:03
[debug] Invoking downloader on u'https://r6---sn-vgqsrn76.googlevideo.com/videoplayback?expire=1584058327&ei=d3tqXufRL6eAir4PgqKCmAc&ip=73.246.88.217&id=o-AAwhxXZHB4Fu4tAhuuIw0ZPcYz9fvLJ7xErMTQM7Uy_T&itag=140&source=youtube&requiressl=yes&mh=lg&mm=31%2C26&mn=sn-vgqsrn76%2Csn-ab5sznld&ms=au%2Conr&mv=m&mvi=5&pl=15&initcwndbps=1990000&vprv=1&mime=audio%2Fmp4&gir=yes&clen=1591397&dur=98.290&lmt=1583634771771627&mt=1584036643&fvip=5&keepalive=yes&fexp=23842630&c=WEB&txp=2311222&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=ADKhkGMwRgIhALrD5UI9loHyeld_y0SuuQFYh12AetZpHCacXwRNZK23AiEA8u9qKZiX6jtmlcV-wbxjUpQKRh6EotPvR3oIAl_MELE%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=ABSNjpQwRQIhAPHN2n9BGJfaw5fBMMmObucwwEf4I9kHrN1DS8MvgTmUAiBT-3Vt5Mwhl-X-Cgwv9N_D8B2ZpbHIZwEu2pBPMQgcDA%3D%3D&ratebypass=yes'
[download] Destination: /Volumes/Godzilla/Video/YouTube Video/Ryan Murphy posts his fastest time since 2016 _ Mens 100m Back A Final _ 2020 TYR Pro Swim Series-HlB3noj2xsI.f140.m4a
[download]   0.1% of 1.52MiB at 245.94KiB/s ETA 00:06
[download]   0.2% of 1.52MiB at 722.53KiB/s ETA 00:02
[download]   0.5% of 1.52MiB at  1.62MiB/s ETA 00:00
[download]   1.0% of 1.52MiB at  3.42MiB/s ETA 00:00
[download]   2.0% of 1.52MiB at  1.83MiB/s ETA 00:00
[download]   4.1% of 1.52MiB at  1.94MiB/s ETA 00:00
[download]   8.2% of 1.52MiB at  2.91MiB/s ETA 00:00
[download]  16.4% of 1.52MiB at  4.15MiB/s ETA 00:00
[download]  32.9% of 1.52MiB at  6.46MiB/s ETA 00:00
[download]  65.8% of 1.52MiB at  9.74MiB/s ETA 00:00
[download] 100.0% of 1.52MiB at 12.61MiB/s ETA 00:00
[download] 100% of 1.52MiB in 00:00
[ffmpeg] Merging formats into "/Volumes/Godzilla/Video/YouTube Video/Ryan Murphy posts his fastest time since 2016 _ Mens 100m Back A Final _ 2020 TYR Pro Swim Series-HlB3noj2xsI.mp4"
[debug] ffmpeg command line: /usr/local/bin/ffmpeg -y -loglevel 'repeat+info' -i 'file:/Volumes/Godzilla/Video/YouTube Video/Ryan Murphy posts his fastest time since 2016 _ Mens 100m Back A Final _ 2020 TYR Pro Swim Series-HlB3noj2xsI.f137.mp4' -i 'file:/Volumes/Godzilla/Video/YouTube Video/Ryan Murphy posts his fastest time since 2016 _ Mens 100m Back A Final _ 2020 TYR Pro Swim Series-HlB3noj2xsI.f140.m4a' -c copy -map '0:v:0' -map '1:a:0' 'file:/Volumes/Godzilla/Video/YouTube Video/Ryan Murphy posts his fastest time since 2016 _ Mens 100m Back A Final _ 2020 TYR Pro Swim Series-HlB3noj2xsI.temp.mp4'
Deleting original file /Volumes/Godzilla/Video/YouTube Video/Ryan Murphy posts his fastest time since 2016 _ Mens 100m Back A Final _ 2020 TYR Pro Swim Series-HlB3noj2xsI.f137.mp4 (pass -k to keep)
Deleting original file /Volumes/Godzilla/Video/YouTube Video/Ryan Murphy posts his fastest time since 2016 _ Mens 100m Back A Final _ 2020 TYR Pro Swim Series-HlB3noj2xsI.f140.m4a (pass -k to keep)

I attempted to add encoding to the configuration and received an error.

youtube-dl: error: no such option: --encoding-US-ASCII

I don't see a list of acceptable encodings

[dstftw]

This has nothing to do with youtube-dl and its options as ’ is outputted by youtube-dl without any problem.
The problem is somewhere outside. You can try setting up proper UTF-8 locale or restrict filenames to ASCII --restrict-filenames.

[jprokos]

Thanks for the tip. I am new to all of this and I am not sure how to "setup proper UTF-8 locale". Is this within youtube-dl?
I don't know how to manipulate what I am seeing in here:

[debug] Encodings: locale US-ASCII, fs utf-8, out None, pref utf-8

I did figure out how to use the encoding property. For those who follow there is no hyphen required between --encoding and type. After adding --encoding utf-8 to the youtube-dl configuration the missing comma issue seems to have been solved. I think this was the correct forum for getting my question answered—appreciate the help. I found this friendly youtube-dl exchange helpful but incomplete.