Double-quotes causing invalid URL error ?

[section83]

Checklist

• I'm asking a question
• I've looked through the README and FAQ for similar questions
• I've searched the bugtracker for similar questions including closed ones

Question

Context: macOS 10.13, youtube-dl v.2019.09.28 (and earlier – not confirmed in latest)

I am invoking youtube-dl programmatically in AppleScript which sends the command to sh shell. Because of possible special characters in URLs, the code always puts single quotes around URLs. It has been working for 18 months or so. But, on rare occasions, youtube-dl reports an invalid URL. The debug report shows double-quotes around the URL. The debug reports look like this:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--restrict-filenames', u'--ignore-errors', u'--prefer-ffmpeg', u'--no-overwrites', u'--verbose', u'-o', u'%(title)s.%(ext)s', u"'https://www.sbs.com.au/ondemand/video/1214953027556/great-continental-railway-journeys-kiev-to-odessa'"]
[debug] Encodings: locale US-ASCII, fs utf-8, out None, pref US-ASCII
[debug] youtube-dl version 2019.09.28
[debug] Python version 2.7.10 (CPython) - Darwin-17.7.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.2.1-tessus, ffprobe 4.2.1-tessus
[debug] Proxy map: {}
ERROR: u"'https://www.sbs.com.au/ondemand/video/1214953027556/great-continental-railway-journeys-kiev-to-odessa'" is not a valid URL. Set --default-search "ytsearch" (or run youtube-dl "ytsearch:'https://www.sbs.com.au/ondemand/video/1214953027556/great-continental-railway-journeys-kiev-to-odessa'" ) to search YouTube
Traceback (most recent call last):
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info
ie_result = ie.extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 530, in extract
ie_result = self._real_extract(url)
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 2263, in _real_extract
% (url, url), expected=True)
ExtractorError: u"'https://www.sbs.com.au/ondemand/video/1214953027556/great-continental-railway-journeys-kiev-to-odessa'" is not a valid URL. Set --default-search "ytsearch" (or run youtube-dl "ytsearch:'https://www.sbs.com.au/ondemand/video/1214953027556/great-continental-railway-journeys-kiev-to-odessa'" ) to search YouTube

The double-quotes can be seen in the "Command-line args:" line. After encountering this error, I have re-run the script to download the same video and there has been no error.

I have tested various formulations of the youtube-dl command trying to get double-quotes to show as they do in the above log, but failed:

youtube-dl -v https://www.sbs.com.au/ondemand/video/1559892547836/the-last-man-on-earth-alive-in-tucson
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.sbs.com.au/ondemand/video/1559892547836/the-last-man-on-earth-alive-in-tucson']

youtube-dl -v "https://www.sbs.com.au/ondemand/video/1559892547836/the-last-man-on-earth-alive-in-tucson"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.sbs.com.au/ondemand/video/1559892547836/the-last-man-on-earth-alive-in-tucson']

youtube-dl -v 'https://www.sbs.com.au/ondemand/video/1559892547836/the-last-man-on-earth-alive-in-tucson'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.sbs.com.au/ondemand/video/1559892547836/the-last-man-on-earth-alive-in-tucson']

youtube-dl -v ''https://www.sbs.com.au/ondemand/video/1559892547836/the-last-man-on-earth-alive-in-tucson''
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.sbs.com.au/ondemand/video/1559892547836/the-last-man-on-earth-alive-in-tucson']

youtube-dl -v 'https:'//'www.sbs.com.au'/'ondemand'/'video'/'1559892547836'/'the-last-man-on-earth-alive-in-tucson'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.sbs.com.au/ondemand/video/1559892547836/the-last-man-on-earth-alive-in-tucson']

youtube-dl copes extremely well with the variety of formulations but, as mentioned, my code always encloses URLs in single quotes.

I'm not seeking help with my script but, I do need help to understand what could cause the double-quotes to appear in the log and/or whether the double-quotes indicate something else is going on.

Thanks.

[dstftw]

URL string literal must not contain any quotes. Yours contains single quotes.

[section83]

OK many thanks. The youtube-dl/README.md says:

Video URL contains an ampersand and I'm getting some strange output [1] 2839 or 'v' is not recognized as an internal or external command
That's actually the output from your shell. Since ampersand is one of the special shell characters it's interpreted by the shell preventing you from passing the whole URL to youtube-dl. To disable your shell from interpreting the ampersands (or any other special characters) you have to either put the whole URL in quotes or escape them with a backslash (which approach will work depends on your shell).

For example if your URL is https://www.youtube.com/watch?t=4&v=BaW_jenozKc you should end up with following command:

youtube-dl 'https://www.youtube.com/watch?t=4&v=BaW_jenozKc'

I thought I was following that advice. My script uses the third formulation above i.e. I put the whole URL in quotes. Perhaps my mistake is to use quotes in all cases instead of just those where the URL contains an ampersand. I'll try to limit using quotes to the ampersand case.

Cheers.

[dstftw]

Again: this is for passing arguments from shell. You must not use any quotation in string literals in code.