Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

macos https url request with http proxy set results in http proxy being used to tunnel https traffic even when no https proxy is configured #26387

Open
5 of 6 tasks
grooveuser opened this issue Aug 20, 2020 · 0 comments · May be fixed by #29371

Comments

@grooveuser
Copy link

Checklist

  • I'm reporting a broken site support issue
  • I've verified that I'm running youtube-dl version 2020.07.28
  • I've checked that all provided URLs are alive and playable in a browser
  • I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • I've searched the bugtracker for similar bug reports including closed ones
  • I've read bugs section in FAQ

Verbose log

output is normal if proxy is running. youtube-dl gets an error if proxy is not running
both are pasted below

verbose output 

hostname:~user$ youtube-dl -v https://www.youtube.com/watch?v=LuN24Dd47Zw
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.youtube.com/watch?v=LuN24Dd47Zw']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.07.28
[debug] Python version 3.8.5 (CPython) - macOS-10.15.6-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {'http': 'http://127.0.0.1:3128', 'https': 'http://127.0.0.1:3128'}
[youtube] LuN24Dd47Zw: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on 'https://r5---sn-vgqsrnek.googlevideo.com/videoplayback?expire=1597924586&ei=ihA-X6atGsXSDOmhltgB&ip=64.46.159.200&id=o-AHASGxmyRfGKBUxrQqKYbz2srs6MpApRsStkcX4VAb-w&itag=136&aitags=133%2C134%2C135%2C136%2C160%2C242%2C243%2C244%2C247%2C278&source=youtube&requiressl=yes&mh=SN&mm=31%2C26&mn=sn-vgqsrnek%2Csn-q4fl6nez&ms=au%2Conr&mv=u&mvi=5&pl=24&vprv=1&mime=video%2Fmp4&gir=yes&clen=107048255&dur=410.475&lmt=1542194436436665&mt=1597902352&fvip=5&keepalive=yes&c=WEB&txp=5432432&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhAPThXLMlsQ2lOGob_XxOnLvk6RvVO_1lqLNfA3lIaDCQAiA8wK10drpv-krtfwPFKv3S_bLi47r0LYQy_YVIemWjew%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl&lsig=AG3C_xAwRQIgVUKMR3xpM3zTv6om4mdGJFHe6B2REXXEY2q-hKCITxgCIQDpPwCJyd89r4o-9Nd7WM2XzIVJYQb_LAbZLjgTwnEgjQ%3D%3D&ratebypass=yes'
[download] Destination: Shooglenifty - Da Eye Wifey-LuN24Dd47Zw.f136.mp4
[download]  10.4% of 102.09MiB at  1.60MiB/s ETA 00:57^C
ERROR: Interrupted by user

without proxy running:

hostname:~user$ youtube-dl -v https://www.youtube.com/watch?v=LuN24Dd47Zw
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.youtube.com/watch?v=LuN24Dd47Zw']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.07.28
[debug] Python version 3.8.5 (CPython) - macOS-10.15.6-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {'http': 'http://127.0.0.1:3128', 'https': 'http://127.0.0.1:3128'}
[youtube] LuN24Dd47Zw: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [Errno 61] Connection refused> (caused by URLError(ConnectionRefusedError(61, 'Connection refused')))
  File "/opt/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/opt/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2238, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/opt/local/bin/youtube-dl/youtube_dl/utils.py", line 2734, in https_open
    return self.do_open(functools.partial(
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py", line 1353, in do_open
    raise URLError(err)

output of scutil:

hostname:~user$ scutil --proxy
<dictionary> {
  ExceptionsList : <array> {
    0 : *.local
    1 : 169.254/16
  }
  FTPPassive : 1
  HTTPEnable : 1
  HTTPPort : 3128
  HTTPProxy : 127.0.0.1
  HTTPSEnable : 0
}


Description

under macos, an https url request when an http proxy is set results in the http proxy being used to tunnel https traffic even when no https proxy is configured.

under macos (specifically 10.15.6 and 10.15.5) when using youtube-dl to retrieve an https url, the http proxy is used if set.
no https proxy is configured or set, as verified via system preferences and scutil
an http proxy (squid) is set, and running on localhost:3128

when the following command is issued:
youtube-dl https://www.youtube.com/watch?v=LuN24Dd47Zw (or any other youtube video)
an https TUNNEL is issued through the http proxy running on localhost:3128
this happens for any video. I haven't tried anything except youtube so far. (but I will...)

I've dug through the youtube extractor, and this doesn't seen to be an issue there.
I'll dig through the cli parser and see if I can pinpoint why/where this is happening.
(note that I'm not certain this is a youtube-dl issue, just that it is specifically happening with youtube-dl, and not, for example, wget, curl, or various other python scripts I've tried. looking at the python errors, I suspect it may be a python urllib/request issue.)
"[debug] Proxy map: {'http': 'http://127.0.0.1:3128', 'https': 'http://127.0.0.1:3128'}" is particularly suspicious.

I don't think this should be happening.

@ghost ghost linked a pull request Jun 23, 2021 that will close this issue
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant