urllib.request.Request.set_proxy doesn't (necessarily) replace type

[tipabu]

BPO	34698
Nosy	@orsenthil, @tipabu

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-09-15.17:17:31.863>
labels = ['3.8', 'library']
title = "urllib.request.Request.set_proxy doesn't (necessarily) replace type"
updated_at = <Date 2019-04-10.12:38:01.516>
user = 'https://github.com/tipabu'

bugs.python.org fields:

activity = <Date 2019-04-10.12:38:01.516>
actor = 'cheryl.sabella'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-09-15.17:17:31.863>
creator = 'tburke'
dependencies = []
files = []
hgrepos = []
issue_num = 34698
keywords = []
message_count = 1.0
messages = ['325449']
nosy_count = 2.0
nosy_names = ['orsenthil', 'tburke']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue34698'
versions = ['Python 3.8']

[tipabu]

Not sure if this is a documentation or behavior bug, but... the docs for urllib.request.Request.set_proxy (https://docs.python.org/3/library/urllib.request.html#urllib.request.Request.set_proxy) say

Prepare the request by connecting to a proxy server. *The host and type will replace those of the instance*, and the instance’s selector will be the original URL given in the constructor.

(Emphasis mine.) In practice, behavior is more nuanced than that:

>>> from urllib.request import Request
>>> req = Request('http://hostame:port/some/path')
>>> req.host, req.type
('hostame:port', 'http')
>>> req.set_proxy('proxy:other-port', 'https')
>>> req.host, req.type # So far, so good...
('proxy:other-port', 'https')
>>>
>>> req = Request('https://hostame:port/some/path')
>>> req.host, req.type
('hostame:port', 'https')
>>> req.set_proxy('proxy:other-port', 'http')
>>> req.host, req.type # Type doesn't change!
('proxy:other-port', 'https')

Looking at the source (https://github.com/python/cpython/blob/v3.7.0/Lib/urllib/request.py#L397) it's obvious why https is treated specially.

The behavior is consistent with how things worked on py2...

>>> from urllib2 import Request
>>> req = Request('http://hostame:port/some/path')
>>> req.get_host(), req.get_type()
('hostame:port', 'http')
>>> req.set_proxy('proxy:other-port', 'https')
>>> req.get_host(), req.get_type()
('proxy:other-port', 'https')
>>>
>>> req = Request('https://hostame:port/some/path')
>>> req.get_host(), req.get_type()
('hostame:port', 'https')
>>> req.set_proxy('proxy:other-port', 'http')
>>> req.get_host(), req.get_type()
('proxy:other-port', 'https')

... but only if you're actually inspecting host/type along the way!

>>> from urllib2 import Request
>>> req = Request('https://hostame:port/some/path')
>>> req.set_proxy('proxy:other-port', 'http')
>>> req.get_host(), req.get_type()
('proxy:other-port', 'http')

(FWIW, this came up while porting an application from py2 to py3; there was a unit test expecting that last behavior of proxying a https connection through a http proxy.)