URL Parse Split incorreclty interpret proxy pw, which include '?'

[NamUKF]

Hi,

I have exported the env var for http proxy, where the Username and password is included in the URL.
The password of the user includes an '?', which is the reason, why the URL string is not correctly processed.

The URL
http_proxy='http://<username>:<pass_with_?>@<server>:<port>'

When the method splits:

cpython/Lib/urllib/parse.py

Line 477 in 53d9cd9

if '?' in url:

It splits at the ? of the password.

Therefore resulting into

 File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 950, in proxy_info_from_url
    port = int(port)
ValueError: invalid literal for int() with base 10: 'p'

if the URL would be:

http://user:p?assword@server:port

Regards
Nam

[theWiseAman]

@NamUKF I think you can encode ? as %3F in the url.
So it would look like this:-
Before encoding:- http://user:p?assword@server:port
After encoding:- http://user:p%3Fassword@server:port

Hope this helps!

[takis]

Hi @NamUKF,

I was curious as to how other languages/runtimes would handle this, so I tried something similar in Javascript, Go, Ruby and PHP. They reject the URL which made me curious as to whether including the password is even considered valid.

So I searched for an RFC on URLs, and found the 1994 RFC1738, which notes:

The use of URLs containing passwords that should be secret is clearly
unwise.
So, unwise, but not invalid?

According to the 1994 W3 spec (which appears to be derived from RFC1738) the password part of a URL apparently can only contain alphanumeric characters, the minus, underscore, dot and plus sign:

password alphanum2 [ password ]
alphanum2 alpha | digit | - | _ | . | +
alpha a | b | c | d | e | f | g | h | i | j | k |
l | m | n | o | p | q | r | s | t | u | v |
w | x | y | z | A | B | C | D | E | F | G |
H | I | J | K | L | M | N | O | P | Q | R |
S | T | U | V | W | X | Y | Z
digit 0 |1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
So, it seems invalid.

The more recent (2005) RFC3986 on URIs states in section 7.5:

URI producers should not provide a URI that contains a username or
password that is intended to be secret. URIs are frequently
displayed by browsers, stored in clear text bookmarks, and logged by
user agent history and intermediary applications (proxies). A
password appearing within the userinfo component is deprecated and
should be considered an error (or simply ignored) except in those
rare cases where the 'password' parameter is intended to be public.

WHATWG states that the password should be an ASCII string:

A URL’s password is an ASCII string identifying a password. It is initially the empty string.

Furthermore, it shows a table in section 4. URLs with URL examples in which a URL including a username and password is marked as being invalid: https://user:password@example.org/

But then it does refer to parsing the password from a URL in section 4.4 URL Parsing:

To set the password given a url and password, set url’s password
to the result of running UTF-8 percent-encode on password
using the userinfo percent-encode set.

The examples in various other programming languages/runtimes:

Javascript snippet:

const url = new URL('http://user:p?assword@example.com');
console.log(url.password);

Node v18.13.0:

node:internal/url:565
  throw new ERR_INVALID_URL(input);
  ^

TypeError [ERR_INVALID_URL]: Invalid URL
    at new NodeError (node:internal/errors:400:5)
    at URL.onParseError (node:internal/url:565:9)
    at new URL (node:internal/url:645:5)
    at Object.<anonymous> (/home/ic/src/js/url/main.js:1:13)
    at Module._compile (node:internal/modules/cjs/loader:1218:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1272:10)
    at Module.load (node:internal/modules/cjs/loader:1081:32)
    at Module._load (node:internal/modules/cjs/loader:922:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:23:47 {
  input: 'http://user:p?assword@example.com',
  code: 'ERR_INVALID_URL'
}

Go snippet:

package main
import (
	"fmt"
	"net/url"
)
func main() {
	s := "http://user:p?assword@www.example.com/"
	u, err := url.Parse(s)
	if err != nil {
		panic(err)
	}
	p, _ := u.User.Password()
	fmt.Println(p)
}

Go output:

panic: parse "http://user:p?assword@www.example.com/": invalid port ":p" after host

Ruby snippet:

require 'uri'
uri = URI.parse('http://user:pa?ssword@www.example.com/')
v = uri.password
print(v)

Output using Ruby 3.1:

/usr/lib/ruby/3.1.0/uri/rfc3986_parser.rb:67:in `split': bad URI(is not URI?): "http://user:pa?ssword@www.example.com/" (URI::InvalidURIError)
	from /usr/lib/ruby/3.1.0/uri/rfc3986_parser.rb:72:in `parse'
	from /usr/lib/ruby/3.1.0/uri/common.rb:188:in `parse'
	from main.rb:3:in `<main>'

PHP snippet:

<?php
$url = "http://user:p?assword@example.com";
$result = parse_url($url);
var_dump($result);
?>

Output using PHP 8.2:

bool(false)

[NamUKF]

Hi

thank you for your answers,

i tried the percent-encoding, both answer mentioned, but it still does run into errors, and isn't able to parse the URL. I actually don't think i have any other choice, than to use username + pw, for the proxy, because in this case add-apt-repository, doesn't give me any other option of adding the env vars http(s)_proxy.