(or RFC 3986 vs WHATWG URL Specification vs the real world)
This document is an attempt to describe where and how RFC 3986 (86), RFC 3987 (87) and the WHATWG URL Specification (TWUS) differ. This might be useful input when trying to interop with URLs on the modern Internet.
"URL" (in the scope of this document) refers to the sequence of characters passed into APIs, sent over the network and passed from machine to machine. It does not necessarily match what typical browsers accept or support in their GUI address bars.
This document focuses on network-using URL schemes (http, https, ftp, etc) as well as 'file'.
This is a best-effort and describes the situation as we understand it at the time of writing. Since the TWUS document is "living" and software keep getting updated, chances are conditions have changed since we last updated this document!
Put simply, a URL may consist of the following components - many of them are optional:
Each component is separated from the following component with a divider character.
Which in an example could look like
|Component||Value||Known interop issues exist|
There are no known interop issues.
86: specifies this to be exactly "://" for all network-using (hierarchical) schemes.
TWUS: says a parser must accept zero to an infinite amount of slashes but a producer should use two.
Real world: one and three slash URLs occur, possibly a few using even more. 'file' URLs are notoriously often malformed.
The userinfo field can be used to set user name and password to pass on to the server. The use of this field is discouraged since it often means passing around the password in plain text and is thus a security risk.
86: specifies that '@' is the separator between the userinfo field and the host name. The first '@' character really.
TWUS: instead takes the last '@' before the host name to be the separator
This is an interop collision
Numerical IP addresses
86: mentions how IPv4 addresses with a dot-notation are valid
TWUS: specifies that both 32bit numbers ("12345677") as well as partial dot-addresses ("127.0") are valid.
Real world: 32 bit numbers occur, and are automagically supported if typical OS level name resolver funcitons are used since they often support this out of the box.
Hostnames were traditionally ASCII based. When introducing IDN hostnames, it has caused problems to the specifications and they are lacking.
86: Is written to work without IDN (ASCII characters), so basically it works with already punycoded domain names.
87: Specifies IDNA 2003 to be used
TWUS: Doesn't specify IDNA 2003 nor 2008, but somehow that's still clear
Real world: A total mess. Some national registries (the german DENIC for example) require IDNA 2008, which makes user-agents treating the host name according to IDNA 2003 and IDNA 2008 TR46/transitional to fail or even to resolve the wrong IP address. Some user-agents use IDNA 2003, some do IDNA 2008 TR46/transitional and some do IDNA 2008 TR46/non-transitional.
This is an interop collision This is a security issue
The port number is a TCP port number between 0 and 65535.
86 and TWUS agree that this is a base-10 number that is virtually unbounded in length. 00000000000000000000000000000000000080 means 80 in both specs.
Real world: at least curl and wget2 ignore "rubbish" entered after the number all the way to the next component divider (a slash, a pound sign, or a question mark). That seems to be a bug according go both 86 and TWUS.
86: says that a URL path is specified as ASCII characters or need to be URL encoded. That makes 8-bit characters illegal.
Real world: 8bit characters are occasionally seen in URLs in the wild, and when used in redirects, browsers are known to URL-encode them in the next outgoing request.
86: does not allow spaces (U+0020) to be part of the path. A space instead ends the URL.
TWUS: allows spaces in URLs and will instead URL-encode it to %20 when sent in a request. A TWUS URL thus needs to end on another character or have another method to know the end.
Real world: Spaces are occasionally seen in URLs in the wild, and when used in redirects, browsers are known to URL-encode them in the next outgoing request.
There are no known interop problems with fragments. The fragment part is not included in protocol data that goes over the network.
TWUS has a test suite: