Link header parsing breaks if attribute value has commas

[hariharshankar]

The function parse_header_links does not parse link headers correctly if the attribute value for a link has commas or semi-colons in them.
https://github.com/kennethreitz/requests/blob/master/requests/utils.py#L562

The Memento protocol (RFC 7089) allows HTTP datetime in the link attributes and the link header parser cannot parse this reliably.
For example:

>>> r = requests.get('http://mementoweb.org/timegate/http://www.google.com', allow_redirects=False)
>>> r.headers['link']
"""<http://www.google.com>;rel="original" ,
<http://mementoweb.org/timemap/link/1/http://www.google.com>;rel="timemap"; type="application/link-format",
<http://mementoweb.org/timegate/http://www.google.com>;rel="timegate",
<http://web.archive.org/web/20140911065756/http://www.google.com/>;rel="memento"; datetime="Thu, 11 Sep 2014 06:57:56 GMT",
<http://web.archive.org/web/20131004112325/http://www.google.com/>;rel="memento first"; datetime="Sat, 04 Oct 1997 22:20:27 GMT",
<http://web.archive.org/web/20121011204401/https://www.google.com/>;rel="memento last"; datetime="Sat, 11 Oct 2014 10:05:51 GMT""""
>>> r.links
{'04 Oct 1997 22:20:27 GMT': {'url': '04 Oct 1997 22:20:27 GMT'},
 '11 Oct 2014 10:05:51 GMT': {'url': '11 Oct 2014 10:05:51 GMT'},
 '11 Sep 2014 06:57:56 GMT': {'url': '11 Sep 2014 06:57:56 GMT'},
 'memento': {'datetime': 'Thu',
  'rel': 'memento',
  'url': 'http://web.archive.org/web/20140911065756/http://www.google.com/'},
 'memento first': {'datetime': 'Sat',
  'rel': 'memento first',
  'url': 'http://web.archive.org/web/20131004112325/http://www.google.com/'},
 'memento last': {'datetime': 'Sat',
  'rel': 'memento last',
  'url': 'http://web.archive.org/web/20121011204401/https://www.google.com/'},
 'original': {'rel': 'original', 'url': 'http://www.google.com'},
 'timegate': {'rel': 'timegate',
  'url': 'http://mementoweb.org/timegate/http://www.google.com'},
 'timemap': {'rel': 'timemap',
  'type': 'application/link-format',
  'url': 'http://mementoweb.org/timemap/link/1/http://www.google.com'}}

There are third party link header parsers in the URLs below, but it would be very convenient if the requests' parse_header_links function could parse correctly.

https://bitbucket.org/azaroth42/linkheaderparser/src/c2321bf3349b94a12a37ed8c41d4e4785006ada7/parse_link.py?at=default
https://gist.github.com/mnot/210535

[Lukasa]

Yeah, we're aware of this and have had an open issue in the past for it. It's not high-priority work for us, so I'm tagging it as 'Contributor Friendly' and leaving it open for others.

If someone finds this, it was last looked at in part of #1980.

[sigmavirus24]

Actually your bug report is inaccurate and a bit misleading. You'll note that we accurately (and correctly) parse Link headers that conform to RFC 5988 (and only have one link per relation). You can verify this with your own example, we parse:

<http://mementoweb.org/timegate/http://www.google.com>;rel="timegate",

into

 'timegate': {'rel': 'timegate',
  'url': 'http://mementoweb.org/timegate/http://www.google.com'},

and

<http://mementoweb.org/timemap/link/1/http://www.google.com>;rel="timemap"; type="application/link-format",

into

 'timemap': {'rel': 'timemap',
  'type': 'application/link-format',
  'url': 'http://mementoweb.org/timemap/link/1/http://www.google.com'}

Memento (RFC 7089) is an extension of 5988 (or so it would seem) and I'm not entirely convinced that we're required to support it. This also doesn't seem related to #1980 since this isn't in relation to having multiple links for a single relation.

That said, I agree our parsing could be better here (e.g., we shouldn't create entries in links for dates). We should parse out what we are supposed to [1] and only that.

1: "supposed to" meaning what is supported in 5988 (on a quick scan I don't see support for datetimes but I think we're already sort of parsing them and I can think of a fix for this)

[hariharshankar]

My bug report indeed focused on problems in parsing attribute values that contain commas and semicolons. Having looked at https://github.com/kennethreitz/requests/pull/1980 , I now see that there are several reasons why Link headers that are used in the Memento protocol will not be parsed correctly by the Requests Link header parser, as they often contain:

[a] Multiple links with the same relation type
[b] Multiple relation types for a link
[c] Links with extension attributes

A Memento Link header that exemplifies all three is shown in the Memento RFC at http://www.mementoweb.org/guide/rfc/#Pattern1.2

[a], [b], and [c] are fully compliant with RFC 5988 (Web Linking). Mark Nottingham, author of RFC 5988 was actually a reviewer of the Memento RFC on behalf of the IETF. While [a] and [b] are
straightforward uses of RFC 5988, admittedly, [c] builds on the extension mechanism provided by the BNF of section "5. The Link Header Field" of RFC 5988. In addition, in Memento, these extension
attributes are used to provide HTTP datetime values, which unfortunately contain a comma. But since Memento is all about resources and time, and since it is an extension of HTTP, there was no
way to avoid this in the protocol specification. Note, BTW, that the use of commas and semicolons in attributes is legitimate in RFC 5988 as long as the attribute values are quoted.

It seems beneficial to the community to make the Requests Link header parser compliant with RFC 5988. Once that is done, Link headers as used in the Memento protocol will also parse correctly. I am willing to provide help to achieve that goal.

[sigmavirus24]

@hariharshankar if you can solve the sole issue here without also addressing #1980 that will be fine. If you can fix the parsing so extensions of 5988 to handle case [c], then that's great. As you can tell from #1980 cases [a] and [b] are not cases we're willing to tackle as a core part of the library.

[mikecool1000]

Simple fix:
change the line:
for val in value.split(","):
to:
for val in re.split(",\ *<",value):

I have tested this on the given cases with success