LWPCookieJar load() set domain_specifed wrong

[BKyven]

BPO	17251

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-02-20.06:48:33.664>
labels = ['3.8', 'type-bug', 'library', '3.9', '3.10']
title = 'LWPCookieJar load() set domain_specifed wrong'
updated_at = <Date 2020-11-06.22:57:48.373>
user = 'https://bugs.python.org/BKyven'

bugs.python.org fields:

activity = <Date 2020-11-06.22:57:48.373>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2013-02-20.06:48:33.664>
creator = 'B. Kyven'
dependencies = []
files = []
hgrepos = []
issue_num = 17251
keywords = ['patch']
message_count = 8.0
messages = ['182476', '183068', '183099', '183117', '183123', '183686', '183775', '183880']
nosy_count = 2.0
nosy_names = ['maxy@debian.org', 'B. Kyven']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17251'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[BKyven]

Hello,

I am using LWPCookieJar to store cookies. But I am having trouble.
Saving is fine, load is wrong. I use Cookie.domain_specified to judge if domain exist.

save the following to test.lwp
-----------------
#LWP-Cookies-2.0
Set-Cookie3: name=value; path="/ddd/"; domain=".domain.com"; path_spec; domain_dot; secure; expires="2030-05-09 14:25:11Z"; version=0
Set-Cookie3: name=value; path="/ddd/"; domain="www.domain.com"; path_spec; secure; expires="2030-05-09 14:25:11Z"; version=0
-----------------

cj = LWPCookieJar('test.lwp').load()
for c in cj:
print c.domain, c.domain_specified, c.domain_initial_dot

output:
.domain.com True True
www.domain.com **False** True

If understood correctly, domain_specified should equal bool(c.domain ="").

This is seen on 2.7 and 2.6.

[maxydebianorg]

Hi,

This is still present in the current mercurial.

I'm attaching a patch that fixes the issue.

Thanks.

[demianbrecht]

According to some digging around that I've done, this issue may be invalid:

(I couldn't find an RFC or detailed spec of the LWP format, so reading from libwww-perl source @ http://cpansearch.perl.org/src/GAAS/libwww-perl-5.836/lib/HTTP/Cookies.pm)

# Try with a more general domain, alternately stripping
# leading name components and leading dots. When this
# results in a domain with no leading dot, it is for
# Netscape cookie compatibility only:

# a.b.c.net Any cookie
# .b.c.net Any cookie
# b.c.net Netscape cookie only
# .c.net Any cookie

So, www.domain.com is not a valid LWP domain and therefore, unless I'm missing something, the module is functioning as expected.

[demianbrecht]

That was silly of me. What I /meant/ to say was that, for this specific report, it's functioning as expected. However, the logic in LWPCookieJar isn't entirely correct. As noted in the comments from libwww-perl, the reported URL is in fact, an invalid LWP cookie. What's missing is the logic to deal with other, valid cookies.

domain_specified = domain.starts_with('.') is incorrect as a four part domain name (a.b.c.d) /is/ a valid LWP domain.

This should likely be patched.

Another question that I have though, is why is LWPCookieJar even part of the stdlib? It's relatively well documented that it is not known to be compatible with any browser. I'm curious as to how heavily used it is and what the rational was to include it (dev might be a better place to ask this, I'm not sure).

[maxydebianorg]

I've deleted my previous patch, as I found the code working as intended.

The domain_specified signals whether the domain stores came from a Domain: tag inside a Set-Cookie request or is taken from the hostname of the request.

The rfc2965 dictates that a value taken from a Domain: tag should be
prepended with a "." if the values doesn't include it. Once stored in a LWPCookieJar the same logic is used to signal if the domain_specified is true or false. Thus the observed behaviour.

The LWP-Cookies-2.0 format is an extension to the perl format, that seeks compatibility adding some features.

About the domain matching, the rfc2965 documents this. I think the perl comment is an example for a.b.c.net, so that matchs with .b.c.net but not with b.c.net.

[BKyven]

I now realized LWPCookieJar is a subclass of CookieJar but it behaves differently. I believe there are other quirks I haven't discovered, like expire=None which cause exception in LWPCookieJar, but works fine for CookieJar. Sadly the doc didn't mention them.

The official python document introduced 2 file cookiejar, one is Mozilla's FileCookieJar, which is explicitly advised as depricated.
LWPCookieJar which is human-readable seems the only advisable chooice to me. But it turns out not that simple.
My guess is that not many people use this module, or the expire=None problem and this dot question should be quite easy to spot on google.

So forks, could your suggest how does other python user deal with cookie storage. Do they just pickle the cookie objects and save to file?
Or any via 3rd party cookie batteries? What's their pro/cons ?

LWPCookie's usage is strage to me and ,I believe, many average users, I guess many people like me will be willing to know alternatives.

[demianbrecht]

@b. Kyven: What are you trying to achieve?

LWP is intended to be used with libwww-perl libary, which is not known to be compatible with any browsers (not sure whether or not this has any bearing on what you're doing).

Really, IMHO, this entire module is in need of a whole lot of love. Unfortunately, I don't think there's much interest in it, so I'm unsure of whether or not that love will happen any time soon.

https://github.com/jjlee/mechanize looks like it has its own implementation of the MozillaCookieJar (extending on the FileCookieJar). If/how it differs from the stdlib implementation I'm not sure.

[BKyven]

@demian Brecht
Um, I do realize the lack of popularity of this module, now.

What I try to achieve is simple. store persistent cookies in a way, that's told to be standard in python ?

Actually, I was trying to sync QtCookiesJar to CookieJar to make urllib2 works with cookie sessions opened in QtWebKit to skip the heavy javascript powered login process.(now it works either way not both) And I need a way to save persistent cookie. Maybe I need to rethink my strategy. Any thoughts?