[CVE-2018-20852] Cookie domain check returns incorrect results

[bobunderson]

BPO	35121
Nosy	@orsenthil, @vstinner, @larryhastings, @benjaminp, @ned-deily, @ambv, @vadmium, @serhiy-storchaka, @miss-islington, @Windsooon, @tirkarthi, @bobunderson, @ret2libc
PRs	bpo-35121: prefix dot in domain for proper subdomain validation #10258 [3.6] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) #12260 [3.7] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) #12261 [3.4] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) #12279 [3.5] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) #12281 [2.7] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) #13426

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/benjaminp'
closed_at = <Date 2019-06-15.16:38:00.513>
created_at = <Date 2018-10-31.06:52:48.946>
labels = ['type-security', '3.8', '3.7', 'library', 'release-blocker']
title = '[CVE-2018-20852] Cookie domain check returns incorrect results'
updated_at = <Date 2019-07-15.09:42:01.659>
user = 'https://github.com/bobunderson'

bugs.python.org fields:

activity = <Date 2019-07-15.09:42:01.659>
actor = 'vstinner'
assignee = 'benjamin.peterson'
closed = True
closed_date = <Date 2019-06-15.16:38:00.513>
closer = 'xtreak'
components = ['Library (Lib)']
creation = <Date 2018-10-31.06:52:48.946>
creator = '\xe8\xa5\xbf\xe7\x94\xb0\xe9\x9b\x84\xe6\xb2\xbb'
dependencies = []
files = []
hgrepos = []
issue_num = 35121
keywords = ['patch', 'security_issue']
message_count = 30.0
messages = ['328973', '328975', '328981', '328985', '329176', '329179', '332299', '332576', '332583', '332920', '335386', '337588', '337590', '337592', '337593', '337598', '337600', '337601', '338109', '338114', '338152', '344555', '344556', '344560', '345689', '345700', '345736', '346748', '346749', '347951']
nosy_count = 13.0
nosy_names = ['orsenthil', 'vstinner', 'larry', 'benjamin.peterson', 'ned.deily', 'lukasz.langa', 'martin.panter', 'serhiy.storchaka', 'miss-islington', 'Windson Yang', 'xtreak', '\xe8\xa5\xbf\xe7\x94\xb0\xe9\x9b\x84\xe6\xb2\xbb', 'rschiron']
pr_nums = ['10258', '12260', '12261', '12279', '12281', '13426']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue35121'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8']

[bobunderson]

http.cookiejar.DefaultPolicy.domain_return_ok returns incorrect results.

So, HTTP clients send cookies which issued from wrong server.

policy = http.cookiejar.DefaultCookiePolicy()
req = urllib.request.Request('https://xxxfoo.co.jp/')
print(policy.domain_return_ok('foo.co.jp', req)   # should be False, but it returns True

[tirkarthi]

The current set of tests are at

cpython/Lib/test/test_http_cookiejar.py

Line 406 in 0353b4e

def test_domain_return_ok(self):

. A simple set of tuple that can be added based on the report as below :

("http://barfoo.com", ".foo.com", False)
("http://barfoo.com", "foo.com", False) # Fails on master

The check is done at

cpython/Lib/http/cookiejar.py

Line 1176 in 0353b4e

if not (req_host.endswith(domain) or erhn.endswith(domain)):

. There is no check to add '.' before domain if absent. Hence it performs a substring match with the values req_host = ".barfoo.com" and erhn = ".barfoo.com" and domain = "foo.com" so the condition not (req_host.endswith(domain) or erhn.endswith(domain)) fails and doesn't return False. I would suggest adding a check to make sure domain also starts with '.' similar to req_host and erhn thus fixing the issue. I tried the fix and existing tests along with the reported case works fine.

diff --git a/Lib/http/cookiejar.py b/Lib/http/cookiejar.py
index 0ba8200f32..da7462701b 100644
--- a/Lib/http/cookiejar.py
+++ b/Lib/http/cookiejar.py
@@ -1173,6 +1173,8 @@ class DefaultCookiePolicy(CookiePolicy):
             req_host = "."+req_host
         if not erhn.startswith("."):
             erhn = "."+erhn
+        if not domain.startswith("."):
+            domain = "."+domain
         if not (req_host.endswith(domain) or erhn.endswith(domain)):
             #_debug("   request domain %s does not match cookie domain %s",
             #       req_host, domain)

("http://barfoo.com", ".foo.com", False)
("http://barfoo.com", "foo.com", False) # Tests pass with fix

Also tried the script attached in the report

$ cat ../backups/bpo35121.py

import urllib
from http.cookiejar import DefaultCookiePolicy

policy = DefaultCookiePolicy()
req = urllib.request.Request('https://xxxfoo.co.jp/')
print(policy.domain_return_ok('foo.co.jp', req))

# without fix

$ ./python.exe ../backups/bpo35121.py
True

# With domain fix

$ ./python.exe ../backups/bpo35121.py
False

The check was added in 2004 with commit 2a6ba90 . If my fix is correct I am willing to raise a PR for this with test.

Hope it helps!

[bobunderson]

I think that is desired result. thanks!

[tirkarthi]

Thanks for the confirmation. I have created a PR (#10258) with test and added 3.8 as affected version. Please add in if I have missed anything in the PR.

[Windsooon]

I wonder https://github.com/python/cpython/blob/master/Lib/test/test_http_cookiejar.py#L420

("http://foo.bar.com/", "com", True),
("http://foo.com/", "com", True),

are expected behavior?

[tirkarthi]

Good catch Windson! I overlooked the tests. There is also a comment that it's liberal in the test function. Since the code was added in 2006 I don't if it's ok broken to fix this or not. I will let the reviewers take a call then. There is also domain_match and user_domain_match that perform more tests.

This is only a rough check for performance reasons, so it's not too critical as long as it's sufficiently liberal.

Thanks for your input!

[tirkarthi]

Looking further into this the domain validation makes it little more stricter and can have wider implications. For example requests library uses cookiejar to maintain cookies between sessions. One more case is that domain can be empty so only non-empty domains can be prefixed with dot.

A simple server that sets Cookie with value A=LDJDSFLKSDJLDSF

import SimpleHTTPServer
import logging

class MyHTTPRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
    def do_GET(self):
        self.cookieHeader = self.headers.get('Cookie')
        SimpleHTTPServer.SimpleHTTPRequestHandler.do_GET(self)

    def end_headers(self):
        self.send_my_headers()
        SimpleHTTPServer.SimpleHTTPRequestHandler.end_headers(self)

    def send_my_headers(self):
        self.send_header('Set-Cookie', 'A=LDJDSFLKSDJLDSF')

if __name__ == '__main__':
    SimpleHTTPServer.test(HandlerClass=MyHTTPRequestHandler)

Add below host entry to /etc/hosts

127.0.0.1 test.com
127.0.0.1 1.test.com
127.0.0.1 footest.com

Sample script to demonstrate requests behavior change

import requests

with requests.Session() as s:
    cookies = dict(cookies_are='working')
    m = s.get("http://test.com:8000", cookies=cookies)
    print(m.request.headers)
    m = s.get("http://1.test.com:8000", cookies=cookies)
    print(m.request.headers)
    m = s.get("http://footest.com:8000", cookies=cookies)
    print(m.request.headers)

Before patch :

{'User-Agent': 'python-requests/2.11.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/', 'Connection': 'keep-alive', 'Cookie': 'cookies_are=working'}
{'User-Agent': 'python-requests/2.11.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/', 'Connection': 'keep-alive', 'Cookie': 'A=LDJDSFLKSDJLDSF; cookies_are=working'}
{'User-Agent': 'python-requests/2.11.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Cookie': 'A=LDJDSFLKSDJLDSF; cookies_are=working'}

After patch :

{'User-Agent': 'python-requests/2.11.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/', 'Connection': 'keep-alive', 'Cookie': 'cookies_are=working'}
{'User-Agent': 'python-requests/2.11.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/', 'Connection': 'keep-alive', 'Cookie': 'A=LDJDSFLKSDJLDSF; cookies_are=working'}
{'User-Agent': 'python-requests/2.11.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Cookie': 'cookies_are=working'}

As with my patch since the cookie is set on test.com while making a request to footest.com the cookie is skipped as part of the patch since footest is not a subdomain of test.com but 1.test.com is a subdomain. This is a behavior change to be decided whether worth doing or to document this since in a client with session like requests module connecting to lot of hosts this can potentially pass cookies of test.com to footest.com. A discussion on requests repo on providing the option for user to set a stricter cookie policy : psf/requests#2576

On testing with curl cookie-jar it seems that the cookies are passed even for the subdomain only when it's set and not as part of top level domain.

[tirkarthi]

Also looking at the docs for different frameworks like Flask and Django they recommend setting Domain attribute only for cross-domain cookies.

From Django docs

Use domain if you want to set a cross-domain cookie. For example, domain="example.com" will set a cookie that is readable by the domains www.example.com, blog.example.com, etc. Otherwise, a cookie will only be readable by the domain that set it.

When there is no domain specified then the frameworks seem to set the cookie only for the host only as per RFC 6265. So domain attribute is set all the time and it's just that if the domain attribute is set explicitly by the server with the set_cookie function in the frameworks then the cookiejar has domain_specified set along with dot prefixed for the domain enabling stricter validations. I don't know about the metrics of setting the domain attribute vs not setting it. Checking with a simple Flask app and set_cookie without domain parameter the cookies are passed to suffix domains. With domain passed to set_cookie has dot prefixed and the cookies are not passed to suffix domains.

I also looked into other implementations

• aiohttp - uses cookiejar but has custom domain checks and update cookie methods at https://github.com/aio-libs/aiohttp/blob/49261c192ff225372dffb39056c3c311714b12c5/aiohttp/cookiejar.py#L141 . Thus it's not affected when tested.
• golang implementation - https://github.com/golang/go/blob/50bd1c4d4eb4fac8ddeb5f063c099daccfb71b26/src/net/http/cookiejar/jar.go#L123

[tirkarthi]

I have come across another behavior change between path checks while using the cookie jar implementation available in Python. This is related to incorrect cookie validation but with respect to path so let me know if this needs a separate ticket. I observed the following difference :

1. Make a request to "/" that sets a cookie with "path=/any"
2. Make a request to "/any" and the set cookie is passed since the path matches
3. Make a request to "/anybad" and cookie with "path=/any" is also passed too.

On using golang stdlib implementation of cookiejar the cookie "path=/any" is not passed when a request to "/anybad" is made. So I checked with RFC 6265 where the path match check is defined in section-5.1.4 . RFC 6265 also obsoletes RFC 2965 upon which cookiejar is based I hope since original implementation of cookiejar is from 2004 and RFC 6265 was standardized later. So I think it's good to enforce RFC 6265 since RFC 2965 is obsolete at least in Python 3.8 unless this is considered as a security issue. I think this is a security issue. The current implementation can potentially cause cookies to be sent to incorrect paths in the same domain that share the same prefix. This is a behavior change with more strict checks but I could see no tests failing with RFC 6265 implementation too.

RFC 2965 also gives a loose definition of path-match without mentioning about / check in the paths based on which Python implementation is based as a simple prefix match.

For two strings that represent paths, P1 and P2, P1 path-matches P2
if P2 is a prefix of P1 (including the case where P1 and P2 string-
compare equal). Thus, the string /tec/waldo path-matches /tec.

RFC 6265 path-match definition : https://tools.ietf.org/html/rfc6265#section-5.1.4

A request-path path-matches a given cookie-path if at least one of
the following conditions holds:

o The cookie-path and the request-path are identical.

o The cookie-path is a prefix of the request-path, and the last
character of the cookie-path is %x2F ("/").

o The cookie-path is a prefix of the request-path, and the first
character of the request-path that is not included in the cookie-
path is a %x2F ("/") character.

The current implementation in cookiejar is as below :

def path_return_ok(self, path, request):
    _debug("- checking cookie path=%s", path)
    req_path = request_path(request)
    if not req_path.startswith(path):
        _debug("  %s does not path-match %s", req_path, path)
        return False
    return True

Translating the RFC 6265 steps (a literal translation of go implementation) would have something like below and no tests fail on master. So the python implementation goes in line with the RFC not passing cookies of "path=/any" to /anybody

def path_return_ok(self, path, request):
    req_path = request_path(request)
    if req_path == path:
        return True
    elif req_path.startswith(path) and ((path.endswith("/") or req_path[len(path)] == "/")):
        return True
        
    return False

The golang implementation is as below which is a literal translation of RFC 6265 steps at https://github.com/golang/go/blob/50bd1c4d4eb4fac8ddeb5f063c099daccfb71b26/src/net/http/cookiejar/jar.go#L130

// pathMatch implements "path-match" according to RFC 6265 section 5.1.4.
func (e *entry) pathMatch(requestPath string) bool {
if requestPath == e.Path {
return true
}
if strings.HasPrefix(requestPath, e.Path) {
if e.Path[len(e.Path)-1] == '/' {
return true // The "/any/" matches "/any/path" case.
} else if requestPath[len(e.Path)] == '/' {
return true // The "/any" matches "/any/path" case.
}
}
return false
}

Feel free to correct me if I am wrong on the above.

[tirkarthi]

I have opened bpo-35647 for path related checks as a separate report.

[tirkarthi]

This issue affects 2.7 as well along with 3.4 and 3.5. The initial report was notified to security@python.org . 2.7.16 release candidate dates were announced at https://mail.python.org/pipermail/python-dev/2019-February/156266.html. I have prepared an initial backport of this with tests for 2.7 at 2.7...tirkarthi:bpo-35121-27 . Serhiy has approved the PR for master. I have added notes here and on the PR about the issue and implementation in other languages. It would be helpful if someone can double check my analysis since cookiejar has not received much change over the years.

If this is a potential candidate for 2.7 release I can help with that once the changes are merged to master. Adding Benjamin Peterson to this issue to take a call on if it needs to be backported to 2.7. If it's planned for a backport then also to decide on priority if this needs to be part of 2.7.16 or later release.

[ned-deily]

New changeset ca7fe50 by Ned Deily (Xtreak) in branch 'master':
bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258)
ca7fe50

[ned-deily]

New changeset e5123d8 by Ned Deily (Miss Islington (bot)) in branch '3.7':
bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) (GH-12261)
e5123d8

[ned-deily]

New changeset b241af8 by Ned Deily (Miss Islington (bot)) in branch '3.6':
bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) (GH-12260)
b241af8

[ned-deily]

OK, thanks for the initial report and a big thank you to @XTreak for the PR and following up on things. The PR is now merged to master for 3.8.0 and backported as a security fix for release in 3.7.3 and 3.6.9. Reassigning to Benjamin for a decision on backporting to 2.7.

[serhiy-storchaka]

What about 3.4 and 3.5?

[tirkarthi]

From my initial tests 3.4 and 3.5 were also affected. 3.4 is going EoL and RC1 is out but there is one another security issue (bpo-36216) fixed last week with a PR open. If the merge window is open and Larry is okay then I can raise backport PRs if needed. There are less changes made to cookiejar and cherry_picker would also work fine as I tried it locally.

cherry_picker --no-push ca7fe50 3.5
🐍 🍒 ⛏

Now backporting 'ca7fe5063593958e5efdf90f068582837f07bd14' into '3.5'
Switched to a new branch 'backport-ca7fe50-3.5'
Branch 'backport-ca7fe50-3.5' set up to track remote branch '3.5' from 'upstream'.

[backport-ca7fe50-3.5 fcb2dd85a0] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258)
Author: Xtreak <tir.karthi@gmail.com>
Date: Sun Mar 10 07:39:48 2019 +0530
3 files changed, 45 insertions(+), 2 deletions(-)
create mode 100644 Misc/NEWS.d/next/Security/2018-10-31-15-39-17.bpo-35121.EgHv9k.rst

Finished cherry-pick ca7fe50 into backport-ca7fe50-3.5 😀

cherry_picker --no-push ca7fe50 3.4
🐍 🍒 ⛏

Now backporting 'ca7fe5063593958e5efdf90f068582837f07bd14' into '3.4'
Switched to a new branch 'backport-ca7fe50-3.4'
Branch 'backport-ca7fe50-3.4' set up to track remote branch '3.4' from 'upstream'.

Performing inexact rename detection: 100% (639108/639108), done.
[backport-ca7fe50-3.4 46ea57d6b3] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258)
Author: Xtreak <tir.karthi@gmail.com>
Date: Sun Mar 10 07:39:48 2019 +0530
3 files changed, 45 insertions(+), 2 deletions(-)
create mode 100644 Misc/NEWS.d/next/Security/2018-10-31-15-39-17.bpo-35121.EgHv9k.rst

Finished cherry-pick ca7fe50 into backport-ca7fe50-3.4 😀

[tirkarthi]

There are many libraries that use DefaultCookiePolicy and requests library uses it for client where session state needs to be maintained across different requests. Currently, requests doesn't have a documented API to change to cookiejar policy and were not keen on introducing a custom one since this might introduce maintenance burden over keeping it in sync with other changes when made upstream. The team have been informed about this when the issue was created and I also updated the maintainers now about the fix being merged since it's a highly popular library.

So requests will remain affected when ran on versions where this patch is not available in CPython standard library as of now. A potentially simple workaround in the absence of patch on affected versions is to set DomainStrict in the cookiepolicy that would make sure a literal match against domain is made at [0] . The disadvantage I guess would be that cookie set on example.com would not be shared with subdomain which might break workflow. aio-http was not affected since it uses custom cookiejar policy. scrapy also seems to be not affected since they custom policies. Most of the web frameworks don't recommend setting domain explicitly and set them implicitly so it can be reproduced in the default setup of frameworks and Flask was the one I tested which makes me assume this could be easily exploited.

[0]

cpython/Lib/http/cookiejar.py

Line 1158 in ca7fe50

(self.strict_ns_domain & self.DomainStrictNonDomain) and

[larryhastings]

New changeset 42ad410 by larryhastings (Xtreak) in branch '3.4':
[3.4] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) (bpo-12279)
42ad410

[larryhastings]

New changeset 4749f1b by larryhastings (Xtreak) in branch '3.5':
[3.5] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) (bpo-12281)
4749f1b

[tirkarthi]

Larry, I am reopening this since this seems to affects 2.7 and would wait for Benjamin's call on backporting this.

[vstinner]

I added this issue to my security website:
https://python-security.readthedocs.io/vuln/cookie-domain-check.html

So it's fixed in Python 3.4.10, 3.5.7 and 3.7.3. Right now, 2.7 and 3.6 are vulnerable (but 3.6 branch is fixed).

[vstinner]

Can someone try to backport the fix to Python 2.7?

[tirkarthi]

Can someone try to backport the fix to Python 2.7?

The backport to 2.7 PR 13426 is open. It would be helpful if someone can review it. I am not sure of the commit review process and who needs to review and approve it since this is assigned to Benjamin Peterson. It's a fairly straightforward backport except that http.cookiejar is cookielib in Python 2.

[miss-islington]

New changeset 979daae by Miss Islington (bot) (Xtreak) in branch '2.7':
[2.7] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) (GH-13426)
979daae

[tirkarthi]

Closing this as resolved since the fix was merged to all branches. Thank you all.

[vstinner]

Again, well done Karthikeyan Singaravelan!

[ret2libc]

Did anybody request a CVE for this issue? I think it deserves one as it is a security issue and it may leak cookies to wrong domains. Does anybody have anything against assigning a CVE to this issue? If not, I would try to get one from MITRE.

[tirkarthi]

I also reported it to security@python.org . Please check with them too to see if there is a CVE request already made. Thanks.

[ret2libc]

CVE-2018-20852 has been assigned to this flaw.