Cookies with "Domain=localhost" aren't getting stored in session file

[pluttrell]

If I use httpie to make a call that returns cookies to a localhost address, such as:

http --session=./session.json POST http://localhost:9081/cookieTest

which sends back a header like this:

Set-Cookie: c1=test; Max-Age=298; Expires=Tue, 15-Aug-2017 19:17:58 GMT; Domain=localhost; Path=/; HttpOnly

but the session.json only includes the following:

{
    "__meta__": {
        "about": "HTTPie session file",
        "help": "https://httpie.org/docs#sessions",
        "httpie": "0.9.9"
    },
    "auth": {
        "password": null,
        "type": null,
        "username": null
    },
    "cookies": {},
    "headers": {
        "Accept": "application/json, */*"
    }
}

If I try this exact same process with the same code hosted at somedomain.com, it works perfectly. For example:

http --session=./session.json POST http://somedomain.com/cookieTest

which sends back a header like this:

Set-Cookie: c1=test; Max-Age=298; Expires=Tue, 15-Aug-2017 19:17:58 GMT; Domain=somedomain.com; Path=/; HttpOnly

and the session.json includes the following:

{
    "__meta__": {
        "about": "HTTPie session file",
        "help": "https://httpie.org/docs#sessions",
        "httpie": "0.9.9"
    },
    "auth": {
        "password": null,
        "type": null,
        "username": null
    },
    "cookies": {
        "c1": {
            "expires": 1502824948,
            "path": "/",
            "secure": false,
            "value": "test"
        }
    },
    "headers": {
        "Accept": "application/json, */*"
    }
}

Is there any way to get it to work with localhost? Or is this a bug?

[makarchuk]

It seems to be a behavior inherited from requests. Alsocookie is being recognized and parsed if you don't setDomain, but not if domain is explicitly set to localhost.

[jkbrzt]

Related #143

[quinncomendant]

I just ran into the same problem (httpie not saving cookie in session when domain is set to localhost).

Is there a workaround within httpie?

The only way I can fix it is by not sending a domain in the cookie when serving from localhost:

$domain = (getenv('HTTP_HOST') == 'localhost' ? null : getenv('HTTP_HOST'));

[Ousret]

'localhost' isn't valid for cookies. A valid domain name should be set.
HTTPie use (internally) a standard implementation for Cookies (CookieJar).

See https://curl.se/rfc/cookie_spec.html for more information. Short story, localhost is not a valid hostname.

What I would suggest is to avoid using "localhost" and rather set a valid domain like "project.dev" in your hosts file.

Hope that clarify.

regards,

edit: answer is partially wrong, see bellow for more.

[quinncomendant]

Hi Ousret,

The Netscape cookie spec doesn't mention localhost. The IETF RFC 6265 that supersedes it also doesn't mention localhost, but it does go into more detail regarding how the Domain in a Set-Cookie header should be handled by user agents:

• First of all, in section 5.2: “[…] the user agent MAY ignore the Set-Cookie header field in its entirety.” So, if httpie wants to ignore or block a cookie for any reason, it may.
• There are two parts of the spec related to the definition of valid domains:

  • Section 4.1.2.3 refers to the requirements for servers using the Set-Cookie header. Related to this issue is this note:

    For security reasons, many user agents are configured to reject Domain attributes that correspond to "public suffixes". For example, some user agents will reject Domain attributes of "com" or "co.uk". (See Section 5.3 for more information.)

  • Section 5.3.5 goes into more detail:

    A "public suffix" is a domain that is controlled by a public registry, such as "com", "co.uk", and "pvt.k12.wy.us". This step is essential for preventing attacker.com from disrupting the integrity of example.com by setting a cookie with a Domain attribute of "com". Unfortunately, the set of public suffixes (also known as "registry controlled domains") changes over time. If feasible, user agents SHOULD use an up-to-date public suffix list, such as the one maintained by the Mozilla project at http://publicsuffix.org/.

If you check the list of public suffixes, localhost is not included. The security concern that a cookie set for localhost may result in an attack from a subdomain of localhost is not valid, because localhost is inherently designed for local network use only.

Httpie may decide to block cookies for localhost at its discretion (perhaps because of design principles), but it shouldn't do so for the reason that “localhost isn't valid for cookies,” because that statement is false. The curl CLI supports using localhost for cookie domains. Chrome supports using localhost for cookie domains. Firefox supports using localhost for cookie domains. httpie is the only user agent I know that doesn't.

You can test that curl supports localhost cookies like this:

1. Set up a server on localhost that responds with a cookie with domain=localhost, e.g., using a PHP script containing:

setcookie("testcookie", "testvalue", time() + 3600, "/", "localhost", false, true);

2. Use curl to make an HTTP request while using a CookieJar:

curl -c cookies.txt http://localhost/set_cookie.php

3. Confirm the CookieJar contains the saved cookie:

❯ cat cookies.txt 
# Netscape HTTP Cookie File
# https://curl.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

#HttpOnly_.localhost TRUE / FALSE 1716167795 testcookie testvalue

4. Test the cookie is sent to the server by using a script that reads cookies and prints them, e.g., a PHP script containing echo json_encode($_COOKIE);:

curl -b cookies.txt http://localhost/read_cookie.php
{"testcookie":"testvalue"}

---

TL;DR:

1. httpie should block cookies with a domain set to any public suffixes.
2. localhost is not a public suffix.
3. All user agents (except httpie) support cookies with a domain set to localhost.

Sorry for the long comment, I wanted to get it right. ☺️

PS: .dev should not be used for local development because it is a TLD. .local should not be used because it conflicts with Multicast DNS. If you don't want to use localhost to access a locally-running server, you should use .test, which is a reserved TLD intended for usage in software testing.

[Ousret]

Great & pertinent analysis. I was wrong then, Indeed localhost is not a public suffix and it "could" be considered for cookie grabbing.

Now, as I said, HTTPie depends on external libraries (incl base for standard library Python) for this:

from requests.cookies import RequestsCookieJar, remove_cookie_by_name

And the standard library does handle this case. Either HTTPie drop the standard library or somehow "circumvent" the said limitation.

Now as you said, HTTPie is free of "not carrying" about this domain. I don't have strong opinion on the topic.

.dev should not be used for local development because it is a TLD. .local should not be used because it conflicts with Multicast DNS.

I merely said that as a solution to unblock this. Not to state a standard way of doing things. .test it is then.
So, given the tangible case, I am reopening it.
The project owner can decide whether he want to pursue this or not.

regards,

[Ousret]

Do you have any evidence that HTTPie is the only user agent not accepting this?
And this what version Chrome and Firefox allowed this? because I firmly remember them to ignore cookie with domain=localhost.

[quinncomendant]

I created a test case in pure python using the requests module, and indeed it does not accepts cookies with domain=localhost. So, it's not a bug in httpie!

There is some discussion at pfs/requests issue #5977 which passes the blame to Python's CookieJar.extract_cookies:

I don't think this is actually request's bug. It looks like Python's CookieJar.extract_cookies doesn't parse them properly.

The python/cpython issue #90233 is tracking this:

Apparently, CookieJar.extract_cookies doesn't process cookies form local domains which explicitly set domain in Set-Cookie header. That means that headers with domain specified, like "Set-Cookie: foo=baz; Domain=localhost;", are ignored.

And has apparently been resolved with PR #30108: “ bpo-46075: Allow for explicit domains in CookieJar” in 2022, included in python v3.11. However, I'm using v3.12 and the issue still exists, so perhaps I've followed up the wrong branch of issues?

Once I realized this is a python bug, it's easy to find much discussion of it online, and everyone has different ways of resolving it. One example is this code in the Pylons project that takes localhost as a special case. (I'm not familiar with python enough to know if this is a good example, just a guess). Perhaps httpie can do something similar?

Do you have any evidence that HTTPie is the only user agent not accepting this?

No, all I can say is that I've been developing on localhost since 2003, and cookies always work fine in every user agent I tried, until now.

[Ousret]

1. The PR (in cpython) mentioned does not fix the "issue".
2. The example code does fix this properly by subclassing the original cookie policy, but just for the "localhost" case. It is worthwhile to be considered.

I may consider adding this to Niquests directly, and thus making this issue fixed in #1531

regards,

[Ousret]

It should be published in the next minor of Niquests.
Then, if linked PR is merged, it will be fixed / closed automatically.