Cookies aren't saved for hostnames

[jay]

I did this

Reported by @deepakhj in the offending PR and Pierre Brico in curl-users (the latter version below):

My issue is related to the cookies usage. I've written a script which 
emulates a simple user who logs into a web site and execute a certain 
action. With curl version 7.40, everything was working as expected... With 
the new version of curl, no cookie is sent when executing the POST. 
I've searched the documentation but found no changes related to the cookie 
engine. That's the reason why I've post my issue to this mailing list...

> POST /auth/realms/fhome/login-actions/authenticate?session_code=anEjPMdR-9_Xl0t68_vDaT5yu6JoXAizauJ3rtrgcPw&execution=f65ccedb-cc7a-4f46-b386-f59408012698&client_id=XXXXXXXXXXXXX&tab_id=SwFAuhzueT0 HTTP/2 
> Host: fronteo-keycloak-rmm:8443 
> User-Agent: curl/7.64.0 
> Accept: */* 
> Content-Length: 27 
> Content-Type: application/x-www-form-urlencoded 
> 

And this output of the old version: 

> POST /auth/realms/fhome/login-actions/authenticate?session_code=yRyjuGxKaOLa-628N1ZLQi4l39Yc1g7BcBa1EwTgJvo&execution=f65ccedb-cc7a-4f46-b386-f59408012698&client_id=XXXXXXXXXXXXX&tab_id=episi_krhDY HTTP/1.1 
> User-Agent: curl/7.40.0 
> Host: fronteo-keycloak-rmm:8443 
> Accept: */* 
> Cookie: AUTH_SESSION_ID=bd7a2b29-f09d-4950-adb4-733c2d2f8cfb.fronteo-keycloak-rmm; KC_RESTART=eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyOTNiZDllMy0wMDJlLTQ4OTYtYjg0Ny0xODdlMDkzNjdmZGQifQ.eyJjaWQiOiIzNDY2Njg3YS03ZTNmLTRjOGYtYjRhMy1iZjE1ZmIxMWFhYTQiLCJwdHkiOiJvcGVuaWQtY29ubmVjdCIsInJ1cmkiOiJodHRwOi8vbG9jYWxob3N0OjgwODAiLCJhY3QiOiJBVVRIRU5USUNBVEUiLCJub3RlcyI6eyJzY29wZSI6ImFpc3AiLCJpc3MiOiJodHRwczovL2Zyb250ZW8ta2V5Y2xvYWstcm1tOjg0NDMvYXV0aC9yZWFsbXMvZmhvbWUiLCJyZXNwb25zZV90eXBlIjoiY29kZSIsImNvZGVfY2hhbGxlbmdlX21ldGhvZCI6InBsYWluIiwicmVkaXJlY3RfdXJpIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwIiwic3RhdGUiOiIxMjM0NSJ9fQ.JhMJmSKLESr_flL3SRZgmaFw4T1e5ueg-1z9OEFoRX8 
> Content-Length: 27 
> Content-Type: application/x-www-form-urlencoded 
> 

Note the host fronteo-keycloak-rmm is not a DNS name so what he's seeing is likely due to #2964 which "extended domain checks to non psl builds", namely such a hostname will be unable to set cookies.

I expected the following

Non-DNS cookies to be saved... maybe. I'm not sure if this is right or not.

curl/libcurl version

curl 7.64.0 (x86_64-pc-msys) libcurl/7.64.0 OpenSSL/1.1.1b zlib/1.2.11 brotli/1.0.7 libidn2/2.1.1 libpsl/0.20.2 (+libidn2/2.1.1) libssh2/1.8.0 nghttp2/1.36.0
Release-Date: 2019-02-06
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS Debug TrackMemory IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz brotli TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL Metalink

operating system

Windows

/cc @danielgustafsson

[bagder]

likely due to #2964

For non-PSL builds yes, as PSL builds already had that logic. Both cases now have stricter host name requirements than curl did back in the 7.40 days.

I'm not sure if this is right or not.

This is a lovely in-between case. The cookie specs are clearly moving towards making this sort of cookie illegal, as they tend to focus on how cookies (should) work on the big Internet. In this case we're talking cookies in a more closed environment.

It's a tricky situation. I would love to be able to fix this without adding another option, but can we?

[jay]

For non-PSL builds yes, as PSL builds already had that logic. Both cases now have stricter host name requirements than curl did back in the 7.40 days.

Firefox and Chrome allow it. Is there a disadvantage of allowing hostname regardless of libpsl other than (I assume) some rfc violation?

/cc @rockdaboot

[bagder]

Firefox and Chrome allow it

You mean they allow this dot-less name? Then we really should too!

Is there a disadvantage of allowing hostname regardless of libpsl other than (I assume) some rfc violation?

Yes, it's a security thing. If "example.co.uk" can set a cookie for "co.uk", then that cookie will then also match "secrets.co.uk" etc. "co.uk" being a PSL. The RFC on this is actually really vague and PSL as a concept is weird and fragile, but that's the best there is right now...

[rockdaboot]

On a dot-less domain (in this case it is a hostname), the "prevailing star rule" of the PSL is applied. In the internet you normally want it for cookie checking, else you open up a security hole.

But on local networks, you might not want it. So you can use psl_is_public_suffix2() with the PSL_TYPE_NO_STAR_RULE as third argument (see docs for details).

The same for the psl utility when using --no-star-rule.

I can currently not say if there is a way to automatically detect the environment. If there is no safe way, it smells for another curl option.

[danielgustafsson]

I can currently not say if there is a way to automatically detect the environment. If there is no safe way, it smells for another curl option.

It seems rather complicated to detect that in way that it will a) catch all the cases and b) be leakproof wrt cookie jars.

If we add an option to this, then we need to make the option apply to PSL enabled builds too I would assume.

(sorry for being slow to respond, travelling put me offline for a bit)

[rockdaboot]

then we need to make the option apply to PSL enabled builds too

That makes sense. Let me know how you name it (if it comes to that point). I would like to see something similar for wget/wget2. Can't hurt if the option names match.

[rockdaboot]

If you use curl with libpsl, you could also add your local domain/host name as exception in the used PSL file.

[bagder]

We should support this without any extra options since the browsers do that. I can go to http://moo/cookie with Firefox and it can set cookies - for that host name.

[bagder]

I noticed the bug is only present in non-PSL builds. I think I'll go with reverting 3773de3. PR coming up.

[bagder]

This bug was introduced in 7.64.0.

[jay]

Are you sure this is only non-psl? The guy who reported it seems to be using libpsl/0.20.2

[bagder]

My new test case in #3659 worked fine with my PSL build when I tried it locally, and the commit I reverted to get it working on a non-PSL build was only affecting non-PSL.

If the problem exists on a PSL build, then I suspect my test case isn't good enough. Unfortunately I don't think we have the full URL and its corresponding response headers to know for sure?

[Largo2005]

For you information, I've noticed the problem using the latest version of MSYS2 which embeds curl 7.64.0. It seems that this curl was built with PSL feature as mentioned by the --version option:
$ curl --version
curl 7.64.0 (x86_64-pc-msys) libcurl/7.64.0 OpenSSL/1.1.1b zlib/1.2.11 brotli/1.0.7 libidn2/2.1.1 libpsl/0.20.2 (+libidn2/2.1.1) libssh2/1.8.0 nghttp2/1.36.0
Release-Date: 2019-02-06
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS Debug TrackMemory IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz brotli TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL Metalink

[bagder]

I've noticed the problem

Then please detail the problem for us. The problem as I reproduced (and fixed) was not present in PSL builds. How do you reproduce it?

[bagder]

Primarily, I would like to know the URL used and what the (cookie) response headers looked like that curl didn't accept.

[Largo2005]

OK, as I already told, I use a binary version of curl embedded in MSYS2 (freely downloadable at https://www.msys2.org/). This binary is built with the features mentioned above.
My problem is linked to local hostname. The URL I would like curl to connect to is "https://fronteo-keycloak-rmm:8443" (a Redhat SSO server instance). Here is a log (I've removed some useless lines):

> GET /auth/realms/fhome/protocol/openid-connect/auth?response_type=code&scope=aisp&client_id=3466687a-7e3f-4c8f-b4a3-bf15fb11aaa4&state=12345&redirect_uri=http://localhost:8080 HTTP/2
> Host: fronteo-keycloak-rmm:8443
> User-Agent: curl/7.64.0
> Accept: */*
>
* STATE: DO => DO_DONE handle 0x600055128; line 1716 (connection #0)
* multi changed, check CONNECT_PEND queue!
* STATE: DO_DONE => WAITPERFORM handle 0x600055128; line 1840 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x600055128; line 1855 (connection #0)
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
} [5 bytes data]
* multi changed, check CONNECT_PEND queue!
{ [5 bytes data]
* HTTP/2 found, allow multiplexing
< HTTP/2 200
< cache-control: no-store, must-revalidate, max-age=0
* Added cookie AUTH_SESSION_ID="8696e8d7-3b93-4687-a9a4-12dc9ece0cb4.fronteo-keycloak-rmm" for domain fronteo-keycloak-rmm, path /auth/realms/fhome/, expire 0
< set-cookie: AUTH_SESSION_ID=8696e8d7-3b93-4687-a9a4-12dc9ece0cb4.fronteo-keycloak-rmm; Version=1; Path=/auth/realms/fhome/; HttpOnly
* Added cookie KC_RESTART="eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyOTNiZDllMy0wMDJlLTQ4OTYtYjg0Ny0xODdlMDkzNjdmZGQifQ.eyJjaWQiOiIzNDY2Njg3YS03ZTNmLTRjOGYtYjRhMy1iZjE1ZmIxMWFhYTQiLCJwdHkiOiJvcGVuaWQtY29ubmVjdCIsInJ1cmkiOiJodHRwOi8vbG9jYWxob3N0OjgwODAiLCJhY3QiOiJBVVRIRU5USUNBVEUiLCJub3RlcyI6eyJzY29wZSI6ImFpc3AiLCJpc3MiOiJodHRwczovL2Zyb250ZW8ta2V5Y2xvYWstcm1tOjg0NDMvYXV0aC9yZWFsbXMvZmhvbWUiLCJyZXNwb25zZV90eXBlIjoiY29kZSIsImNvZGVfY2hhbGxlbmdlX21ldGhvZCI6InBsYWluIiwicmVkaXJlY3RfdXJpIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwIiwic3RhdGUiOiIxMjM0NSJ9fQ.JhMJmSKLESr_flL3SRZgmaFw4T1e5ueg-1z9OEFoRX8" for domain fronteo-keycloak-rmm, path /auth/realms/fhome/, expire 0
< set-cookie: KC_RESTART=eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyOTNiZDllMy0wMDJlLTQ4OTYtYjg0Ny0xODdlMDkzNjdmZGQifQ.eyJjaWQiOiIzNDY2Njg3YS03ZTNmLTRjOGYtYjRhMy1iZjE1ZmIxMWFhYTQiLCJwdHkiOiJvcGVuaWQtY29ubmVjdCIsInJ1cmkiOiJodHRwOi8vbG9jYWxob3N0OjgwODAiLCJhY3QiOiJBVVRIRU5USUNBVEUiLCJub3RlcyI6eyJzY29wZSI6ImFpc3AiLCJpc3MiOiJodHRwczovL2Zyb250ZW8ta2V5Y2xvYWstcm1tOjg0NDMvYXV0aC9yZWFsbXMvZmhvbWUiLCJyZXNwb25zZV90eXBlIjoiY29kZSIsImNvZGVfY2hhbGxlbmdlX21ldGhvZCI6InBsYWluIiwicmVkaXJlY3RfdXJpIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwIiwic3RhdGUiOiIxMjM0NSJ9fQ.JhMJmSKLESr_flL3SRZgmaFw4T1e5ueg-1z9OEFoRX8; Version=1; Path=/auth/realms/fhome/; HttpOnly
< x-xss-protection: 1; mode=block
< x-frame-options: SAMEORIGIN
< content-security-policy: frame-src 'self'; frame-ancestors 'self'; object-src 'none';
< date: Tue, 12 Mar 2019 14:22:06 GMT
< x-robots-tag: none
< strict-transport-security: max-age=31536000; includeSubDomains
< x-content-type-options: nosniff
< content-type: text/html;charset=utf-8
< content-language: fr
<
{ [15509 bytes data]
* nread <= 0, server closed connection, bailing
* STATE: PERFORM => DONE handle 0x600055128; line 2052 (connection #0)
* multi_done
* Connection #0 to host fronteo-keycloak-rmm left intact
* Expire cleared (transfer 0x600055128)
POST URL https://fronteo-keycloak-rmm:8443/auth/realms/fhome/login-actions/authenticate?session_code=GJrBBqxLoIhXdOGItEhE3qPOFX380SelqRveHwaYBxA&execution=f65ccedb-cc7a-4f46-b386-f59408012698&client_id=3466687a-7e3f-4c8f-b4a3-bf15fb11aaa4&tab_id=Y4joE-fW6mE
* Expire in 0 ms for 6 (transfer 0x600055128)
* STATE: INIT => CONNECT handle 0x600055128; line 1443 (connection #-5000)
* Added connection 0. The cache now contains 1 members
* Expire in 1 ms for 1 (transfer 0x600055128)
* STATE: CONNECT => WAITRESOLVE handle 0x600055128; line 1484 (connection #0)
* Expire in 0 ms for 1 (transfer 0x600055128)
* Expire in 1 ms for 1 (transfer 0x600055128)
*   Trying 192.100.2.31...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x600055128)
* STATE: WAITRESOLVE => WAITCONNECT handle 0x600055128; line 1565 (connection #0)
* Connected to fronteo-keycloak-rmm (192.100.2.31) port 8443 (#0)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x600055128; line 1619 (connection #0)
* Marked for [keep alive]: HTTP default
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /usr/ssl/certs/ca-bundle.crt
  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x600055128; line 1633 (connection #0)
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [94 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [815 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=BE; L=LLN; O=Mainsys; CN=keycloak
*  start date: Feb 11 14:23:59 2019 GMT
*  expire date: Feb 11 14:23:59 2039 GMT
*  issuer: C=BE; L=LLN; O=Mainsys; CN=keycloak
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* STATE: PROTOCONNECT => DO handle 0x600055128; line 1654 (connection #0)
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x600055128)
} [5 bytes data]
> POST /auth/realms/fhome/login-actions/authenticate?session_code=GJrBBqxLoIhXdOGItEhE3qPOFX380SelqRveHwaYBxA&execution=f65ccedb-cc7a-4f46-b386-f59408012698&client_id=3466687a-7e3f-4c8f-b4a3-bf15fb11aaa4&tab_id=Y4joE-fW6mE HTTP/2
> Host: fronteo-keycloak-rmm:8443
> User-Agent: curl/7.64.0
> Accept: */*
> Content-Length: 27
> Content-Type: application/x-www-form-urlencoded
>
* STATE: DO => DO_DONE handle 0x600055128; line 1716 (connection #0)
* multi changed, check CONNECT_PEND queue!
* STATE: DO_DONE => WAITPERFORM handle 0x600055128; line 1840 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x600055128; line 1855 (connection #0)
} [5 bytes data]
* We are completely uploaded and fine
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
} [5 bytes data]
* multi changed, check CONNECT_PEND queue!
{ [5 bytes data]
* HTTP/2 found, allow multiplexing
< HTTP/2 500
< cache-control: no-store, must-revalidate, max-age=0
< content-length: 0
< date: Tue, 12 Mar 2019 14:22:06 GMT
<

Here is the file containing the cookies:
cookies-in.txt

Hope this helps,
Pierre

[bagder]

I turned that into a "real" issue: #3676 so that it gets the correct attention.