Skip to content

Cookies aren't saved for hostnames #3649

Closed
Closed
Cookies aren't saved for hostnames#3649
Labels
HTTPcookiesregression

Description

@jay
jay
opened on Mar 6, 2019

I did this

Reported by @deepakhj in the offending PR and Pierre Brico in curl-users (the latter version below):

My issue is related to the cookies usage. I've written a script which 
emulates a simple user who logs into a web site and execute a certain 
action. With curl version 7.40, everything was working as expected... With 
the new version of curl, no cookie is sent when executing the POST. 
I've searched the documentation but found no changes related to the cookie 
engine. That's the reason why I've post my issue to this mailing list...

> POST /auth/realms/fhome/login-actions/authenticate?session_code=anEjPMdR-9_Xl0t68_vDaT5yu6JoXAizauJ3rtrgcPw&execution=f65ccedb-cc7a-4f46-b386-f59408012698&client_id=XXXXXXXXXXXXX&tab_id=SwFAuhzueT0 HTTP/2 
> Host: fronteo-keycloak-rmm:8443 
> User-Agent: curl/7.64.0 
> Accept: */* 
> Content-Length: 27 
> Content-Type: application/x-www-form-urlencoded 
> 

And this output of the old version: 

> POST /auth/realms/fhome/login-actions/authenticate?session_code=yRyjuGxKaOLa-628N1ZLQi4l39Yc1g7BcBa1EwTgJvo&execution=f65ccedb-cc7a-4f46-b386-f59408012698&client_id=XXXXXXXXXXXXX&tab_id=episi_krhDY HTTP/1.1 
> User-Agent: curl/7.40.0 
> Host: fronteo-keycloak-rmm:8443 
> Accept: */* 
> Cookie: AUTH_SESSION_ID=bd7a2b29-f09d-4950-adb4-733c2d2f8cfb.fronteo-keycloak-rmm; KC_RESTART=eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyOTNiZDllMy0wMDJlLTQ4OTYtYjg0Ny0xODdlMDkzNjdmZGQifQ.eyJjaWQiOiIzNDY2Njg3YS03ZTNmLTRjOGYtYjRhMy1iZjE1ZmIxMWFhYTQiLCJwdHkiOiJvcGVuaWQtY29ubmVjdCIsInJ1cmkiOiJodHRwOi8vbG9jYWxob3N0OjgwODAiLCJhY3QiOiJBVVRIRU5USUNBVEUiLCJub3RlcyI6eyJzY29wZSI6ImFpc3AiLCJpc3MiOiJodHRwczovL2Zyb250ZW8ta2V5Y2xvYWstcm1tOjg0NDMvYXV0aC9yZWFsbXMvZmhvbWUiLCJyZXNwb25zZV90eXBlIjoiY29kZSIsImNvZGVfY2hhbGxlbmdlX21ldGhvZCI6InBsYWluIiwicmVkaXJlY3RfdXJpIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwIiwic3RhdGUiOiIxMjM0NSJ9fQ.JhMJmSKLESr_flL3SRZgmaFw4T1e5ueg-1z9OEFoRX8 
> Content-Length: 27 
> Content-Type: application/x-www-form-urlencoded 
>

Note the host fronteo-keycloak-rmm is not a DNS name so what he's seeing is likely due to #2964 which "extended domain checks to non psl builds", namely such a hostname will be unable to set cookies.

I expected the following

Non-DNS cookies to be saved... maybe. I'm not sure if this is right or not.

curl/libcurl version

curl 7.64.0 (x86_64-pc-msys) libcurl/7.64.0 OpenSSL/1.1.1b zlib/1.2.11 brotli/1.0.7 libidn2/2.1.1 libpsl/0.20.2 (+libidn2/2.1.1) libssh2/1.8.0 nghttp2/1.36.0
Release-Date: 2019-02-06
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS Debug TrackMemory IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz brotli TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL Metalink

operating system

Windows

/cc @danielgustafsson

Activity

  1. bagder commented on Mar 6, 2019

    @bagder
    bagder
    on Mar 6, 2019
    Member

    likely due to #2964

    For non-PSL builds yes, as PSL builds already had that logic. Both cases now have stricter host name requirements than curl did back in the 7.40 days.

    I'm not sure if this is right or not.

    This is a lovely in-between case. The cookie specs are clearly moving towards making this sort of cookie illegal, as they tend to focus on how cookies (should) work on the big Internet. In this case we're talking cookies in a more closed environment.

    It's a tricky situation. I would love to be able to fix this without adding another option, but can we?

  2. jay commented on Mar 7, 2019

    @jay
    jay
    on Mar 7, 2019
    MemberAuthor

    For non-PSL builds yes, as PSL builds already had that logic. Both cases now have stricter host name requirements than curl did back in the 7.40 days.

    Firefox and Chrome allow it. Is there a disadvantage of allowing hostname regardless of libpsl other than (I assume) some rfc violation?

    /cc @rockdaboot

  3. bagder commented on Mar 7, 2019

    @bagder
    bagder
    on Mar 7, 2019
    Member

    Firefox and Chrome allow it

    You mean they allow this dot-less name? Then we really should too!

    Is there a disadvantage of allowing hostname regardless of libpsl other than (I assume) some rfc violation?

    Yes, it's a security thing. If "example.co.uk" can set a cookie for "co.uk", then that cookie will then also match "secrets.co.uk" etc. "co.uk" being a PSL. The RFC on this is actually really vague and PSL as a concept is weird and fragile, but that's the best there is right now...

  4. rockdaboot commented on Mar 7, 2019

    @rockdaboot
    rockdaboot
    on Mar 7, 2019
    Contributor

    On a dot-less domain (in this case it is a hostname), the "prevailing star rule" of the PSL is applied. In the internet you normally want it for cookie checking, else you open up a security hole.

    But on local networks, you might not want it. So you can use psl_is_public_suffix2() with the PSL_TYPE_NO_STAR_RULE as third argument (see docs for details).

    The same for the psl utility when using --no-star-rule.

    I can currently not say if there is a way to automatically detect the environment. If there is no safe way, it smells for another curl option.

  5. danielgustafsson commented on Mar 7, 2019

    @danielgustafsson
    danielgustafsson
    on Mar 7, 2019
    Contributor

    I can currently not say if there is a way to automatically detect the environment. If there is no safe way, it smells for another curl option.

    It seems rather complicated to detect that in way that it will a) catch all the cases and b) be leakproof wrt cookie jars.

    If we add an option to this, then we need to make the option apply to PSL enabled builds too I would assume.

    (sorry for being slow to respond, travelling put me offline for a bit)

  6. rockdaboot commented on Mar 7, 2019

    @rockdaboot
    rockdaboot
    on Mar 7, 2019
    Contributor

    then we need to make the option apply to PSL enabled builds too

    That makes sense. Let me know how you name it (if it comes to that point). I would like to see something similar for wget/wget2. Can't hurt if the option names match.

  7. rockdaboot commented on Mar 7, 2019

    @rockdaboot
    rockdaboot
    on Mar 7, 2019
    Contributor

    If you use curl with libpsl, you could also add your local domain/host name as exception in the used PSL file.

  8. bagder commented on Mar 7, 2019

    @bagder
    bagder
    on Mar 7, 2019
    Member

    We should support this without any extra options since the browsers do that. I can go to http://moo/cookie with Firefox and it can set cookies - for that host name.

  9. bagder commented on Mar 8, 2019

    @bagder
    bagder
    on Mar 8, 2019
    Member

    I noticed the bug is only present in non-PSL builds. I think I'll go with reverting 3773de3. PR coming up.

  10. bagder
    added a commit that references this issue on Mar 8, 2019

    test331: verify set-cookie for dot-less host name

    912caf1
  11. bagder
    added
    regression
    HTTP
    on Mar 8, 2019

  12. bagder commented on Mar 8, 2019

    @bagder
    bagder
    on Mar 8, 2019
    Member

    This bug was introduced in 7.64.0.

  13. jay commented on Mar 8, 2019

    @jay
    jay
    on Mar 8, 2019
    MemberAuthor

    Are you sure this is only non-psl? The guy who reported it seems to be using libpsl/0.20.2

  14. bagder commented on Mar 8, 2019

    @bagder
    bagder
    on Mar 8, 2019
    Member

    My new test case in #3659 worked fine with my PSL build when I tried it locally, and the commit I reverted to get it working on a non-PSL build was only affecting non-PSL.

    If the problem exists on a PSL build, then I suspect my test case isn't good enough. Unfortunately I don't think we have the full URL and its corresponding response headers to know for sure?

  15. bagder
    closed this as completedin 299d966on Mar 10, 2019
  16. bagder
    added a commit that references this issue on Mar 10, 2019

    test331: verify set-cookie for dotless host name

    ff7e5a2

  17. Largo2005 commented on Mar 11, 2019

    @Largo2005
    Largo2005
    on Mar 11, 2019

    For you information, I've noticed the problem using the latest version of MSYS2 which embeds curl 7.64.0. It seems that this curl was built with PSL feature as mentioned by the --version option:
    $ curl --version
    curl 7.64.0 (x86_64-pc-msys) libcurl/7.64.0 OpenSSL/1.1.1b zlib/1.2.11 brotli/1.0.7 libidn2/2.1.1 libpsl/0.20.2 (+libidn2/2.1.1) libssh2/1.8.0 nghttp2/1.36.0
    Release-Date: 2019-02-06
    Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
    Features: AsynchDNS Debug TrackMemory IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz brotli TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL Metalink

  18. bagder commented on Mar 11, 2019

    @bagder
    bagder
    on Mar 11, 2019
    Member

    I've noticed the problem

    Then please detail the problem for us. The problem as I reproduced (and fixed) was not present in PSL builds. How do you reproduce it?

  19. bagder commented on Mar 12, 2019

    @bagder
    bagder
    on Mar 12, 2019
    Member

    Primarily, I would like to know the URL used and what the (cookie) response headers looked like that curl didn't accept.

  20. Largo2005 commented on Mar 12, 2019

    @Largo2005
    Largo2005
    on Mar 12, 2019

    OK, as I already told, I use a binary version of curl embedded in MSYS2 (freely downloadable at https://www.msys2.org/). This binary is built with the features mentioned above.
    My problem is linked to local hostname. The URL I would like curl to connect to is "https://fronteo-keycloak-rmm:8443" (a Redhat SSO server instance). Here is a log (I've removed some useless lines):

    > GET /auth/realms/fhome/protocol/openid-connect/auth?response_type=code&scope=aisp&client_id=3466687a-7e3f-4c8f-b4a3-bf15fb11aaa4&state=12345&redirect_uri=http://localhost:8080 HTTP/2
> Host: fronteo-keycloak-rmm:8443
> User-Agent: curl/7.64.0
> Accept: */*
>
* STATE: DO => DO_DONE handle 0x600055128; line 1716 (connection #0)
* multi changed, check CONNECT_PEND queue!
* STATE: DO_DONE => WAITPERFORM handle 0x600055128; line 1840 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x600055128; line 1855 (connection #0)
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
} [5 bytes data]
* multi changed, check CONNECT_PEND queue!
{ [5 bytes data]
* HTTP/2 found, allow multiplexing
< HTTP/2 200
< cache-control: no-store, must-revalidate, max-age=0
* Added cookie AUTH_SESSION_ID="8696e8d7-3b93-4687-a9a4-12dc9ece0cb4.fronteo-keycloak-rmm" for domain fronteo-keycloak-rmm, path /auth/realms/fhome/, expire 0
< set-cookie: AUTH_SESSION_ID=8696e8d7-3b93-4687-a9a4-12dc9ece0cb4.fronteo-keycloak-rmm; Version=1; Path=/auth/realms/fhome/; HttpOnly
* Added cookie KC_RESTART="eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyOTNiZDllMy0wMDJlLTQ4OTYtYjg0Ny0xODdlMDkzNjdmZGQifQ.eyJjaWQiOiIzNDY2Njg3YS03ZTNmLTRjOGYtYjRhMy1iZjE1ZmIxMWFhYTQiLCJwdHkiOiJvcGVuaWQtY29ubmVjdCIsInJ1cmkiOiJodHRwOi8vbG9jYWxob3N0OjgwODAiLCJhY3QiOiJBVVRIRU5USUNBVEUiLCJub3RlcyI6eyJzY29wZSI6ImFpc3AiLCJpc3MiOiJodHRwczovL2Zyb250ZW8ta2V5Y2xvYWstcm1tOjg0NDMvYXV0aC9yZWFsbXMvZmhvbWUiLCJyZXNwb25zZV90eXBlIjoiY29kZSIsImNvZGVfY2hhbGxlbmdlX21ldGhvZCI6InBsYWluIiwicmVkaXJlY3RfdXJpIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwIiwic3RhdGUiOiIxMjM0NSJ9fQ.JhMJmSKLESr_flL3SRZgmaFw4T1e5ueg-1z9OEFoRX8" for domain fronteo-keycloak-rmm, path /auth/realms/fhome/, expire 0
< set-cookie: KC_RESTART=eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyOTNiZDllMy0wMDJlLTQ4OTYtYjg0Ny0xODdlMDkzNjdmZGQifQ.eyJjaWQiOiIzNDY2Njg3YS03ZTNmLTRjOGYtYjRhMy1iZjE1ZmIxMWFhYTQiLCJwdHkiOiJvcGVuaWQtY29ubmVjdCIsInJ1cmkiOiJodHRwOi8vbG9jYWxob3N0OjgwODAiLCJhY3QiOiJBVVRIRU5USUNBVEUiLCJub3RlcyI6eyJzY29wZSI6ImFpc3AiLCJpc3MiOiJodHRwczovL2Zyb250ZW8ta2V5Y2xvYWstcm1tOjg0NDMvYXV0aC9yZWFsbXMvZmhvbWUiLCJyZXNwb25zZV90eXBlIjoiY29kZSIsImNvZGVfY2hhbGxlbmdlX21ldGhvZCI6InBsYWluIiwicmVkaXJlY3RfdXJpIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwIiwic3RhdGUiOiIxMjM0NSJ9fQ.JhMJmSKLESr_flL3SRZgmaFw4T1e5ueg-1z9OEFoRX8; Version=1; Path=/auth/realms/fhome/; HttpOnly
< x-xss-protection: 1; mode=block
< x-frame-options: SAMEORIGIN
< content-security-policy: frame-src 'self'; frame-ancestors 'self'; object-src 'none';
< date: Tue, 12 Mar 2019 14:22:06 GMT
< x-robots-tag: none
< strict-transport-security: max-age=31536000; includeSubDomains
< x-content-type-options: nosniff
< content-type: text/html;charset=utf-8
< content-language: fr
<
{ [15509 bytes data]
* nread <= 0, server closed connection, bailing
* STATE: PERFORM => DONE handle 0x600055128; line 2052 (connection #0)
* multi_done
* Connection #0 to host fronteo-keycloak-rmm left intact
* Expire cleared (transfer 0x600055128)
POST URL https://fronteo-keycloak-rmm:8443/auth/realms/fhome/login-actions/authenticate?session_code=GJrBBqxLoIhXdOGItEhE3qPOFX380SelqRveHwaYBxA&execution=f65ccedb-cc7a-4f46-b386-f59408012698&client_id=3466687a-7e3f-4c8f-b4a3-bf15fb11aaa4&tab_id=Y4joE-fW6mE
* Expire in 0 ms for 6 (transfer 0x600055128)
* STATE: INIT => CONNECT handle 0x600055128; line 1443 (connection #-5000)
* Added connection 0. The cache now contains 1 members
* Expire in 1 ms for 1 (transfer 0x600055128)
* STATE: CONNECT => WAITRESOLVE handle 0x600055128; line 1484 (connection #0)
* Expire in 0 ms for 1 (transfer 0x600055128)
* Expire in 1 ms for 1 (transfer 0x600055128)
*   Trying 192.100.2.31...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x600055128)
* STATE: WAITRESOLVE => WAITCONNECT handle 0x600055128; line 1565 (connection #0)
* Connected to fronteo-keycloak-rmm (192.100.2.31) port 8443 (#0)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x600055128; line 1619 (connection #0)
* Marked for [keep alive]: HTTP default
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /usr/ssl/certs/ca-bundle.crt
  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x600055128; line 1633 (connection #0)
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [94 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [815 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=BE; L=LLN; O=Mainsys; CN=keycloak
*  start date: Feb 11 14:23:59 2019 GMT
*  expire date: Feb 11 14:23:59 2039 GMT
*  issuer: C=BE; L=LLN; O=Mainsys; CN=keycloak
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* STATE: PROTOCONNECT => DO handle 0x600055128; line 1654 (connection #0)
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x600055128)
} [5 bytes data]
> POST /auth/realms/fhome/login-actions/authenticate?session_code=GJrBBqxLoIhXdOGItEhE3qPOFX380SelqRveHwaYBxA&execution=f65ccedb-cc7a-4f46-b386-f59408012698&client_id=3466687a-7e3f-4c8f-b4a3-bf15fb11aaa4&tab_id=Y4joE-fW6mE HTTP/2
> Host: fronteo-keycloak-rmm:8443
> User-Agent: curl/7.64.0
> Accept: */*
> Content-Length: 27
> Content-Type: application/x-www-form-urlencoded
>
* STATE: DO => DO_DONE handle 0x600055128; line 1716 (connection #0)
* multi changed, check CONNECT_PEND queue!
* STATE: DO_DONE => WAITPERFORM handle 0x600055128; line 1840 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x600055128; line 1855 (connection #0)
} [5 bytes data]
* We are completely uploaded and fine
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
} [5 bytes data]
* multi changed, check CONNECT_PEND queue!
{ [5 bytes data]
* HTTP/2 found, allow multiplexing
< HTTP/2 500
< cache-control: no-store, must-revalidate, max-age=0
< content-length: 0
< date: Tue, 12 Mar 2019 14:22:06 GMT
<

    Here is the file containing the cookies:
    cookies-in.txt

    Hope this helps,
    Pierre

  21. bagder commented on Mar 12, 2019

    @bagder
    bagder
    on Mar 12, 2019
    Member

    I turned that into a "real" issue: #3676 so that it gets the correct attention.

  22. lock
    locked as resolved and limited conversation to collaborators on Jun 10, 2019
  23. bagder
    added
    cookies
    on Nov 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    HTTPcookiesregression

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions