Skip to content

DNS_CACHE_TIMEOUT == 0 but address might still be used for second transfer! #2169

Closed
@p

Description

@p

Consider this:

>>> c=pycurl.Curl()
# disable dns cache
>>> c.setopt(c.DNS_CACHE_TIMEOUT,0)
>>> c.setopt(c.VERBOSE,1)
# set ip to .102
>>> c.setopt(c.RESOLVE,['ya.ru:80:127.0.0.102'])
>>> c.setopt(c.URL,'http://ya.ru')
>>> c.perform()
* Added ya.ru:80:127.0.0.102 to DNS cache
# ^ ok
* Rebuilt URL to: http://ya.ru/
* Hostname ya.ru was found in DNS cache
*   Trying 127.0.0.102...
# ^ ok
* TCP_NODELAY set
* connect to 127.0.0.102 port 80 failed: Connection refused
* Failed to connect to ya.ru port 80: Connection refused
* Closing connection 0
Traceback (most recent call last):
  File "", line 1, in 
pycurl.error: (7, 'Failed to connect to ya.ru port 80: Connection refused')
# clear fake dns
>>> c.setopt(c.RESOLVE,['-ya.ru:80'])
# set fake dns to another ip, .103
>>> c.setopt(c.RESOLVE,['ya.ru:80:127.0.0.103'])
>>> c.perform()
* Added ya.ru:80:127.0.0.103 to DNS cache
# ^ ok
* Rebuilt URL to: http://ya.ru/
* Hostname ya.ru was found in DNS cache
*   Trying 127.0.0.102...
# ^ ??? uses old ip
* TCP_NODELAY set
* connect to 127.0.0.102 port 80 failed: Connection refused
* Failed to connect to ya.ru port 80: Connection refused
* Closing connection 1
Traceback (most recent call last):
  File "", line 1, in 
pycurl.error: (7, 'Failed to connect to ya.ru port 80: Connection refused')
>>> 

If I remove fake dns entry a request is made to the real ip:

# continuing with the same curl handle
>>> c.setopt(c.RESOLVE,['-ya.ru:80'])
>>> c.perform()
* Rebuilt URL to: http://ya.ru/
*   Trying 87.250.250.242...
* TCP_NODELAY set
* Connected to ya.ru (87.250.250.242) port 80 (#2)
> GET / HTTP/1.1
Host: ya.ru
User-Agent: PycURL/7.43.0.1 libcurl/7.56.1 OpenSSL/1.0.2l zlib/1.2.8 libidn2/2.0.2 libpsl/0.18.0 (+libidn2/2.0.2) libssh2/1.8.0 nghttp2/1.26.0 librtmp/2.3
Accept: */*

< HTTP/1.1 302 Found
< Date: Sat, 09 Dec 2017 05:52:48 GMT
< Cache-Control: no-cache,no-store,max-age=0,must-revalidate
< Location: https://ya.ru/
< Expires: Sat, 09 Dec 2017 05:52:49 GMT
< Last-Modified: Sat, 09 Dec 2017 05:52:49 GMT
< P3P: policyref="/w3c/p3p.xml", CP="NON DSP ADM DEV PSD IVDo OUR IND STP PHY PRE NAV UNI"
< Set-Cookie: yandexuid=4809395281512798769; Expires=Tue, 07-Dec-2027 05:52:48 GMT; Domain=.ya.ru; Path=/
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Content-Length: 0
< 
* Connection #2 to host ya.ru left intact
>>> 

But now if I set another fake ip, curl makes a connection to the real ip:

# still using the same handle
>>> c.setopt(c.RESOLVE,['ya.ru:80:127.0.0.104'])
>>> c.perform()
* Added ya.ru:80:127.0.0.104 to DNS cache
# ^ ok
* Rebuilt URL to: http://ya.ru/
* Found bundle for host ya.ru: 0x559567269060 [can pipeline]
* Re-using existing connection! (#2) with host ya.ru
* Connected to ya.ru (87.250.250.242) port 80 (#2)
> GET / HTTP/1.1
Host: ya.ru
User-Agent: PycURL/7.43.0.1 libcurl/7.56.1 OpenSSL/1.0.2l zlib/1.2.8 libidn2/2.0.2 libpsl/0.18.0 (+libidn2/2.0.2) libssh2/1.8.0 nghttp2/1.26.0 librtmp/2.3
Accept: */*

< HTTP/1.1 302 Found
< Date: Sat, 09 Dec 2017 05:53:51 GMT
< Cache-Control: no-cache,no-store,max-age=0,must-revalidate
< Location: https://ya.ru/
< Expires: Sat, 09 Dec 2017 05:53:52 GMT
< Last-Modified: Sat, 09 Dec 2017 05:53:52 GMT
< P3P: policyref="/w3c/p3p.xml", CP="NON DSP ADM DEV PSD IVDo OUR IND STP PHY PRE NAV UNI"
< Set-Cookie: yandexuid=5327304491512798831; Expires=Tue, 07-Dec-2027 05:53:51 GMT; Domain=.ya.ru; Path=/
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Content-Length: 0
< 
* Connection #2 to host ya.ru left intact
>>> 

OK, maybe I need to also set FORBID_REUSE. Let's try that.

# still on the same curl handle
>>> c.setopt(c.FORBID_REUSE,1)
>>> c.perform()
* Rebuilt URL to: http://ya.ru/
* Found bundle for host ya.ru: 0x559567269060 [can pipeline]
* Re-using existing connection! (#2) with host ya.ru
# ^ ??? Didn't I forbid reuse?
* Connected to ya.ru (87.250.250.242) port 80 (#2)
> GET / HTTP/1.1
Host: ya.ru
User-Agent: PycURL/7.43.0.1 libcurl/7.56.1 OpenSSL/1.0.2l zlib/1.2.8 libidn2/2.0.2 libpsl/0.18.0 (+libidn2/2.0.2) libssh2/1.8.0 nghttp2/1.26.0 librtmp/2.3
Accept: */*

< HTTP/1.1 302 Found
< Date: Sat, 09 Dec 2017 05:55:05 GMT
< Cache-Control: no-cache,no-store,max-age=0,must-revalidate
< Location: https://ya.ru/
< Expires: Sat, 09 Dec 2017 05:55:05 GMT
< Last-Modified: Sat, 09 Dec 2017 05:55:05 GMT
< P3P: policyref="/w3c/p3p.xml", CP="NON DSP ADM DEV PSD IVDo OUR IND STP PHY PRE NAV UNI"
< Set-Cookie: yandexuid=3103926351512798905; Expires=Tue, 07-Dec-2027 05:55:05 GMT; Domain=.ya.ru; Path=/
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Content-Length: 0
< 
* Closing connection 2
>>> 

Let's try this all again:

>>> c=pycurl.Curl()
>>> c.setopt(c.VERBOSE,1)
>>> c.setopt(c.FORBID_REUSE,1)
>>> c.setopt(c.DNS_CACHE_TIMEOUT,0)
>>> c.setopt(c.RESOLVE,['ya.ru:80:127.0.0.104'])
>>> c.setopt(c.URL,'http://ya.ru')
>>> c.perform()
* Added ya.ru:80:127.0.0.104 to DNS cache
* Rebuilt URL to: http://ya.ru/
* Hostname ya.ru was found in DNS cache
*   Trying 127.0.0.104...
* TCP_NODELAY set
* connect to 127.0.0.104 port 80 failed: Connection refused
* Failed to connect to ya.ru port 80: Connection refused
* Closing connection 0
Traceback (most recent call last):
  File "", line 1, in 
pycurl.error: (7, 'Failed to connect to ya.ru port 80: Connection refused')
>>> c.setopt(c.RESOLVE,['-ya.ru:80'])
>>> c.setopt(c.RESOLVE,['ya.ru:80:127.0.0.105'])
>>> c.perform()
* Added ya.ru:80:127.0.0.105 to DNS cache
* Rebuilt URL to: http://ya.ru/
* Hostname ya.ru was found in DNS cache
*   Trying 127.0.0.104...
#  ^ shouldn't it try to connect to .105?
* TCP_NODELAY set
* connect to 127.0.0.104 port 80 failed: Connection refused
* Failed to connect to ya.ru port 80: Connection refused
* Closing connection 1
Traceback (most recent call last):
  File "", line 1, in 
pycurl.error: (7, 'Failed to connect to ya.ru port 80: Connection refused')
>>> 

I think something in the DNS cache/connection reuse does not interact properly with RESOLVE option.

This came out of me investigating pycurl/pycurl#443 which, strictly speaking, appears to be a user error as the user did not disable DNS cache and connection reuse.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions