New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What is the valid value format for the resolve option? #95

Closed
tobischo opened this Issue Nov 27, 2014 · 11 comments

Comments

Projects
None yet
3 participants
@tobischo
Copy link

tobischo commented Nov 27, 2014

Curl has an option which allows me to specify to which IP a domain should be resolved
e.g. curl --resolve example.com:443:1.2.3.4 https://example.com/foo
to make sure that a very specific server is called
(e.g. when multiple servers have the same vhost with a load balancer usually in front of it and there are multiple applications running on the same port with different vhosts)

How do I set this value?
This is how I'd expect it to work
e = Ethon::Easy.new(url: "https://example.com/foo", :resolve => "example.com:443:1.2.3.4")
but I'm getting an invalid value exception
Ethon::Errors::InvalidValue: The value: example.com:443:1.2.3.4 is invalid for option: resolve.

I took a look at the code but couldn't figure out how I'd have to provide the value - and the documentation on this is a bit scarce

Thanks in advance for any reply that might point me in the right direction

@i0rek

This comment has been minimized.

Copy link
Member

i0rek commented Nov 27, 2014

@tobischo thanks for asking! The libcurl docs for that option are here: http://curl.haxx.se/libcurl/c/CURLOPT_RESOLVE.html. There is currently no straight forward way to specify that in ethon.

@martinseener

This comment has been minimized.

Copy link

martinseener commented Nov 28, 2014

@i0rek is there an non-straight forward way? or do you know how we can somehow manage this? net/http also doesn't seem to have such a resolve method. This resolve is quite important if you have a load-balancer behind a domain-name/dns name but want to connect to one of the servers behind the load-balancer directly. but when using https with TLSv1 and SNI (multiple ssl vhosts) you need to call that server with the correct host-name set. therefore you need to override the dns resolved name from the load-balancer IP to the host you want to connect (we need that for smoke testing the application)

any other ideas how we could manage that?

@i0rek

This comment has been minimized.

Copy link
Member

i0rek commented Nov 28, 2014

You could use headers= as a blueprint: https://github.com/typhoeus/ethon/blob/master/lib/ethon/easy/header.rb#L23-32.

resolve = nil
Ethon::Curl.slist_append(resolve, "example.com:443:1.2.3.4")
e = Ethon::Easy.new(url: "https://example.com/foo", :resolve => resolve)
#=> #<Ethon::Easy:0x007faca2574f30 @url="https://example.com/foo", ...>
@i0rek

This comment has been minimized.

Copy link
Member

i0rek commented Nov 28, 2014

That was not too hard... Thanks for asking!

@martinseener

This comment has been minimized.

Copy link

martinseener commented Nov 28, 2014

Looks good! Thanks alot :)

@i0rek

This comment has been minimized.

Copy link
Member

i0rek commented Nov 28, 2014

I noticed one last gotcha: what I wrote down might leak memory. https://github.com/typhoeus/ethon/blob/master/lib/ethon/easy/header.rb#L31 prevents that. If you are going to use it you should take care of that too.

@tobischo

This comment has been minimized.

Copy link
Author

tobischo commented Nov 28, 2014

Thanks! that actually helped me a lot.
Also thanks for the hint on memory leaks. :)

@tobischo tobischo closed this Nov 28, 2014

@tobischo tobischo reopened this Dec 1, 2014

@tobischo

This comment has been minimized.

Copy link
Author

tobischo commented Dec 1, 2014

OK, after playing around a bit now I noticed that the resolve setting does not appear to be working properly

require 'ethon'

resolve = nil
r = "example.com:443:1.2.3.4"
uri = "https://example.com/foo"
Ethon::Curl.slist_append(resolve, r)
e = Ethon::Easy.new(url: uri, resolve: resolve, timeout: 4)
e.perform
=> :operation_timedout 

The Ethon version resulted in a timeout from not being able to resolve it while the curl version did not have any issues and returned the expected response body.
We turned off the SLB to make sure it is going to the correct host on its own

curl https://example.com/foo --resolve example.com:443:1.2.3.4 --max-time 4

I am using Ethon 0.7.1 and curl 7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8z zlib/1.2.5

Now I am wondering whether it is a layer 8 problem (me) or whether the issue can be found in Ethon

@i0rek

This comment has been minimized.

Copy link
Member

i0rek commented Dec 1, 2014

@tobischo my best bet atm is your curl version. 7.24.0 is pretty ancient. Are you able to upgrade? How could I reproduce the problem? I guess I need my own dns for that, no?

@tobischo

This comment has been minimized.

Copy link
Author

tobischo commented Dec 1, 2014

Well, I could upgrade my OS X...
Nonetheless we also tested it with a newer version (7.37.1) which also successfully managed to resolve the host to the IP provided

As for reproducing the problem: you actually don't need a DNS, you just need a server with a vhost somewhere which would respond to the given domain.
The resolve flag should tell curl to not resolve the domain via DNS, but rather just open a connection to the given IP for requests made to a specific domain+port.
In fact using a domain which is not resolved over DNS is probably the best way here because it will just time out if the resolve flag does not work (as it did when I tested it)

Also while playing around and trying to build you a test scenario I actually noticed the mistake preventing me from a successful request:

I set up a vhost on one of my servers (which certainly would not respond when calling that domain otherwise)
curl http://resolvetest.anotherdomain.de --resolve resolvetest.anotherdomain.de:80:1.2.3.4
which works also with my very old curl version

Now to the mistake:

require 'ethon'

resolve = nil
r = "resolvetest.anotherdomain.de:80:1.2.3.4"
uri = "http://resolvetest.anotherdomain.de"
Ethon::Curl.slist_append(resolve, r)
e = Ethon::Easy.new(url: uri, resolve: resolve, timeout: 4)
e.perform

was what I tried first but I should have assigned the result from slist_append to resolve

resolve = Ethon::Curl.slist_append(nil, r)

So yeah, correct for this case would be

require 'ethon'

r = "resolvetest.anotherdomain.de:80:1.2.3.4"
uri = "http://resolvetest.anotherdomain.de"
resolve = Ethon::Curl.slist_append(nil, r)
e = Ethon::Easy.new(url: uri, resolve: resolve, timeout: 4)
e.perform

Anyways, thanks for pointing me in the right direction and taking another look at it although it was just dumb luck that I noticed it now :D

@tobischo tobischo closed this Dec 1, 2014

@i0rek

This comment has been minimized.

Copy link
Member

i0rek commented Dec 1, 2014

@tobischo thanks for looking into that! I guess that was my mistake, I am sorry!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment