UnknownHostException when behind a proxy

[michahirsch]

I'm using the cloudfoundry client behind an proxy and I get problems since the change to the reactor Netty HttpClient. The Netty HttpClient is using a io.netty.resolver.DefaultNameResolver to resolve the hostname. The hostname should be resolved by the proxy which actually is working but using the InetAddress.getByName(inetHost) will always through an UnknownHostException.

Is there anyway to inject an own implementation of a io.netty.resolver.InetNameResolver?

Caused by: java.net.UnknownHostException: xxxxxxxxxxxxx
    at java.net.InetAddress.getAllByName0(InetAddress.java:1259)
    at java.net.InetAddress.getAllByName(InetAddress.java:1171)
    at java.net.InetAddress.getAllByName(InetAddress.java:1105)
    at java.net.InetAddress.getByName(InetAddress.java:1055)
    at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:40)
    at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:62)
    at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53)
    at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55)
    at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31)
    at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106)
    at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:174)
    at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:142)

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/119193483

The labels on this github issue will be updated when the story is started.

[nebhale]

@michahirsch We've never really had full proxy support and lucked out with the support that Spring's RestTemplate provided. It's definitely on the roadmap, but not currently at the top. I'll have a look today if there is something that we can do easily to get you back up and running.

[nebhale]

@michahirsch I've added support for proxies to the Reactor side of the implementation. Configuration can be done as shown in our integration tests. Note that despite what it appears, this will not configure the Spring bit of the implementation and you'll have to set both sets of configuration parameters until the Spring implementation is completely removed.

Let me know how the latest snapshot works for you.

[michahirsch]

@nebhale thanks for the quick reply. Yes with the RestTemplate it worked quite good with the proxy :). I tried setting the proxy configuration as in your integration test. Unfortunate it didn't work out. Still retrieving the UnknownHostException due the proxy needs to resolve the hostname and it won't work with calling the InetAddress.getByName(inetHost) in the io.netty.resolver.DefaultNameResolver.doResolve(String, Promise<InetAddress>).

Java won't be able to resolve the hostname, this needs to be done by the proxy which e.g. the URLConnection of java is doing if there is an http-proxy set.

[nebhale]

Right, I'm surprised that it's resolving the host name at all. The addition of the Netty HttpProxyHandler should mean that the hostname is placed in the CONNECT call and resolved there. Let me look over some code.

[nebhale]

I just wanted to give you a heads up that we're waiting on a change to Reactor Netty that will give you the functionality that you need. Should arrive in the next couple of days. At that point, the hybrid version of proxy configuration should work properly for you. After that it'll just be us remove the rest of the Spring networking stuff and then you'll be down to the single configuration.

[michahirsch]

@nebhale thanks for the update, sounds great, I looking forward to try it out then :)

[s-bortolussi]

Hello @nebhale . Do you have any update about this issue ?

[smaldini]

@s-bortolussi we are updating some internals in reactor to support it and hopefully come to a resolution later this week that will sync with the CF java client team. I don't think the client opens the low level config of the client yet to the user for now so it will be hard until you get our own workaround.

[s-bortolussi]

@smaldini thanks for the quick reply. Looking forward to getting your workaround.

[nebhale]

@s-bortolussi I'm back at work after a holiday, and I'll make sure that I have a crack at this as soon as I can.

[Zteve]

@nebhale: Is this the same issue as #516?

[omsoni]

@s-bortolussi did you find a workaround ? I am assuming 516 is same issue .

[s-bortolussi]

@omsoni - No workaround found yet.

[nebhale]

@michahirsch Still on the backlog to get done for the 2.0.0 release. Hoping to have Reactor updated for this soon.

[nebhale]

@michahirsch @s-bortolussi this should now be fixed in v2.0.0.M9. Can you please take a look to see if solves your issues?

[michahirsch]

@nebhale awesome thanks a lot, I'll take a look and will try it out.

[smaldini]

@michahirsch It might present an issue in some circumstances that can be manifested by a wrong HOST header when sent out (using the proxy host vs the target host). Let us know if you face that situation.

[nebhale]

Since there's been no response on this issue in a couple of weeks, I'm going to close it. If you'd like to see it re-opened, please comment on the issue and I'll reopen it.

[adrian-herscu]

Using 2.2.0.RELEASE -- I am getting java.nio.channels.UnresolvedAddressException

I am using these builders:

DefaultCloudFoundryOperations.builder()
            .cloudFoundryClient(getCloudFoundryClientFor(configuration))
            .dopplerClient(getDopplerClientFor(configuration))
            .uaaClient(getUaaClientFor(configuration))
            .organization(configuration.organization)
            .space(configuration.space)
            .build()

ReactorCloudFoundryClient.builder()
            .connectionContext(getConnectionContextFor(configuration))
            .tokenProvider(getTokenProviderFor(configuration))
            .build()

DefaultConnectionContext.builder()
            .apiHost(configuration.apiHost)
            .proxyConfiguration(getProxyConfigurationFor(configuration))
            .build()

ProxyConfiguration.builder()
            .host(configuration.proxyHost)
            .port(configuration.proxyPort)
            .build()

[adrian-herscu]

Tried that right now. The same exception is now thrown on different thread (reactor-tcp-nio-1):

java.nio.channels.UnresolvedAddressException
	at sun.nio.ch.Net.checkAddress(Net.java:101)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
	at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:331)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1266)
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:556)
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:541)
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:523)
	at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:985)
	at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:255)
	at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:454)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
	at java.lang.Thread.run(Thread.java:745)

[tareksha]

Hi,

Found the issue. It is acutally in netty's code (or at least how it is used in cf client).

In CF's _DefaultConnectionContext.getHttpClient() the proxy is configured by calling Netty's HttpClientOptions.proxy(). Looking at ClientOptions.proxy() :

https://github.com/reactor/reactor-netty/blob/v0.6.0.RELEASE/src/main/java/reactor/ipc/netty/options/ClientOptions.java#L394

it explicitly resets Netty's configured address resolver to NoopAddressResolver for some reason. This no-op resolver leaves the addresses unresolved during runtime:

https://github.com/netty/netty/blob/netty-4.1.6.Final/resolver/src/main/java/io/netty/resolver/NoopAddressResolver.java#L27

The eventual result is that cf client operations will fail due to unresolved hosts when a proxy is configured. I tried searching around for any code that calls Netty's relevant configuration method Bootstrap.resolver(resolver) but found nothing. Seems Netty always disables resolving when it is configured to go through a proxy.

This pull request should provide the needed API in Netty: reactor/reactor-netty#29

@nebhale

[tareksha]

Hi again,

I think another simpler solution is to simply resolve the proxy host as InetAddress and setting a full InetSocketAddress for the proxy in Netty's HttpClientOptions.proxy(). This should be enough because resolving addresses other than the proxy itself is not necessary - the proxy is used for that. No Netty modifications are needed here..

pull request: #614

@michahirsch
@nebhale

[michahirsch]

@tareqhs I've checked out your branch and use it, but seems like is the same problem, or maybe I did something wrong.

java.nio.channels.UnresolvedAddressException
	at sun.nio.ch.Net.checkAddress(Net.java:123)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:621)
	at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:331)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1266)
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:556)
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:541)
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:523)
	at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:985)
	at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:255)
	at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:454)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
	at java.lang.Thread.run(Thread.java:745)

[varanasipavan6]

Any solution when this occurs in case of direct connection when manual ping successful to the server?

[twoseat]

@varanasipavan6 Please open a new issue with more details of your issue.