-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using forced Round Robin resolver with async-http-client #1285
Comments
Hi @slandelle, I will take a look if I can help with any of this. WS is not my proficiency for now, maybe would take a look at second one on a free time. But I'm here mostly not about absence of forced RoundRobin in netty-bp, it's very easy to add with just public class RoundRobinInetAddressResolver extends InetNameResolver {
private NameResolver<InetAddress> nameResolver;
/**
* @param executor the {@link EventExecutor} which is used to notify the listeners of the {@link Future} returned by
* {@link #resolve(java.net.SocketAddress)}
* @param nameResolver the {@link NameResolver} used for name resolution
*/
public RoundRobinInetAddressResolver(EventExecutor executor, NameResolver<InetAddress> nameResolver) {
super(executor);
this.nameResolver = nameResolver;
}
@Override
protected void doResolve(String inetHost, Promise<InetAddress> promise) throws Exception {
// hijack the doResolve request, but do a doResolveAll request under the hood
// Note that InetSocketAddress.getHostName() will never incur a reverse lookup here,
// because an unresolved address always has a host name.
resolveAll(inetHost)
.addListener((FutureListener<List<InetAddress>>) future -> {
if (future.isSuccess()) {
List<InetAddress> inetAddresses = future.getNow();
int numAddresses = inetAddresses.size();
if (numAddresses == 0) {
promise.setFailure(new UnknownHostException(inetHost));
} else {
// if there are multiple addresses: we shall pick one at random
// this is to support the round robin distribution
int index =
(numAddresses == 1)? 0 : ThreadLocalRandom.current().nextInt(numAddresses);
promise.setSuccess(inetAddresses.get(index));
}
} else {
promise.setFailure(future.cause());
}
});
}
@Override
protected void doResolveAll(String inetHost, Promise<List<InetAddress>> promise) throws Exception {
nameResolver.resolveAll(inetHost, promise);
}
} Problem is that async-http-client will not pick up this logic at all because of the method how it works with NameResolver. It would just call I will try to think and prepare some PR today to move to the |
Why not just pass a custom |
@slandelle This is what I do! But async-http-client only cares about |
Only The problem is with your "round robin" strategy. You shouldn't be picking a random address, you should be shuffling the whole list. |
This could be done in another way. For example, call resolve() for the next address and use resolveAll only if
It's not mine, it's from netty's master. So, for now this forced round robin thing from netty is incompatible with async-http-client. WDYT? |
No, we don't want to perform name resolution again. We need the information that the name is mapped to multiple IPs. That's how proper user agents work.
Netty DNS stuff is experimental and unstable.
Contribs welcome. I guess
You can plug your own logic and decorate the way you want. That's what I do in Gatling. |
Motivation: Now the ```resolveAll``` method of RoundRobinInetAddressResolver returns results without any rotation and shuffling. As a result, it doesn't force any round-robin for clients that get a result of ```resolveAll``` and use addresses from the result one by one for a connection establishing until success. This commit implements round-robin in RoundRobinInetAddressResolver#resolveAll. These improvements inspired by the discussion here: AsyncHttpClient/async-http-client#1285 Modifications: Rotate collection from internal ```resolveAll``` call by index, which is incremented every call to RoundRobinInetAddressResolver#resolveAll method. Random replaced by an incrementing counter, which makes code cheaper and guarantees predictable address order in tests. Result: Improved ```RoundRobinInetAddressResolver``` is compatible with clients that use ```resolveAll``` result.
Motivation: Now the ```resolveAll``` method of RoundRobinInetAddressResolver returns results without any rotation and shuffling. As a result, it doesn't force any round-robin for clients that get a result of ```resolveAll``` and use addresses from the result one by one for a connection establishing until success. This commit implements round-robin in RoundRobinInetAddressResolver#resolveAll. These improvements inspired by the discussion here: AsyncHttpClient/async-http-client#1285 Modifications: Rotate collection from internal ```resolveAll``` call by index, which is incremented every call to RoundRobinInetAddressResolver#resolveAll method. Random replaced by an incrementing counter, which makes code cheaper and guarantees predictable address order in tests. Result: Improved ```RoundRobinInetAddressResolver``` is compatible with clients that use ```resolveAll``` result.
Closed by netty/netty#5972 |
Motivation: Now the ```resolveAll``` method of RoundRobinInetAddressResolver returns results without any rotation and shuffling. As a result, it doesn't force any round-robin for clients that get a result of ```resolveAll``` and use addresses from the result one by one for a connection establishing until success. This commit implements round-robin in RoundRobinInetAddressResolver#resolveAll. These improvements inspired by the discussion here: AsyncHttpClient/async-http-client#1285 Modifications: Rotate collection from internal ```resolveAll``` call by index, which is incremented every call to RoundRobinInetAddressResolver#resolveAll method. Random replaced by an incrementing counter, which makes code cheaper and guarantees predictable address order in tests. Result: Improved ```RoundRobinInetAddressResolver``` is compatible with clients that use ```resolveAll``` result.
Hi,
We need to have DNS resolution logic for our http client like here https://github.com/netty/netty/pull/5915/files#diff-0babf93c8bd17964680042f4442f38b5R36 (already merged to master), this is my PR to expose it as a NameResolver netty/netty#5915
So, we need to have an even distribution of connections/requests between addresses from DNS. But now async-http-client has very limited ability to implement this logic because it calls
resolveAll()
on NameResolver and pass it in NettyChannelConnector, where always starts with the top of the list. So this submitted RoundRobinInetAddressResolver or already existing RoundRobinInetSocketAddressResolver would not work as expected if passed to async-http-client request.I think it would be great and flexible to rework this approach to always call
resolve()
in async-http-client before attempting to create one more connection, so a logic of connection distribution and host selection would be fully in a NameResolver implementation.WDYT? Maybe you can recommend another way to fix/implement it?
The text was updated successfully, but these errors were encountered: