Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is there a way to reuse AsyncHttpClient asynchronously for different proxy servers? #1880

Closed
couldbejake opened this issue May 9, 2023 · 2 comments

Comments

@couldbejake
Copy link

I'm needing to create a large amount of clients / accommodate a large number of requests using different proxy servers.

I notice the object immutable, would reflection be required to update the client, or is there a solution I'm not seeing.

Thanks!

@couldbejake
Copy link
Author

I have updated the code by doing some research, but now continually get the error

"java.util.concurrent.atomic.AtomicReferenceFieldUpdater.compareAndSet(Object, Object, Object)" because "org.asynchttpclient.netty.NettyResponseFuture.EX_EX_UPDATER" is null

[AsyncHttpClient-4-27] ERROR org.asynchttpclient.netty.handler.HttpHandler - Cannot invoke "java.util.concurrent.atomic.AtomicReferenceFieldUpdater.compareAndSet(Object, Object, Object)" because "org.asynchttpclient.netty.NettyResponseFuture.EX_EX_UPDATER" is null
java.lang.NullPointerException: Cannot invoke "java.util.concurrent.atomic.AtomicReferenceFieldUpdater.compareAndSet(Object, Object, Object)" because "org.asynchttpclient.netty.NettyResponseFuture.EX_EX_UPDATER" is null
	at org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:239)
	at org.asynchttpclient.netty.request.NettyRequestSender.abort(NettyRequestSender.java:392)
	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.exceptionCaught(AsyncHttpClientHandler.java:211)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:230)
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:209)
	at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:230)
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:209)
	at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:230)
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:209)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:416)
	at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:79)
	at io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:144)
	at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
	at io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:223)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:230)
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:209)
	at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:79)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:230)
	at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:209)
	at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:950)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:87)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:447)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:401)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:371)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.base/java.lang.Thread.run(Thread.java:1589)

I will continue investigating, but if anyone has a quick fix, please share!


/* Thread class */

public TaskProcessor() {

    AsyncHttpClientConfig config = new DefaultAsyncHttpClientConfig.Builder().build();
    this.c = asyncHttpClient(config);

}

public void processNextTask() {

    ProxyServer proxy1 = new ProxyServer.Builder("proxy1.example.com", 8080).build();
    
    Request request1 = new RequestBuilder("GET")
            .setUrl("http://example.com/")
            //.setProxyServer(proxy1)
            .build();

        c.executeRequest(request1, new handler(c, this));
}

/* consumer class */

import org.asynchttpclient.*;

import java.io.IOException;

public class handler implements AsyncHandler<Integer> {
    
    private final AsyncHttpClient c;
    private final TaskProcessor processor;
    private Integer status;

    public handler(AsyncHttpClient c, TaskProcessor TaskProcessor) {
        this.c = c;
        this.processor = TaskProcessor;
    }


    @Override
        public AsyncHandler.State onStatusReceived(HttpResponseStatus responseStatus) throws Exception {
            status = responseStatus.getStatusCode();
            if(status >= 200 && status < 400){
                this.processor.getScraper().logger.increaseSuccessRequestCount();
            } else {
                this.processor.getScraper().logger.increaseFailedRequestCount();
            }
            return AsyncHandler.State.ABORT;
        }

        @Override
        public AsyncHandler.State onHeadersReceived(HttpResponseHeaders headers) throws Exception {
            return AsyncHandler.State.ABORT;
        }


        @Override
        public AsyncHandler.State onBodyPartReceived(HttpResponseBodyPart bodyPart) throws Exception {
            return AsyncHandler.State.ABORT;
        }

        @Override
        public Integer onCompleted() throws Exception{
            return status;

        }

        @Override
        public void onThrowable(Throwable t) {
            
        }

}

@couldbejake
Copy link
Author

Looks like this was patched in the latest version!

@hyperxpro hyperxpro self-assigned this Nov 11, 2023
@hyperxpro hyperxpro added Defect and removed Defect labels Nov 11, 2023
@hyperxpro hyperxpro removed their assignment Nov 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants