Jersey Client API with Apache connector hangs on 2nd retry when using retryIf

[bennybauer]

Hi,

When I'm using retryIf in the RetryPolicy to make http calls with Jersey Client API and the apache connector the second retry will make the http call but it will hang and no response will be retrieved. However, it will work as expected if this is done without providing the connector, i.e. Jersey Client API will use the default HttpUrlConnection.

The apache connector works well on other retry policies, as long as retryIf isn't used.

Here is a test which reproduces the issue:

// MyTest.java
import net.jodah.failsafe.Failsafe;
import net.jodah.failsafe.RetryPolicy;
import net.jodah.failsafe.function.Predicate;
import org.glassfish.jersey.apache.connector.ApacheConnectorProvider;
import org.glassfish.jersey.client.ClientConfig;
import org.glassfish.jersey.client.ClientProperties;
import org.junit.Test;

import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.core.Response;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.concurrent.Callable;
import java.util.concurrent.TimeUnit;

import static org.junit.Assert.assertEquals;

public class MyTest {
   @Test
    public void testRetryIfHangs() {
        RetryPolicy retryPolicy = new RetryPolicy()
                .withDelay(500, TimeUnit.MILLISECONDS)
                .withMaxRetries(2)                          // works with one retry
                .retryIf(new Predicate<Response>() {
                    public boolean test(Response response) {
                        return response != null && response.getStatus() == 429;
                    }
                });

        ClientConfig clientConfig = new ClientConfig();
        clientConfig.property(ClientProperties.CONNECT_TIMEOUT, 1000);
        clientConfig.property(ClientProperties.READ_TIMEOUT, 500);
        clientConfig.connectorProvider(new ApacheConnectorProvider());   // works well when this line is commented out
        final Client httpClient = ClientBuilder.newClient(clientConfig);

        Response response = Failsafe.with(retryPolicy).get(new Callable<Response>() {
            public Response call() {
                String url = "http://httpstat.us/429";
                System.out.println(new SimpleDateFormat("HH:mm:ss.SSS").format(new Date()) + " Executing " + url);
                return httpClient.target(url)
                        .request("text/plain")
                        .accept("text/plain")
                        .get();
            }
        });

        // will never get here
        assertEquals(429, response.getStatus()); 
    }
}

<!-- pom.xml dependencies -->
<dependencies>
        <!-- Http client -->
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.5</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.core</groupId>
            <artifactId>jersey-client</artifactId>
            <version>${jersey-version}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.connectors</groupId>
            <artifactId>jersey-apache-connector</artifactId>
            <version>${jersey-version}</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
</dependencies>
<properties>
     <jersey-version>2.25</jersey-version>
</properties>

I'm Using Java 7.

Besides that, Failsafe is great! easy, well documented and works well.

Thanks!
Benny

[jhalterman]

Hi @bennybauer

I'm not seeing anything suspicious going on with Failsafe here. It seems like the retry attempts, when using ApacheConnectorProvider just cause the connection attempt to hang. There's not much Failsafe can do about this. It might be that clients created with ApacheConnectorProvider just aren't reusable? Or maybe some other bug going on?

At any rate, a quick workaround I found with your example code is to move the ClientBuilder.newClient(clientConfig) line inside of the Callable, so a new client is created on each connection attempt.

[bennybauer]

Thanks @jhalterman. I've configured it to use connection pooling and it worked:

    @Test
    public void testRetryIfHangs() throws InterruptedException, ExecutionException, IOException {
        RetryPolicy retryPolicy = new RetryPolicy()
                .withDelay(500, TimeUnit.MILLISECONDS)
                .withMaxRetries(2)                        
                .retryIf(new Predicate<Response>() {
                    public boolean test(Response response) {
                        return response != null && response.getStatus() == 429;
                    }
                });

        PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
        cm.setMaxTotal(3);
        cm.setDefaultMaxPerRoute(3);

        ClientConfig clientConfig = new ClientConfig();
        clientConfig.property(ClientProperties.CONNECT_TIMEOUT, 1000);
        clientConfig.property(ClientProperties.READ_TIMEOUT, 500);
        clientConfig.connectorProvider(new ApacheConnectorProvider());
        clientConfig.property(ApacheClientProperties.CONNECTION_MANAGER, cm);
        final Client httpClient = ClientBuilder.newClient(clientConfig);

        Response response = Failsafe.with(retryPolicy).get(new Callable<Response>() {
            public Response call() {
                String url = "http://httpstat.us/429";
                System.out.println(new SimpleDateFormat("HH:mm:ss.SSS").format(new Date()) + " Executing " + url);
                Response result = httpClient.target(url)
                        .request("text/plain")
                        .accept("text/plain")
                        .get();

                result.getEntity();
                return result;
            }
        });

        assertEquals(429, response.getStatus());
    }

[jhalterman]

👍