Apache HttpClient interception setup failed

[mivanov]

Hi Tim, I've worked a bit with HTTP Toolkit a while back, and now trying to use it in a different context.

The issue I am seeing is that, after attaching to a running JVM process, I am able to see some HTTP traffic being intercepted, but HTTPS traffic fails with the error below. This Apache HttpClient is being used by the AWS SDK.

I thought this had something to do with the root CA so I imported the HTTP Toolkit cert, but this did not seem to change anything. Not sure this is even needed since from my understanding HTTP Toolkit is doing this part automatically when setting up the interception.

Here is my java version:

openjdk version "11.0.11" 2021-04-20 LTS
OpenJDK Runtime Environment Corretto-11.0.11.9.1 (build 11.0.11+9-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.11.9.1 (build 11.0.11+9-LTS, mixed mode)

And full log with stack trace:

HTTP Toolkit interception active
java.lang.IllegalStateException: Apache HttpClient interception setup failed
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:393) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.upgrade(DefaultHttpClientConnectionOperator.java:191) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.upgrade(PoolingHttpClientConnectionManager.java:392) ~[httpclient-4.5.13.jar:4.5.13]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) ~[aws-java-sdk-core-1.11.1034.jar:?]
	at com.amazonaws.http.conn.$Proxy33.upgrade(Unknown Source) ~[?:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:428) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar:4.5.13]
	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[aws-java-sdk-core-1.11.1034.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1331) ~[aws-java-sdk-core-1.11.1034.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) ~[aws-java-sdk-core-1.11.1034.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) ~[aws-java-sdk-core-1.11.1034.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) ~[aws-java-sdk-core-1.11.1034.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) ~[aws-java-sdk-core-1.11.1034.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) ~[aws-java-sdk-core-1.11.1034.jar:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) ~[aws-java-sdk-core-1.11.1034.jar:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) ~[aws-java-sdk-core-1.11.1034.jar:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) ~[aws-java-sdk-core-1.11.1034.jar:?]
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:4243) ~[aws-java-sdk-dynamodb-1.11.550.jar:?]
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:4210) ~[aws-java-sdk-dynamodb-1.11.550.jar:?]
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeQuery(AmazonDynamoDBClient.java:2626) ~[aws-java-sdk-dynamodb-1.11.550.jar:?]
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.query(AmazonDynamoDBClient.java:2593) ~[aws-java-sdk-dynamodb-1.11.550.jar:?]
	at com.amazonaws.services.dynamodbv2.document.internal.QueryCollection.firstPage(QueryCollection.java:53) ~[aws-java-sdk-dynamodb-1.11.550.jar:?]
	at com.amazonaws.services.dynamodbv2.document.internal.PageIterator.next(PageIterator.java:45) ~[aws-java-sdk-dynamodb-1.11.550.jar:?]
	at com.amazonaws.services.dynamodbv2.document.internal.IteratorSupport.nextResource(IteratorSupport.java:87) ~[aws-java-sdk-dynamodb-1.11.550.jar:?]
	at com.amazonaws.services.dynamodbv2.document.internal.IteratorSupport.hasNext(IteratorSupport.java:55) ~[aws-java-sdk-dynamodb-1.11.550.jar:?]
	[application code]

I'm using HTTP Toolkit Version 1.4.4 (1.4.4) on macos.

[pimterry]

Hi @mivanov. Thanks for reporting this, that's one I haven't seen before. Is there a demo somewhere you can share that I could use to reproduce this? Which version of the AWS SDK & Apache HttpClient are being used here? If you launch the process from an intercepted terminal, instead of attaching to it when it's already running, do you see the same error?

You are right that you shouldn't need to import the certificate, and this should work automatically. It looks like something unexpected is happening in some of the low-level APIs that HTTP Toolkit is hooking into, but it's hard to debug further without reproducing it myself.

[mivanov]

I haven't had success reproducing this in isolation yet. Trying to use this version of Apache HttpClient directly it seems to work for simple use cases. So I suspect it may have something to do with the ConnectionManager used by the AWS SDK.

I will attempt to reproduce this with minimal code and report back.

You can see the versions of the libs in the stacktrace above:

aws-java-sdk-core-1.11.1034
httpclient-4.5.13

[pimterry]

Ok, thanks, that's useful. Unfortunately I think without reproducing the issue locally it's going to be very difficult for me to fix this... If you do manage to find any kind of setup that I could use to reproduce this and debug it myself though, I'm very happy to investigate further! Let me know how it goes.

[davidhsv]

I'm having the same issue but for a different environment (JAX-RS + apache httpclient). For what I saw, the Object thisFactory in ApacheSetSslSocketFactoryAdvice is not what the code expect. It's an anonymous class, and thus the getDeclaredField don't work.

I would suggest to change ApacheSetSslSocketFactoryAdvice to

package tech.httptoolkit.javaagent.advice.apacheclient;

import net.bytebuddy.asm.Advice;

import javax.net.ssl.SSLContext;
import java.lang.reflect.Field;
import java.util.Arrays;

public class ApacheSetSslSocketFactoryAdvice {

    @Advice.OnMethodEnter
    public static void beforeCreateSocket(@Advice.This Object thisFactory) throws Exception {
        // Before creating the socket - replace the SSL context so the new socket trusts us.

        boolean intercepted = false;
        for (String factoryFieldName : Arrays.asList("socketfactory", "socketFactory")) {
            try {
                // Detect which field(s) are present on this class
                Field field = getDeclaredFieldInClassTree(thisFactory.getClass(), factoryFieldName);

                // Allow ourselves to change the socket factory value
                field.setAccessible(true);

                // Overwrite the socket factory with our own:
                field.set(thisFactory, SSLContext.getDefault().getSocketFactory());
                intercepted = true;
            } catch (NoSuchFieldException ignored) { }
        }

        if (!intercepted) {
            throw new IllegalStateException("Apache HttpClient interception setup failed");
        }
    }

    public static Field getDeclaredFieldInClassTree(Class<?> type, String fieldName) throws NoSuchFieldException {
        for (Class<?> clazz = type; clazz != null; clazz = clazz.getSuperclass()) {
            try {
                return clazz.getDeclaredField(fieldName);
            } catch (NoSuchFieldException ignored) { }
        }
        throw new NoSuchFieldException();
    }
    
}

I could not build the project, nor I found instructions to build it (I'm having unresolved references import com.sun.tools.attach.*)

Can you please build and send me a jar, so I can test in my environment.
Sorry for the inconvenience.

[davidhsv]

The code that is triggering the issue:

Invocation.Builder webTarget = ClientBuilder.newClient().target("https://google.com")
                .request(MediaType.APPLICATION_JSON_TYPE);
        return webTarget.buildPost(construirEntityFormUrlEncoded(requisicao)).invoke();

[pimterry]

Hi @davidhsv

I'm having unresolved references import com.sun.tools.attach.*

This usually means you're using a JVM that isn't fully supported, e.g. Java 8. The agent can run in Java 8 when attaching at startup with -javaagent but attaching externally at runtime requires Java 11+, as the APIs aren't available in older JVMs. That's for OpenJDK & the Oracle JVM, but I suspect other runtimes will be similar.

I could not build the project, nor I found instructions to build it

If you're using a recent JVM, the setup steps should be pretty standard - you can look at the GitHub CI job as an example: https://github.com/httptoolkit/jvm-http-proxy-agent/blob/main/.github/workflows/ci.yml. It should just be a matter of cloning the repo and running ./gradlew quickTest with a recent JVM. Or if you use an IDE like Intellij it should detect the Gradle tasks available automatically.

Does that work for you?

Thanks for pointing this out btw, I'll add some notes to the README about this now.

I would suggest to change ApacheSetSslSocketFactoryAdvice to...

At a glance, this looks sensible to me, but I don't have time in the very short term to set this up and fully test it myself. If you're interested in this, could you create a new test case in https://github.com/httptoolkit/jvm-http-proxy-agent/tree/main/test-app/src/main/java/tech/httptoolkit/testapp/cases, reference it here, check that fails with the current code, and then test that your change fixes it?

If you can open a PR that fixes this issue without breaking the existing cases, I'm very happy to review & ship it. By the way, in case you're not aware, HTTP Toolkit Pro is totally free for all contributors - I'd be happy to set you up with a free account if you can help fix this issue.

[davidhsv]

Thanks for the explanation, I will get back to it Monday and I send you a PR them. Have a nice weekend!

[pimterry]

OK, great! You too 👍

[davidhsv]

I added the test case and the fix. I was not able to get the file generated from github action, put in /Applications/HTTP Toolkit.app/Contents/Resources/httptoolkit-server/overrides/java-agent.jar and make it work. I always get

Caused by: java.lang.ClassNotFoundException: tech.httptoolkit.javaagent.advice.apacheclient.ApacheSetSslSocketFactoryAdvice from [Module "org.apache.httpcomponents.core" from local module loader @5c42d2b7 (finder: local module finder @625abb97 (roots: /usr/local/pje/wildfly-26.0.1.Final/modules,/usr/local/pje/wildfly-26.0.1.Final/modules/system/layers/base))]
	at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:200)

Any tips how to make it work? I'm using the file generated here https://github.com/davidhsv/jvm-http-proxy-agent/actions/runs/2771461785

Thanks!

[pimterry]

Hi @davidhsv - that change looks good, so I'm not sure what could cause the error you're seeing there. Just copying the agent JAR into place should be all the setup that's required.

I've just given this a quick test myself, and it all seems fine AFAICT. How exactly are you testing this? That class is indeed present in your JAR, so there must be something unusual happening in the classloader in your specific setup to stop it being loaded.

[davidhsv]

I tried the .jar with my change on a demo project using Spring Boot and everything worked fine. After, I tried the same project in .war + Wildfly 26.0.1.Final and I was able to replicate the ClassNotFoundException bug. Same JDK.

So I searched and I found that you should start Wildfly with -Djboss.modules.system.pkgs=tech.httptoolkit to be able to attach the agent in a running Wildfly... Not sure if we can bypass it in HTTP Toolkit.

[davidhsv]

So yes, everything is working fine after my fix + -Djboss.modules.system.pkgs=tech.httptoolkit. Thank for your help 🙇

[davidhsv]

Ideally, we would crash and print in the console that we need to use this configuration above when attaching to a non-configured Wildfly. What do you think?