Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ProcessingException: RESTEASY004655: Unable to invoke request: java.net.SocketException: Connection reset #8917

Closed
myifeng opened this issue Nov 25, 2021 · 17 comments
Labels
area/admin/api kind/bug Categorizes a PR related to a bug

Comments

@myifeng
Copy link

myifeng commented Nov 25, 2021

Describe the bug

I get a reset error when I use keycloak to search for users.
This error does not appear every time, only occasionally, but I don't know how to fix it.
Please help~

private boolean verifyAccountFromKeycloak(String account, String email) {
        final UsersResource usersResource = keycloak.realm(currentRealm).users();
        if (CollectionUtils.isNotEmpty(usersResource.search(account, true))) {
            return false;
        }
        List<UserRepresentation> users2 = usersResource.search(null, null, null, email, 0, 20);
        return CollectionUtils.isEmpty(users2) || !users2.stream().anyMatch(s -> email.equals(s.getEmail()));
    }

Version

15.0.2

Expected behavior

No errors or exceptions

Actual behavior

javax.ws.rs.ProcessingException: RESTEASY004655: Unable to invoke request: java.net.SocketException: Connection reset
        at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:328)
        at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:443)
        at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:149)
        at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:112)
        at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:76)
        at com.sun.proxy.$Proxy196.refreshToken(Unknown Source)
        at org.keycloak.admin.client.token.TokenManager.refreshToken(TokenManager.java:111)
        at org.keycloak.admin.client.token.TokenManager.getAccessToken(TokenManager.java:72)
        at org.keycloak.admin.client.token.TokenManager.getAccessTokenString(TokenManager.java:65)
        at org.keycloak.admin.client.resource.BearerAuthFilter.filter(BearerAuthFilter.java:52)
        at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.filterRequest(ClientInvocation.java:579)
        at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:440)
        at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:149)
        at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:112)
        at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:76)
        at com.sun.proxy.$Proxy249.search(Unknown Source)
        at com.commercial.provider.service.impl.UserServiceImpl.lambda$verifyAccountFromKeycloak$0(UserServiceImpl.java:133)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
        ... 1 common frames omitted
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
        at sun.security.ssl.InputRecord.read(InputRecord.java:503)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute$original$3AyxwFtD(InternalHttpClient.java:185)
        at org.apache.http.impl.client.InternalHttpClient.doExecute$original$3AyxwFtD$accessor$QdjClIoY(InternalHttpClient.java)
        at org.apache.http.impl.client.InternalHttpClient$auxiliary$wWrcgHZe.call(Unknown Source)
        at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:323)
        ... 18 common frames omitted

How to Reproduce?

Sorry,This error does not appear every time, only occasionally.

Anything else?

Spring Boot: 2.3.11.RELEASE
JDK Image: openjdk:8u212-jre-slim

@myifeng myifeng added kind/bug Categorizes a PR related to a bug status/triage labels Nov 25, 2021
@stianst
Copy link
Contributor

stianst commented Nov 25, 2021

Looks like a networking issue to me? Can't see anything that would indicate this is a bug in Keycloak.

@myifeng
Copy link
Author

myifeng commented Nov 26, 2021

@stianst In the error log I saw jbossTokenManager.refreshTokenBearerAuthFilter.filter .Is this related to token expiration?

I highly suspect that there is an error in'TokenManager.refreshToken(TokenManager.java:111)':
currentToken = tokenService.refreshToken(config.getRealm(), form.asMap());

An unknown error occurred when the client requested the token, which may be on server or during network transmission.

After the token expired, when the client tried to obtain the token again, the error occurred?

The following is how I use it:

KeycloakClientConfig.java

@RefreshScope
@Configuration
public class KeycloakClientConfig {
    @Value("${project.keycloak.server-url}")
    private String serverUrl;
    @Value("${project.keycloak.realm}")
    private String realm;
    @Value("${project.keycloak.client-id}")
    private String clientId;
    @Value("${project.keycloak.username}")
    private String username;
    @Value("${project.keycloak.password}")
    private String password;

    @Bean
    public Keycloak keycloak() {
        return KeycloakBuilder.builder()
                .serverUrl(serverUrl)
                .realm(realm)
                .clientId(clientId)
                .username(username)
                .password(password)
                .build();
    }
}

UserServiceImpl.java

@Resource
    Keycloak keycloak;

    @Value("${project.keycloak.current-realm}")
    private String currentRealm;

    private boolean verifyAccountFromKeycloak(String account, String email) {
        final UsersResource usersResource = keycloak.realm(currentRealm).users();
        if (CollectionUtils.isNotEmpty(usersResource.search(account, true))) {
            return false;
        }
        List<UserRepresentation> users2 = usersResource.search(null, null, null, email, 0, 20);
        return CollectionUtils.isEmpty(users2) || !users2.stream().anyMatch(s -> email.equals(s.getEmail()));
    }

@abstractj
Copy link
Contributor

@myifeng SocketException is often a network environment issue, instead of a bug on Keycloak. If the issue happens occasionally, we need a reproducible step, otherwise it will be really hard for us to guess what's going on.

@myifeng
Copy link
Author

myifeng commented Dec 23, 2021

@abstractj see #9269 Both of these situations occur when the token expires and the token is re-acquired

@abstractj
Copy link
Contributor

Thanks for taking the time to submit this issue. However, we were unable to reproduce the mentioned steps. We strongly recommend that people upgrade to the latest releases of Keycloak. The WildFly distribution was discontinued and no longer supported by our team.

If you can reproduce the issue using the latest releases of Keycloak, please reopen this issue, including a reproducer.

@abstractj abstractj closed this as not planned Won't fix, can't repro, duplicate, stale Dec 22, 2022
@ghost ghost removed the status/triage label Dec 22, 2022
@roberto-cisternino
Copy link

This error happen also with latest 21.1.1 version and it is during long processes such as the user migration from another system and using the admin client Java APIs.

@transducer
Copy link
Contributor

We have the same issue with long running processes.

@bakoorjakkals
Copy link

Also have the same issue in 21.1.1

@myifeng
Copy link
Author

myifeng commented Aug 31, 2023 via email

@tramiaczek
Copy link

tramiaczek commented Sep 10, 2023

I have the same issue, but possibly found the workaround (it is under tests now). KeycloakBuilder has a method: .resteasyClient(..) to provide custom implementation of javax.ws.rs.client.Client and that implementation could have some settings about connection/read timeouts, evicting idle connections policy and retry handler implementation. Spring configuration for Keycloak bean could look like the following:

    @Bean
    public Client customizedResteasyClient() {
        CloseableHttpClient httpClient = createHttpClient();

        ApacheHttpClient43Engine engine = new ApacheHttpClient43Engine(httpClient);
        return ((ResteasyClientBuilder) ClientBuilder.newBuilder())
                .httpEngine(engine)
                .connectTimeout(10000, TimeUnit.MILLISECONDS)
                .readTimeout(7000, TimeUnit.MILLISECONDS)
                .connectionTTL(-1, TimeUnit.MILLISECONDS)
                .disableTrustManager()
                .build();
    }

    private CloseableHttpClient createHttpClient() {
        PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
        cm.setValidateAfterInactivity(1000);
        cm.setMaxTotal(200);
        cm.setDefaultMaxPerRoute(20);
        return HttpClients.custom()
                .setConnectionManager(cm)
                .evictExpiredConnections()
                .evictIdleConnections(10000, TimeUnit.MILLISECONDS)
                .setRetryHandler(DefaultHttpRequestRetryHandler.INSTANCE)
                .build();
    }

    @Bean
     public Keycloak customizedKeycloakAdminClient() {
        return KeycloakBuilder.builder()
                .grantType(OAuth2Constants.CLIENT_CREDENTIALS)
                .realm("admin")
                .clientId("clientId")
                .clientSecret("clientSecret")
                .serverUrl("http://127.0.0.1:8080")
                .resteasyClient(customizedResteasyClient())
                .build();
    }

I think that pooling settings like retry handler and idle connections eviction mechanism could cause that sockets used to connect through HTTP protocol should be tested before use and connections closed by some external mechanisms like firewall would be rejected by KeycloakAdmin client.

@transducer
Copy link
Contributor

@tramiaczek did this solution work?

I tried reproducing the issue using calls to groups and members periodically (tried a period longer and shorter than Access Token Lifetime), but failed to make a reproduction scenario.

@tramiaczek
Copy link

tramiaczek commented Oct 5, 2023

We have the problem reported from the client's environment - random java.net.SocketException: Connection reset while executing get user operation (in most cases). Our scenario was" create a user (through Keycloak API) and generate some OTP for it (OTP generation code was not related to the Keycloak API, but we needed to get user Id from Keycloak to put it as a reference in our database). Keycloak and the service that invoked the API were deployed as separate pod's in the same Kubernetes namespace - the client's TEST/UAT env and our DEV environments were pretty the same (client followed our instructions to prepare env).

Of course as we could not reproduce the issue, we tried to seek for the solution in a blind manner - we tried to use the latest possible version of keycloak-admin-client library, but it did not help. So we thought that we should have as wide control over the HTTP client as possible and prepared the workaround described above. Our first idea was to turn on some DEBUG logs and observe how the connection pool or timeouts behave on client's env, especially when the problem occurs again. But it did not occur... and the client closed the issue in Jira :) As client swears that he did not change enything in the environment, we suppose that some idle connection eviction/validation mechanims resolved the problem.

@transducer
Copy link
Contributor

@tramiaczek thanks for clarification. Do I understand correctly that when passing the custom client the error did not occur again?

@tramiaczek
Copy link

@tramiaczek thanks for clarification. Do I understand correctly that when passing the custom client the error did not occur again?

Yes, it currently works almost 2 months and the error was not reopened by the client.

@transducer
Copy link
Contributor

I've implemented the solution (here's the Clojure version). Where we first got roughly 30 RESTEASY004655 error messages a day, we now average 3 per day. Mainly coming from the health check that calls the service every 15 seconds thus 5760 times per day. So tenfold decrease in error messages, but there are still some. Maybe can use some further tuning, but already great reduction. Thanks a lot for this improvement @tramiaczek.

@Felk
Copy link

Felk commented Nov 7, 2023

I was able to reliable reproduce the issue using the latest keycloak and keycloak-admin-client (each 22.0.5).

Reproducer setup

I am running on Ubuntu 20.04 (WSL2 on Windows) and OpenJDK 17.

  • Start a keycloak server: docker run -p 8081:8080 -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin quay.io/keycloak/keycloak:22.0.5 start-dev
  • Have tcpkill send RST packets to connections on port 8081 (you may need to sudo apt install dsniff): sudo tcpkill -1 -i lo port 8081
    • This is to simulate a firewall or other network infrastructure closing the connection. In our case we suspect a firewall to kill connections that are open for too long (10+ hours)
  • Set up a java (e.g. maven) project that uses the keycloak-admin-client and run this reproducer:
    public class App {
        public static void main(String[] args) throws InterruptedException {
            Keycloak kc = KeycloakBuilder.builder()
                    .serverUrl("http://localhost:8081")
                    .realm("master")
                    .username("admin")
                    .password("admin")
                    .clientId("admin-cli")
                    .build();
            while (true) {
                tryTalkToKeycloakWithExpiredToken(kc);
                Thread.sleep(100);
            }
        }
      
        private static void tryTalkToKeycloakWithExpiredToken(Keycloak kc) {
            // force the next request to refresh the token
            kc.tokenManager().invalidate(kc.tokenManager().getAccessToken().getToken());
            kc.realm("master").getAdminEvents().size(); // trigger any request
        }
    }

The reproducer may take a few iterations until it hits this exception and exits:

Exception in thread "main" jakarta.ws.rs.ProcessingException: RESTEASY004655: Unable to invoke request: java.net.SocketException: Connection reset
	at org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine.invoke(ManualClosingApacheHttpClient43Engine.java:361)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:427)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:134)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:103)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:61)
	at jdk.proxy2/jdk.proxy2.$Proxy16.refreshToken(Unknown Source)
	at org.keycloak.admin.client.token.TokenManager.refreshToken(TokenManager.java:121)
	at org.keycloak.admin.client.token.TokenManager.getAccessToken(TokenManager.java:77)
	at org.keycloak.admin.client.token.TokenManager.getAccessTokenString(TokenManager.java:70)
	at org.keycloak.admin.client.resource.BearerAuthFilter.filter(BearerAuthFilter.java:52)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.filterRequest(ClientInvocation.java:644)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:424)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:134)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:103)
	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:61)
	at jdk.proxy2/jdk.proxy2.$Proxy29.getAdminEvents(Unknown Source)
	at org.example.App.tryTalkToKeycloakWithExpiredToken(App.java:33)
	at org.example.App.main(App.java:25)
Caused by: java.net.SocketException: Connection reset
	at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:323)
	at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
	at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
	at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine.invoke(ManualClosingApacheHttpClient43Engine.java:344)
	... 17 more

Some iterations may fully succeed, some may only print this message:

Nov 07, 2023 12:51:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://localhost:8081: Connection reset by peer
Nov 07, 2023 12:51:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://localhost:8081

I believe the differences are due to tcpkill working on a best-effort and timing based approach to inject RST packets, which may arrive at the application sooner or later. The desicionmaking whether a request is retried lives inside the DefaultHttpRequestRetryHandler. I observed that retryRequest (the method that decides whether to retry a request) ends with this snippet of code:

        if (!clientContext.isRequestSent() || this.requestSentRetryEnabled) {
            // Retry if the request has not been sent fully or
            // if it's OK to retry methods that have been sent
            return true;
        }
        // otherwise do not retry
        return false;

The code always reaches that statement. I saw that this.requestSentRetryEnabled is always false, and the deciding factor is clientContext.isRequestSent(). My conclusion is that the POST request to refresh the token has been fully sent before a RST packet was received, resulting in the retry mechanism not retrying the request because it might have been processed by the server and can not be assumed to be idempotent.

Possible solutions

Assuming the troublemaker is some network component closing idle connections, the problem can be avoided by auto-closing idle connections and/or testing pooled connections before use. This is what @tramiaczek suggested in his previous comment #8917 (comment)

A minimal configuration to enable auto-closing of connections may look like this:

Keycloak kc = KeycloakBuilder.builder()
        .serverUrl("http://localhost:8081")
        .realm("master")
        .username("admin")
        .password("admin")
        .clientId("admin-cli")
        .resteasyClient(((ResteasyClientBuilder) ClientBuilder.newBuilder())
                .connectionPoolSize(3)
                .connectionTTL(10, TimeUnit.SECONDS)
                .build())
        .build();

For this specific usecase, namely the keycloak-admin-client transparently refreshing its token, it might be useful for it to automatically retry this too, since other connection failures are typically retried as well. Though I am unsure how this would be done.

Alternatively, application-side retry mechanisms may be considered. Given the keycloak admin client may throw connection errors at any time anyway, applications should probably be robust against this.

@souravs17031999
Copy link
Contributor

I have been able to reproduce this still in KC v23.0.6 (keycloak-admin-client).
From time to time, it throws RESTEASY004655: Unable to invoke request: java.net.SocketException: Connection reset

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/admin/api kind/bug Categorizes a PR related to a bug
Projects
None yet
Development

No branches or pull requests

9 participants