Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

java.net.ConnectException: Cannot assign requested address (connect failed) #2170

Closed
mihirshah91 opened this issue Nov 30, 2020 · 9 comments
Closed

java.net.ConnectException: Cannot assign requested address (connect failed) #2170

mihirshah91 opened this issue Nov 30, 2020 · 9 comments
Labels
guidance Question that needs advice or information.

Comments

@mihirshah91
Copy link

mihirshah91 commented Nov 30, 2020 •
edited
Loading

I am connecting to DDB using sts assume role policy for cross-account access
During our load-test, I am seeing java.net.ConnectException: Cannot assign requested address (connect failed) when executing read query on DDB table. Under normal load, it works fine. Problem happens when load testing the service. Pasted full stack trace in current behaviour section

I am setting required timeouts on DDB client and also have increased max connections to support load test still seeing this error. Following is DDB client and sts client configs

STS and DDB client settings:

import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration;
import software.amazon.awssdk.core.retry.RetryPolicy;
import software.amazon.awssdk.http.SdkHttpConfigurationOption;
import software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.sts.StsClient;
import software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider;
import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;
import software.amazon.awssdk.utils.AttributeMap;



@Bean(name = STS_BEAN)
public StsAssumeRoleCredentialsProvider assumeRole() {

    AssumeRoleRequest assumeRoleRequest = AssumeRoleRequest.builder().roleArn(roleArn).roleSessionName(roleSessionName).build();
    StsClient stsClient = StsClient.builder()
            .region(Region.of(region))
            .credentialsProvider(awsCredentialsProvider)
            .httpClientBuilder(UrlConnectionHttpClient.builder())
            .build();
    return StsAssumeRoleCredentialsProvider.builder().refreshRequest(assumeRoleRequest).stsClient(stsClient).build();
}

@Bean(name = DDB_CLIENT)
public DynamoDbClient dynamoDB(@Qualifier(
STS_BEAN) StsAssumeRoleCredentialsProvider stsCredentials) {

    ClientOverrideConfiguration configuration = ClientOverrideConfiguration.builder()
            .apiCallAttemptTimeout(Duration.ofMillis(1000))
            .apiCallTimeout(Duration.ofMillis(1500))
            .retryPolicy(RetryPolicy.defaultRetryPolicy())
            .build();

    AttributeMap AWS_SDK_HTTP_CONFIG_OPTIONS = AttributeMap.builder()
            .put(SdkHttpConfigurationOption.CONNECTION_TIMEOUT, Duration.ofMillis(500))
            .put(SdkHttpConfigurationOption.READ_TIMEOUT, Duration.ofMillis(1500))
            .put(SdkHttpConfigurationOption.WRITE_TIMEOUT, Duration.ofMillis(1500))
            .put(SdkHttpConfigurationOption.MAX_CONNECTIONS, 450)
            .put(SdkHttpConfigurationOption.CONNECTION_MAX_IDLE_TIMEOUT, Duration.ofSeconds(300))
            .build();

    return DynamoDbClient.builder()
            .credentialsProvider(stsCredentials)
            .region(Region.of(region))
            .httpClient(UrlConnectionHttpClient.builder().buildWithDefaults(AWS_SDK_HTTP_CONFIG_OPTIONS))
            .overrideConfiguration(configuration)
            .build();
}

Expected Behavior

Mentioned exception should not be thrown under high load

Current Behavior

Full stack trace:

Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Cannot assign requested address (connect failed)
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:205)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:66)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:133)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:159)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:112)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:167)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:94)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
at software.amazon.awssdk.services.dynamodb.DefaultDynamoDbClient.getItem(DefaultDynamoDbClient.java:2152)
Caused by: java.net.ConnectException: Cannot assign requested address (connect failed)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:681)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:167)
at software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient$RequestCallable.call(UrlConnectionHttpClient.java:205)
at software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient$RequestCallable.call(UrlConnectionHttpClient.java:193)
at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:64)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:76)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:55)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:39)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)
... 59 more

Possible Solution

Any pointers on how to avoid Cannot assign requested address (connect failed)? Does it mean that there is no available tcp port for opening socket? If yes, how can I fix it?

Steps to Reproduce (for bugs)

N/A

Context

It affects using sdk java v2 client for our new feature since it doesn't work as expected under high load

Your Environment

  • AWS Java SDK version used: aws-sdk-java v2
  • JDK version used: jdk 1.8
  • Operating System and version:
@mihirshah91
Copy link
Author

Can using UrlConnectionHttpClient instead of Apache http client cause this issue?

@debora-ito debora-ito added guidance Question that needs advice or information. needs-triage This issue or PR still needs to be triaged. labels Dec 1, 2020
@debora-ito
Copy link
Member

Can using UrlConnectionHttpClient instead of Apache http client cause this issue?

Perhaps. According to the documentation, UrlConnectionHttpClient builder only takes connection timeout and socket timeout, the other configurations are ignored. Have you tested with ApacheHttpClient?

Are you creating one new DynamoDB client per request or are you sharing the client instance? We recommend to share the instance if possible - for more best practices: https://github.com/aws/aws-sdk-java-v2/blob/master/docs/BestPractices.md

@debora-ito debora-ito added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 10 days. and removed needs-triage This issue or PR still needs to be triaged. labels Dec 2, 2020
@mihirshah91
Copy link
Author

mihirshah91 commented Dec 3, 2020 •
edited
Loading

Tested with ApacheHttpClient and it has solved our issue. We don't create DDB client for every request and reuse across all requests.

Regarding urlconnection client ignoring those configuration, I wanted to highlight code of buildWithDefaults method which does seem to take in those AttributeMap configs as they both look similar

Urlconnection client method - https://github.com/aws/aws-sdk-java-v2/blob/master/http-clients/url-connection-client/src/main/java/software/amazon/awssdk/http/urlconnection/UrlConnectionHttpClient.java#L343

ApacheHttp client method - https://github.com/aws/aws-sdk-java-v2/blob/master/http-clients/apache-client/src/main/java/software/amazon/awssdk/http/apache/ApacheHttpClient.java#L583

If we look at error again - Cannot assign requested address (connect failed) - which I think means here is - there were too many concurrent TCP connections open for the kernel and all available source port numbers were used up.

Reading further on internet regarding implementation of java HttpURLConnection client, it looks like HttpURLConnection may not be supporting HTTP persistence connection feature and it tries to open new socket for every request. Or may be aws sdk's UrlConnectionHttpClient class is trying to create new socket for every request which is causing this error.

I am not fully sure but this might be the root-cause here. May be SDK team can further confirm here if they have more idea around UrlConnectionHttpClient implementation. It would be great to confirm the root-cause here and also if there is a way to solve this issue with UrlConnectionHttpClient

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 10 days. label Dec 3, 2020
@zoewangg
Copy link
Contributor

Hi @mihirshah91, apologies for the delayed response.

Regarding urlconnection client ignoring those configuration, I wanted to highlight code of buildWithDefaults method which does seem to take in those AttributeMap configs as they both look similar

UrlHttpConnectionHttpClient does not support all the configurations available in SdkHttpConfigurationOption. You can find the supported configuration in UrlHttpConnectionHttpClient#builder

Here is an example of how to configure them:

aws-sdk-java-v2/http-clients/url-connection-client/src/main/java/software/amazon/awssdk/http/urlconnection/UrlConnectionHttpClient.java

Lines 240 to 249 in 8728dbb

* A builder for an instance of {@link SdkHttpClient} that uses JDKs build-in {@link java.net.URLConnection} HTTP
* implementation. A builder can be created via {@link #builder()}.
*
* <pre class="brush: java">
* SdkHttpClient httpClient = UrlConnectionHttpClient.builder()
* .socketTimeout(Duration.ofSeconds(10))
* .connectionTimeout(Duration.ofSeconds(1))
* .build();
* </pre>
*/

Reading further on internet regarding implementation of java HttpURLConnection client, it looks like HttpURLConnection may not be supporting HTTP persistence connection feature and it tries to open new socket for every request. Or may be aws sdk's UrlConnectionHttpClient class is trying to create new socket for every request which is causing this error.

The Java SDK's UrlConnectionHttpClient is just a wrapper around HttpURLConnection and doesn't really do anything else.

Per HttpURLConnection doc https://docs.oracle.com/javase/7/docs/technotes/guides/net/http-keepalive.html,

it should by default reuse connections. This can be configured by keep.alive property (default to be true)

The maximum number of connections per destination to be kept alive is default to 5. You can configure it viahttp.maxConnections based on your use-case.

Could you try tuning those configurations to see if it helps?

@debora-ito debora-ito added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 10 days. label Dec 10, 2020
@github-actions
Copy link

It looks like this issue hasn’t been active in longer than a week. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

@github-actions github-actions bot added the closing-soon This issue will close in 4 days unless further comments are made. label Dec 17, 2020
@mihirshah91
Copy link
Author

Hi @zoewangg thanks for the info.

How can we configure http.maxConnections in UrlHttpConnectionHttpClient? In my question, I have given that i tried to set it using .put(SdkHttpConfigurationOption.MAX_CONNECTIONS, 450) on AttributeMap object but sdk ignored it

@github-actions github-actions bot removed closing-soon This issue will close in 4 days unless further comments are made. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 10 days. labels Dec 17, 2020
@zoewangg
Copy link
Contributor

UrlHttpConnectionHttpClient does not have an API level configuration for maxConnection.

You can configure it using System.setProperty("http.maxConnections", "50"); or adding the system property to start your application, eg:java -Dhttp.maxConnections=50 MyService.jar.

@mihirshah91
Copy link
Author

ok, thanks. I am closing this issue. As a resolution we had moved to apache client. But will keep this in mind if we want to switch back

@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

dagnir added a commit that referenced this issue Oct 19, 2022
@dagnir
Merge pull request #2170 from aws/dongie/s3-rules-update
786c5cb 
Update S3 RuleSet
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
guidance Question that needs advice or information.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@debora-ito @mihirshah91 @zoewangg