Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS S3 - Getting java.net.UnknownHostException: document-storage-service.s3.ap-south-1.amazonaws.com intermittently #3122

Closed
1 task done
iLoveOctoCat opened this issue Jun 18, 2024 · 1 comment
Labels
closed-for-staleness guidance Question that needs advice or information.

Comments

@iLoveOctoCat
Copy link

iLoveOctoCat commented Jun 18, 2024

Upcoming End-of-Support

  • I acknowledge the upcoming end-of-support for AWS SDK for Java v1 was announced, and migration to AWS SDK for Java v2 is recommended.

Describe the bug

Hello Team,

We are getting UnknownHostException for amazon s3 bucket with bucket name document-storage-service.s3.ap-south-1.amazonaws.com intermittently in our spring boot application.

Till now we have tried following things:

  1. Amazon s3 tech support but they asked us to raise the issue on github
  2. We tried setting Security.setProperty("networkaddress.cache.ttl", "0"); in main method before spring boot app start
  3. We are able to ping successfully this host from container where this app is running

Let us know in case any more details are required.

Expected Behavior

spring boot app should be able to resolve s3 host name successfully

Current Behavior

Application is facing UnknownHostException intermittently and downloading the document from AWS S3 fails

Stacktrace of exception:
2024-06-18 17:29:12.228 Caused by: java.net.UnknownHostException: document-storage-service.s3.ap-south-1.amazonaws.com 2024-06-18 17:29:12.228 at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801) 2024-06-18 17:29:12.228 at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509) 2024-06-18 17:29:12.228 at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1367) 2024-06-18 17:29:12.228 at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1301) 2024-06-18 17:29:12.228 at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27) 2024-06-18 17:29:12.228 at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38) 2024-06-18 17:29:12.228 at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112) 2024-06-18 17:29:12.228 at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) 2024-06-18 17:29:12.228 at jdk.internal.reflect.GeneratedMethodAccessor132.invoke(Unknown Source) 2024-06-18 17:29:12.228 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2024-06-18 17:29:12.228 at java.base/java.lang.reflect.Method.invoke(Method.java:568) 2024-06-18 17:29:12.228 at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) 2024-06-18 17:29:12.228 at com.amazonaws.http.conn.$Proxy34.connect(Unknown Source) 2024-06-18 17:29:12.228 at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) 2024-06-18 17:29:12.228 at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) 2024-06-18 17:29:12.228 at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) 2024-06-18 17:29:12.228 at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) 2024-06-18 17:29:12.228 at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) 2024-06-18 17:29:12.228 at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) 2024-06-18 17:29:12.228 at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) 2024-06-18 17:29:12.228 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1331) 2024-06-18 17:29:12.228 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) 2024-06-18 17:29:12.228 ... 99 common frames omitted

Reproduction Steps

NA

Possible Solution

No response

Additional Information/Context

We have tried disabling the network address cache in our application with following code
public static void main(String[] args) { log.info("Disabling network address cache"); Security.setProperty("networkaddress.cache.ttl", "0"); SpringApplication.run(BootApplication.class, args); }
And logs for the same stating that this property was initialized before application and tomcat server start
2024-06-18 17:17:35.194 11:47:35.178 [main] INFO com.pchf.documentconsumerservice.BootApplication -- Disabling network address cache 2024-06-18 17:17:37.368 2024-06-18 17:17:37.368 . ____ _ __ _ _ 2024-06-18 17:17:37.368 /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ 2024-06-18 17:17:37.368 ( ( )\___ | '_ | '_| | '_ \/ _ | \ \ \
2024-06-18 17:17:37.368 \/ )| |)| | | | | || (| | ) ) ) )
2024-06-18 17:17:37.368 ' |
| .__|| ||| |_, | / / / /
2024-06-18 17:17:37.368 =========||==============|/=///_/
2024-06-18 17:17:37.368 :: Spring Boot :: (v3.1.6)
2024-06-18 17:17:37.368
2024-06-18 17:17:37.521 2024-06-18 11:47:37 | INFO | [,] | main | c.p.d.BootApplication | Starting BootApplication v1.0.0-SNAPSHOT using Java 17.0.1 with PID 1 (/app/service.jar started by root in /app)
2024-06-18 17:17:37.522 2024-06-18 11:47:37 | INFO | [,] | main | c.p.d.BootApplication | No active profile set, falling back to 1 default profile: "default"`

AWS Java SDK version used

1.12.261

JDK version used

17

Operating System and version

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

@iLoveOctoCat iLoveOctoCat added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jun 18, 2024
@debora-ito
Copy link
Member

The SDK resolves hosts via InetAddress.getAllByName(), so it sounds like a DNS caching issue in the JVM or the OS. Being an intermittent error also points to a caching issue.

Check this comment from a past UnknownHostException issue: #1503 (comment)

@debora-ito debora-ito added closing-soon This issue will close in 2 days unless further comments are made. guidance Question that needs advice or information. and removed bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jun 19, 2024
@github-actions github-actions bot added closed-for-staleness and removed closing-soon This issue will close in 2 days unless further comments are made. labels Jun 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
closed-for-staleness guidance Question that needs advice or information.
Projects
None yet
Development

No branches or pull requests

2 participants