"Bad Request, S3 Extended Request ID: null" Caused by a path that is too long

[mgyboom]

Describe the bug

22/12/19 08:29:58 INFO DAGScheduler: Job 1 finished: csv at CsvHandler.java:33, took 31.664408 s

org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on s3a://pvc-21a0520e-a094-4e65-a1df-1fc5b46bbf77/csi-fs/prepared/f66ed11cfb034476aaddf7d55f04e008/56683cce1f7b4ed4a38e55e0e66ef435/3000/id_md5-administrativeCounterpartName-socialCode-legalPepresentative-IDType-IDCard-administrativeCounterpartType-punishmentDecisionNo-date-punishmentCategory-TypesofIllegalActs-punishmentBasis-illegalFacts-punishmentContent-AmountOfPenalty-ConfiscationOfIllegalIncome-totalFines-punishmentAuthority-dataSourceAuthority/_SUCCESS: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 1732248F3C5056FA; S3 Extended Request ID: null), S3 Extended Request ID: null:400 Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 1732248F3C5056FA; S3 Extended Request ID: null)

at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:224)

at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:151)

at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2198)

at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2163)

at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2102)

at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1683)

at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:2984)

at cn.tongdun.sparkdatahandler.Sharding.main(Sharding.java:90)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.base/java.lang.reflect.Method.invoke(Unknown Source)

at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)

at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)

at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)

at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)

at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)

at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 1732248F3C5056FA; S3 Extended Request ID: null), S3 Extended Request ID: null

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1640)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)

at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)

at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4368)

at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4315)

at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1271)

at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$4(S3AFileSystem.java:1249)

at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)

at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:285)

at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1246)

at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2183)

... 17 more

Expected Behavior

No AmazonS3Exception occurs.

Current Behavior

S3AFileSystem.exists(new Path("")

Reproduction Steps

S3AFileSystem.exists(new Path("")

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

1.12.237

JDK version used

11

Operating System and version

debian

[yasminetalby]

Hello @mgyboom ,

Thank you very much for your submission.
Unfortunately there is not information for me to investigate this submission. Could you please provide more context regarding your use case?

Best,

Yasmine

[github-actions]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.