S3Exception is thrown when using versions above 2.17.295

[ivangfr]

Describe the bug

Hi, I've recently upgraded my project springboot-aws-localstack-opensearch-s3-secretsmanager to LocalStack 1.4.0.

As the project name suggests, it uses: OpenSearch, S3 and SecretsManager.

I am having the S3Exception (see below) when I upgrade software.amazon.awssdk to any version above 2.17.295.

Expected Behavior

The exception should not be thrown.

Current Behavior

Here are the logs

ERROR 7987 --- [           main] o.s.boot.SpringApplication               : Application run failed

java.lang.IllegalStateException: Failed to execute CommandLineRunner
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:772) ~[spring-boot-3.0.3.jar:3.0.3]
	at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:753) ~[spring-boot-3.0.3.jar:3.0.3]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:317) ~[spring-boot-3.0.3.jar:3.0.3]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1304) ~[spring-boot-3.0.3.jar:3.0.3]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1293) ~[spring-boot-3.0.3.jar:3.0.3]
	at com.ivanfranchin.movieapi.MovieApiApplication.main(MovieApiApplication.java:10) ~[classes/:na]
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: exception while calling s3 with unknown operation: Traceback (most recent call last):
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 769, in _parse_xml_string_to_dom
    parser.feed(xml_string)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1, column 4

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/code/localstack/localstack/aws/chain.py", line 90, in handle
    handler(self, self.context, response)
  File "/opt/code/localstack/localstack/aws/handlers/service.py", line 63, in __call__
    return self.parse_and_enrich(context)
  File "/opt/code/localstack/localstack/aws/handlers/service.py", line 76, in parse_and_enrich
    operation, instance = parser.parse(context.request)
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 172, in wrapper
    return func(*args, **kwargs)
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 1059, in parse
    return super().parse(request)
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 172, in wrapper
    return func(*args, **kwargs)
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 567, in parse
    self._parse_payload(request, shape, shape.members, uri_params, final_parsed)
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 603, in _parse_payload
    original_parsed = self._initial_body_parse(request)
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 666, in _initial_body_parse
    return self._parse_xml_string_to_dom(body)
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 772, in _parse_xml_string_to_dom
    raise ProtocolParserError(
localstack.aws.protocol.parser.ProtocolParserError: Unable to parse request (not well-formed (invalid token): line 1, column 4), invalid XML received:
b'3232;chunk-signature=876a557d22b6d07bd822d057f1fb1346cb869872f03ba07cd40f3108173e8475\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xe2\x02(ICC_PROFILE\x00...
 (Service: S3, Status Code: 500, Request ID: MLM9X7RA9P2YCLTQFP4N2OILB0H1EO7U75LG0Q4H4TAE8B7R5Y9I, Extended Request ID: MzRISOwyjmnupUWO8PSXH0V7BRKQDIQQ8RHVH4WAFS7A2QGCIPMIDEGNE59X7PYEP7/JypPGXLh0OVFGcJaaO3KW/hRAqKOpIEEp)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156) ~[aws-xml-protocol-2.20.12.jar:na]
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108) ~[aws-xml-protocol-2.20.12.jar:na]
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85) ~[aws-xml-protocol-2.20.12.jar:na]
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43) ~[aws-xml-protocol-2.20.12.jar:na]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:95) ~[aws-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:270) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[sdk-core-2.20.12.jar:na]
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) ~[aws-core-2.20.12.jar:na]
	at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:9321) ~[s3-2.20.12.jar:na]
	at com.ivanfranchin.movieapi.service.PosterServiceImpl.uploadFile(PosterServiceImpl.java:88) ~[classes/:na]
	at com.ivanfranchin.movieapi.service.PosterServiceImpl.uploadFile(PosterServiceImpl.java:73) ~[classes/:na]
	at com.ivanfranchin.movieapi.runner.S3InitialUploadRunner.run(S3InitialUploadRunner.java:25) ~[classes/:na]
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:769) ~[spring-boot-3.0.3.jar:3.0.3]
	... 5 common frames omitted
	Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: exception while calling s3 with unknown operation: Traceback (most recent call last):
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 769, in _parse_xml_string_to_dom
    parser.feed(xml_string)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1, column 4

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/code/localstack/localstack/aws/chain.py", line 90, in handle
    handler(self, self.context, response)
  File "/opt/code/localstack/localstack/aws/handlers/service.py", line 63, in __call__
    return self.parse_and_enrich(context)
  File "/opt/code/localstack/localstack/aws/handlers/service.py", line 76, in parse_and_enrich
    operation, instance = parser.parse(context.request)
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 172, in wrapper
    return func(*args, **kwargs)
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 1059, in parse
    return super().parse(request)
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 172, in wrapper
    return func(*args, **kwargs)
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 567, in parse
    self._parse_payload(request, shape, shape.members, uri_params, final_parsed)
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 603, in _parse_payload
    original_parsed = self._initial_body_parse(request)
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 666, in _initial_body_parse
    return self._parse_xml_string_to_dom(body)
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 772, in _parse_xml_string_to_dom
    raise ProtocolParserError(
localstack.aws.protocol.parser.ProtocolParserError: Unable to parse request (not well-formed (invalid token): line 1, column 4), invalid XML received:
b'3232;chunk-signature=99a75ea14bcdf935ad5a85690312f72a1c0655a7f03e16d9f530c461e8955ce7\r\n\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xe2\x02(ICC_PROFILE\x00...
 (Service: S3, Status Code: 500, Request ID: 5TO84SGVQ5LO4K8BAB2G6ATUERQHPRVIJNLWPQUCERBTA1EKHFT3, Extended Request ID: MzRISOwyjmnupY5U7KEHU5261QP2JAMG5BFM3K59WBBQY3V1E2TMOU7L74G9OBXLU7/JypPGXLh0OVFGcJaaO3KW/hRAqKOpIEEp)
	Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: exception while calling s3 with unknown operation: Traceback (most recent call last):
  File "/opt/code/localstack/localstack/aws/protocol/parser.py", line 769, in _parse_xml_string_to_dom
    parser.feed(xml_string)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1, column 4

Reproduction Steps

• git clone https://github.com/ivangfr/springboot-aws-localstack-opensearch-s3-secretsmanager.git;
• cd springboot-aws-localstack-opensearch-s3-secretsmanager
• DEBUG=1 docker-compose up -d
• ./init-localstack.sh some-invalid-key
  .... wait the script to finish
• update amazon-awssdk.version in movie.api pom.xml to 2.20.12
• ./mvnw clean spring-boot:run --projects movie-api -Dspring-boot.run.jvmArguments="-Daws.accessKey=key -Daws.secretAccessKey=secret"

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.20.12

JDK version used

openjdk 17.0.6 2023-01-17

Operating System and version

macOS

[debora-ito]

Hey @ivangfr the stacktrace is saying that Localstack was not able to parser your request:
localstack.aws.protocol.parser.ProtocolParserError: Unable to parse request (not well-formed (invalid token): line 1, column 4), invalid XML received

Please reach out to the Localstack support team.

[ivangfr]

Hi @debora-ito Thanks for the help.

Indeed, it's a Localstack issue. I've found this one that explains how to fix. In a nutshell, we need to set forcePathStyle to true while configuring the s3Client.

Best regards,

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[reubenporterjisc]

Setting forcePathStyle: true within the s3 client config fixed this issue in our case as well.