bug: Localstack doesn't work properly with SQS client from AWS SDK v2 version 2.21.18 and above

[mihalyr]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

My issue is described in this issue on AWS SDK v2 aws/aws-sdk-java-v2#4759

I thought it is an SDK bug as it appeared once I updated the SDK to 2.21.18 or above. But it seems that SDK works with SQS service and it is only breaking functionality when used with Localstack. I think there was a protocol change in 2.21.18 on SQS side that might not be implemented on Localstack.

This problem is now causing failures all over the SQS integration tests using Localstack.

Expected Behavior

• SQS SDK SendMessageBatchResponse.hasFailed() should return false when all messages were sent successfully (no failed items) -> this is returning true on SDK 2.21.18+ with Localstack
• SQS SDK ReceiveMessageResponse.hasMessages() should return false when there were no messages received -> this is returning true on SDK 2.21.18+ with Localstack

How are you starting LocalStack?

Custom (please describe below)

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

Using TestContainers

  static final LocalStackContainer localStack = new LocalStackContainer(localStackImage)
      .withEnv("DYNAMODB_IN_MEMORY", "1")
      .withEnv("SQS_DISABLE_CLOUDWATCH_METRICS", "1")
      // Set LOCALSTACK_HOST explicitly to avoid issues running docker-in-docker in CI, where TestContainers
      // might automatically try to map the dind container's hostname to it
      .withEnv("LOCALSTACK_HOST", "localhost")
      // Bind init scripts to localstack container
      // These will ensure all necessary stacks are created before the tests run
      .withCopyFileToContainer(MountableFile.forHostPath(LOCALSTACK_INIT_SCRIPT), LOCALSTACK_INIT_SCRIPT_MOUNT)
      .withCopyFileToContainer(MountableFile.forHostPath(JSON_STORE_STACK_TEMPLATE), JSON_STORE_STACK_TEMPLATE_MOUNT)
      .withCopyFileToContainer(MountableFile.forHostPath(ID_QUEUE_STACK_TEMPLATE), ID_QUEUE_STACK_TEMPLATE_MOUNT)
      .withCopyFileToContainer(MountableFile.forHostPath(TASK_PROCESSING_QUEUE_STACK_TEMPLATE),
          TASK_PROCESSING_QUEUE_STACK_TEMPLATE_MOUNT)
      // Add logging to see what's happening inside the container
      .withLogConsumer(new Slf4jLogConsumer(containerLog))
      .waitingFor(new WaitAllStrategy()
          .withStrategy(new HostPortWaitStrategy())
          .withStrategy(new LogMessageWaitStrategy()
              .withRegEx("localstack\\-init\\.sh finished\\R"))
          .withStartupTimeout(Duration.ofMinutes(1)));

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

For example the following fails with Localstack on 2.21.18+ but works on 2.21.17. In both cases there are no failed items, but somehow the SDK will create a different List instance internally which causes this method to change behavior. This doesn't happen with the SQS service, only with Localstack it seems and only after AWS SDK v2 release 2.21.18 and later after the SQS protocol update.

    final SendMessageBatchResponse response = sqsClient.sendMessageBatch(
          SendMessageBatchRequest.builder()
              .queueUrl(queueUrl)
              .entries(messages)
              .build()
      );

    assertThat(response.hasFailed()).isFalse();

Environment

- OS: Fedora 38
- LocalStack: latest docker hub image as of today (`sha256:ecae56493567b8e115b48a08e1e47a5c5961776c622254ae1eb9fb91cb9ee0ab`)
- JDK 21
- TestContainers 1.19.3

Anything else?

No response

[localstack-bot]

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.

[Nikola-Milovic]

Posted on Slack, but it seems there's an issue already reported. I am experiencing not being able to find the SQS queue, not sure if its related to the update as well

failed to get queue analytics.fifo url, not found, ResolveEndpointV2

func NewQueue(ctx context.Context, cfg SQSConfig) (Queue, error) {
	sqsclient := sqs.NewFromConfig(awsutil.Config)
	output, err := sqsclient.GetQueueUrl(ctx, &sqs.GetQueueUrlInput{
		QueueName: aws.String(cfg.QueueName),
	})
	if err != nil || output.QueueUrl == nil {
		return nil, fmt.Errorf("failed to get queue %s url, %w", cfg.QueueName, err)
	}

	return SQS{
		queueURL: *output.QueueUrl,
		client:   sqsclient,
	}, nil
}

> awslocal sqs get-queue-url --queue-name analytics.fifo
{
    "QueueUrl": "http://sqs.eu-central-1.localhost.localstack.cloud:4566/000000000000/analytics.fifo"
}

func getAWSConfig(ctx context.Context) (aws.Config, error) {
	var opts []func(*config.LoadOptions) error
	opts = append(opts, config.WithLogger(&AWSLogger{}))

	if os.Getenv("ENVIRONMENT") == "local" {
		customResolver := (func(service, region string, opts ...interface{}) (aws.Endpoint, error) {
			return aws.Endpoint{
				PartitionID:   "aws",
				URL:           "http://localstack:4566",
				SigningRegion: "eu-central-1",
			}, nil
		})

		credentialsProvider := aws.CredentialsProviderFunc(func(context.Context) (aws.Credentials, error) {
			return aws.Credentials{
				CanExpire:       false,
				AccessKeyID:     "test",
				SecretAccessKey: "test",
			}, nil
		})

		opts = append(opts, config.WithEndpointResolverWithOptions(aws.EndpointResolverWithOptionsFunc(customResolver)))
		opts = append(opts, config.WithCredentialsProvider(credentialsProvider))
	} else {
		opts = append(opts, config.WithRegion(os.Getenv("AWS_REGION")))
	}

	return config.LoadDefaultConfig(ctx, opts...)
}

EDIT: this issue is for golang for me, not java

github.com/aws/aws-sdk-go-v2 v1.24.0
github.com/aws/aws-sdk-go-v2/config v1.18.42
github.com/aws/aws-sdk-go-v2/service/sqs v1.24.5

[erikv85]

Also having problems with 2.21.18 and above, specifically getting SqsException: Service returned HTTP status code 400 (Service: Sqs, Status Code: 400, Request ID: null) from SqsClient.getQueueUrl(GetQueueUrlRequest).

[thrau]

I believe the general issue of SDK compatibility was fixed in #8268 and released in 3.0. I tested it with latest and it should work.
This includes @erikv85 and @Nikola-Milovic reports. If these are not solved with localstack >=3.0.1, please open new issues with specific reproduction instructions.

The issue reported by Robert @mihalyr related to Java SDK's SendMessageBatchResponse.hasFailed() always returning true is addressed in #9998.

Here's the code i used.

package cloud.localstack;

import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.http.urlconnection.UrlConnectionSdkHttpService; // Import the HTTP service
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.sqs.SqsClient;
import software.amazon.awssdk.services.sqs.model.*;

import java.net.URI;

public class Main {

    public static void main(String[] args) {
        // Specify the region
        Region region = Region.US_EAST_1;

        // Create an SQS client
        SqsClient sqsClient = SqsClient.builder()
                .region(region)
                .credentialsProvider(DefaultCredentialsProvider.create())
                .httpClientBuilder(new UrlConnectionSdkHttpService().createHttpClientBuilder())
                .endpointOverride(URI.create("http://localhost.localstack.cloud:4566"))
                .build();

        // Create an SQS queue
        String queueName = "MyQueue";
        CreateQueueRequest createQueueRequest = CreateQueueRequest.builder()
                .queueName(queueName)
                .build();
        CreateQueueResponse createQueueResponse = sqsClient.createQueue(createQueueRequest);
        String queueUrl = createQueueResponse.queueUrl();
        System.out.println("Queue created: " + queueUrl);

        // get the queue URL (Addresses https://github.com/localstack/localstack/issues/9832#issuecomment-1849989910)
        var getQueueUrlResponse = sqsClient.getQueueUrl(GetQueueUrlRequest.builder().queueName(queueName).build());
        System.out.println("Queue URL from getQueueUrl: " + getQueueUrlResponse.queueUrl());

        // Send a message to the queue
        var sendMessageRequest = SendMessageBatchRequest.builder()
                .queueUrl(queueUrl)
                .entries(
                        SendMessageBatchRequestEntry.builder().id("1").messageBody("message-1").build(),
                        SendMessageBatchRequestEntry.builder().id("2").messageBody("message-2").build()
                )
                .build();

        var response = sqsClient.sendMessageBatch(sendMessageRequest);
        System.out.println(response.hasFailed());
        assert !response.hasFailed();
        System.out.println(response.successful().get(0).messageId());
        System.out.println(response.successful().get(1).messageId());

        // Close the SQS client
        sqsClient.close();
    }

}

output:

Queue created: http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/MyQueue
Queue URL from getQueueUrl: http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/MyQueue
false
0b0996a1-1558-4687-81bd-73068a60b413
6795cb9c-a3ae-4753-b907-118de87c0d80

[thrau]

will be fixed by #9710

[mihalyr]

Wow, didn't know there was such a mess around the protocol changes. Thank you @thrau for the updates, great work, really appreciate the help!