fix SQS JSON protocol support in ASF

[alexrashed]

This PR aims at providing JSON support for SQS. AWS supports both protocols, query and json, for SQS for some time.

• With version 1.27.127 of botocore (the official AWS Python SDK clients) the protocol of the SQS service spec has changed from query to json: boto/botocore@2395012
  • This broke the usage of SQS with the AWS Python SDK for that underlying botocore version as reported in bug: Creating SQS queue fails with botocore==1.29.127 & botocore==1.31.81 #8267.
• With version 1.27.129 this change reverted it again: Revert SQS protocol changes boto/botocore#2931
• With version 1.31.81 the switch was reintroduced and the issue effects us again: boto/botocore@84c7847
  • Since 2023-11-09, the JSON protocol is generally available as announced in this AWS blogpost

Initially explored solution:

• Upgrade to the latest version of botocore internally.
• Implement a parser and serializer facade which can dynamically select the right parser/serializer based on the content type / accept headers of the request.
• Implement error code handling for x-amzn-query-error, as implemented in Fetch query compatible error from header to support service boto/botocore#2768
• Upgrade to a moto which contains a fix for Creating of SQS queues with latest botorecore (1.29.127) is broken getmoto/moto#6286
• Implement tests which ensure that both, query-protocol and json-protocol requests and responses can be sent for SQS.
• Fix all other tests that might break with the upgrade of botocore.
  • Try to make SQS tests protocol agnostic, and the parser tests protocol specific (try not to depend on botocore spec protocol)

As it turns out this solution is not feasible, because the JSON specification removed a lot of information from the spec which is used to properly parse and serialize the requests and responses.
This means that there is no other alternative than handling two different specifications.

New solution

• Upgrade to the latest version of botocore internally.
• Add the latest available version of the SQS query specifications to localstack.aws.data.
  • This is a newly created folder which is configured as extra search path in botocore when loading the specifications.
  • This will add a newly created "alias" service called sqs-query.

What's left to do:

• Check if it is necessary to support JSON requests for SQS Queue URLs (or implement content negotiation).
  • Not necessary per se, it is not used by the json protocol based SDK. Would be nice to implement soon.
• Check if the internal dev endpoints for SQS should be migrated to JSON (or implement content negotiation).
  • those endpoint are not targeted by SDKs, so it is not necessary right now, and can be tackled in a second iteration
• Fix failing tests.
  • Most of them should only fail because they assume an XML response structure, but due to the upgrade of botocore a JSON request (and with that a JSON response) is sent.
• Assert that the implementation works with some tests explicitly using XML
• Ensure compatibility with ext (-ext is mostly green, only 4 tests to fix there with wrong assert)

[github-actions]

LocalStack Community integration with Pro

       2 files  ±0         2 suites  ±0   1h 25m 8s ⏱️ + 18m 49s
2 307 tests ±0  2 010 ✔️  - 1  297 💤 +1  0 ❌ ±0 
2 308 runs  ±0  2 010 ✔️  - 1  298 💤 +1  0 ❌ ±0 

Results for commit 3c5a78f. ± Comparison against base commit 74c336c.

This pull request skips 1 test.

tests.aws.services.sqs.test_sqs.TestSqsProvider ‑ test_delete_message_batch_from_lambda

♻️ This comment has been updated with latest results.

[coveralls]

coverage: 84.017% (+0.002%) from 84.015%
when pulling 3c5a78f on fix-sqs-json
into 74c336c on master.

[alexrashed]

Thanks so much @bentsku for finalizing the PR and figuring out the nitty gritty details of the remaining test failures! 🦸🏽
From my perspective, this PR is ready to be merged as it is, but I would like to get another pair of eyes from @dominikschubert or @thrau before merging this.
I'm not happy with some of the changes in here (like the hacky "provider alias"), but I think we can iron these things out in a follow-up.

[dominikschubert]

LGTM, Thanks so much @alexrashed and @bentsku for getting this over the line so quickly!

It's probably not going to be the only case where they'll make the switch to JSON I guess 😞

As @bentsku said, any typos, etc. can be removed in a follow-up. First let's get this merged 🚀

[ilia-beliaev-miro]

Looks like this change has broke the localstack SQS usage for us due to the service name change from sqs to sqs-query. We receive Service 'sqs-query' is not enabled. now, was it an expected outcome and we should now switch the service name when creating a localstack instance from sqs to sqs-query?

[bentsku]

Hello @ilia-beliaev-miro, this is an unexpected outcome and we will fix it with #9607. Sorry for the issue, you can declare both service names for now, but it should be fixed by the end of the day!