Merge LocalStack changes into upstream moto (#4082)

* fix OPTIONS requests on non-existing API GW integrations

* add cloudformation models for API Gateway deployments

* bump version

* add backdoor to return CloudWatch metrics

* Updating implementation coverage

* Updating implementation coverage

* add cloudformation models for API Gateway deployments

* Updating implementation coverage

* Updating implementation coverage

* Implemented get-caller-identity returning real data depending on the access key used.

* bump version

* minor fixes

* fix Number data_type for SQS message attribute

* fix handling of encoding errors

* bump version

* make CF stack queryable before starting to initialize its resources

* bump version

* fix integration_method for API GW method integrations

* fix undefined status in CF FakeStack

* Fix apigateway issues with terraform v0.12.21
* resource_methods -> add handle for "DELETE" method
* integrations -> fix issue that "httpMethod" wasn't included in body request (this value was set as the value from refer method resource)

* bump version

* Fix setting http method for API gateway integrations (#6)

* bump version

* remove duplicate methods

* add storage class to S3 Key when completing multipart upload (#7)

* fix SQS performance issues; bump version

* add pagination to SecretsManager list-secrets (#9)

* fix default parameter groups in RDS

* fix adding S3 metadata headers with names containing dots (#13)

* Updating implementation coverage

* Updating implementation coverage

* add cloudformation models for API Gateway deployments

* Updating implementation coverage

* Updating implementation coverage

* Implemented get-caller-identity returning real data depending on the access key used.

* make CF stack queryable before starting to initialize its resources

* bump version

* remove duplicate methods

* fix adding S3 metadata headers with names containing dots (#13)

* Update amis.json to support EKS AMI mocks (#15)

* fix PascalCase for boolean value in ListMultipartUploads response (#17); fix _get_multi_param to parse nested list/dict query params

* determine non-zero container exit code in Batch API

* support filtering by dimensions in CW get_metric_statistics

* fix storing attributes for ELBv2 Route entities; API GW refactorings for TF tests

* add missing fields for API GW resources

* fix error messages for Route53 (TF-compat)

* various fixes for IAM resources (tf-compat)

* minor fixes for API GW models (tf-compat)

* minor fixes for API GW responses (tf-compat)

* add s3 exception for bucket notification filter rule validation

* change the way RESTErrors generate the response body and content-type header

* fix lint errors and disable "black" syntax enforcement

* remove return type hint in RESTError.get_body

* add RESTError XML template for IAM exceptions

* add support for API GW minimumCompressionSize

* fix casing getting PrivateDnsEnabled API GW attribute

* minor fixes for error responses

* fix escaping special chars for IAM role descriptions (tf-compat)

* minor fixes and tagging support for API GW and ELB v2 (tf-compat)

* Merge branch 'master' into localstack

* add "AlarmRule" attribute to enable support for composite CloudWatch metrics

* fix recursive parsing of complex/nested query params

* bump version

* add API to delete S3 website configurations (#18)

* use dict copy to allow parallelism and avoid concurrent modification exceptions in S3

* fix precondition check for etags in S3 (#19)

* minor fix for user filtering in Cognito

* fix API Gateway error response; avoid returning empty response templates (tf-compat)

* support tags and tracingEnabled attribute for API GW stages

* fix boolean value in S3 encryption response (#20)

* fix connection arn structure

* fix api destination arn structure

* black format

* release 2.0.3.37

* fix s3 exception tests

see botocore/parsers.py:1002 where RequestId is removed from parsed

* remove python 2 from build action

* add test failure annotations in build action

* fix events test arn comparisons

* fix s3 encryption response test

* return default value "0" if EC2 availableIpAddressCount is empty

* fix extracting SecurityGroupIds for EC2 VPC endpoints

* support deleting/updating API Gateway DomainNames

* fix(events): Return empty string instead of null when no pattern is specified in EventPattern (tf-compat) (#22)

* fix logic and revert CF changes to get tests running again (#21)

* add support for EC2 customer gateway API (#25)

* add support for EC2 Transit Gateway APIs (#24)

* feat(logs): add `kmsKeyId` into `LogGroup` entity (#23)

* minor change in ELBv2 logic to fix tests

* feat(events): add APIs to describe and delete CloudWatch Events connections (#26)

* add support for EC2 transit gateway route tables (#27)

* pass transit gateway route table ID in Describe API, minor refactoring (#29)

* add support for EC2 Transit Gateway Routes (#28)

* fix region on ACM certificate import (#31)

* add support for EC2 transit gateway attachments (#30)

* add support for EC2 Transit Gateway VPN attachments (#32)

* fix account ID for logs API

* add support for DeleteOrganization API

* feat(events): store raw filter representation for CloudWatch events patterns (tf-compat) (#36)

* feat(events): add support to describe/update/delete CloudWatch API destinations (#35)

* add Cognito UpdateIdentityPool, CW Logs PutResourcePolicy

* feat(events): add support for tags in EventBus API (#38)

* fix parameter validation for Batch compute environments (tf-compat)

* revert merge conflicts in IMPLEMENTATION_COVERAGE.md

* format code using black

* restore original README; re-enable and fix CloudFormation tests

* restore tests and old logic for CF stack parameters from SSM

* parameterize RequestId/RequestID in response messages and revert related test changes

* undo LocalStack-specific adaptations

* minor fix

* Update CodeCov config to reflect removal of Py2

* undo change related to CW metric filtering; add additional test for CW metric statistics with dimensions

* Terraform - Extend whitelist of running tests

Co-authored-by: acsbendi <acsbendi28@gmail.com>
Co-authored-by: Phan Duong <duongpv@outlook.com>
Co-authored-by: Thomas Rausch <thomas@thrau.at>
Co-authored-by: Macwan Nevil <macnev2013@gmail.com>
Co-authored-by: Dominik Schubert <dominik.schubert91@gmail.com>
Co-authored-by: Gonzalo Saad <saad.gonzalo.ale@gmail.com>
Co-authored-by: Mohit Alonja <monty16597@users.noreply.github.com>
Co-authored-by: Miguel Gagliardo <migag9@gmail.com>
Co-authored-by: Bert Blommers <info@bertblommers.nl>

.github/workflows/build.yml

@@ -90,7 +90,7 @@ jobs:
       with:
         path: ${{ steps.pip-cache.outputs.dir }}
         key: pip-${{ matrix.python-version }}-${{ hashFiles('**/setup.py') }}-4
-    # Update PIP - recent version does not support PY2 though
+    # Update PIP
     - name: Update pip
       run: |
         # https://github.com/pypa/pip/issues/10201
@@ -136,6 +136,7 @@ jobs:
       run: |
         pip install -r requirements-dev.txt
         pip install pytest-cov
+        pip install pytest-github-actions-annotate-failures
     - name: Test with pytest
       run: |
         make test-coverage
@@ -229,7 +230,7 @@ jobs:
     - name: Run Terraform Tests
       run: |
         cd moto-terraform-tests
-        bin/run-tests -i ../tests/terraform-tests.success.txt
+        bin/run-tests -i ../tests/terraform-tests.success.txt -e ../tests/terraform-tests.failures.txt
         cd ..
     - name: "Create report"
       run: |

.gitignore

@@ -1,4 +1,4 @@
-moto.egg-info/*
+moto*.egg-info/*
 dist/*
 .cache
 .tox

Makefile

@@ -3,8 +3,7 @@ SHELL := /bin/bash
 ifeq ($(TEST_SERVER_MODE), true)
 	# exclude test_kinesisvideoarchivedmedia
 	# because testing with moto_server is difficult with data-endpoint
-
-	TEST_EXCLUDE :=  -k 'not test_kinesisvideoarchivedmedia'
+	TEST_EXCLUDE := -k 'not test_kinesisvideoarchivedmedia'
 else
 	TEST_EXCLUDE :=
 endif

moto/acm/models.py

@@ -448,11 +448,13 @@ def import_cert(self, certificate, private_key, chain=None, arn=None, tags=None)
             else:
                 # Will reuse provided ARN
                 bundle = CertBundle(
-                    certificate, private_key, chain=chain, region=region, arn=arn
+                    certificate, private_key, chain=chain, region=self.region, arn=arn
                 )
         else:
             # Will generate a random ARN
-            bundle = CertBundle(certificate, private_key, chain=chain, region=region)
+            bundle = CertBundle(
+                certificate, private_key, chain=chain, region=self.region
+            )
 
         self._certificates[bundle.arn] = bundle
 

moto/apigateway/exceptions.py

@@ -1,8 +1,16 @@
 from __future__ import unicode_literals
-from moto.core.exceptions import RESTError
+from moto.core.exceptions import JsonRESTError
 
 
-class BadRequestException(RESTError):
+class BadRequestException(JsonRESTError):
+    pass
+
+
+class NotFoundException(JsonRESTError):
+    pass
+
+
+class AccessDeniedException(JsonRESTError):
     pass
 
 
@@ -14,7 +22,7 @@ def __init__(self):
         )
 
 
-class CrossAccountNotAllowed(RESTError):
+class CrossAccountNotAllowed(AccessDeniedException):
     def __init__(self):
         super(CrossAccountNotAllowed, self).__init__(
             "AccessDeniedException", "Cross-account pass role is not allowed."
@@ -71,10 +79,19 @@ def __init__(self):
         )
 
 
-class NoIntegrationDefined(BadRequestException):
+class NoIntegrationDefined(NotFoundException):
     def __init__(self):
         super(NoIntegrationDefined, self).__init__(
-            "BadRequestException", "No integration defined for method"
+            "NotFoundException", "No integration defined for method"
+        )
+
+
+class NoIntegrationResponseDefined(NotFoundException):
+    code = 404
+
+    def __init__(self, code=None):
+        super(NoIntegrationResponseDefined, self).__init__(
+            "NotFoundException", "No integration defined for method, code '%s'" % code
         )
 
 
@@ -85,7 +102,7 @@ def __init__(self):
         )
 
 
-class AuthorizerNotFoundException(RESTError):
+class AuthorizerNotFoundException(NotFoundException):
     code = 404
 
     def __init__(self):
@@ -94,7 +111,7 @@ def __init__(self):
         )
 
 
-class StageNotFoundException(RESTError):
+class StageNotFoundException(NotFoundException):
     code = 404
 
     def __init__(self):
@@ -103,7 +120,7 @@ def __init__(self):
         )
 
 
-class ApiKeyNotFoundException(RESTError):
+class ApiKeyNotFoundException(NotFoundException):
     code = 404
 
     def __init__(self):
@@ -112,7 +129,7 @@ def __init__(self):
         )
 
 
-class UsagePlanNotFoundException(RESTError):
+class UsagePlanNotFoundException(NotFoundException):
     code = 404
 
     def __init__(self):
@@ -121,7 +138,7 @@ def __init__(self):
         )
 
 
-class ApiKeyAlreadyExists(RESTError):
+class ApiKeyAlreadyExists(JsonRESTError):
     code = 409
 
     def __init__(self):
@@ -139,7 +156,7 @@ def __init__(self):
         )
 
 
-class DomainNameNotFound(RESTError):
+class DomainNameNotFound(NotFoundException):
     code = 404
 
     def __init__(self):
@@ -166,7 +183,7 @@ def __init__(self):
         )
 
 
-class RestAPINotFound(RESTError):
+class RestAPINotFound(NotFoundException):
     code = 404
 
     def __init__(self):
@@ -175,7 +192,7 @@ def __init__(self):
         )
 
 
-class ModelNotFound(RESTError):
+class ModelNotFound(NotFoundException):
     code = 404
 
     def __init__(self):
@@ -184,10 +201,19 @@ def __init__(self):
         )
 
 
-class ApiKeyValueMinLength(RESTError):
+class ApiKeyValueMinLength(BadRequestException):
     code = 400
 
     def __init__(self):
         super(ApiKeyValueMinLength, self).__init__(
             "BadRequestException", "API Key value should be at least 20 characters"
         )
+
+
+class MethodNotFoundException(NotFoundException):
+    code = 404
+
+    def __init__(self):
+        super(MethodNotFoundException, self).__init__(
+            "NotFoundException", "Invalid method properties specified"
+        )