Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: make add_transform_test.py always match expected output #3195

Merged
merged 7 commits into from
Jun 1, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 6 additions & 17 deletions DEVELOPMENT_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,27 +134,16 @@ one python version locally and then have our ci (appveyor) run all supported ver

Transform tests ensure a SAM template transforms into the expected CloudFormation template.

When adding new transform tests, we have provided a script to help generate the transform test input
and output files in the correct directory given a `template.yaml` file.
```bash
python3 bin/add_transform_test.py --template-file template.yaml
```

This script will automatically generate the input and output files. It will guarantee that the output
files have the correct AWS partition (e.g. aws-cn, aws-us-gov).
We provide a script to help generate the transform test input
and output files in the correct directory given a SAM template. For example:

For `AWS::ApiGateway::RestApi`, the script will automatically append `REGIONAL` `EndpointConfiguration`.
To disable this feature, run the following command instead.
```bash
python3 bin/add_transform_test.py --template-file template.yaml --disable-api-configuration
```

The script automatically updates hardcoded ARN partitions to match the output partition. To disable this, use:
```bash
python3 bin/add_transform_test.py --template-file template.yaml --disable-update-partition
python3 bin/add_transform_test.py --template-file template.yaml
```

Please always check the generated output is as expected. This tool does not guarantee correct output.
> **Warning**
>
> Always check the generated output is as expected. This tool does not guarantee correct output.

#### Transform failures

Expand Down
52 changes: 8 additions & 44 deletions bin/add_transform_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@
from copy import deepcopy
from pathlib import Path
from typing import Any, Dict
from unittest.mock import patch

import boto3

from samtranslator.translator.arn_generator import ArnGenerator
from samtranslator.translator.managed_policy_translator import ManagedPolicyLoader
from samtranslator.translator.transform import transform
from samtranslator.yaml_helper import yaml_parse
Expand All @@ -28,16 +28,6 @@
type=Path,
default=Path("template.yaml"),
)
parser.add_argument(
"--disable-api-configuration",
help="Disable adding REGIONAL configuration to AWS::ApiGateway::RestApi",
action="store_true",
)
parser.add_argument(
"--disable-update-partition",
help="Disable updating the partition of arn to aws-cn/aws-us-gov",
action="store_true",
)
CLI_OPTIONS = parser.parse_args()


Expand All @@ -51,25 +41,6 @@ def write_json_file(obj: Dict[str, Any], file_path: Path) -> None:
json.dump(obj, f, indent=2, sort_keys=True)


def add_regional_endpoint_configuration_if_needed(template: Dict[str, Any]) -> Dict[str, Any]:
for _, resource in template["Resources"].items():
if resource["Type"] == "AWS::ApiGateway::RestApi":
properties = resource["Properties"]
if "EndpointConfiguration" not in properties:
properties["EndpointConfiguration"] = {"Types": ["REGIONAL"]}
if "Parameters" not in properties:
properties["Parameters"] = {"endpointConfigurationTypes": "REGIONAL"}

return template


def replace_aws_partition(partition: str, file_path: Path) -> None:
template = read_json_file(file_path)
updated_template = json.loads(json.dumps(template).replace("arn:aws:", f"arn:{partition}:"))
file_path.write_text(json.dumps(updated_template, indent=2), encoding="utf-8")
print(f"Transform Test output files generated {file_path}")


def generate_transform_test_output_files(input_file_path: Path, file_basename: str) -> None:
output_file_option = file_basename + ".json"

Expand All @@ -82,20 +53,13 @@ def generate_transform_test_output_files(input_file_path: Path, file_basename: s
"aws-us-gov": ("us-gov-west-1", TRANSFORM_TEST_DIR / "output" / "aws-us-gov" / output_file_option),
}

for partition, (region, output_path) in transform_test_output_paths.items():
# Set Boto Session Region to guarantee the same hash input as transform tests for API deployment id
ArnGenerator.BOTO_SESSION_REGION_NAME = region
# Implicit API Plugin may alter input template file, thus passing a copy here.
output_fragment = transform(deepcopy(manifest), {}, ManagedPolicyLoader(iam_client))

if not CLI_OPTIONS.disable_api_configuration and partition != "aws":
output_fragment = add_regional_endpoint_configuration_if_needed(output_fragment)

write_json_file(output_fragment, output_path)

# Update arn partition if necessary
if not CLI_OPTIONS.disable_update_partition:
replace_aws_partition(partition, output_path)
for _, (region, output_path) in transform_test_output_paths.items():
with patch("samtranslator.translator.arn_generator._get_region_from_session", return_value=region), patch(
"boto3.session.Session.region_name", region
):
# Implicit API Plugin may alter input template file, thus passing a copy here.
output_fragment = transform(deepcopy(manifest), {}, ManagedPolicyLoader(iam_client))
write_json_file(output_fragment, output_path)


def get_input_file_path() -> Path:
Expand Down
54 changes: 54 additions & 0 deletions tests/translator/input/api_with_custom_domains_regional.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
Parameters:
MyRestRegionalDomainName:
Type: String
MyRestRegionalDomainCert:
Type: String
HostedZoneId:
Type: String

Globals:
Api:
Domain:
DomainName:
Ref: MyRestRegionalDomainName
CertificateArn:
Ref: MyRestRegionalDomainCert
EndpointConfiguration: REGIONAL
MutualTlsAuthentication:
TruststoreUri: ${mtlsuri}
TruststoreVersion: 0
SecurityPolicy: TLS_1_2
BasePath:
- /get
- /post
Route53:
HostedZoneId:
Ref: HostedZoneId

Resources:
MyFunction:
Type: AWS::Serverless::Function
Properties:
InlineCode: |
exports.handler = async (event) => {
const response = {
statusCode: 200,
body: JSON.stringify('Hello from Lambda!'),
};
return response;
};
Handler: index.handler
Runtime: nodejs14.x
Events:
ImplicitGet:
Type: Api
Properties:
Method: Get
Path: /get
ImplicitPost:
Type: Api
Properties:
Method: Post
Path: /post
Metadata:
SamTransformTest: true
35 changes: 35 additions & 0 deletions tests/translator/input/basic_state_machine_with_tags.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
Resources:
MyStateMachine:
Type: AWS::Serverless::StateMachine
Properties:
Definition:
Comment: A Hello World example of the Amazon States Language using Pass states
StartAt: Hello
States:
Hello:
Type: Pass
Result: Hello
Next: World
World:
Type: Pass
Result: World
End: true
Policies:
- Version: '2012-10-17'
Statement:
- Effect: Deny
Action: '*'
Resource: '*'
Tags:
TagOne: ValueOne
TagTwo: ValueTwo
Tracing:
Enabled: true

Outputs:
MyStateMachineArn:
Description: ARN of the state machine
Value:
Ref: MyStateMachine
Metadata:
SamTransformTest: true
76 changes: 76 additions & 0 deletions tests/translator/input/state_machine_with_api_combination.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
Resources:

# Create one API resource. This will be referred to by the State machine
ExistingRestApi:
Type: AWS::Serverless::Api
Properties:
StageName: Dev

MyStateMachine:
Type: AWS::Serverless::StateMachine
Properties:
Definition:
Comment: A Hello World example of the Amazon States Language using Pass states
StartAt: Hello
States:
Hello:
Type: Pass
Result: Hello
Next: World
World:
Type: Pass
Result: World
End: true
Policies:
- Version: '2012-10-17'
Statement:
- Effect: Deny
Action: '*'
Resource: '*'

Events:
GetApi:
Type: Api
Properties:
Path: /pathget
Method: get
RestApiId:
Ref: ExistingRestApi

PostApi:
Type: Api
Properties:
Path: /pathpost
Method: post

Outputs:
Region:
Description: Region
Value:
Ref: AWS::Region
Partition:
Description: Partition
Value:
Ref: AWS::Partition
MyStateMachineArn:
Description: ARN of the State Machine
Value:
Ref: MyStateMachine
MyImplicitApiRoleName:
Description: Name of the role created for the implicit Api method
Value:
Ref: MyStateMachinePostApiRole
MyImplicitApiRoleArn:
Description: ARN of the role created for the implicit Api method
Value:
Fn::GetAtt: MyStateMachinePostApiRole.Arn
MyExplicitApiRoleName:
Description: Name of the role created for the explicit Api method
Value:
Ref: MyStateMachineGetApiRole
MyExplicitApiRoleArn:
Description: ARN of the role created for the explicit Api method
Value:
Fn::GetAtt: MyStateMachineGetApiRole.Arn
Metadata:
SamTransformTest: true
41 changes: 41 additions & 0 deletions tests/translator/input/state_machine_with_api_single.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
Transform: AWS::Serverless-2016-10-31
Resources:
MyApi:
Type: AWS::Serverless::Api
Properties:
StageName: Prod
HelloWorldFunction:
Type: AWS::Serverless::Function
Properties:
InlineCode: |
def handler(event, context):
print(event)
return "do nothing"
Handler: index.handler
Runtime: python3.8
Post:
Type: AWS::Serverless::StateMachine
Properties:
Policies:
- arn:aws:iam::aws:policy/AWSLambda_FullAccess
Definition:
StartAt: One
States:
One:
Type: Task
Resource: !GetAtt HelloWorldFunction.Arn
End: true
Events:
PostEcho:
Type: Api
Properties:
RestApiId: !Ref MyApi
Path: /echo
Method: POST
UnescapeMappingTemplate: true

Outputs:
ApiEndpoint:
Value: !Sub "https://${MyApi}.execute-api.${AWS::Region}.${AWS::URLSuffix}/Prod/echo"
Metadata:
SamTransformTest: true
Loading