use aws_api_gateway_stage instead of stage_name from aws_api_gateway_deployment for Terraform

[jrbeilke]

Resolves #1838

Also related to #1816 as a workaround for associating WAF ACL with Chalice APIGW Stages

Bump AWS provider version used for Terraform with fixes for API Gateway and follow latest recommendation using aws_api_gateway_stage instead of stage_name from aws_api_gateway_deployment
ie. hashicorp/terraform-provider-aws#11344 (comment)

This is covered in the latest docs for the Terraform AWS provider:
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/api_gateway_deployment

In addition to switching to aws_api_gateway_stage for Terraform deployment I've also added a Terraform output RestAPIStageArn which can be used to further enhance Chalice deployments of API Gateway (ie. associating a WAF ACL)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[kapilt]

Lgtm, thanks! Still needs to wait on a current amazon employee/maintainer for actual approval/merge...

One question would be understanding how these changes affect an extant deployment with the previous terraform?

[jamesls]

Looks good to me, just want to double check backwards compat for existing chalice apps using terraform deployments.

[jamesls]

A couple questions, when I try this on a brand new app, I'm getting an error about the deployment ID not existing:

╷
│ Error: Error creating API Gateway Stage: BadRequestException: Deployment id does not exist
│
│   with aws_api_gateway_stage.rest_api,
│   on chalice.tf.json line 89, in resource.aws_api_gateway_stage.rest_api:
│   89:       }
│

For reference here's the:

app.py file

from chalice import Chalice

app = Chalice(app_name='tf3')


@app.route('/')
def index():
    return {'hello': 'world'}


@app.lambda_function()
def tf3asdf(event, context):
    return {'foo': 'bar'}

generated chalice.tf.json template

{
  "resource": {
    "aws_iam_role": {
      "default-role": {
        "name": "tf3-dev",
        "assume_role_policy": "{\"Version\": \"2012-10-17\", \"Statement\": [{\"Sid\": \"\", \"Effect\": \"Allow\", \"Principal\": {\"Service\": \"lambda.amazonaws.com\"}, \"Action\": \"sts:AssumeRole\"}]}"
      }
    },
    "aws_iam_role_policy": {
      "default-role": {
        "name": "default-rolePolicy",
        "policy": "{\"Version\": \"2012-10-17\", \"Statement\": [{\"Effect\": \"Allow\", \"Action\": [\"logs:CreateLogGroup\", \"logs:CreateLogStream\", \"logs:PutLogEvents\"], \"Resource\": \"arn:*:logs:*:*:*\"}]}",
        "role": "${aws_iam_role.default-role.id}"
      }
    },
    "aws_lambda_function": {
      "tf3asdf": {
        "function_name": "tf3-dev-tf3asdf",
        "runtime": "python3.9",
        "handler": "app.tf3asdf",
        "memory_size": 128,
        "tags": {
          "aws-chalice": "version=1.26.2:stage=dev:app=tf3"
        },
        "timeout": 60,
        "source_code_hash": "${filebase64sha256(\"${path.module}/deployment.zip\")}",
        "filename": "${path.module}/deployment.zip",
        "role": "${aws_iam_role.default-role.arn}"
      },
      "api_handler": {
        "function_name": "tf3-dev",
        "runtime": "python3.9",
        "handler": "app.app",
        "memory_size": 128,
        "tags": {
          "aws-chalice": "version=1.26.2:stage=dev:app=tf3"
        },
        "timeout": 60,
        "source_code_hash": "${filebase64sha256(\"${path.module}/deployment.zip\")}",
        "filename": "${path.module}/deployment.zip",
        "role": "${aws_iam_role.default-role.arn}"
      }
    },
    "aws_api_gateway_rest_api": {
      "rest_api": {
        "body": "${data.template_file.chalice_api_swagger.rendered}",
        "name": "tf3",
        "binary_media_types": [
          "application/octet-stream",
          "application/x-tar",
          "application/zip",
          "audio/basic",
          "audio/ogg",
          "audio/mp4",
          "audio/mpeg",
          "audio/wav",
          "audio/webm",
          "image/png",
          "image/jpg",
          "image/jpeg",
          "image/gif",
          "video/ogg",
          "video/mpeg",
          "video/webm"
        ],
        "endpoint_configuration": {
          "types": [
            "EDGE"
          ]
        }
      }
    },
    "aws_api_gateway_deployment": {
      "rest_api": {
        "rest_api_id": "${aws_api_gateway_rest_api.rest_api.id}",
        "triggers": {
          "redeployment": "sha1(jsonencode(aws_api_gateway_rest_api.rest_api.body))"
        },
        "lifecycle": {
          "create_before_destroy": true
        }
      }
    },
    "aws_api_gateway_stage": {
      "rest_api": {
        "deployment_id": "aws_api_gateway_deployment.rest_api.id",
        "rest_api_id": "${aws_api_gateway_rest_api.rest_api.id}",
        "stage_name": "api"
      }
    },
    "aws_lambda_permission": {
      "rest_api_invoke": {
        "function_name": "${aws_lambda_function.api_handler.arn}",
        "action": "lambda:InvokeFunction",
        "principal": "apigateway.amazonaws.com",
        "source_arn": "${aws_api_gateway_rest_api.rest_api.execution_arn}/*",
        "depends_on": [
          "aws_api_gateway_deployment.rest_api"
        ]
      }
    }
  },
  "terraform": {
    "required_version": "> 0.11.0, < 1.1.0"
  },
  "provider": {
    "template": {
      "version": "~> 2"
    },
    "aws": {
      "version": ">= 3.25, < 4"
    },
    "null": {
      "version": ">= 2, < 4"
    }
  },
  "data": {
    "aws_caller_identity": {
      "chalice": {}
    },
    "aws_partition": {
      "chalice": {}
    },
    "aws_region": {
      "chalice": {}
    },
    "null_data_source": {
      "chalice": {
        "inputs": {
          "app": "tf3",
          "stage": "dev"
        }
      }
    },
    "template_file": {
      "chalice_api_swagger": {
        "template": "{\"swagger\": \"2.0\", \"info\": {\"version\": \"1.0\", \"title\": \"tf3\"}, \"schemes\": [\"https\"], \"paths\": {\"/\": {\"get\": {\"consumes\": [\"application/json\"], \"produces\": [\"application/json\"], \"responses\": {\"200\": {\"description\": \"200 response\", \"schema\": {\"$ref\": \"#/definitions/Empty\"}}}, \"x-amazon-apigateway-integration\": {\"responses\": {\"default\": {\"statusCode\": \"200\"}}, \"uri\": \"${aws_lambda_function.api_handler.invoke_arn}\", \"passthroughBehavior\": \"when_no_match\", \"httpMethod\": \"POST\", \"contentHandling\": \"CONVERT_TO_TEXT\", \"type\": \"aws_proxy\"}}}}, \"definitions\": {\"Empty\": {\"type\": \"object\", \"title\": \"Empty Schema\"}}, \"x-amazon-apigateway-binary-media-types\": [\"application/octet-stream\", \"application/x-tar\", \"application/zip\", \"audio/basic\", \"audio/ogg\", \"audio/mp4\", \"audio/mpeg\", \"audio/wav\", \"audio/webm\", \"image/png\", \"image/jpg\", \"image/jpeg\", \"image/gif\", \"video/ogg\", \"video/mpeg\", \"video/webm\"]}"
      }
    }
  },
  "output": {
    "EndpointURL": {
      "value": "${aws_api_gateway_deployment.rest_api.invoke_url}"
    },
    "RestAPIId": {
      "value": "${aws_api_gateway_rest_api.rest_api.id}"
    },
    "RestAPIStageArn": {
      "value": "aws_api_gateway_stage.rest_api.arn"
    }
  }
}

I also noticed that when I try to deploy an existing Chalice app that was using the previous tf template generation, I see this in the output:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
+/- create replacement and then destroy

Terraform will perform the following actions:

  # aws_api_gateway_deployment.rest_api must be replaced
+/- resource "aws_api_gateway_deployment" "rest_api" {
      ~ created_date      = "2021-12-08T20:33:23Z" -> (known after apply)
      ~ execution_arn     = "arn:aws:execute-api:us-west-2:760296166491:2orvj7maoi/api" -> (known after apply)
      ~ id                = "xsb3aj" -> (known after apply)
      ~ invoke_url        = "https://2orvj7maoi.execute-api.us-west-2.amazonaws.com/api" -> (known after apply)
      - stage_description = "e298b85a463dbbd0fa086cf759dc8f8a" -> null # forces replacement
      - stage_name        = "api" -> null # forces replacement
      + triggers          = {
          + "redeployment" = "sha1(jsonencode(aws_api_gateway_rest_api.rest_api.body))"
        } # forces replacement
        # (1 unchanged attribute hidden)
    }

  # aws_api_gateway_stage.rest_api will be created
  + resource "aws_api_gateway_stage" "rest_api" {
      + arn           = (known after apply)
      + deployment_id = "aws_api_gateway_deployment.rest_api.id"
      + execution_arn = (known after apply)
      + id            = (known after apply)
      + invoke_url    = (known after apply)
      + rest_api_id   = "2orvj7maoi"
      + stage_name    = "api"
      + tags_all      = (known after apply)
    }

The deployment fails with:

╷
│ Error: Error creating API Gateway Stage: ConflictException: Stage already exists
│
│   with aws_api_gateway_stage.rest_api,
│   on chalice.tf.json line 89, in resource.aws_api_gateway_stage.rest_api:
│   89:       }
│
╵

but assuming it was able to deploy successfully, wouldn't this mean that the rest API would be replaced and therefore have a new Rest API url?

[jrbeilke]

Interesting, I'll have to do some more digging into that Deployment id issue for new deployments. Which version of Terraform and the Terraform AWS provider were you running?

As for existing apps it looks like that would be a pretty big breaking change for anyone using Chalice that has already deployed an API, since Terraform is unaware of the API stages that were deployed before this change. Seems like there would be two options at that point:

1. Import the existing API stages into the tfstate so Terraform is aware of them
2. Instruct users to delete/remove existing API stages so Terraform can re-create them with this change

[jamesls]

Which version of Terraform and the Terraform AWS provider were you running?

$ terraform -version
Terraform v1.0.7
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.69.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/template v2.2.0

Here's the full logs when trying to do the initial deployment:

deployment output

$ terraform apply
provider.aws.region
  The region where AWS operations will take place. Examples
  are us-east-1, us-west-2, etc.

  Enter a value: us-west-2


Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.template_file.chalice_api_swagger will be read during apply
  # (config refers to values not yet known)
 <= data "template_file" "chalice_api_swagger"  {
      + id       = (known after apply)
      + rendered = (known after apply)
      + template = (known after apply)
    }

  # aws_api_gateway_deployment.rest_api will be created
  + resource "aws_api_gateway_deployment" "rest_api" {
      + created_date  = (known after apply)
      + execution_arn = (known after apply)
      + id            = (known after apply)
      + invoke_url    = (known after apply)
      + rest_api_id   = (known after apply)
      + triggers      = {
          + "redeployment" = "sha1(jsonencode(aws_api_gateway_rest_api.rest_api.body))"
        }
    }

  # aws_api_gateway_rest_api.rest_api will be created
  + resource "aws_api_gateway_rest_api" "rest_api" {
      + api_key_source               = (known after apply)
      + arn                          = (known after apply)
      + binary_media_types           = [
          + "application/octet-stream",
          + "application/x-tar",
          + "application/zip",
          + "audio/basic",
          + "audio/ogg",
          + "audio/mp4",
          + "audio/mpeg",
          + "audio/wav",
          + "audio/webm",
          + "image/png",
          + "image/jpg",
          + "image/jpeg",
          + "image/gif",
          + "video/ogg",
          + "video/mpeg",
          + "video/webm",
        ]
      + body                         = (known after apply)
      + created_date                 = (known after apply)
      + description                  = (known after apply)
      + disable_execute_api_endpoint = (known after apply)
      + execution_arn                = (known after apply)
      + id                           = (known after apply)
      + minimum_compression_size     = -1
      + name                         = "testtf4"
      + policy                       = (known after apply)
      + root_resource_id             = (known after apply)
      + tags_all                     = (known after apply)

      + endpoint_configuration {
          + types            = [
              + "EDGE",
            ]
          + vpc_endpoint_ids = (known after apply)
        }
    }

  # aws_api_gateway_stage.rest_api will be created
  + resource "aws_api_gateway_stage" "rest_api" {
      + arn           = (known after apply)
      + deployment_id = "aws_api_gateway_deployment.rest_api.id"
      + execution_arn = (known after apply)
      + id            = (known after apply)
      + invoke_url    = (known after apply)
      + rest_api_id   = (known after apply)
      + stage_name    = "api"
      + tags_all      = (known after apply)
    }

  # aws_iam_role.default-role will be created
  + resource "aws_iam_role" "default-role" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRole"
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "lambda.amazonaws.com"
                        }
                      + Sid       = ""
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = "testtf4-dev"
      + name_prefix           = (known after apply)
      + path                  = "/"
      + tags_all              = (known after apply)
      + unique_id             = (known after apply)

      + inline_policy {
          + name   = (known after apply)
          + policy = (known after apply)
        }
    }

  # aws_iam_role_policy.default-role will be created
  + resource "aws_iam_role_policy" "default-role" {
      + id     = (known after apply)
      + name   = "default-rolePolicy"
      + policy = jsonencode(
            {
              + Statement = [
                  + {
                      + Action   = [
                          + "logs:CreateLogGroup",
                          + "logs:CreateLogStream",
                          + "logs:PutLogEvents",
                        ]
                      + Effect   = "Allow"
                      + Resource = "arn:*:logs:*:*:*"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + role   = (known after apply)
    }

  # aws_lambda_function.api_handler will be created
  + resource "aws_lambda_function" "api_handler" {
      + architectures                  = (known after apply)
      + arn                            = (known after apply)
      + filename                       = "./deployment.zip"
      + function_name                  = "testtf4-dev"
      + handler                        = "app.app"
      + id                             = (known after apply)
      + invoke_arn                     = (known after apply)
      + last_modified                  = (known after apply)
      + memory_size                    = 128
      + package_type                   = "Zip"
      + publish                        = false
      + qualified_arn                  = (known after apply)
      + reserved_concurrent_executions = -1
      + role                           = (known after apply)
      + runtime                        = "python3.7"
      + signing_job_arn                = (known after apply)
      + signing_profile_version_arn    = (known after apply)
      + source_code_hash               = "pw8U6yDNXo4FZ9oB7/5+6RsUbs/ZWxlUIsaBae2McQ8="
      + source_code_size               = (known after apply)
      + tags                           = {
          + "aws-chalice" = "version=1.26.2:stage=dev:app=testtf4"
        }
      + tags_all                       = {
          + "aws-chalice" = "version=1.26.2:stage=dev:app=testtf4"
        }
      + timeout                        = 60
      + version                        = (known after apply)

      + tracing_config {
          + mode = (known after apply)
        }
    }

  # aws_lambda_function.tf3asdf will be created
  + resource "aws_lambda_function" "tf3asdf" {
      + architectures                  = (known after apply)
      + arn                            = (known after apply)
      + filename                       = "./deployment.zip"
      + function_name                  = "testtf4-dev-tf3asdf"
      + handler                        = "app.tf3asdf"
      + id                             = (known after apply)
      + invoke_arn                     = (known after apply)
      + last_modified                  = (known after apply)
      + memory_size                    = 128
      + package_type                   = "Zip"
      + publish                        = false
      + qualified_arn                  = (known after apply)
      + reserved_concurrent_executions = -1
      + role                           = (known after apply)
      + runtime                        = "python3.7"
      + signing_job_arn                = (known after apply)
      + signing_profile_version_arn    = (known after apply)
      + source_code_hash               = "pw8U6yDNXo4FZ9oB7/5+6RsUbs/ZWxlUIsaBae2McQ8="
      + source_code_size               = (known after apply)
      + tags                           = {
          + "aws-chalice" = "version=1.26.2:stage=dev:app=testtf4"
        }
      + tags_all                       = {
          + "aws-chalice" = "version=1.26.2:stage=dev:app=testtf4"
        }
      + timeout                        = 60
      + version                        = (known after apply)

      + tracing_config {
          + mode = (known after apply)
        }
    }

  # aws_lambda_permission.rest_api_invoke will be created
  + resource "aws_lambda_permission" "rest_api_invoke" {
      + action        = "lambda:InvokeFunction"
      + function_name = (known after apply)
      + id            = (known after apply)
      + principal     = "apigateway.amazonaws.com"
      + source_arn    = (known after apply)
      + statement_id  = (known after apply)
    }

Plan: 8 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + EndpointURL     = (known after apply)
  + RestAPIId       = (known after apply)
  + RestAPIStageArn = "aws_api_gateway_stage.rest_api.arn"
╷
│ Warning: Version constraints inside provider configuration blocks are deprecated
│
│   on chalice.tf.json line 108, in provider.template:
│  108:       "version": "~> 2"
│
│ Terraform 0.13 and earlier allowed provider version constraints inside the provider configuration block, but that is now deprecated and will be removed in a future version of Terraform. To silence this warning, move the provider version constraint
│ into the required_providers block.
│
│ (and 2 more similar warnings elsewhere)
╵
╷
│ Warning: Deprecated Resource
│
│   with data.null_data_source.chalice,
│   on chalice.tf.json line 133, in data.null_data_source.chalice:
│  133:       }
│
│ The null_data_source was historically used to construct intermediate values to re-use elsewhere in configuration, the same can now be achieved using locals
│
│ (and one more similar warning elsewhere)
╵

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_iam_role.default-role: Creating...
aws_iam_role.default-role: Creation complete after 0s [id=testtf4-dev]
aws_iam_role_policy.default-role: Creating...
aws_lambda_function.tf3asdf: Creating...
aws_lambda_function.api_handler: Creating...
aws_iam_role_policy.default-role: Creation complete after 0s [id=testtf4-dev:default-rolePolicy]
aws_lambda_function.api_handler: Still creating... [10s elapsed]
aws_lambda_function.tf3asdf: Still creating... [10s elapsed]
aws_lambda_function.api_handler: Creation complete after 17s [id=testtf4-dev]
data.template_file.chalice_api_swagger: Reading...
data.template_file.chalice_api_swagger: Read complete after 0s [id=90e1aaf9b378b69eacb2084c4f0ee36dd01612f6125a5fe076ffe211161cc47b]
aws_api_gateway_rest_api.rest_api: Creating...
aws_lambda_function.tf3asdf: Still creating... [20s elapsed]
aws_api_gateway_rest_api.rest_api: Creation complete after 3s [id=nx0v3ghb25]
aws_api_gateway_deployment.rest_api: Creating...
aws_api_gateway_stage.rest_api: Creating...
aws_api_gateway_deployment.rest_api: Creation complete after 1s [id=9k6u7g]
aws_lambda_permission.rest_api_invoke: Creating...
aws_lambda_permission.rest_api_invoke: Creation complete after 1s [id=terraform-20211213224149419900000001]
aws_lambda_function.tf3asdf: Creation complete after 24s [id=testtf4-dev-tf3asdf]
╷
│ Error: Error creating API Gateway Stage: BadRequestException: Deployment id does not exist
│
│   with aws_api_gateway_stage.rest_api,
│   on chalice.tf.json line 89, in resource.aws_api_gateway_stage.rest_api:
│   89:       }
│
╵

We'd need to come up with something so that users with existing chalice apps deployed with terraform can upgrade Chalice without the new template replacing any resources, and without any additional actions on their part (so any existing CI/CD setups continue to work).

Worst case scenario, we may need to toggle this updated tf template generation behind a config file option until we get to a 2.0 release, but ideally we can come up with a way to do this in a backwards compatible way.

[jrbeilke]

Closing this PR for now as there is more work needed with regards to testing and the ability to have some sort of feature flag to avoid disrupting existing Chalice projects