awscc_lambda_function cannot assume aws_role in one apply (running apply twice works)

[ansgarm]

The first terraform apply fails with:

│ Error: AWS SDK Go Service Operation Incomplete
│ 
│   with awscc_lambda_function.fails,
│   on main.tf line 28, in resource "awscc_lambda_function" "fails":
│   28: resource "awscc_lambda_function" "fails" {
│ 
│ Waiting for Cloud Control API service CreateResource operation completion returned: waiter state transitioned to FAILED. StatusMessage: The role defined for the function cannot be assumed by
│ Lambda. (Service: Lambda, Status Code: 400, Request ID: 7696307b-0818-4ca0-8042-210de7796e06, Extended Request ID: null). ErrorCode: InvalidRequest
╵

A second apply (creating only the missing awscc_lambda_function) works.
I suspect a bug in the internals of the awscc API, but that's just a gut feeling.

It works with an aws_lambda_function (also included, does not affect the failure of the awscc_lambda_function)

Short example below. Terminal output here: https://gist.github.com/ansgarm/1ead0cf84837130d9405d8370b9187d6

terraform {
  required_providers {
    awscc = {
      source  = "local/hashicorp/awscc"
      version = "0.0.11" # I used commit a0d053e99fcfe here (also tagged as 0.0.11)
    }

    aws = {
      source  = "hashicorp/aws"
      version = "3.59.0"
    }
  }
}

provider "awscc" {
}
provider "aws" {
  region = "us-west-2" # same region as awscc in preview
}

resource "aws_iam_role" "lambda_service_role" {
  assume_role_policy = "{\"Statement\":[{\"Action\":\"sts:AssumeRole\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"lambda.amazonaws.com\"}}],\"Version\":\"2012-10-17\"}"
  managed_policy_arns = [
    "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
  ]
}

resource "awscc_lambda_function" "fails" {
  code = {
    zip_file = "def main(event, context):\n    print(\"I'm running!\")"
  }
  handler = "index.main"
  role    = aws_iam_role.lambda_service_role.arn
  runtime = "python3.6"
  timeout = 300
}

resource "aws_lambda_function" "works" {
  function_name = "test-lambda-which-works"
  filename         = "${path.root}/.archive_files/aws_lambda.zip"
  source_code_hash = data.archive_file.aws_lambda_code.output_base64sha256
  handler = "index.main"
  role    = aws_iam_role.lambda_service_role.arn
  runtime = "python3.6"
  timeout = 300
}

data "archive_file" "aws_lambda_code" {
  type        = "zip"
  output_path = "${path.root}/.archive_files/aws_lambda.zip"
  source {
    filename = "index.py"
    content  = "def main(event, context):\n    print(\"I'm running!\")"
  }
}

[ewbankkit]

@ansgarm

Yes, this definitely looks like a problem with the Cloud Control API's handling of IAM eventual consistency for the Lambda Function resource.
In terraform-provider-aws, a retry loop is put around creation to deal with this (and other similar) errors:

https://github.com/hashicorp/terraform-provider-aws/blob/36596b88a1b1a8b73759c54d00624e7a3192ed90/aws/resource_aws_lambda_function.go#L517-L523.

Let's leave this issue open for now - We should consider how to report such errors (nudge @breathingdust).

[ewbankkit]

Closing in favor of #221.