Failing to run in eu-west-1

[crablab]

General information:

• Operating System: MacOS Big Sur
• Terracognita version / tag: v0.6.2
• Did you build Terracognita from sources or did you use the Docker image: installed from Brew

Describe the bug:

When running against our AWS account with resources in eu-west-1, we get a 403 when terracognita tries to call endpoints in API Gateway. Not only are there no resources in API Gateway in this account, but this is being run with AWSAdministratorAccess, so no permissions issues should occur.

I think this could be related to #171, where assumptions about certain resources existing in all regions is not true.

Log message

ts=2021-04-01T10:54:41.332899Z caller=provider.go:39 func=reader.New msg="configuring aws Reader"
ts=2021-04-01T10:54:42.021977Z caller=provider.go:52 func=aws.NewProvider msg="configuring TF Client"
ts=2021-04-01T10:54:42.995209Z caller=aws.go:99 func=cmd.aws.RunE msg="initializing HCL writer"
ts=2021-04-01T10:54:42.995278Z caller=aws.go:108 func=cmd.aws.RunE msg=importing
ts=2021-04-01T10:54:42.99529Z caller=aws.go:111 func=cmd.aws.RunE msg="starting terracognita" version=v0.6.2
ts=2021-04-01T10:54:42.995316Z caller=import.go:66 func=provider.Import filters="\n\tTags:    [],\n\tInclude: [],\n\tExclude: [],\n\tTargets: [],\n"
ts=2021-04-01T10:54:42.996259Z caller=import.go:82 func=provider.Import resource=aws_instance msg="fetching the list of resources"
ts=2021-04-01T10:54:43.073336Z caller=import.go:168 func=provider.Import resource=aws_instance msg="importing done"
ts=2021-04-01T10:54:43.073514Z caller=import.go:82 func=provider.Import resource=aws_alb msg="fetching the list of resources"
ts=2021-04-01T10:54:43.22091Z caller=import.go:168 func=provider.Import resource=aws_alb msg="importing done"
ts=2021-04-01T10:54:43.221132Z caller=import.go:82 func=provider.Import resource=aws_alb_listener msg="fetching the list of resources"
ts=2021-04-01T10:54:43.221206Z caller=import.go:168 func=provider.Import resource=aws_alb_listener msg="importing done"
ts=2021-04-01T10:54:43.221245Z caller=import.go:82 func=provider.Import resource=aws_alb_listener_certificate msg="fetching the list of resources"
ts=2021-04-01T10:54:43.221288Z caller=import.go:168 func=provider.Import resource=aws_alb_listener_certificate msg="importing done"
ts=2021-04-01T10:54:43.221323Z caller=import.go:82 func=provider.Import resource=aws_alb_listener_rule msg="fetching the list of resources"
ts=2021-04-01T10:54:43.221357Z caller=import.go:168 func=provider.Import resource=aws_alb_listener_rule msg="importing done"
ts=2021-04-01T10:54:43.22139Z caller=import.go:82 func=provider.Import resource=aws_alb_target_group msg="fetching the list of resources"
ts=2021-04-01T10:54:43.258598Z caller=import.go:168 func=provider.Import resource=aws_alb_target_group msg="importing done"
ts=2021-04-01T10:54:43.258847Z caller=import.go:82 func=provider.Import resource=aws_alb_target_group_attachment msg="fetching the list of resources"
ts=2021-04-01T10:54:43.296053Z caller=import.go:168 func=provider.Import resource=aws_alb_target_group_attachment msg="importing done"
ts=2021-04-01T10:54:43.296187Z caller=import.go:82 func=provider.Import resource=aws_api_gateway_deployment msg="fetching the list of resources"
Error: could not import from AWS: AccessDeniedException:
        status code: 403, request id: 55c361c4-2b0e-4cd9-996f-26ed909f7a49
error while reading from resource "aws_api_gateway_deployment"
github.com/cycloidio/terracognita/aws.(*aws).Resources
        github.com/cycloidio/terracognita/aws/provider.go:93
github.com/cycloidio/terracognita/provider.Import
        github.com/cycloidio/terracognita/provider/import.go:91
github.com/cycloidio/terracognita/cmd.glob..func2
        github.com/cycloidio/terracognita/cmd/aws.go:112
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra@v1.0.0/command.go:842
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra@v1.0.0/command.go:950
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra@v1.0.0/command.go:887
main.main
        github.com/cycloidio/terracognita/main.go:11
runtime.main
        runtime/proc.go:225
runtime.goexit
        runtime/asm_amd64.s:1371
github.com/cycloidio/terracognita/provider.Import
        github.com/cycloidio/terracognita/provider/import.go:98
github.com/cycloidio/terracognita/cmd.glob..func2
        github.com/cycloidio/terracognita/cmd/aws.go:112
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra@v1.0.0/command.go:842
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra@v1.0.0/command.go:950
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra@v1.0.0/command.go:887
main.main
        github.com/cycloidio/terracognita/main.go:11
runtime.main
        runtime/proc.go:225
runtime.goexit
        runtime/asm_amd64.s:1371

[xescugc]

Sorry for the delay 🙏

Could be related but the main issue is the AccessDeniedException which means that this account has no access to aws_api_gateway_deployment for that specific region (eu-west-1).

Could you confirm that your authentication does not allow that?

[crablab]

Hey 👋

No worries!

As I say, this was from an Administrator account which can ordinarily manage aws_api_gateway_deployment resources 😄

[xescugc]

The error on #171 is not that the resource does not exists, but that the Service (ec2, sqs, sms, ses, etc) does not exist on that region and I'm quite sure that eu-west-1 has "everything" as is the one I always use too haha.

The error AccessDeniedException is 100% from AWS and we are just forwarding it up, no modification. Everything I'm searching about it is related to IAM.

Using "admin" you should never get one of those errors as you should have access to everything.

Either way, speaking with the OPS team we think that a good solution for this would be to log the error but continue importing, so the user knows that some resources did not import due to missing permissions but at least we import everything we can with the given permissions :)

That would be a solution that would work in your usecase no?

[crablab]

a good solution for this would be to log the error but continue importing, so the user knows that some resources did not import due to missing permissions but at least we import everything we can with the given permissions :)

Yeah I think this is a good solution 🙌

I'm not quite sure why this error is occurring either. I had the same thought process as you, so assumed it might be something to do with the specific request you were making. I'm definitely using API Gateway in that account anyway and Terraform has no issues with aws_api_gateway_deployment resources! 😅

[xescugc]

The only think we are fetching before the aws_api_gateway_deployment is the aws_api_gateway_rest_api, we need the ID to fetch the gateways from those API if any.

I cannot tell if the error is from that one or not, but it has to be from one of those 2 and the AWS error is just a general AccessDeniedException so no more info.

We'll go for the solution I mentioned and I'll also open another issue to improve errors when fetching the resources IDs that are more complex than just one call.

But what we can agree is that somehow you do not have access to something on that context haha.

[xescugc]

New release https://github.com/cycloidio/terracognita/releases/tag/v0.6.4 fixes it :)