unable to import pre-existing s3 bucket

[b3nnb]

So first, i know im behind on the versions, i have to do this to be able to update everything all at once.
running
terragrunt 0.18.7
terrafrom 0.11.14

and trying to import a pre-existing s3 bucket with

terragrunt import aws_s3_bucket.module_bucket_name bucket_to_import

and im getting

Did not find any Terraform files (*.tf) in ............... aws_s3_bucket.module_bucket_name

any advice? im hoping im missing something small

my file structure is

- project
    - dev
        - config
    - prod
        - config
-modules
    - module

[brikis98]

Could you share see your terragrunt.hcl files and the full log output?

[b3nnb]

you mean the tfvars files in the live project repo of the structure?

[b3nnb]

Log output:

[terragrunt] [/Users/MyUser/Documents/code-repo/terraform-infra/terraform-biztech_live/bi/dev/matillion] 2020/09/29 09:32:33 Running command: terraform --version
[terragrunt] 2020/09/29 09:32:50 Reading Terragrunt config file at /Users/MyUser/Documents/code-repo/terraform-infra/terraform-biztech_live/bi/dev/matillion/terraform.tfvars
[terragrunt] 2020/09/29 09:32:50 The --terragrunt-source-update flag is set, so deleting the temporary folder /Users/MyUser/Documents/code-repo/terraform-infra/terraform-biztech_live/bi/dev/matillion/.terragrunt-cache/2GNCiwHeWw3NWmw09ueMMSaORU8/Dk3vJCkbUgoeM-kj-hrDkhmCvu8 before downloading source.
[terragrunt] 2020/09/29 09:32:50 Downloading Terraform configurations from file:///Users/MyUser/Documents/code-repo/terraform-infra/modules into /Users/MyUser/Documents/code-repo/terraform-infra/terraform-biztech_live/bi/dev/matillion/.terragrunt-cache/2GNCiwHeWw3NWmw09ueMMSaORU8/Dk3vJCkbUgoeM-kj-hrDkhmCvu8 using terraform init
[terragrunt] [/Users/MyUser/Documents/code-repo/terraform-infra/terraform-biztech_live/bi/dev/matillion] 2020/09/29 09:32:50 Initializing remote state for the s3 backend
[terragrunt] [/Users/MyUser/Documents/code-repo/terraform-infra/terraform-biztech_live/bi/dev/matillion] 2020/09/29 09:32:51 Running command: terraform init -backend-config=bucket=terraform-biztech-live -backend-config=key=dev/matillion/terraform.tfstate -backend-config=region=us-west-2 -backend-config=encrypt=true -get=false -get-plugins=false -backend=false -from-module=file:///Users/MyUser/Documents/code-repo/terraform-infra/modules -no-color /Users/MyUser/Documents/code-repo/terraform-infra/terraform-biztech_live/bi/dev/matillion/.terragrunt-cache/2GNCiwHeWw3NWmw09ueMMSaORU8/Dk3vJCkbUgoeM-kj-hrDkhmCvu8
Copying configuration from "file:///Users/MyUser/Documents/code-repo/terraform-infra/modules"...
Terraform initialized in an empty directory!

The directory has no Terraform configuration files. You may begin working
with Terraform immediately by creating Terraform configuration files.
[terragrunt] 2020/09/29 09:33:08 Copying files from /Users/MyUser/Documents/code-repo/terraform-infra/terraform-biztech_live/bi/dev/matillion into /Users/MyUser/Documents/code-repo/terraform-infra/terraform-biztech_live/bi/dev/matillion/.terragrunt-cache/2GNCiwHeWw3NWmw09ueMMSaORU8/Dk3vJCkbUgoeM-kj-hrDkhmCvu8/aws_s3_bucket.import_bucket
[terragrunt] 2020/09/29 09:33:08 Setting working directory to /Users/MyUser/Documents/code-repo/terraform-infra/terraform-biztech_live/bi/dev/matillion/.terragrunt-cache/2GNCiwHeWw3NWmw09ueMMSaORU8/Dk3vJCkbUgoeM-kj-hrDkhmCvu8/aws_s3_bucket.import_bucket
[terragrunt] 2020/09/29 09:33:08 Did not find any Terraform files (*.tf) in /Users/MyUser/Documents/code-repo/terraform-infra/terraform-biztech_live/bi/dev/matillion/.terragrunt-cache/2GNCiwHeWw3NWmw09ueMMSaORU8/Dk3vJCkbUgoeM-kj-hrDkhmCvu8/aws_s3_bucket.import_bucket
[terragrunt] 2020/09/29 09:33:08 Unable to determine underlying exit code, so Terragrunt will exit with error code 1

Stack terrform.tfvars

terragrunt = {
  // bring in the base terraform.tfvars file, which sets some globals as well as the remote state bucket
  include {
    path = "${find_in_parent_folders()}"
  }

  dependencies {
    // this allows plan-all and apply-all to work, plus I explicitly data source the common
    paths = ["../common"]
  }

  terraform {
    // This takes ?ref= pointers to sha, branch or tag, defaulting to master
    // the double slash divides the repo from the module subdirectory
    // you can override the source path with a --terragrunt-source file path (with a similar double slash) during development
    // source = "git::ssh://git@github.com/IT/terraform-infra_modules.git//matillion"
    source = "git::ssh://git@github.com/IT/terraform-infra_modules.git//matillion"
  }
}

stack = "matillion" // I'll generally use this name in resource we create

zone = "us-west-2a"

environment.tfvars:

# variables set in this file will be automatically included by any project subdir

environment = "dev"

# this mirrors this repos directory structure back to a root terraform.tfvars, confirm via the s3 bucket if in doubt
terragrunt_remote_common_state_key = "dev/common/terraform.tfstate"

project terraform.tfvars:

// We consider this file to be the root terraform.tfvars file for this project
// We expect all leaf directories (stacks) to terragrunt include this file

terragrunt = {
  // remote state is stored in s3, locking is included with this backend
  remote_state {
    backend = "s3"

    config {
      bucket  = "terraform-biztech-live"
      // the path to tfstate inside the above bucket will match the path to leaf terraform.tfvars which include {} this file
      key  = "${path_relative_to_include()}/terraform.tfstate"
      region  = "us-west-2"
      encrypt = true
      // we could supply a key we get from vault as another way to restrict access to the remote state
      // encryption_key = A 32 byte base64 encoded 'customer supplied encryption key'

      s3_bucket_tags {
        owner = "bizops"
        name  = "Terraform state storage"
        control = "terraform-biztech_live"
      }

      dynamodb_table_tags {
        owner = "bizops"
        name  = "Terraform lock table"
        control = "terraform-biztech_live"
      }
    }
  }

  // make all the commands in the list below automatically pull in the project.tfvars
  terraform {
    # we are not yet ready for terraform 0.12.
    required_version = "< 0.12"

    extra_arguments "conditional_vars" {
      commands = ["${get_terraform_commands_that_need_vars()}"]

      required_var_files = [
        "${get_parent_tfvars_dir()}/project.tfvars",
        "${get_tfvars_dir()}/../environment.tfvars"
      ]
    }
  }
}

[b3nnb]

any suggestions for this?

[brikis98]

Sorry for the delay.

    source = "git::ssh://git@github.com/IT/terraform-infra_modules.git//matillion"

What is in matillion?

[b3nnb]

Just the name of the project im playing with

…

On Thu, Nov 12, 2020, 12:47 AM Yevgeniy Brikman ***@***.***> wrote: Sorry for the delay. source = ***@***.***/IT/terraform-infra_modules.git//matillion" What is in matillion? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1371 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABDWLH75VR3JGNTJ4OGXK3TSPOOITANCNFSM4RY52CTQ> .

[brikis98]

Right, but what is in that project? Because the code you showed is going to clone that project and try to run Terragrunt/Terraform in the matillion folder, so whatever error you're getting is most likely related to what's in that folder.

[b3nnb]

Its everything from an ALB to instances, IAM policies, and a bunch of stuff that's been successfully deployed. And I just added this s3 bucket manually, then added the code for it in this module but can't import it.

Sorry if that's not helpful. Im not sure how to answer that

[brikis98]

Hm, the log output shows:

/Users/MyUser/Documents/code-repo/terraform-infra/terraform-biztech_live/bi/dev/matillion/.terragrunt-cache/2GNCiwHeWw3NWmw09ueMMSaORU8/Dk3vJCkbUgoeM-kj-hrDkhmCvu8/aws_s3_bucket.import_bucket

Why is it running in the folder aws_s3_bucket.import_bucket instead of matillion? Is there something else in the config I'm not seeing? What is the full command you are running on the CLI?

[b3nnb]

Oh I get ya. The command I run has is just terragrunt import and then the details of the s3 bucket. Im guessing im doing something incorrect. Im just trying to treat terragrunt as a wrapper and subbing terraform for terragrunt to get a resource imported

[brikis98]

Could you copy/paste the exact command you're running? E.g., I see in the logs that there is a --terragrunt-source-update flag set, but that wasn't in your original message. Seeing the whole command you're running would help troubleshoot this.

[b3nnb]

Ok so i got it sorted out.
The command i was running was this.
terragrunt import --terragrunt-source ~/Documents/code-repo/terraform-infra/modules/matillion/ aws_s3_bucket.bucket matillion-dev

but the issue was in my provider resource

provider "aws" {
  region = "${local.region}"
}

if i change tis to a hardcoded value it works for the import. then i can change it back and everything that wa simported still works fine.

is this normal? that you have to hardcode that value? It works fine when running terragrunt plan && apply

[brikis98]

Ah, you may be hitting a Terraform import bug: hashicorp/terraform#13018 or hashicorp/terraform#26211. Hard-coding is, sadly, the workaround. We actually built a hacky command into Terragrunt to work around this import bug: https://terragrunt.gruntwork.io/docs/reference/cli-options/#aws-provider-patch.

[arik-porat]

Just to be familiar in case you're not, there's this project which can help you to import pre existing stuff to terraform with state file https://github.com/GoogleCloudPlatform/terraformer.
I've tested it for s3 with 0.12.20 and it work like a charm

[b3nnb]

Ya, i used something like that. (or that exact one) years ago but it only exported the resources into terraform code and didnt import to the state file. looks like it has come a long way. thanks for the tip!!