0.12.0-rc1 - import fails if default provider is missing

[a13xb]

Summary

Import of an AWS resource will fail if a default (non-aliased) provider is not present, even if all resources are using a valid aliased provider.

Import of the same resource succeeds if:

1. a default provider is added, even if it is completely unrelated and unused, e.g. configured with a different region than the resource imported
2. AWS_DEFAULT_REGION environment variable is set to any region value, even if it's a different region than the resource imported

0.11 has a slightly different behaviour in that it's blocking on an interactive prompt asking for a region value. I can specify any region value and the import will proceed. I am assuming it's related, but in 0.12 it's immediate "hard" failure.

Terraform Version

Terraform v0.12.0-rc1
+ provider.aws v1.60.0-dev20190216h00-dev

Terraform Configuration Files

provider aws {
  alias = "sydney"
  region = "ap-southeast-2"
  profile = "development"
}

resource aws_vpc main {
  provider = "aws.sydney"
  cidr_block = "10.0.0.0/24"
}

resource aws_route_table main {
  provider = "aws.sydney"
  vpc_id = aws_vpc.main.id
}

Steps to Reproduce

1. Create resources outside of Terraform
2. terraform init
3. terraform import -provider=aws.sydney aws_route_table.main rtb-XXX

Expected Behavior

Import finishes successfully.

Actual Behavior

After refreshing the resource state correctly, import fails anyway.

aws_route_table.main: Importing from ID "rtb-XXX"...
aws_route_table.main: Import complete!
  Imported aws_route_table
  Imported aws_route
  Imported aws_route
  Imported aws_route
  Imported aws_route
  Imported aws_route_table_association
aws_route_table_association.main: Refreshing state... [id=rtbassoc-YYY]
aws_route.main: Refreshing state... [id=r-rtb-XXX1111111111]
aws_route.main-2: Refreshing state... [id=r-rtb-XXX222222222]
aws_route.main-3: Refreshing state... [id=r-rtb-XXX3333333333]
aws_route_table.main: Refreshing state... [id=rtb-XXX]
aws_route.main-1: Refreshing state... [id=r-rtb-XXX555555555]

Error: Missing required argument

The argument "region" is required, but was not set.

Debug Output

The only excerpt that seems to be relevant happens before the import starts:

...
BuiltinEvalContext: Initialized "aws" provider for provider.aws
<root>: eval: *terraform.EvalOpFilter
<root>: eval: *terraform.EvalSequence
<root>: eval: *terraform.EvalGetProvider
 [INFO]  plugin: configuring client automatic mTLS
...
BuiltinEvalContext: Initialized "aws" provider for provider.aws.sydney
<root>: eval: *terraform.EvalOpFilter
<root>: eval: *terraform.EvalSequence
<root>: eval: *terraform.EvalGetProvider
...
<root>: eval: *terraform.EvalSequence
<root>: eval: *terraform.EvalConfigProvider
buildProviderConfig for provider.aws: no configuration at all
GRPCProvider: GetSchema
buildProviderConfig for provider.aws.sydney: using explicit config only
GRPCProvider: GetSchema
<root>: eval: *terraform.EvalConfigProvider, non-fatal err: Missing required argument: The argument "region" is required, but was not set.
<root>: eval: *terraform.EvalSequence, err: Missing required argument: The argument "region" is required, but was not set.
<root>: eval: *terraform.EvalOpFilter, err: Missing required argument: The argument "region" is required, but was not set.
<root>: eval: *terraform.EvalSequence, err: Missing required argument: The argument "region" is required, but was not set.
[walkImport] Exiting eval tree: provider.aws
vertex "provider.aws": visit complete
dag/walk: upstream of "aws_vpc.main" errored, so skipping
dag/walk: upstream of "aws_route_table.main" errored, so skipping
dag/walk: upstream of "provider.aws (close)" errored, so skipping

And then a few seconds later after the import has finished:

[walkImport] Exiting eval tree: provider.aws.sydney (close)
vertex "provider.aws.sydney (close)": visit complete
dag/walk: upstream of "root" errored, so skipping

[a13xb]

Additionally, if I add a default provider block without region configured:

provider aws {
}

Then the error becomes more clear:

Error: Missing required argument

  on /path/to/main.tf line 1, in provider "aws":
   1: provider aws {

The argument "region" is required, but no definition was found.

But it still happens at the same stage: after the import process seems to have succeeded and all child resources have been found.

[apparentlymart]

Hi @a13xb! Thanks for reporting this.

Indeed, it looks like for some reason the import process is unnecessarily adding the default (unaliased) aws provider into the graph even though the configuration doesn't use it. Off the top of my head I can't think of why it would need to add that provider node, so I think we ought to be able to arrange for it to not be added at all and thus not need any configuration.

The import graph builder seems to be using the same TransformProviders function for its provider node management that all of the other walks do. Its intended behavior is to walk the configuration looking for provider blocks and provider meta-arguments in resources and create nodes for each of them. It also adds the default nodes for each provider in the root module, but is also intended to prune them back out once it detects that a particular one isn't used, so possibly that last step isn't functioning correctly in the import scenario for some reason.

Since this bug is pre-existing in Terraform v0.11 (albeit with some slightly different symptoms, because the prompting behavior was refactored for v0.12), it's likely that this will require some digging to fully diagnose and fix. Because we're close to v0.12.0 final now we're going to hold this for a follow-up v0.12.x patch release so we can make sure we have enough time for full debugging and testing since this will be a change to behavior that has probably been present since the -provider option was added back in Terraform 0.8.

Thanks for reporting this!

[uovobw]

this issue is still very much present in the final terraform version, i currently have:

Terraform v0.12.0
+ provider.acme v1.3.1
+ provider.aws v2.12.0
+ provider.template v2.1.2
+ provider.tls v2.0.1

and with the provider configuration

provider "acme" {
  alias      = "prod"
  server_url = "https://acme-v02.api.letsencrypt.org/directory"
}

provider "acme" {
  alias      = "staging"
  server_url = "https://acme-staging-v02.api.letsencrypt.org/directory"
}

the import of a very simple AWS EBS volume with

terraform import aws_ebs_volume.wordpress_root vol-01234567890

fails with

aws_ebs_volume.wordpress_root: Importing from ID "vol-01234567890"...
aws_ebs_volume.wordpress_root: Import complete!
  Imported aws_ebs_volume
aws_ebs_volume.wordpress_root: Refreshing state... [id=vol-01234567890]

Error: Missing required argument

The argument "server_url" is required, but was not set.

(volume id edited)

are there any workarounds?

[uovobw]

replying to my previous comment here, it appears the issue can be solved by specifying a default provider for each provider type. i have moved the acme.staging provider to be the acme one (so instead of provider="acme.prod" and provider="acme.staging" i now have provider="acme" and provider="acme.prod") and now the import process works.

[ozbillwang]

Any work around of this issue? Can't pass the validate check

$ terraform12 validate

Error: Missing required argument

The argument "region" is required, but was not set.

Got the same today when convert a module to 0.12.1

Updates

A quick fix, add below code.

variable "region" {
  description = "region"
  type        = string
  default     = "us-east-2"
}

provider "aws" {
  region = var.region
}

[WraithCadmus]

I'm having what I believe is a similar issue with a vmware provider. We leave user and password blank so they don't get committed to source. However I don't get that prompt when I try to import a vm

terraform import module.foobar.vsphere_virtual_machine.vmware_vm 1234-6578-90ab

Error: Missing required argument

  on /home/warith.cadmus/terraform/qa/provider.tf line 1, in provider "vsphere":
   1: provider "vsphere" {

The argument "password" is required, but no definition was found.

[jonathortense]

Workaround that worked for me

I have two AWS providers:

provider "aws" {
  alias      = "aws_root_account"
  version    = "~> 2.0"
  access_key = "${var.aws_access_key}"
  secret_key = "${var.aws_secret_key}"
  region     = "eu-west-1"
}

provider "aws" {
  alias      = "aws_new_subaccount"
  version    = "~> 2.0"
  access_key = "${var.aws_access_key}"
  secret_key = "${var.aws_secret_key}"
  region     = "eu-west-1"
  assume_role {
    role_arn     = "arn:aws:iam::XXXXXXXXXXX:role/OrganizationAccountAccessRole"
  }
}

On terraform 0.11.14 I can run this without problems:

terraform import -provider=aws.aws_new_subaccount ...

Now, on terraform 0.12.2 I had to add an empty AWS provider like this:

provider "aws" {
  region = "${var.region}"
}

And it worked.

[meschansky]

The same happens on terraform 0.12.3 with aliased mysql providers when no default one is present:

$ terraform import aws_ecr_repository.repo1 repo1
aws_ecr_repository.repo1: Importing from ID "repo1"...
aws_ecr_repository.repo1: Import complete!
  Imported aws_ecr_repository

Error: Missing required argument

The argument "username" is required, but was not set.


Error: Missing required argument

The argument "endpoint" is required, but was not set.```

and resources aren't imported eventually.

A workaround is to add the default mysql provider:

provider "mysql" {
  endpoint = ""
  username = ""
}

[AirCombat]

I'm having what I believe is a similar issue with a vmware provider. We leave user and password blank so they don't get committed to source. However I don't get that prompt when I try to import a vm

terraform import module.foobar.vsphere_virtual_machine.vmware_vm 1234-6578-90ab

Error: Missing required argument

  on /home/warith.cadmus/terraform/qa/provider.tf line 1, in provider "vsphere":
   1: provider "vsphere" {

The argument "password" is required, but no definition was found.

@WraithCadmus did you find a solution to this? I had an issue and lost my state and now can't import.

[pythian-ciciliani]

Also seeing this problem in 0.12.5, with AWS multiple providers. The workaround proposed by @meschansky of adding a default provider pointing to the needed region temporarily worked for me

[hex2a]

this seems related to #13018, still occurring in 0.12.5

when specifying the AWS_DEFAULT_REGION environment variable, the import fails because it is missing the profile:

Error: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider

there is only a single non-aliased provider defined in the root module, none in child modules.

[fatred]

@AirCombat I just had the same thing. Had to set the default provider to point to the same settings as the aliased provider for the entity I was importing.

e.g.
before:

$ terraform import -provider=vsphere.sv1001 vsphere_virtual_machine.jump /ha-datacenter/vm/myvm 
vsphere_virtual_machine.jump: Importing from ID "/ha-datacenter/vm/myvm"...
vsphere_virtual_machine.jump: Import complete!
  Imported vsphere_virtual_machine
vsphere_virtual_machine.jump: Refreshing state... [id=564d58b1-41be-fb61-f839-2345ae60c2a8]

Error: Missing required argument

The argument "user" is required, but was not set.


Error: Missing required argument

The argument "password" is required, but was not set.

Added identical (minus alias line) default provider:

provider "vsphere" {
  user           = "root"
  password       = "password"
  vsphere_server = "1.1.1.1"
  allow_unverified_ssl = true
}

provider "vsphere" {
  alias          = "sv1001"
  user           = "root"
  password       = "password"
  vsphere_server = "1.1.1.1"
  allow_unverified_ssl = true
}

Gives me output:

$ terraform import -provider=vsphere.sv1001 vsphere_virtual_machine.jump /ha-datacenter/vm/myvm 
vsphere_virtual_machine.jump: Importing from ID "/ha-datacenter/vm/myvm"...
vsphere_virtual_machine.jump: Import complete!
  Imported vsphere_virtual_machine
vsphere_virtual_machine.jump: Refreshing state... [id=564d58b1-41be-fb61-f839-2345ae60c2a8]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

Basically, i guess you have to setup the default to point to whatever, import all your entities on that aliased thing, and then rotate over all your different aliases until you're all done.

Maybe in your case set an empty var that gets prompted on runtime, and set the providers to use that var? or maybe do it locally and make sure you remove those files before check-in?

Can we make the state refresh thing use the -provider=blah settings please?

[javierbeaumont]

Same issue if you defined as variable:

provider "aws" {
  region = "${var.region}"
}

in environment and modules and only a default value in environment:

variable "region" {
  type     = string
  default = "us-east-2"
}

and empty in modules:

variable "region" {
  type = string
}

[Satak]

I can't run terraform validate against my vSphere module because I get an error message:

Error: Missing required argument
The argument "password" is required, but was not set.

Error: Missing required argument
The argument "user" is required, but was not set.

The module itself doesn't have provider because the end user is expected to add it. Validation works if I add this to my module main.tf:

provider "vsphere" {
  user     = "test"
  password = "test"
}

How can one skip provider check for vSphere module validation?

[Satak]

you need run terraform init before validate

Yes I did run that.

[mildwonkey]

Hi folks! Much of (possibly all) the relevant code paths have changed since 0.12.0, and we've fixed various issues in this area (specifically with import), so I am going to close this issue. If you are still seeing problems in v0.13.0, please open a new issue. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.