terraform init -from-module errors or crashes if that module references module's with local relative directories

[rmt]

Terraform Version

[/tmp/test] $ terraform-0.12.29 version
Terraform v0.12.29

Terraform Output (Error Scenario)

[/tmp/test] $ mkdir toplevel
[/tmp/test] $ mkdir submodule
[/tmp/test] $ echo 'output "pi" { value = 3.142 }' > submodule/output.tf
[/tmp/test] $ echo 'module "pi" { source = "../submodule" }' > toplevel/pi.tf
[/tmp/test] $ mkdir tftempdir; cd tftempdir; TF_LOG=trace terraform-0.12.29 init -from-module /tmp/test/toplevel
2020/07/30 15:59:22 [INFO] Terraform version: 0.12.29  
2020/07/30 15:59:22 [INFO] Go runtime version: go1.12.13
2020/07/30 15:59:22 [INFO] CLI args: []string{"/home/rmt/bin/terraform-0.12.29", "init", "-from-module", "/tmp/test/toplevel"}
2020/07/30 15:59:22 [DEBUG] Attempting to open CLI config file: /home/rmt/.terraformrc
2020/07/30 15:59:22 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/07/30 15:59:22 [INFO] CLI command args: []string{"init", "-from-module", "/tmp/test/toplevel"}
2020/07/30 15:59:22 [DEBUG] installing modules in .terraform/init-from-module to initialize working directory from "/tmp/test/toplevel"
2020/07/30 15:59:22 [DEBUG] Module installer: begin root
2020/07/30 15:59:22 [TRACE] ModuleInstaller: cleaning directory .terraform/init-from-module/root prior to install of root
2020/07/30 15:59:22 [TRACE] ModuleInstaller: root address "/tmp/test/toplevel" will be handled by go-getter
2020/07/30 15:59:22 [DEBUG] will download "/tmp/test/toplevel" to .terraform/init-from-module/root
2020/07/30 15:59:22 [TRACE] go-getter detectors rewrote "/tmp/test/toplevel" to "file:///tmp/test/toplevel"
2020/07/30 15:59:22 [TRACE] fetching "file:///tmp/test/toplevel" to ".terraform/init-from-module/root"
2020/07/30 15:59:22 [TRACE] ModuleInstaller: root "/tmp/test/toplevel" was downloaded to .terraform/init-from-module/root
2020/07/30 15:59:22 [DEBUG] Module installer: root installed at .terraform/init-from-module/root
2020/07/30 15:59:22 [DEBUG] Module installer: begin root.pi
2020/07/30 15:59:22 [TRACE] ModuleInstaller: cleaning directory .terraform/init-from-module/root.pi prior to install of root.pi
2020/07/30 15:59:22 [TRACE] ModuleInstaller: root.pi has local path "../submodule"
2020/07/30 15:59:22 [TRACE] ModuleInstaller: root.pi uses directory from parent: .terraform/init-from-module/submodule
2020/07/30 15:59:22 [DEBUG] Module installer: root.pi installed at 
2020/07/30 15:59:22 [TRACE] modsdir: writing modules manifest to .terraform/init-from-module/modules.json
Copying configuration from "/tmp/test/toplevel"...

Error: Unreadable module directory

Unable to evaluate directory symlink: lstat
.terraform/init-from-module/submodule: no such file or directory


Error: Failed to read module directory

Module directory  does not exist or cannot be read.


Error: Unreadable module directory

Unable to evaluate directory symlink: lstat
.terraform/init-from-module/submodule: no such file or directory


Error: Failed to read module directory

Module directory  does not exist or cannot be read.

Terraform Output (Crash Scenario)

If you instead try to use a relative subdir, terraform will crash.

[/tmp/test] $ mkdir toplevel
[/tmp/test] $ mkdir toplevel/submodule
[/tmp/test] $ echo 'output "pi" { value = 3.142 }' > toplevel/submodule/output.tf
[/tmp/test] $ echo 'module "pi" { source = "./submodule" }' > toplevel/pi.tf
2020/07/30 16:05:03 [INFO] Terraform version: 0.12.29  
2020/07/30 16:05:03 [INFO] Go runtime version: go1.12.13
2020/07/30 16:05:03 [INFO] CLI args: []string{"/home/rmt/bin/terraform-0.12.29", "init", "-from-module", "/tmp/test/toplevel"}
2020/07/30 16:05:03 [DEBUG] Attempting to open CLI config file: /home/rmt/.terraformrc
2020/07/30 16:05:03 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/07/30 16:05:03 [INFO] CLI command args: []string{"init", "-from-module", "/tmp/test/toplevel"}
2020/07/30 16:05:03 [DEBUG] installing modules in .terraform/init-from-module to initialize working directory from "/tmp/test/toplevel"
2020/07/30 16:05:03 [DEBUG] Module installer: begin root
2020/07/30 16:05:03 [TRACE] ModuleInstaller: cleaning directory .terraform/init-from-module/root prior to install of root
2020/07/30 16:05:03 [TRACE] ModuleInstaller: root address "/tmp/test/toplevel" will be handled by go-getter
2020/07/30 16:05:03 [DEBUG] will download "/tmp/test/toplevel" to .terraform/init-from-module/root
2020/07/30 16:05:03 [TRACE] go-getter detectors rewrote "/tmp/test/toplevel" to "file:///tmp/test/toplevel"
2020/07/30 16:05:03 [TRACE] fetching "file:///tmp/test/toplevel" to ".terraform/init-from-module/root"
2020/07/30 16:05:03 [TRACE] ModuleInstaller: root "/tmp/test/toplevel" was downloaded to .terraform/init-from-module/root
2020/07/30 16:05:03 [DEBUG] Module installer: root installed at .terraform/init-from-module/root
2020/07/30 16:05:03 [DEBUG] Module installer: begin root.pi
2020/07/30 16:05:03 [TRACE] ModuleInstaller: cleaning directory .terraform/init-from-module/root.pi prior to install of root.pi
2020/07/30 16:05:03 [TRACE] ModuleInstaller: root.pi has local path "./submodule"
2020/07/30 16:05:03 [TRACE] ModuleInstaller: root.pi uses directory from parent: .terraform/init-from-module/root/submodule
2020/07/30 16:05:03 [DEBUG] Module installer: root.pi installed at /tmp/test/toplevel/submodule
2020/07/30 16:05:03 [TRACE] modsdir: writing modules manifest to .terraform/init-from-module/modules.json
2020/07/30 16:05:03 [TRACE] copying new root module from .terraform/init-from-module/root to .
Copying configuration from "/tmp/test/toplevel"...
panic: Rel: can't make /tmp/test/toplevel/submodule relative to 

goroutine 1 [running]:
github.com/hashicorp/terraform/internal/initwd.DirFromModule(0x1f13f6d, 0x1, 0xc000040600, 0x12, 0x7ffd2937933f, 0x12, 0xc00048dc00, 0x244aec0, 0xc00048dba0, 0x18, ...)
	/home/circleci/project/project/internal/initwd/from_module.go:269 +0x36ab
github.com/hashicorp/terraform/command.(*Meta).initDirFromModule(0xc000468700, 0x1f13f6d, 0x1, 0x7ffd2937933f, 0x12, 0x244aec0, 0xc00048dba0, 0x2, 0x0, 0x0)
	/home/circleci/project/project/command/meta_config.go:255 +0x19a
github.com/hashicorp/terraform/command.(*InitCommand).Run(0xc000468700, 0xc00003a0a0, 0x2, 0x2, 0xc000074e10)
	/home/circleci/project/project/command/init.go:148 +0x2af0
github.com/mitchellh/cli.(*CLI).Run(0xc0000d4640, 0xc0000d4640, 0xc000533d90, 0x1)
	/go/pkg/mod/github.com/mitchellh/cli@v1.0.0/cli.go:255 +0x1f1
main.wrappedMain(0x0)
	/home/circleci/project/project/main.go:251 +0xc8a
main.realMain(0x0)
	/home/circleci/project/project/main.go:106 +0xb4
main.main()
	/home/circleci/project/project/main.go:42 +0x3b
panic: Rel: can't make /tmp/test/toplevel/submodule relative to 

goroutine 1 [running]:
github.com/hashicorp/terraform/internal/initwd.DirFromModule(0x1f13f6d, 0x1, 0xc000040600, 0x12, 0x7ffd2937933f, 0x12, 0xc00048dc00, 0x244aec0, 0xc00048dba0, 0x18, ...)
	/home/circleci/project/project/internal/initwd/from_module.go:269 +0x36ab
github.com/hashicorp/terraform/command.(*Meta).initDirFromModule(0xc000468700, 0x1f13f6d, 0x1, 0x7ffd2937933f, 0x12, 0x244aec0, 0xc00048dba0, 0x2, 0x0, 0x0)
	/home/circleci/project/project/command/meta_config.go:255 +0x19a
github.com/hashicorp/terraform/command.(*InitCommand).Run(0xc000468700, 0xc00003a0a0, 0x2, 0x2, 0xc000074e10)
	/home/circleci/project/project/command/init.go:148 +0x2af0
github.com/mitchellh/cli.(*CLI).Run(0xc0000d4640, 0xc0000d4640, 0xc000533d90, 0x1)
	/go/pkg/mod/github.com/mitchellh/cli@v1.0.0/cli.go:255 +0x1f1
main.wrappedMain(0x0)
	/home/circleci/project/project/main.go:251 +0xc8a
main.realMain(0x0)
	/home/circleci/project/project/main.go:106 +0xb4
main.main()
	/home/circleci/project/project/main.go:42 +0x3b



!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

SECURITY WARNING: the "crash.log" file that was created may contain 
sensitive information that must be redacted before it is safe to share 
on the issue tracker.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Expected Behavior

I would expect terraform to resolve the relative directories before the copy, resulting in a successful terraform init with the downloaded modules.

Actual Behavior

Cryptic error message in the case of module foo { source = "../whatever" } and a crash in the case of module foo { source = "./whatever" }

Steps to Reproduce

As above.

Additional Context

I realize that you can avoid these errors by not using -from-module, so we do have a workaround.. We were using -from-module so that we have a chance to do some out-of-band TF templating between the terraform init & terraform apply commands, although we managed to strip most of this requirement thanks to 0.12.x.

References

[rmt]

When trying to work around this by just not using -from-module, I encountered what looks like #21966 (backend initialization required issues) ... so my current workaround is to copy the source folder to a new directory (minus .terraform & any other artifacts from local testing) and then do a run from the directory with pristine source. It does mean that module { source = "../sistermodule" } doesn't work out of the box, but module { source = "./submodule") does at least.

[danieldreier]

@rmt I have reproduced this issue on 0.12.29, but using 0.13.0 RC1 I cannot reproduce it, so I think this is fixed in the latest build. Can you try it with 0.13.0 RC1 (or GA, after Monday) and see if it's still a problem on your side?

[danieldreier]

@rmt I haven't been able to reproduce this in 0.13.x, and I haven't heard back from you, so I'm going to go ahead and close this. If you're still seeing this problem in 0.13.x, please comment here or file a new issue and link this one.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.