S3 Backend fails to resolve mDNS address

[eaglesemanation]

Terraform Version

Terraform v1.3.8
on linux_amd64

Terraform Configuration Files

terraform {
  backend "s3" {
    endpoint = "https://emnt-nas.local:9000/"
    region = "us-west-1"
    bucket = "terraform-state"
    key = "talos-cluster.tfstate"
    shared_credentials_file = "./s3creds.toml"
    skip_credentials_validation = true
    skip_metadata_api_check = true
    force_path_style = true
  }
}

Debug Output

https://gist.github.com/eaglesemanation/65ae127ad4fd928302c460cc8bdb8e67

Expected Behavior

Connect to MinIO S3 server hosted on my TrueNAS server

Actual Behavior

Terraform refuses to resolve mDNS address. Replacing it with IP address and falling back to HTTP works.

Steps to Reproduce

terraform init

Additional Context

No response

References

No response

[userbradley]

Can you try remove the trailing slash?

I doubt it would make a difference, but looking at this post here they dont have a / in their endpoint

Also other thing, is that SSL cert valid for https://emnt-nas.local:9000?

Finally, I know this is beating a dead horse, can you do an nslookup emnt-nas.local for the nas?

[eaglesemanation]

• Trailing slash does not have any effect. With domain name it will fail regardless of slash, with IP - it will work.
• SSL certificate is the default one provided with TrueNAS, so it's self-signed, and it's issued for localhost. So it's not fully valid, but I think that will be unrelated issue.
• nslookup emnt-nas.local does not return an IP, because it does not use /etc/nsswitch.conf and instead only uses DNS servers specified in /etc/resolv.conf. So for example ping emnt-nas.local does resolve domain to correct IP address.

After some troubleshooting, I found a way to configure my system to make this work. I'm using systemd-resolved in combination with NetworkManager, which are defaults on Fedora 37. I had to change /etc/systemd/resolved.conf with this:

[Resolve]
...
MulticastDNS=yes
...

And add this to /etc/NetworkManager/NetworkManager.conf:

[connection]
connection.mdns=2

Which made nslookup resolve IP correctly. Now I'm getting errors from terraform about certificate being not valid, but that's something that I can fix on my own.

I would still consider this a bug, because terraform shouldn't necessarily rely only on DNS to work correctly.

[jbardin]

Hi @eaglesemanation,

I'm glad you found a solution to your problem. It would be hard to come up with a universal solution to this for Terraform becauseTerraform isn't directly involved in the name resolution here at all. The AWS SDK is handling the connection, which in turn relies on the standard library for name resolution. In most cases it is up to the host operating system to be configured such that name resolution falls back to the correct underlying method, which is appears is what you have done (I know macOS does this by default already).

I suspect that linking Terraform to the host libc might resolve what you expect given that the ping command resolved for you. You can test this if you want by building Terraform locally with gcc installed, which will automatically link to the host resolver by default. Compiling Terraform in this manner is something we have considered, but we lose some of the benefits of a static binary making it less portable.

[bflad]

It may also be worth mentioning here that darwin binaries built with Go 1.20 are more integrated with the macOS DNS resolution system without needing CGO, etc. golang/go#12524

[eaglesemanation]

There is an issue for netdns to do mDNS resolution, and it's marked as Unplanned since 2019. Looks like to solve this somebody needs to either fix Go net lib, or AWS SDK, and either options sound quite daunting. So instead I'll probably request Fedora maintainers to enable mDNS by default.

[gdavison]

Thanks for reporting this, @eaglesemanation.

Is there any difference in the behaviour of name resolution if you use a fully-qualified domain name, that is https://emnt-nas.local.:9000/ (with a . after local)? That may prompt the resolver to behave differently.

Also, can you re-run the Terraform operation with the environment variable GODEBUG=netdns=1 set? That should give some extra information from Go's resolver, as described at https://pkg.go.dev/net#hdr-Name_Resolution

[jbardin]

@gdavison, Terraform is distributed as a statically linked binary, so will always be using netdns=go.

[tristanmorgan]

@eaglesemanation Is this now resolved by #32688 ?
my test with minio works with an extra resolver in /etc/resolver/consul

[eaglesemanation]

It's been a while, and I'm not 100% sure that I've correctly reverted my fix to test it, but it seems to be working! Thanks for reminding me

[crw]

@eaglesemanation Given your feedback, I am going to close this issue. If I am mistaken in my understanding that this issue is resolved, please let me know. Thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.