Understanding the Role of Kubernetes Operator for Netbird

[bgurgoiener]

The most likely issue is that I do not understand the role of the Kubernetes operator for Netbird.

My use case:

• I have 60 laptops and 30 mobile devices that connect to various applications across multiple Kubernetes clusters.
• I want to migrate the connection to use WireGuard, with Netbird coordinating it.

What I expect:

• Install the Kubernetes operator in the cluster.

• Add annotations to services such as:

  netbird.io/expose: "true"
  netbird.io/policy: default
  netbird.io/resource-name: my-intranet-web

• Configure something in the Netbird admin panel.

• Ensure that the 60 laptops and 30 mobile devices can access the service via a specific domain, e.g., my-intranet-web.vpn.my-company.com.

What happens:

• In the Netbird admin panel under Networks > Kubernetes, I see an entry for my-intranet-web with the address:
  my-intranet-web-service.default.svc.cluster.local
• The clients do not seem to resolve the domain my-intranet-web-service.default.svc.cluster.local.
• I do not know which IP I should use to connect from the laptops to the service.

I am testing this locally with a Kubernetes cluster in KIND and a Docker image simulating a laptop. The issue might be related to my testing environment. I need guidance on how to diagnose it.

Optional extra requirements:

• The domain name used should belong to our company. For usability, we want a web page to appear when users are outside the VPN, informing them that they need to connect.
• To promote a security culture, we want the connection to be via HTTPS with a valid certificate. We would use our own domain for this purpose.

I believe this Kubernetes operator might not be designed for my use case, but I am not sure.

[Neks3s]

You need to use the same cluster DNS in your Helm values as the one configured in your Kubernetes cluster. If you haven’t changed anything, it should be set to the default.

What can be changed is the cluster name. Also, make sure you’re using the same resource name as the service name—this allowed me to resolve the service.

However, I’m still stuck because the actual connection isn’t going through.

[mohamed-essam]

Hello,

There are some points I would like to make sure are communicated correctly:

1. For the time being, domain routing is not supported by Android AFAIK, so this is a limitation from the main Android application implementation.
2. To correctly route the connection, DNS Wildcard Routing and a valid Nameserver must be configured (the Nameserver should be configured for the clients only not the routing peers), as mentioned in the Usage docs.
3. For the time being, the only possible way to address services internally is through the Kubernetes DNS (which is as you mentioned *.svc.cluster.local), there are discussion for better addressing of services in progress.

As for the optional extra requirements:

1. Currently custom DNS records for Resources are not supported, but they may be supported in future releases.
2. Since the routing peers are not privy to the actual HTTP communications, I do not think it would be possible to upgrade the connection to HTTPS with a custom domain name, HTTPS would need to be implemented at the Service level.

Please let me know if you're still having issues with connection from laptop to kubernetes after enabling DNS settings as mentioned above 🙏

[HurricanKai]

After spending the better part of a day debugging this, this comment finally solved it. Of course Peers have to be excluded from the DNS config! This never mentioned anywhere, or I didn't notice it, even after reading it like 10 times.
Thank you @mohamed-essam!

[2WeltenChris]

3. For the time being, the only possible way to address services internally is through the Kubernetes DNS (which is as you mentioned *.svc.cluster.local), there are discussion for better addressing of services in progress.

It would be great if cluster ressources could be addressed either through the netbird domain configured under Settings > Networks or a custom one.

Changing the domain within k8s can be tricky or even impossible. And using the default one makes problems with issuing certificates.

Would it be possible to automatically rewrite the domain defined in cluster > dns (within the values.yaml for the helm chart) to whatever the cluster has set as a domain?

Thanks!

[mohamed-essam]

It would be great if cluster ressources could be addressed either through the netbird domain configured under Settings > Networks or a custom one.

This is currently outside the scope of the operator as it would need core functionality changes within the NetBird repository, but rest assured it's in the works.

Changing the domain within k8s can be tricky or even impossible. And using the default one makes problems with issuing certificates.

That's completely understandable, improvements to connectivity, security, and ease of use will be added with time.

Would it be possible to automatically rewrite the domain defined in cluster > dns (within the values.yaml for the helm chart) to whatever the cluster has set as a domain?

I'm not sure if there's a way for the operator to find the cluster DNS automatically from the cluster itself, but will check if it's possible and get back to you

I will be closing the issue as the root issue is resolved, feel free to open any other issues as needed