Question: How to tell clients to connect to a different STUNner IP with the stunner-gateway-operator?

[calexandre]

I have a scenario where we have an internal network where our Kubernetes exposed services are behind 1:1 NAT IPs...
This means that the LoadBalancer's IP addresses that Kubernetes knows are not the same that the clients use to connect to them.

For example:

• Suppose we expose STUNner's LoadBalancer service on external-ip address 10.0.0.1:3478/udp but our clients will reach it on 10.0.20.1:3478/udp as configured on our internal firewall.

On the Kurento One2one Call example I need to be able to configure the webrtc-server frontend to expose the TURN URI to something like turn:10.0.20.1:3478?transport=UDP, which is the IP address that the client will be able to reach.

Using STUNner in standalone mode I believe I can tweak the config STUNNER_PUBLIC_ADDR to point to the correct IPAddress that the client can reach; but I was not able to figure it out with the stunner-gateway-operator.

Maybe you can shed some light?

[nmate]

Hi,
Currently the operator fetches the public IP of the LoadBalancer service and in the examples we give this address to the clients via the stunner-auth-lib (https://github.com/l7mp/stunner#configuring-webrtc-clients). And if I get it right, you would like to override this value. Would you try to use an explicit ICE config in the app server instead of getting it from the configuration? (i.e. instead of using auth.getIceConfig() in auth-lib, providing the exact values you need)

[calexandre]

Hi, Currently the operator fetches the public IP of the LoadBalancer service and in the examples we give this address to the clients via the stunner-auth-lib (https://github.com/l7mp/stunner#configuring-webrtc-clients). And if I get it right, you would like to override this value. Would you try to use an explicit ICE config in the app server instead of getting it from the configuration? (i.e. instead of using auth.getIceConfig() in auth-lib, providing the exact values you need)

I could, but I'm trying to make this work on the existing examples such as CloudRetro and Kurento...
Is there any way to do it on the server-side?

I tried to change the ConfigMap manually, but it gets reconciled after some time...

[nmate]

You are supposed to change the stunner config via Kubernetes objects. These are then inspected by the operator and rendered into stunner, and in our example we use the same config for the webrtc-server application as well - since we do not have the special setup you have.

I assume you do not want to change the stunner config itself, but only to be able to specify the IP address that the clients can reach regardless the stunner config. And you would like to do it without touching the webrtc-server Docker image. If this is the case, a possible solution could be to introduce a deployment time user parameter (IP address) to the webrtc-server yamls that will be parsed and used as ICE config towards the clients (similar to what I proposed previously).

On the other hand, there seems to be a future track in the operator that allows users to define LB service that is then exposed for a gateway (i.e. not automatically filled) (see the comments of https://github.com/l7mp/stunner-gateway-operator/blob/main/internal/renderer/service_util.go). This does not work right now for sure, and I have to discuss the plans with this in the team.

[calexandre]

If this is the case, a possible solution could be to introduce a deployment time user parameter (IP address) to the webrtc-server yamls that will be parsed and used as ICE config towards the clients (similar to what I proposed previously).

Will this work on the CloudRetro examples aswell? Namely, on the coordinator-deployment deployment.

[bbalint105]

Hi,
In the CloudRetro demo You can easily configure which address it should return as a TURN server to the clients, its in the ConfigMap of the coordinator.

Namely the cloudretro-config-c, under the CLOUD_GAME_WEBRTC_ICESERVERS_0_URL env. variable. This will overwrite the default coordinator config upon launch, and this is the address the coordinator will share with the clients as the TURN server. You can edit this value to whatever You would like, so when clients connect to the coordinator, they will receive your address.
Don't forget to rollout restart the deployment to make changes take effect!

If You can't reach the coordinator through TCP, now that will be one pleasant ride.

[calexandre]

Thank you @bbalint105!
I will try to use this issue to get some answers regarding this issue (#24) aswell, which is about connecting clients to CloudRetro demo using TCP instead of UDP.

Assuming it is possible,, in order for clients to connect to STUNner via TCP I will need the following:

• Using the stunner-gateway-operator, create a TCP Gateway resource
• Create a UDPRoute resource referencing the TCP Gateway I created above
• Change the config CLOUD_GAME_WEBRTC_ICESERVERS_0_URL to something like turn:<tcp-gateway-public-ipaddress>:3478
  • Question: Do I need any kind of notation here to let the browser know that it must connect using TCP transport? Like some querystring ?transport=tcp or equivalent...?

The TCP Gateway and the UDPRoute would be something like this:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  name: tcp-gateway
  namespace: stunner
spec:
  gatewayClassName: stunner-gatewayclass
  listeners:
    - name: tcp-listener
      port: 3478
      protocol: TCP
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: UDPRoute
metadata:
  name: worker-udp-route
  namespace: stunner
spec:
  parentRefs:
    - name: tcp-gateway
  rules:
    - backendRefs:
        - name: worker-ci-udp-svc
          namespace: cloudretro

Am I going the right direction?

[bbalint105]

Assuming it is possible,, in order for clients to connect to STUNner via TCP I will need the following:

(...)

Am I going the right direction?

Yes, You quite got everything (as well the ?transport=tcp after the TURN server address)!
Although, You will still need a new ClusterIP service for the workers, because the worker-ci-udp-svc is set as UDP.

Adding it to your list should make it work;

apiVersion: v1
kind: Service
metadata:
  name: worker-ci-tcp-svc
  namespace: cloudretro
spec:
  selector:
    app: worker
  ports:
    - protocol: TCP
      port: 8443
      targetPort: 8443
  type: ClusterIP
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  name: tcp-gateway
  namespace: stunner
spec:
  gatewayClassName: stunner-gatewayclass
  listeners:
    - name: tcp-listener
      port: 3478
      protocol: TCP
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: UDPRoute
metadata:
  name: worker-tcp-route
  namespace: stunner
spec:
  parentRefs:
    - name: tcp-gateway
  rules:
    - backendRefs:
        - name: worker-ci-tcp-svc
          namespace: cloudretro

I've tested it on my cluster, and double-checked with wireshark, it was indeed running on TCP.
Let me know if it works, I could have missed to write down something.

[rg0now]

Quick clarification to the above:

• inside the cluster STUNner always uses UDP, no matter whether you expose the gateway on UDP or TCP (STUNner will automatically convert TCP to UDP if needed), that's why STUNner implements only UDP routes,
• so you need only a single target/backend service and the protocol should always be UDP (port number does not matter), even if the gateway itself uses TCP.
  We are still hesitating whether to implement TCPRoutes as well to allow TCP intra-cluster traffic, but so far we haven't seen strong use cases for this: UDP seems to be constantly better for live media than TCP

[calexandre]

I confirm the above setup works and was able to expose STUNner with TCP.
Unfortunately, the cloudretro demo is still pretty much unplayable on my corporate cluster... This is probably something on the network and we're still trying to figure what is causing the bad experience... :(

On the other side, the Kurento examples don't have any issues and are streaming just fine, so I really don't know how to debug this...

[calexandre]

My problems have been fixed. I was able to make this solution work inside a corporate network!