Integrity check fails on passwords containing $ [was: Auth server return bad url]
#102
Comments
rg0now
added
priority: high
type: bug
|
An immediate problem is that your UDPRoute cannot attach to the Gateway: You can see why this in a problem the If you want to attach the UDPRoute to a specific listener on the gateway, you must specify both the Gateway name and the listener name in the This though still doesn't explain the other weirdness, the TURN errors: This means the client sends truncated packets or some middlebox chops off the meaningful part of our TURN packets or this can be the wildest case of an MTU issue I've ever seen. I must see a tcpdump to know what's wrong here, but it is definitely not a STUNner issue. |
|
Well, and then there is the auth issue. This is weird: we fail to generate a turn protocol scheme and a transport protocol, that is clearly a bug. Did you install from the stable channel or the latest version from the dev channel? There has been a massive rewrite lately that might have broken the dev installs, we'll look into that ASAP. |
|
@rg0now Thank you for your quick response. I found I can't connect the turnserver, it return 401, so I change the auth type from ephemeral to static, The problem remains. Then I found your response, And change the config. It works. Then I change the auth type back, I can't get to the server again. In the log, I found: What is it means?
I install the stable operotor weeks ago. At first it return good url. Today I found the url is broken. I have not reinstalled the operator. I change the auth type back to static, and I can connect to server again, Here is the log: By the way, there are logs of similar errors such as "not enough bytes to read header" and "BadFormat for message/cookie: 34353637 is invalid magic cookie (should be 2112a442)". It seems that the stunner still work well with these errors. |
|
This was me breaking the auth service, which, combined with anyone bug in our helm charts that defaults the auth-service container image version to the latest dev version even in the stable install, caused that everyone was immediately affected. We'll try to settle this ASAP. Meanwhile you can manually downgrade the auth-service container image version to v0.15.0, that should fix this until we stabilize the dev version and roll a new release. Or you can use a fix TURN credential for now. As per the truncated TURN messages, that's still unexplainable. I doubt this issue has anything to do with the auth service bug, maybe a misbehaving client? We'll see once we fix the auth-service. |
|
Here is my stunner image: I have change docker.io/l7mp/stunner-auth-server:dev to docker.io/l7mp/stunner-auth-server:0.15.0 The url is right now. But ephemeral auth type still not work. And the error not change. |
|
From the below it seems that the auth-credentials are accepted by STUNner.: The subsequent error is then caused by that your client generates a wrong integrity hash into the packets: Can you please confirm that it works with static username/password pairs, only the ephemeral auth credentials are what trigger this issue? Judging from the weird errors you get, it can be some misbehaving TURN client. Are you trying to connect from a browser? |
|
I cannot reproduce the ephemeral auth issue with the latest dev version. Please uninstall stunnerd and the gateway-operator and reinstall from the dev channel and report back any issue you find. Thx! As per the "truncated TURN packets problem: please use the simple-tunnel tutorial to check whether your cluster is OK. That demo uses our own TURN client so at least the "misbehaving client" problem will go away. |
I have tried this many times and only ephemeral auth triggers this issue. I can't say for sure if this has been an issue before, as this seems to be the first time I've tried to use ephemeral auth. it's always been static auth before, I'll do some further research next Monday. |
|
Ephemeral auth still not work. Here are the errors: Change to static auth, error disappeared, and it works. |
|
I cannot reproduce this issue with latest stable. I tested with simple-tunnel, changed the authentication to ephemeral: Then tested with |
|
@rg0now I try again, still not work, same error. Is there a docker image bundled with turncat, so I can use it to test? |
|
Hi @vipcxj !
l7mp/net-debug has |
|
@levaitamas I found the problem~ It's secret key. apiVersion: v1
kind: Secret
metadata:
name: stunner-study-ai-auth
namespace: stunner
type: Opaque
stringData:
type: ephemeral
secret: 4h$sdgh[k)tAgjTR54gfjus3wayrt
# secret: my-shared-secret |
|
This is (hopefully) resolved now. Feel free to reopen if problem persists. |
|
@rg0now which commit solve it? I have shorten the secret as the workaround. So it seems that I will not trigger this issue any more. |
|
Sorry for the confusion, I thought we resolved it here. Can you please confirm that the secret |
|
Can't replace secret right now, but I confirm that I did have this issue when I commented earlier, and it's not clear if you've done any fixes since then. I fixed it by use a shorter secret. So it seems that too long secret will tirgger the issue. |
|
Thx for the report, I'm trying to track this down now. My bet would be that it's the symbols like |
rg0now
added
type: bug
|
So this seems like a good old UNIX/Bash gotcha. If you use the below to set the secret, then any value you try to use that is containing a symbol like The good news is that the corrupted secret nicely finds its way into the STUNner dataplane config, so at least we don't mess things up during the base64 encode/decode roundtrip: However, place your Secret into a YAML file, say, And then kubectl-apply it and all is fine: So I guess shell-escaping bites us here: the shell interprets So this fixes it, and this is indeed a shell escaping thingie... Today I (re)learned something important again: just use standard YAML manifests that do not get processed by the shell and then all is fine. |
|
@rg0now I got the auth secret from stunner auth server, so if the secret is changed by something, it not matter what I gotten. Because I do not achieve the secret from k8s directly. |
|
@rg0now And I have tried config secret both directly in GatewayConfig and an authRef of secret, nothing changed. The issue does not disappeared until I changed the secret. By the way, the new secret does not contain '$', but still contain '@', and it is 13 bytes long. |
|
Can you please specify exactly what didn't work for you? Just post all YAMLs and command lines what you did so that I can reproduce the problem. |
Then my webrtc client achieve the auth config just from the stunner auth server, and unable to connect the peer. After I change authType to plaintext, the issue disappears. Of couse, change the secret to a shorter one also works. |
rg0now
changed the title
$ [was: Auth server return bad url]
|
This should be fixed in 1bb46b7 and will be available in dev once the CI pipeline has finished building the image. The bug was in STUNner: we apply environment substitution on the config file while parsing it (this allows to customize per-gateway configs to per-pod context locally) and this plays badly with TURN credential parsing if the password or the secret contains a Thanks a lot @vipcxj for helping us tracking this down, this was a particularly ugly one. Feel free to reopen if the problem persists. |
it return:
{ "iceServers":[ { "username":"1695376501:user1", "credential":"EPddI2tMN9vtfGMhup1RYE5nSkA=", "urls":[":192.168.0.247:3478?transport="] } ], "iceTransportPolicy":"all" }Here is the log of auth server:
Here is the log of the stunner pod:
Here is my config:
The text was updated successfully, but these errors were encountered: