Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

URL encode / decode for uid #22

Open
tftm opened this issue Sep 30, 2022 · 6 comments
Open

URL encode / decode for uid #22

tftm opened this issue Sep 30, 2022 · 6 comments

Comments

@tftm
Copy link

tftm commented Sep 30, 2022 •
edited
Loading

Hi. We are trying to use the UserPrincipalName entry from MS AD as the UID field. In this case Synapse replaces the @ in the username with a slash (/) and performs the url_encode operation when accessing the API. Thus, the string /%40name%2Fdomain%3Adomain is formed in the URL.
However, matrix_corporal does not do this, and when we using name/domain format names, it substitutes them directly into the URL. As a result, for "id": "@name/domain:domain" record in policy.json the API is accessed at /_synapse/admin/v1
/users/@name/domain:domain/login which results in the error code=400 wrapped=M_UNRECOGNIZED: Unrecognized request
@spantaleev
Copy link
Member

This should be fixed in 2.3.1.

Would Synapse like you throwing things like @name/domain:domain there anyway?

@tftm
Copy link
Author

tftm commented Sep 30, 2022

We have many numeric accounts in AD, but Synapse has the specifics of working with them matrix-org/matrix-synapse-ldap3#55
Using UserPrincipalName as uid is the only alternative I see.

@tftm
Copy link
Author

tftm commented Sep 30, 2022

Hi, @spantaleev
There seems to be two encodings going on
In logs I see records like uri='/_synapse/admin/v1/users/%2540user%252Fdomain.net%253Adomain.net/login'
but it is url_encode() of uri='/_synapse/admin/v1/users/%40user%2Fdomain.net%3Adomain.net/login'

Logs for 2.3.0 (without encoding, error bc username seems like a url)

Sep 30 13:48:54 matrix matrix-corporal: time="2022-09-30T13:48:54Z" level=info msg="Retrying reconciliation.."
Sep 30 13:48:54 matrix matrix-synapse: 2022-09-30 13:48:54,240 - shared_secret_authenticator - 102 - INFO - POST-25 - Authenticating user @matrix-corporal:domain.net with login type m.login.password
Sep 30 13:48:54 matrix matrix-synapse: 2022-09-30 13:48:54,240 - shared_secret_authenticator - 121 - INFO - POST-25 - Authenticated user: @matrix-corporal:domain.net
Sep 30 13:48:54 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:48:54 +0000] "POST /_matrix/client/r0/login HTTP/1.1" 200 212 "-" "Go-http-client/1.1" "-"
Sep 30 13:48:54 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:48:54 +0000] "GET /_matrix/client/r0/account/whoami HTTP/1.1" 200 113 "-" "Go-http-client/1.1" "-"
Sep 30 13:48:54 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:48:54 +0000] "GET /_synapse/admin/v2/users?deactivated=true&guests=false&limit=100000000000 HTTP/1.1" 200 233 "-" "Go-http-client
/1.1" "-"
Sep 30 13:48:54 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:48:54 +0000] "POST /_synapse/admin/v1/users/@user/domain.net:domain.net/login HTTP/1.1" 400 70 "-" "Go-http-client/1.1" "-"
Sep 30 13:48:54 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:48:54 +0000] "POST /_matrix/client/r0/logout HTTP/1.1" 200 33 "-" "Go-http-client/1.1" "-"
Sep 30 13:48:54 matrix matrix-corporal: time="2022-09-30T13:48:54Z" level=warning msg="Reconciliation failed: Failure determining current state: contents=[123 34 101 114 114 99 111 100 101 34 58 34 77 95
85 78 82 69 67 79 71 78 73 90 69 68 34 44 34 101 114 114 111 114 34 58 34 85 110 114 101 99 111 103 110 105 122 101 100 32 114 101 113 117 101 115 116 34 125] msg=Failed to POST JSON to /_synapse/admin/v1
/users/@user/domain.net:domain.net/login code=400 wrapped=M_UNRECOGNIZED: Unrecognized request"

Logs for 2.3.1 (with encoding, error bc username seems like twice encoded)

Sep 30 13:50:45 matrix matrix-corporal: time="2022-09-30T13:50:45Z" level=info msg="Retrying reconciliation.."
Sep 30 13:50:45 matrix matrix-synapse: 2022-09-30 13:50:45,255 - shared_secret_authenticator - 102 - INFO - POST-1 - Authenticating user @matrix-corporal:domain.net with login type m.login.password
Sep 30 13:50:45 matrix matrix-synapse: 2022-09-30 13:50:45,258 - shared_secret_authenticator - 121 - INFO - POST-1 - Authenticated user: @matrix-corporal:domain.net
Sep 30 13:50:45 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:50:45 +0000] "POST /_matrix/client/r0/login HTTP/1.1" 200 215 "-" "Go-http-client/1.1" "-"
Sep 30 13:50:45 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:50:45 +0000] "GET /_matrix/client/r0/account/whoami HTTP/1.1" 200 113 "-" "Go-http-client/1.1" "-"
Sep 30 13:50:45 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:50:45 +0000] "GET /_synapse/admin/v2/users?deactivated=true&guests=false&limit=100000000000 HTTP/1.1" 200 233 "-" "Go-http-client
/1.1" "-"
Sep 30 13:50:45 matrix matrix-synapse: 2022-09-30 13:50:45,309 - shared_secret_authenticator - 102 - INFO - POST-4 - Authenticating user @matrix-corporal:domain.net with login type m.login.password
Sep 30 13:50:45 matrix matrix-synapse: 2022-09-30 13:50:45,309 - shared_secret_authenticator - 121 - INFO - POST-4 - Authenticated user: @matrix-corporal:domain.net
Sep 30 13:50:45 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:50:45 +0000] "POST /_matrix/client/r0/login HTTP/1.1" 200 201 "-" "Go-http-client/1.1" "-"
Sep 30 13:50:45 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:50:45 +0000] "GET /_matrix/client/r0/account/whoami HTTP/1.1" 200 100 "-" "Go-http-client/1.1" "-"
Sep 30 13:50:45 matrix matrix-synapse: 2022-09-30 13:50:45,331 - synapse.http.server - 183 - ERROR - POST-6 - Failed handle request via 'UserTokenRestServlet': <XForwardedForRequest at 0x7f30febd0910 meth
od='POST' uri='/_synapse/admin/v1/users/%2540user%252Fdomain.net%253Adomain.net/login' clientproto='HTTP/1.0' site='8008'>
Sep 30 13:50:45 matrix matrix-synapse: Traceback (most recent call last):
Sep 30 13:50:45 matrix matrix-synapse: File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
Sep 30 13:50:45 matrix matrix-synapse: result = current_context.run(gen.send, result)
Sep 30 13:50:45 matrix matrix-synapse: StopIteration: 1
Sep 30 13:50:45 matrix matrix-synapse: During handling of the above exception, another exception occurred:
Sep 30 13:50:45 matrix matrix-synapse: Traceback (most recent call last):
Sep 30 13:50:45 matrix matrix-synapse: File "/usr/local/lib/python3.9/site-packages/synapse/http/server.py", line 366, in _async_render_wrapper
Sep 30 13:50:45 matrix matrix-synapse: callback_return = await self._async_render(request)
Sep 30 13:50:45 matrix matrix-synapse: File "/usr/local/lib/python3.9/site-packages/synapse/http/server.py", line 572, in _async_render
Sep 30 13:50:45 matrix matrix-synapse: callback_return = await raw_callback_return
Sep 30 13:50:45 matrix matrix-synapse: File "/usr/local/lib/python3.9/site-packages/synapse/rest/admin/users.py", line 926, in on_POST
Sep 30 13:50:45 matrix matrix-synapse: if not self.is_mine_id(user_id):
Sep 30 13:50:45 matrix matrix-synapse: File "/usr/local/lib/python3.9/site-packages/synapse/server.py", line 341, in is_mine_id
Sep 30 13:50:45 matrix matrix-synapse: return string.split(":", 1)[1] == self.hostname
Sep 30 13:50:45 matrix matrix-synapse: IndexError: list index out of range
Sep 30 13:50:45 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:50:45 +0000] "POST /_synapse/admin/v1/users/%2540user%252Fdomain.net%253Adomain.net/login HTTP/1.1" 500 66 "-" "Go-http-client/1.1" "-
"
Sep 30 13:50:45 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:50:45 +0000] "POST /_matrix/client/r0/logout HTTP/1.1" 200 33 "-" "Go-http-client/1.1" "-"
Sep 30 13:50:45 matrix matrix-corporal: time="2022-09-30T13:50:45Z" level=warning msg="Reconciliation failed: Failure determining current state: contents=[123 34 101 114 114 99 111 100 101 34 58 34 77 95
85 78 75 78 79 87 78 34 44 34 101 114 114 111 114 34 58 34 73 110 116 101 114 110 97 108 32 115 101 114 118 101 114 32 101 114 114 111 114 34 125] msg=Failed to POST JSON to /_synapse/admin/v1/users/%40gi
i%2Fdomain.net%3Adomain.net/login code=500 wrapped=M_UNKNOWN: Internal server error"

Logs from synapse when user login by Element (with encoding no errors)

Sep 30 13:54:21 matrix matrix-nginx-proxy: 10.0.241.61 - - [30/Sep/2022:13:54:21 +0000] "OPTIONS /_matrix/client/r0/user/%40user%2Fdomain.net%3Adomain.net/filter HTTP/2.0" 200 0 "-" "Mozilla/5.0 (Macinto
sh; Intel Mac OS X 10.15; rv:104.0) Gecko/20100101 Firefox/104.0" "-"
Sep 30 13:54:21 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:54:21 +0000] "POST /_matrix/client/r0/user/%40user%2Fdomain.net%3Adomain.net/filter HTTP/1.1" 200 48 "-" "Mozilla/5.0 (Macintosh
; Intel Mac OS X 10.15; rv:104.0) Gecko/20100101 Firefox/104.0" "10.0.241.61, 172.18.0.11"
Sep 30 13:54:21 matrix matrix-nginx-proxy: 10.0.241.61 - - [30/Sep/2022:13:54:21 +0000] "POST /_matrix/client/r0/user/%40user%2Fdomain.net%3Adomain.net/filter HTTP/2.0" 200 37 "-" "Mozilla/5.0 (Macintosh
; Intel Mac OS X 10.15; rv:104.0) Gecko/20100101 Firefox/104.0" "-"
....
Sep 30 13:54:21 matrix matrix-nginx-proxy: 10.0.241.61 - - [30/Sep/2022:13:54:21 +0000] "OPTIONS /_matrix/client/r0/sync?filter=0&timeout=30000&since=s1_2_0_1_1_1_1_737_0 HTTP/2.0" 200 0 "-" "Mozilla/5.0
(Macintosh; Intel Mac OS X 10.15; rv:104.0) Gecko/20100101 Firefox/104.0" "-"
Sep 30 13:54:21 matrix matrix-nginx-proxy: 10.0.241.61 - - [30/Sep/2022:13:54:21 +0000] "OPTIONS /_matrix/client/unstable/keys/signatures/upload HTTP/2.0" 200 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10.15; rv:104.0) Gecko/20100101 Firefox/104.0" "-"
Sep 30 13:54:21 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:54:21 +0000] "GET /_matrix/client/r0/capabilities HTTP/1.1" 200 289 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:104.0)
Gecko/20100101 Firefox/104.0" "10.0.241.61, 172.18.0.11"
Sep 30 13:54:21 matrix matrix-nginx-proxy: 10.0.241.61 - - [30/Sep/2022:13:54:21 +0000] "GET /_matrix/client/r0/capabilities HTTP/2.0" 200 277 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:104.0)
Gecko/20100101 Firefox/104.0" "-"
Sep 30 13:54:21 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:54:21 +0000] "GET /_matrix/client/r0/profile/%40user%2Fdomain.net%3Adomain.net HTTP/1.1" 200 104 "-" "Mozilla/5.0 (Macintosh; In
tel Mac OS X 10.15; rv:104.0) Gecko/20100101 Firefox/104.0" "10.0.241.61, 172.18.0.11"
Sep 30 13:54:21 matrix matrix-nginx-proxy: 10.0.241.61 - - [30/Sep/2022:13:54:21 +0000] "GET /_matrix/client/r0/profile/%40user%2Fdomain.net%3Adomain.net HTTP/2.0" 200 93 "-" "Mozilla/5.0 (Macintosh; Int
el Mac OS X 10.15; rv:104.0) Gecko/20100101 Firefox/104.0" "-"
Sep 30 13:54:21 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:54:21 +0000] "POST /_matrix/client/unstable/keys/signatures/upload HTTP/1.1" 200 161 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10.15; rv:104.0) Gecko/20100101 Firefox/104.0" "10.0.241.61, 172.18.0.11"
Sep 30 13:54:21 matrix matrix-nginx-proxy: 10.0.241.61 - - [30/Sep/2022:13:54:21 +0000] "POST /_matrix/client/unstable/keys/signatures/upload HTTP/2.0" 200 150 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10.15; rv:104.0) Gecko/20100101 Firefox/104.0" "-"
Sep 30 13:54:21 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:54:21 +0000] "POST /_matrix/client/r0/keys/upload HTTP/1.1" 200 78 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:104.0) G
ecko/20100101 Firefox/104.0" "10.0.241.61, 172.18.0.11"
Sep 30 13:54:21 matrix matrix-nginx-proxy: 10.0.241.61 - - [30/Sep/2022:13:54:21 +0000] "POST /_matrix/client/r0/keys/upload HTTP/2.0" 200 67 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:104.0) G
ecko/20100101 Firefox/104.0" "-"
Sep 30 13:54:21 matrix matrix-synapse: 2022-09-30 13:54:21,204 - shared_secret_authenticator - 102 - INFO - POST-75 - Authenticating user @user/domain.net:domain.net with login type m.login.password
Sep 30 13:54:21 matrix matrix-synapse: 2022-09-30 13:54:21,204 - shared_secret_authenticator - 113 - INFO - POST-75 - Bad hmac value for user: @user/domain.net:domain.net
Sep 30 13:54:21 matrix matrix-synapse: 2022-09-30 13:54:21,204 - rest_auth_provider - 48 - INFO - POST-75 - Got password check for @user/domain.net:domain.net
Sep 30 13:54:21 matrix matrix-corporal: time="2022-09-30T13:54:21Z" level=info msg="HTTP gateway: internal REST authentication" method=POST uri=/_matrix/corporal/_matrix-internal/identity/v1/check_credent
ials
Sep 30 13:54:21 matrix matrix-synapse: 2022-09-30 13:54:21,214 - rest_auth_provider - 60 - INFO - POST-75 - User not authenticated
Sep 30 13:54:21 matrix matrix-synapse: 2022-09-30 13:54:21,234 - ldap_auth_provider - 178 - INFO - POST-75 - User authenticated against LDAP server: ldap://10.0.x.xx:389 - cleartext - user: CN=User U
,OU=Test,DC=domain,DC=net - not lazy - bound - open - <local: 172.18.0.12:59782 - remote: 10.0.xx.xx:389> - tls not started - listening - SyncStrategy - internal decoder
Sep 30 13:54:21 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:54:21 +0000] "POST /_matrix/client/r0/keys/upload HTTP/1.1" 200 79 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:104.0) G
ecko/20100101 Firefox/104.0" "10.0.241.61, 172.18.0.11"
Sep 30 13:54:21 matrix matrix-nginx-proxy: 10.0.241.61 - - [30/Sep/2022:13:54:21 +0000] "POST /_matrix/client/r0/keys/upload HTTP/2.0" 200 68 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:104.0) G
ecko/20100101 Firefox/104.0" "-"
Sep 30 13:54:21 matrix matrix-nginx-proxy: 172.18.0.16 - - [30/Sep/2022:13:54:21 +0000] "POST /_matrix/client/unstable/keys/device_signing/upload HTTP/1.1" 200 33 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS
X 10.15; rv:104.0) Gecko/20100101 Firefox/104.0" "10.0.241.61, 172.18.0.11"
Sep 30 13:54:21 matrix matrix-nginx-proxy: 10.0.241.61 - - [30/Sep/2022:13:54:21 +0000] "POST /_matrix/client/unstable/keys/device_signing/upload HTTP/2.0" 200 22 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS
X 10.15; rv:104.0) Gecko/20100101 Firefox/104.0" "-"

@spantaleev
Copy link
Member

You're right.. 2.3.1 made things worse - completely breaking reconcilliation 🤦‍♂️

I've since reverted this URL-encoding change. 2.3.2 should be pretty much the same as 2.3.0.

People with servers with normal mxids should no longer see errors during reconcilliation.

Perhaps your problem is not with matrix-corporal's reconcilliation though? But with you hitting /_synapse/admin/v1/users/<user_id>/login yourself from your own scripts?

Or is reconcilliation failing on 2.3.0 (and now on 2.3.2)?

@spantaleev spantaleev reopened this Oct 2, 2022
@tftm
Copy link
Author

tftm commented Oct 3, 2022

Reconcilliation fails on 2.3.0 without creating first user in list.
On 2.3.1 the reconcilliation fails too, but the first user is created.
M.b. I'm wrong, but I guess, that on 2.3.1 matrix-corporal sends the correct uid first (encoded it on POST /login), than get response with same encoded uid as confirmation.
For the second time matrix-corporal takes the already encoded uid and encoded it once more on POST /login.

M.b. something like this can solve the issue:

if strings.Contains(userId, "/") {
               userId  = url.QueryEscape(userId)
}
err = client.MakeRequest(
 		"POST",
 		buildPrefixlessURL(client, fmt.Sprintf("/_synapse/admin/v1/users/%s/login", userId), map[string]string{}),
 		requestPayload,
 		&response,
 	)

but I wouldn't break master once more)

@spantaleev
Copy link
Member

You can try that patch yourself and see if it helps with all of your users.

Something is definitely fishy with these users containing /. I suspect it may be an upstream problem too. Perhaps the /_synapse/admin/v1/users/<user_id>/login API needs to be adjusted so that it handles URL-encoded values or something. It seems to hate URL-encoding in the general case, but like it in case there are things like /. Hmm..

Have you tried calling these APIs manually (with an admin token) to see what values they like and don't like? You could then report the issue to the Synapse repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@spantaleev @tftm