401 Unauthorized error on a request to "/1.5/{uid}/*" when using a node-url with added path like "http://localhost/fxs"

[LuckyTeran]

TL;DR:
When i add a node into the database that has a added path like "http://localhost/fxs" auth seems to fail and the server returns 401. Without the added path everything works. Even changing different hostname (like test.tld) or port does not produce an error. It seems somehow the url is used in auth but not the hostname?

Long version with steps i tried to find and diagnose the error:

Since #1189 is fixed i was able to run a working local instance of syncstorage_rs. I then tried to deploy syncstorage_rs on a dedicated server with public ip and for added security behind Apache2.4 as proxy. The connection with the tokenserver part (/1.0/sync/1.5) works without problems but every time firefox tries to connect to the syncstorage part (/1.5/{uid}/*) the server returns 401 as status. I suspected that apache as a proxy is somehow causing issues.

After investigating further i could replicate the error in a local setup with the following configs:

firefox-syncstorage.toml:

host="127.0.0.1"
port=8000

database_url = "mysql://sample_user:sample_password@localhost/syncstorage_rs"

master_secret = "mysecret"

human_logs = 1

enable_quota = 0

disable_syncstorage = false
tokenserver.database_url = "mysql://sample_user:sample_password@localhost/tokenserver_rs"
tokenserver.enabled = true
tokenserver.fxa_email_domain = "api.accounts.firefox.com"
tokenserver.fxa_metrics_hash_secret = "mysecret"
tokenserver.fxa_oauth_server_url = "https://oauth.accounts.firefox.com"
tokenserver.test_mode_enabled = false

apache vhost:

<VirtualHost *:80>
        ServerName test.tld
        ServerAlias test.tld

        AllowEncodedSlashes On
        ProxyPreserveHost On

        <LocationMatch "/fxs">
                ProxyPass http://localhost:8000
                ProxyPassReverse http://localhost:8000
        </LocationMatch>

</VirtualHost>

When i directly connect to syncserver_rs with Firefox ("identity.sync.tokenserver.uri=http://locahost:8000/1.0/sync/1.5") and add a node into the tokenserver_rs database with "node=http://localhost:8000" everyting works as expected.
Here is the relevant part of the log of syncstorage_rs (everything after the "/1.5/13/storage/meta/global" request is removed to shorten the log to be readable):

RUST_LOG="trace" /usr/bin/firefox-syncstorage --config=/etc/firefox-syncstorage.toml
Feb 06 23:13:28.330 INFO Starting 32 workers
Feb 06 23:13:28.339 INFO Starting "actix-web-service-127.0.0.1:8000" service on 127.0.0.1:8000
Feb 06 23:13:28.340 INFO Server running on http://127.0.0.1:8000 (mysql) No quota
Feb 06 23:13:39.429 INFO {"first_seen_at":"1644177403218","ua.browser.family":"Firefox","ua.name":"Firefox","ua.os.family":"Linux","ua.browser.ver":"96.0","uri.method":"GET","ua":"96.0","ua.os.ver":"UNKNOWN","uri.path":"/1.0/sync/1.5","uid":"uuuuuuuu","metrics_uid":"xxxxxxx"}
Feb 06 23:13:39.447 INFO {"ua.os.ver":"UNKNOWN","ua.os.family":"Linux","uri.method":"GET","ua.name":"Firefox","ua.browser.family":"Firefox","uri.path":"/1.5/13/info/collections","ua":"96.0.3","ua.browser.ver":"96.0.3"}
Feb 06 23:13:39.453 INFO {"ua.os.ver":"UNKNOWN","ua":"96.0.3","ua.browser.ver":"96.0.3","uri.method":"GET","uri.path":"/1.5/13/info/configuration","ua.os.family":"Linux","ua.name":"Firefox","ua.browser.family":"Firefox"}
Feb 06 23:13:39.463 INFO {"ua.name":"Firefox","uri.method":"GET","ua.browser.ver":"96.0.3","uri.path":"/1.5/13/storage/meta/global","ua.os.ver":"UNKNOWN","ua.os.family":"Linux","ua":"96.0.3","ua.browser.family":"Firefox"}

If i connect with "identity.sync.tokenserver.uri=http://locahost/fxs/1.0/sync/1.5" with a modified node db entry "node=http://localhost/fxs" and over Apache as a proxy, i get the 401 error on the request to "/1.5/13/info/collections". This should mean that Apache is forwarding the request correctly to syncstorage_rs.
Here is the log of syncstorage_rs:

RUST_LOG="trace" /usr/bin/firefox-syncstorage --config=/etc/firefox-syncstorage.toml

Feb 06 23:18:35.409 INFO Starting 32 workers
Feb 06 23:18:35.418 INFO Starting "actix-web-service-127.0.0.1:8000" service on 127.0.0.1:8000
Feb 06 23:18:35.419 INFO Server running on http://127.0.0.1:8000 (mysql) No quota
Feb 06 23:18:46.206 INFO {"ua.os.ver":"UNKNOWN","ua.browser.family":"Firefox","first_seen_at":"1644177403218","ua.name":"Firefox","uri.method":"GET","uri.path":"/1.0/sync/1.5","uid":"uuuuuuuu","metrics_uid":"xxxxxxx","ua.browser.ver":"96.0","ua":"96.0","ua.os.family":"Linux"}
Feb 06 23:18:46.222 INFO {"ua.os.ver":"UNKNOWN","ua.browser.ver":"96.0.3","uri.path":"/1.5/13/info/collections","uri.method":"GET","ua.name":"Firefox","ua.os.family":"Linux","ua.browser.family":"Firefox","ua":"96.0.3"}
Feb 06 23:18:46.987 INFO {"uri.method":"GET","uri.path":"/1.0/sync/1.5","metrics_uid":"xxxxxxx","ua.browser.family":"Firefox","ua.name":"Firefox","ua.os.family":"Linux","uid":"uuuuuuuu","first_seen_at":"1644177403218","ua":"96.0","ua.browser.ver":"96.0","ua.os.ver":"UNKNOWN"}

The logs tell me that Apache strips the "/fxs" part from the path correctly and forwards the requests to syncstorage_rs.

To narrow it down more i removed the path "/fxs" from the apache config and tried to connect with "identity.sync.tokenserver.uri=http://locahost/1.0/sync/1.5" and node db entry of "node=http://localhost" over Apache and it worked.
Here are the syncstorage_rs logs:
RUST_LOG="trace" /usr/bin/firefox-syncstorage --config=/etc/firefox-syncstorage.toml

Feb 06 23:08:11.706 INFO Starting 32 workers
Feb 06 23:08:11.715 INFO Starting "actix-web-service-127.0.0.1:8000" service on 127.0.0.1:8000
Feb 06 23:08:11.715 INFO Server running on http://127.0.0.1:8000 (mysql) No quota
Feb 06 23:08:26.287 INFO {"ua.browser.family":"Firefox","ua.os.family":"Linux","ua":"96.0","first_seen_at":"1644177403218","uid":"uuuuuuuu","ua.name":"Firefox","uri.path":"/1.0/sync/1.5","ua.os.ver":"UNKNOWN","uri.method":"GET","metrics_uid":"xxxxxxx","ua.browser.ver":"96.0"}
Feb 06 23:08:26.308 INFO {"ua.name":"Firefox","ua.browser.ver":"96.0.3","ua.browser.family":"Firefox","ua.os.ver":"UNKNOWN","ua.os.family":"Linux","uri.method":"GET","ua":"96.0.3","uri.path":"/1.5/13/info/collections"}
Feb 06 23:08:26.317 INFO {"ua.browser.family":"Firefox","ua.os.ver":"UNKNOWN","uri.path":"/1.5/13/info/configuration","ua.name":"Firefox","ua":"96.0.3","ua.browser.ver":"96.0.3","uri.method":"GET","ua.os.family":"Linux"}
Feb 06 23:08:26.329 INFO {"ua.browser.family":"Firefox","ua.os.ver":"UNKNOWN","uri.path":"/1.5/13/storage/meta/global","ua.name":"Firefox","ua.os.family":"Linux","uri.method":"GET","ua":"96.0.3","ua.browser.ver":"96.0.3"}

At that point i thought that somehow the hostname and/or url is used in auth so i added a host entry "test.tld" to point to my lan ip (192.168.x.x) instead of localhost (127.0.0.1). I ran the same test this time with "identity.sync.tokenserver.uri=http://test.tld/1.0/sync/1.5" and node db entry of "node=http://test.tld" and it still worked.
Here is the log:

RUST_LOG="trace" /usr/bin/firefox-syncstorage --config=/etc/firefox-syncstorage.toml

Feb 07 00:23:35.153 INFO Starting 32 workers
Feb 07 00:23:35.161 INFO Starting "actix-web-service-127.0.0.1:8000" service on 127.0.0.1:8000
Feb 07 00:23:35.161 INFO Server running on http://127.0.0.1:8000 (mysql) No quota
Feb 07 00:23:44.184 INFO {"ua.os.ver":"UNKNOWN","ua":"96.0","uid":"uuuuuuuu","uri.path":"/1.0/sync/1.5","ua.name":"Firefox","ua.os.family":"Linux","ua.browser.ver":"96.0","metrics_uid":"xxxxxxx","uri.method":"GET","first_seen_at":"1644177403218","ua.browser.family":"Firefox"}
Feb 07 00:23:44.213 INFO {"uri.path":"/1.5/13/info/collections","ua.browser.family":"Firefox","ua.browser.ver":"96.0.3","ua.name":"Firefox","uri.method":"GET","ua.os.family":"Linux","ua":"96.0.3","ua.os.ver":"UNKNOWN"}
Feb 07 00:23:44.219 INFO {"ua.os.family":"Linux","ua.browser.ver":"96.0.3","uri.path":"/1.5/13/info/configuration","uri.method":"GET","ua.os.ver":"UNKNOWN","ua":"96.0.3","ua.browser.family":"Firefox","ua.name":"Firefox"}
Feb 07 00:23:44.231 INFO {"ua.browser.family":"Firefox","ua.os.family":"Linux","uri.method":"GET","ua.name":"Firefox","ua.browser.ver":"96.0.3","uri.path":"/1.5/13/storage/meta/global","ua":"96.0.3","ua.os.ver":"UNKNOWN"}

It seems the hostname and Apache as proxy are not the problem. Only if i add a path like "/fxs" ("identity.sync.tokenserver.uri=http://locahost/fxs" with node db entry "node=http://localhost/fxs") i get the auth error.
I started to capture the traffic with wireshark to check the requests and the auth data. The value of the Authorization header on a request to "/1.0/sync/1.5" is always exactly the same in every constellation (Authorization: Bearer xxx). The Authorization header on a request to "/1.5/13/info/collection" is different in every constellation, but that is expected because the hawk-id includes the node-url (every other field inside the ID is identical except some binary part at the end). I did not see anything else, and I don't know enough about hawk to judge if the auth strings are correct :(

Out of options i tried other combinations:

"identity.sync.tokenserver.uri=http://locahost/fxs/1.0/sync/1.5" with "node=http://localhost:8000" -> works
"identity.sync.tokenserver.uri=http://localhost:8000/1.0/sync/1.5" with "node=http://localhost/fxs" -> fails with 401
"identity.sync.tokenserver.uri=http://localhost:8000/1.0/sync/1.5" with "node=http://localhost" -> works

The tokenserver url inside firefox apparently does not matter. This further shows that Apache as proxy is not a problem, only using a node url like "http://localhost/fxs" is somehow causing the auth to fail. I could not find in the code if and where the node url gets used in the auth.
Maybe this is related to #671 ?

I don't know enough to further narrow down where exactly the error comes from but i need the added "/fxs" path in my setup so i hope someone else is able to find the problem or point me in the right direction where to look or what i'm doing wrong.

[ethowitz]

I think what's happening is that, when your browser sends a request to your Sync node, it generates a MAC using the path that includes the /fxs, but since the proxy removes the /fxs from the path, the Sync node tries to match the MAC it gets from your browser with a MAC it generates from the path that doesn't include the /fxs, resulting in a 401.

The whole flow would look like:

• Your browser sends a request to Tokenserver, and it responds with an API endpoint like "http://localhost/fxa/1.5/your_uid"
• Your browser sends a request to "http://localhost/fxs/1.5/your_uid" with an auth header that includes a MAC that includes the path from that API endpoint as an input (note that this path includes /fxs)
• The request hits your Apache proxy, it strips the /fxs part from the URL, and it forwards the request to http://localhost:8000/1.5/your_uid
• The Sync node generates a MAC using a bunch of info from the request, including the path, which does not include /fxs and tries to match this MAC against the MAC in the Hawk header your browser sent it
• Since the MAC on Tokenserver was generated with the path "/fxa/1.5/your_uid" and the MAC on the Sync node was generated with the path from the request (which was "/1.5/your_uid"), the MACs don't match, and the Sync node returns a 401

This is expected behavior, but you may be able to work around it using a different Apache configuration.

[LuckyTeran]

I already had the same idea and modified my Apache config to include the /fxs part when Firefox is accessing the storage part of the server:

<VirtualHost *:80>
        ServerName test.tld
        ServerAlias test.tld

        AllowEncodedSlashes On
        ProxyPreserveHost On
        RewriteEngine On

        <LocationMatch "/fxs">
                ProxyPass http://localhost:8000
                ProxyPassReverse http://localhost:8000
        </LocationMatch>

         <LocationMatch "/fxs/1.5">
                ProxyPass http://localhost:8000/fxs/1.5
                ProxyPassReverse http://localhost:8000/fxs/1.5
        </LocationMatch>
</VirtualHost>

The problem with that is, that the actix webserver is expecting /1.5/your_uid as and endpoint for storage. So if i forward the /fxs with Apache the request path is now correct but i never reach the endpoint. I don't know of a way to access a path like /1.5/your_uid but with a request path for the server like /fxa/1.5/your_uid.
If i look at

syncstorage-rs/src/server/mod.rs

Line 81 in 04b2437

web::resource(&cfg_path("/info/collections"))

i can see that a function cfg_path is used to assemble the api endpoint path. This function (

syncstorage-rs/src/server/mod.rs

Lines 52 to 60 in 04b2437

	pub fn cfg_path(path: &str) -> String {
	let path = path
	.replace(
	"{collection}",
	&format!("{{collection:{}}}", COLLECTION_ID_REGEX),
	)
	.replace("{bso}", &format!("{{bso:{}}}", BSO_ID_REGEX));
	format!("/{}/{{uid:{}}}{}", SYNC_VERSION_PATH, MYSQL_UID_REGEX, path)
	}

) replaces some variable parts of the path and outputs a string with fixed /1.5/your_uid prefix which is used by actix as the api endpoint. So appending my /fxs will not work :(

I'm still trying to wrap my head around rust syntax and how everything is connected to better help to find a solution.

As far as i understand the source code, https://github.com/mozilla-services/syncstorage-rs/blob/aa93312a1c1e7c1e102ad38a1ff935518e437cb4/src/web/extractors.rs and https://github.com/mozilla-services/syncstorage-rs/blob/04b2437816b2b653f0143fa333d1e61230466cb3/src/web/auth.rs are used to generate and check the hawk secret + mac during a request. The actual path of the node from the database is never used besides sending it to Firefox so it knows how to contact the storage system.
One solution could be to find the assigned node of the user, extract the path and prepend it to every instance where the hawk secret + mac is check and generated.
Another solution could be to add an api endpoint for every node with their added path. (I can already see problems with that ... )
Solution number 3, document that added paths are forbidden for nodes and add a check that throws an error when a path is found in the database ;)

Any other ideas?

[LuckyTeran]

I now tried the server again on my dedicated server(public IP) without the \fxs path with a normal subdomain in the form of https://token.test.tld and with debug enabled (server was compiled in release mode before). The server is still running behind Apache and even with this configuration it does not work and i get the 401 error. Debug output confirms the mac is wrong with DEBG calculated mac doesn't match header.

My Apache config:

<VirtualHost *:80>
        ServerName token.test.tld
        ServerAlias token.test.tld

        AllowEncodedSlashes On
        ProxyPreserveHost On

        ProxyPass / http://localhost:8000
        ProxyPassReverse http://localhost:8000
</VirtualHost>

Syncstorage config is the same. Api endpoint for the node inside the DB is http://token.test.tld, Firefox has http://token.test.tld/1.0/sync/1.5 Running the server without Apache as proxy works. The only reason i use Apache as proxy is to get HTTPS.

Is it possible to add a parameter to the config to give a public url like the old syncserver had? Or is there a possibility to run the server with HTTPS without a proxy?

[jewelooper]

@LuckyTeran I think you have to prevent ssl offloading. so the origin server understand that the request is made over https.
In apache you can achieve this by adding RequestHeader set X-Forwarded-Proto https to your virtual host config.
This directive is provided by apaches headers_module, so don't forget to uncomment LoadModule headers_module modules/mod_headers.so in apache config file.
By the way: I think the directive AllowEncodedSlashes is not necessary.

Another thing i have to do was to allow all origins by adding cors_allowed_origin = "null" in firefox-syncstorage.toml.
I don't know if it is a security issue, but without this parameter I was not able to manage a successfully login.

[ethowitz]

Is it possible to add a parameter to the config to give a public url like the old syncserver had?

This would probably be the path we'd take to support reverse proxies. In the meantime, I've added a ticket to explore supporting HTTPS directly on the server it self to eliminate the need for a reverse proxy.