Installation in docker?

[WAdama]

Hi all,

I tried to install syncstorage-rs via docker, but I can't get my head around it with this documentation.

For one it seems the prerequisite mozilla-rust-sdk is now google-cloud-rust, is that correct?

Maybe someone has already got this to run and can provide me with a more step by step documentation...

Thanks
Ingo

┆Issue is synchronized with this Jira Task

[ictabc]

Same here, managed to compile the syncstorage-rs with a bit of a strugle, but after that just a big black hole. No good documentation, missing a lot of parts in the documentation to get things working. Guess this documentation is build by people that forgot the parts that other people need to get things running. Don't want to use the antiquated syncserver code again. And I'm not allowed to sync my data to the cloud, specially if it's in the US.

Found a docker image in the docker hub, but absolutely no data on how to use it. https://hub.docker.com/r/mozilla/syncstorage-rs

As nobody has documented anything yet on how to use it with docker compile or docker cli, totally lost on what to do. I'm not a noob, but but with current documentation it's impossible to do anything.

Maybe someone from the community can write something, how to use the docker container. Or point me to a page where it is described, at least better than the non information that's in the readme posted in this repository.

[jakobkukla]

I managed to get it working a few days ago. See my docker-compose.yml file down below. Don't forget to change the secrets, mysql credentials/urls and paths.

version: "3.8"

services:
  firefox-sync:
    image: mozilla/syncstorage-rs:0.12.4
    container_name: firefox-sync
    depends_on:
      - firefox-sync-syncstorage-db
      - firefox-sync-tokenserver-db
    environment:
      SYNC_HOST: 0.0.0.0
      SYNC_HUMAN_LOGS: 1
      SYNC_MASTER_SECRET: MY_SECRET
      SYNC_DATABASE_URL: mysql://MY_SYNC_MYSQL_USER:MY_SYNC_MYSQL_USER_PASSWORD@firefox-sync-syncstorage-db:3306/syncstorage
      SYNC_TOKENSERVER__ENABLED: "true"
      SYNC_TOKENSERVER__RUN_MIGRATIONS: "true"
      SYNC_TOKENSERVER__NODE_TYPE: mysql
      SYNC_TOKENSERVER__DATABASE_URL: mysql://MY_TOKEN_MYSQL_USER:MY_TOKEN_MYSQL_USER_PASSWORD@firefox-sync-tokenserver-db:3306/tokenserver
      SYNC_TOKENSERVER__FXA_EMAIL_DOMAIN: api.accounts.firefox.com
      SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL: https://oauth.accounts.firefox.com/v1
      SYNC_TOKENSERVER__FXA_METRICS_HASH_SECRET: MY_OTHER_SECRET
      # I don't really know what this is doing
      SYNC_TOKENSERVER__ADDITIONAL_BLOCKING_THREADS_FOR_FXA_REQUESTS: 2
    ports:
      - 5000:8000
    restart: always
  firefox-sync-syncstorage-db:
    image: mysql:5.7
    container_name: firefox-sync-syncstorage-db
    environment:
      MYSQL_ROOT_PASSWORD: MY_SYNC_MYSQL_ROOT_PASSWORD
      MYSQL_DATABASE: syncstorage
      MYSQL_USER: MY_SYNC_MYSQL_USER
      MYSQL_PASSWORD: MY_SYNC_MYSQL_USER_PASSWORD
    volumes:
      - path/to/appdata/firefox-sync/syncstorage-db:/var/lib/mysql
    ports:
      - 3306
    restart: always
  firefox-sync-tokenserver-db:
    image: mysql:5.7
    container_name: firefox-sync-tokenserver-db
    environment:
      MYSQL_ROOT_PASSWORD: MY_TOKEN_MYSQL_ROOT_PASSWORD
      MYSQL_DATABASE: tokenserver
      MYSQL_USER: MY_TOKEN_MYSQL_USER
      MYSQL_PASSWORD: MY_TOKEN_MYSQL_USER_PASSWORD
    volumes:
      - path/to/appdata/firefox-sync/tokenserver-db:/var/lib/mysql
    ports:
      - 3306
    restart: always

After running docker-compose up to let mysql set itself up, you need to insert the rows below into the tokenserver db as described here. Change the mydomain.tld to your domain. IP address with port also works fine but afaik it must be the same domain that you intend to use for identity.sync.tokenserver.uri in firefox later (Only the domain and protocol as seen below though, not the entire url).

INSERT INTO `services` (`id`, `service`, `pattern`) VALUES ('1', 'sync-1.5', '{node}/1.5/{uid}');
INSERT INTO `nodes` (`id`, `service`, `node`, `available`, `current_load`, `capacity`, `downed`, `backoff`) VALUES ('1', '1', 'https://mydomain.tld', '1', '0', '1', '0', '0');

Since the port configuration in the docker-compose file is not explicit, you will need to look up the tokenserver-db container port with docker-compose ps when connecting to the db to insert the sql.

[WAdama]

Hi jakobkukla,

thanks very much for that!

[jakobkukla]

Keep in mind that I have no idea what I'm doing, so not sure if this configuration is suitable/save to be used in public. Maybe someone from the dev team could chime in :).

Especially the value for SYNC_TOKENSERVER__ADDITIONAL_BLOCKING_THREADS_FOR_FXA_REQUESTS is just some random number. I don't really know what the setting is doing...

[WAdama]

Just good to know it's really working.

I will also try something a little different as I have already a working instance of MariaDB.

[jakobkukla]

@WAdama MariaDB was not working for me for some reason. Had to specifically use mysql 5.7.

But if you can get it to run with MariaDB, I'd like to know how :). It should be working in theory.

[jrconlin]

Thanks @jakobkukla!

I think that's got most of the args folk should need. @ethowitz can say definitively, but I believe this comment describes what SYNC_TOKENSERVER__ADDITIONAL_BLOCKING_THREADS_FOR_FXA_REQUESTS does.

In short, it's a bit of extra thread count buffer that the TokenServer needs to talk to the FxA servers. A low count should be fine for small, stand alone installations.

[jakobkukla]

@jrconlin Thanks for the quick reply!

I have one more question. Would it be possible to create the service and node entry at first startup? Using something like tokenserver.node_domain as a setting. That would greatly improve the setup experience for self hosting. Or is there some technical reason that's currently not possible?

And maybe providing a reasonable default to tokenserver.additional_blocking_threads_for_fxa_requests would be a good idea? I think it's kind of an odd setting to leave uninitialized.

[WAdama]

If I try your compose file I get for the MySQL containers the following error:
mysqld: Can't create directory '/var/lib/mysql/' (Errcode: 17 - File exists)

I use - of course - a folder which exists...

Edit: Found it. Folder has to be set to executable.. (chmod +x ...)

[jrconlin]

@jakobkukla: I think @ethowitz might be able to provide better guidance about creating the service and node entry at first start-up, since he's responsible for that code. He's currently very heads down on the crate re-org which will help a good deal in making the stand-alone side easier to build and maintain, so he may have that as a TODO item.

I suspect that both of these items might be on his task list.

[WAdama]

@jakobkukla I now have the containers up. But I get sync errors in about:sync-log

Did you add something else in the environment of the Sync container?

How is your Token Server in Firefox itself formatted?

[jakobkukla]

@WAdama No, I didn't need anything else. What kind of error are you getting?

My identity.sync.tokenserver.uri setting is set to https://mydomain.tld/1.0/sync/1.5.

[WAdama]

@jakobkukla I should have tried first without my reverse proxy inbetween...

Adressing my docker instance directly worked.

By the way I changed the installation to using only one mysql container. Of course I had to make the changes manually, but at least I need only one database container...

Edit: The problem with the reverse proxy is also solved - more or less. I have tried a subfolder in domain. If I don't use a subfolder but only the domain it works.

[WAdama]

By the way, is there a reason why you used version 0.12.4 and not latest?

[jakobkukla]

Yes, because latest is for some reason an image from 3 years ago.

[WAdama]

Ah ok, that's a reason.. ;-)

Don't want to bother you again, but after running with a test user and a test profile I now tried to change my existing Firefox profile to the new sync server and I got errors again.

Have attached two error logs

error-sync-1666896951311.txt
error-sync-1666897119826.txt

[ictabc]

This helps a lot, but I still have some issues, although I'm close. (Running this on my synology docker until I get it working, then converting it to my Kubernetes cluster (just started with k8s), but want to get it work work first with docker.)

I'm also using an Apache as reverse proxy to handle the SSL part. But connecting directly to docker gives the same error.

My first mistake was to use :latest, because there is an issue where the latest version on docker hub isn't updated to the latest version available. Been open since Juli 15th, not fixed yet. #1362

Second mistake was not feeding the correct variables to docker, for some reason my node_type was set to MySQL URL. Finally figured that out. And everything seems to be working.

Now with the the correct docker images I get the following error. And can't seem to figure out what goes wrong.

docker-0.12.4-sync-error-last-lines.txt

1666953384868 Sync.Resource DEBUG GET fail 401 https://sync.my.domain/1.5/4/info/collections
1666953384868 Sync.Resource WARN GET request to https://sync.my.domain/1.5/4/info/collections failed with status 401
1666953384868 Sync.Service WARN 401: login failed.

For some reason I get an 401 when accessing my collections.

User data is filled in the database when logging in.

Using an newer docker images (Newer then 0.12.4), always gives an database error, doesn't seem to get the database_url settings for tokenstorage.

When I try to sync again I get the following errors in sync-log, still 401's

docker-0.12.4-sync-error.txt

[WAdama]

@jakobkukla It may be a dumb question, but you're using more than one user with your sync server I guess?

When I try a second user, the user doesn't register to my sync server, I tried even a totally new one. The first user register to the server and works, the second one not.

[jdarmetzki]

I have the same issues as @ictabc , but i am still looking into my setup.
In the meantime i overhauled the compose-file so that the credentials are stored in a separate file, there is only one mysql-instance needed, and you don't need to manually insert stuff into the database.

version: "3.8"

services:
  firefox-sync:
    image: mozilla/syncstorage-rs:0.12.4
    container_name: firefox-sync
    depends_on:
      - firefox-sync-db
    environment:
      SYNC_HOST: 0.0.0.0
      SYNC_HUMAN_LOGS: 1
      SYNC_MASTER_SECRET: ${SYNC_MASTER_SECRET}
      SYNC_DATABASE_URL: mysql://${MYSQL_USER}:${MYSQL_PASS}@firefox-sync-db:3306/syncstorage
      SYNC_TOKENSERVER__ENABLED: "true"
      SYNC_TOKENSERVER__RUN_MIGRATIONS: "true"
      SYNC_TOKENSERVER__NODE_TYPE: mysql
      SYNC_TOKENSERVER__DATABASE_URL: mysql://${MYSQL_USER}:${MYSQL_PASS}@firefox-sync-db:3306/tokenserver
      SYNC_TOKENSERVER__FXA_EMAIL_DOMAIN: api.accounts.firefox.com
      SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL: https://oauth.accounts.firefox.com/v1
      SYNC_TOKENSERVER__FXA_METRICS_HASH_SECRET: ${METRICS_HASH_SECRET}
      # I don't really know what this is doing
      SYNC_TOKENSERVER__ADDITIONAL_BLOCKING_THREADS_FOR_FXA_REQUESTS: 2
    ports:
      - 5000:8000
    restart: always
  firefox-sync-db:
    image: mysql:5.7
    container_name: firefox-sync-db
    environment:
      MYSQL_ROOT_PASSWORD: ${MSYQL_SYNC_ROOT_PASS}
      MYSQL_DATABASE: syncstorage
      MYSQL_USER: ${MYSQL_USER}
      MYSQL_PASSWORD: ${MYSQL_PASS}
    entrypoint:
      sh -c "
      echo 'CREATE DATABASE IF NOT EXISTS syncstorage; CREATE DATABASE IF NOT EXISTS tokenserver;' > /docker-entrypoint-initdb.d/init.sql;
      echo 'GRANT ALL PRIVILEGES ON syncstorage.* TO `${MYSQL_USER}`@`%`;' >> /docker-entrypoint-initdb.d/init.sql;
      echo 'GRANT ALL PRIVILEGES ON tokenserver.* TO `${MYSQL_USER}`@`%`;' >> /docker-entrypoint-initdb.d/init.sql;
      /usr/local/bin/docker-entrypoint.sh --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
      "
    volumes:
      - ./syncstorage-db:/var/lib/mysql
    ports:
      - 3306
    restart: always
  db-setup-sidecar:
    image: mysql:5.7
    depends_on:
      - firefox-sync-db
      - firefox-sync
    entrypoint:
      bash -c " 
      IS_DONE=10;      
      while [ $$IS_DONE -gt 0 ]; do
        echo \"USE tokenserver; INSERT IGNORE INTO services (id, service, pattern) VALUES ('1', 'sync-1.5', '{node}/1.5/{uid}'); 
        INSERT INTO nodes (id, service, node, available, current_load, capacity, downed, backoff)  
        VALUES ('1', '1', '${DOMAIN}', '1', '0', '1', '0', '0') ON DUPLICATE KEY UPDATE node='${DOMAIN}';\"|/usr/bin/mysql -h firefox-sync-db --user=${MYSQL_USER} -p${MYSQL_PASS} ;
        RC=$$?;
        echo \"mysql return code was $$RC\";
        if [ $$RC == 0 ] ; then
          IS_DONE=0;
          echo 'Done!';
          exit 0;
        else
          echo 'Waiting for tables...';
          sleep 5;
          ((IS_DONE--));                  
        fi;
      done;
      echo 'Giving up, sorry';
      exit 42;
      "
    restart: "no"

For the config you just need a simple .env-File next to docker-compose.yml

MYSQL_USER=sync
MYSQL_PASS=<your_password>
SYNC_MASTER_SECRET=<your_master_password>
METRICS_HASH_SECRET=<your_hash_secret>
MSYQL_SYNC_ROOT_PASS=<your_mysql_root_password>

DOMAIN=https://<example.your.domain>

EDIT: Works for me now - DOMAIN in .env has to be prefixed with https://

[jakobkukla]

@WAdama No I've only tried with one user. Could the capacity field in the nodes table be the problem?

[WAdama]

@jakobkukla I will have a look and test it.

[WAdama]

@jakobkukla You're my hero...
That was the solution.
To be on the sure side I have set it to 5 and - shazam - the second user could attach and was created in the database, too...

[WAdama]

@jdarmetzki I used your compose file, it worked like a charm, great work. Thanks.

After having a working instance I will try to use the MariaDB on my Synology NAS.

[WAdama]

I got it running with MariaDB and Docker on my Synology NAS. I used jdarmetzki's work as blueprint.

First I connect on command line to the database instance: mysql -u root -p
I then created the user, the dabases and give the user the rights:
`CREATE USER sync_rs@"172.%" IDENTIFIED BY '';

CREATE DATABASE IF NOT EXISTS syncstorage_rs;
CREATE DATABASE IF NOT EXISTS tokenserver_rs;

GRANT ALL PRIVILEGES ON syncstorage_rs.* TO sync_rs@"172.%";
GRANT ALL PRIVILEGES ON tokenserver_rs.* TO sync_rs@"172.%";`

For the container I used this compose file:
`version: "3.8"

services:
firefox-sync:
image: mozilla/syncstorage-rs:0.12.5
network_mode: bridge
container_name: FirefoxSync_RS
environment:
SYNC_HOST: 0.0.0.0
SYNC_HUMAN_LOGS: 1
SYNC_MASTER_SECRET: ${SYNC_MASTER_SECRET}
SYNC_DATABASE_URL: mysql://${MYSQL_USER}:${MYSQL_PASS}@${DATABASE_SERVER}:${DATABASE_PORT}/syncstorage_rs
SYNC_TOKENSERVER__ENABLED: "true"
SYNC_TOKENSERVER__RUN_MIGRATIONS: "true"
SYNC_TOKENSERVER__NODE_TYPE: mysql
SYNC_TOKENSERVER__DATABASE_URL: mysql://${MYSQL_USER}:${MYSQL_PASS}@${DATABASE_SERVER}:${DATABASE_PORT}/tokenserver_rs
SYNC_TOKENSERVER__FXA_EMAIL_DOMAIN: api.accounts.firefox.com
SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL: https://oauth.accounts.firefox.com/v1
SYNC_TOKENSERVER__FXA_METRICS_HASH_SECRET: ${METRICS_HASH_SECRET}
SYNC_TOKENSERVER__ADDITIONAL_BLOCKING_THREADS_FOR_FXA_REQUESTS: 2
ports:
- ${EXTERNAL_PORT}:8000
restart: always
and this .env file:COMPOSE_PROJECT_NAME=ffsync_rs
MYSQL_USER=sync_rs
MYSQL_PASS=
SYNC_MASTER_SECRET=
METRICS_HASH_SECRET=
DATABASE_SERVER=
DATABASE_PORT=
EXTERNAL_PORT=`

After created and started the container attach again to the database and create the nodes and service entries:
USE tokenserver_rs; INSERT IGNORE INTO services (id, service, pattern) VALUES ('1', 'sync-1.5', '{node}/1.5/{uid}'); INSERT INTO nodes (id, service, node, available, current_load, capacity, downed, backoff) VALUES ('1', '1', 'https://<your_server>:<port>', '1', '0', '5', '0', '0');

I've set the capacity entry to 5 in my case.

The sync was running at once. There's only one problem shown in the log. I've got several entries like Nov 03 18:14:51.367 ERRO Lost connection to MySQL server during query. But the logs in "about:sync-logs" are all success logs.
I'm still investigating this.

The container was created with 0.12.4 but as you see I have already updated it with 0.12.5.

[ictabc]

Running basically the same config settings.

Usually I do the following to start cleanly
Stopped, the docker container, dropped every table in current database. Start docker container again, database is refilled. Then add the services and node configuration. And start the sync, tokens part goes ok, collections keep getting an 401.

Keep getting:

1667554182564 Sync.Resource DEBUG GET fail 401 https://sync.my.domain/1.5/4/info/collections
1667554182564 Sync.Resource WARN GET request to https://sync.my.domain/1.5/4/info/collections failed with status 401
1667554182564 Sync.Service WARN 401: login failed.

Very strange that it works for some and not for others.

[WAdama]

Hi @ictabc,

The domain in Firefox and the database are the same? For example
https://sync.mydomain.de/1.0/sync/1.5 in Firefox and
https://sync.mydomain.de in database?

[ictabc]

Hi WAdama,

Yup, those are the same. Do have an Apache reverse proxy config in between. But even without the reverse proxy config, it doesn't work.

Don't get the 401 now, but still an auth error, when going directly to the server. But that is without SSL, so prefer the Apache reverse proxy option.

=================================================
1667562062183 Sync.SyncAuthManager ERROR Non-authentication error in _fetchTokenForUser: TokenServerClientNetworkError({"error":{}})(resource://services-common/tokenserverclient.js:39:36) JS Stack trace: TokenServerClientNetworkError@tokenserverclient.js:62:16
_tokenServerExchangeRequest@tokenserverclient.js:241:13
1667562062183 Sync.Status DEBUG Status.login: success.status_ok => error.login.reason.network
1667562062183 Sync.Status DEBUG Status.service: error.login.failed => error.login.failed
1667562062184 Sync.SyncAuthManager INFO Failed to fetch the cluster URL: TokenServerClientNetworkError({"error":{}})(resource://services-common/tokenserverclient.js:39:36) JS Stack trace: TokenServerClientNetworkError@tokenserverclient.js:62:16
_tokenServerExchangeRequest@tokenserverclient.js:241:13
1667562062184 Sync.Service DEBUG verifyLogin failed: TokenServerClientNetworkError({"error":{}})(resource://services-common/tokenserverclient.js:39:36) JS Stack trace: TokenServerClientNetworkError@tokenserverclient.js:62:16
_tokenServerExchangeRequest@tokenserverclient.js:241:13
1667562062184 Sync.Status DEBUG Status.login: error.login.reason.network => error.login.reason.network
1667562062184 Sync.Status DEBUG Status.service: error.login.failed => error.login.failed
1667562062184 Sync.ErrorHandler ERROR Sync encountered a login error
1667562062184 Sync.SyncScheduler DEBUG Clearing sync triggers and the global score.
1667562062185 Sync.SyncScheduler DEBUG Next sync in 3600000 ms. (why=schedule)
1667562062186 FirefoxAccounts TRACE not checking freshness of profile as it remains recent
1667562062186 Sync.Service DEBUG Exception calling WrappedLock: Error: Login failed: error.login.reason.network(resource://services-sync/service.js:1039:15) JS Stack trace: onNotify@service.js:1039:15
1667562062187 Sync.Service DEBUG Not syncing: login returned false.
1667562062187 FirefoxAccounts TRACE not checking freshness of profile as it remains recent

[ictabc]

Changed both to :

https://sync.mydomain.de:8000/1.0/sync/1.5 in Firefox and
https://sync.mydomain.de:8000/ in database?

As port 5000 is not available on a synology.

[WAdama]

I know, have running it on a Syno myself. Using myself a port in the higher region (xxxxx)..

Did you check if the server is running correct with https://sync.mydomain.de:8000/__heartbeat__?

[ictabc]

{"status":"Ok","tokenserver":{"database":"Ok","status":"Ok"},"quota":{"enabled":false,"size":0},"database":"Ok","version":"0.12.5"}

Looks ok.

The token server works, but the collection part fails.

[jdarmetzki]

Did you also change the ports in SYNC_TOKENSERVER__DATABASE_URL and SYNC_TOKENSERVER__DATABASE_URL ? Otherwise the syncserver tries to connect to the homassistant-db, which will fail

[jakobkukla]

No, home assistant is a seperate installation via home assistant supervisor and also a different user.

Ah sorry. I totally misunderstood your problem haha. If you change the database urls like @jdarmetzki suggested, it should just work.

[Micha-Btz]

hm, the strage thing is, that non of my entrys in the .env file are used. o the user is not created and permissions on the table not being granted and therefore the connection don't work.

The docker compose file and the .env file are in the same dir, don't know why its not working.
Linux debian stable

[ictabc]

Been trying off and on to get things working, but through Apache reverse proxy or direct I always get an stacktrace warning, would be nice if the documentation is updated so I can finally get it to work.

Got the docker container up and running, sync-log sees succesfull login to auth url, but firefox is unable to update to connect and the sync-log doesn't give me any info I can use to troubleshoot why.

Would love to get this working for firefox again. I know it can work but can't get it to work on my end.

And why do we need to fill de database with data, this should be done by syncstorage-rs when first starting.

Hopefully there will be some progress soon to get it running correct always. Or at least better documentation and would like to know what extra steps arei needed for an apache reverseproxy config or nginx reverse proxy.

[djusHa]

Big thx at all!

Got it running now, on Docker, behind a nginx reverse Proxy.

Have modified the docker-compose.yml from jakobkukla slightly:

Added RUST_LOG: debug env. variable to firefox-sync, it's really helpfull for troubleshooting

Added:

depends_on:
      firefox-sync-db:
        condition: service_healthy

to firefox-sync, so it starts right after firefox-sync-db

And at least, added:

healthcheck:
      test: ["CMD", "mysqladmin" ,"ping", "-h", "localhost", "-p${MSYQL_SYNC_ROOT_PASS}"]
      timeout: 20s
      retries: 10

to firefox-sync-db.

[godfuture]

Been trying off and on to get things working

Exact same thing for me. Starting, failing badly, getting confused, leaving it, coming back...see my postings here: #1186

I have created the DB tables, applied permissions, created docker container but then my firefox sync login does not persist. I guess my db init is missing. So I am basically stuck at:
3. Change cargo.toml mozilla-rust-sdk entry to point to "path = "mozilla-rust-sdk/googleapis-raw" instead of the parent dir.

I have no idea what Mozilla wants to tell me to do. Where is "/vendor"? It is nowhere in the image or here in the repository. Cargo or diesel is not installed inside the container...why being so cryptic?

Got it running now, on Docker, behind a nginx reverse Proxy.

As you made it, could you maybe please give instructions on this rust-sdk db migration? And also interesting for me...did someone migrate from old sync selfhosted server? How to upgrade to sync rs?

Many thanks (getting crazy again)

[djusHa]

@godfuture:
No need to compile the image yourself, there is already an image.

Just use provided docker compose config and .env from here: #1428 (comment)

Replace the env var as described here: #1428 (comment)

And it should run...

[godfuture]

No need to compile the image yourself, there is already an image.

I did not compile myself, I am using the docker image from mozilla docker hub.

Just use provided docker compose config and .env from here: #1428 (comment)

My setup is slightly different. I have one mariadb instance for all my apps and databases in it and I am using portainer with stacks to set it up (no compose). I tried to insert into services table, but my tokenserver_rs database does not contain any table at all. Syncstorage_rs does. As said, I think I missed the db init somehow.

And it should run...

Should diesel or cargo (aka google cloud sdk, right?) be available on the mozilla image out of box?

Many thanks

[kimberlyeet]

No need to compile the image yourself, there is already an image.

I did not compile myself, I am using the docker image from mozilla docker hub.

Just use provided docker compose config and .env from here: #1428 (comment)

My setup is slightly different. I have one mariadb instance for all my apps and databases in it and I am using portainer with stacks to set it up (no compose). I tried to insert into services table, but my tokenserver_rs database does not contain any table at all. Syncstorage_rs does. As said, I think I missed the db init somehow.

And it should run...

Should diesel or cargo (aka google cloud sdk, right?) be available on the mozilla image out of box?

Many thanks

make sure to use the version "0.12.4" as in all above examples.
I've just set syncserver up myself and had the same issue until I noticed that I was using the latest tag.

Any ideas here why its not working on latest? (currently 0.13.6)

[WAdama]

You can use 13.6. This is my docker compose and env file:

sync.zip

[fwillo]

Following the thread and using @WAdama 's latest compose file results for me in an error with a freshly set up database:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ApiError { kind: Db(DbError { kind: Pool(Error(None)), status: 500, backtrace:    0: <syncstorage_db_common::error::DbError as core::convert::From<syncstorage_db_common::error::DbErrorKind>>::from
             at syncstorage-db-common/src/error.rs:119:24

Any idea what I might be doing wrong?

[jrconlin]

Looks like the error is the result of a SyncstorageDbErrorKind::BatchNotFound. There are a few possible reasons you could get that error, but they all revolve around the problem that the database can't find the batch that this is referring to.

(The UserAgent can collect up a bunch of changes and submit them as a single "batch" of changes. Since there can be a lot of records, it can break up the "batch" into multiple POST requests and then set a "commit" flag to say "Ok, I'm done. Go ahead and commit all this."

It's very unusual for this to break, but there might be a few reasons:

1. there's a problem connecting up to the database. (e.g. the server asked for more connections than the database can handle during the process, the database restarted mid-operation, etc.) Sometimes comparing the database log times to see if there's some clue in there can help.
2. There's a bug in the MySQL code. We still haven't had a lot of time to work on that, so there's possibly some odd bug about how those bits are being handled.
3. (exceptionally rare) A client might be doing something odd like sending in an old batch request.

You can up the level of logging by setting the "RUST_LOG=info" environment variable.
If you want even more logging info, set the slog feature max_level_trace instead of max_level_info, use "RUST_LOG=trace" and you'll get a firehose of logging information.

[fwillo]

Thanks for the detailed answer. I tried to look at the behavior with different RUST_LOG settings. However, none of them changed the output. I made sure that the containers were set up cleanly after changing the parameters.

Looking at the log output of mariadb, however, revealed some sort of network error. The following messages were printed while the container is in "health: starting":

mariadbd[813102]: 2023-08-24 16:50:44 3465 [Warning] IP address '172.19.0.2' could not be resolved: Temporary failure in name resolution

This most likely seems to be a firewall/networking issue, which is weird because I have other containers working fine.

[ggrzeczkowicz]

It works perfectly for me with MardiaDB 10.11 LTS and syncstorage-rs 13.6, on windows and android.

From @jdarmetzki and @WAdama :

docker-compose.yml :

version: '3.8'

services:
    syncstorage:
        container_name: firefox_syncstorage
        image: mozilla/syncstorage-rs:0.13.6
        environment:
            SYNC_HOST: 0.0.0.0
            SYNC_HUMAN_LOGS: 1
            SYNC_MASTER_SECRET: ${SYNC_MASTER_SECRET}
            SYNC_SYNCSTORAGE__DATABASE_URL: mysql://${MARIADB_USER}:${MARIADB_PASSWORD}@syncstorage_db:3306/syncstorage
            SYNC_TOKENSERVER__ENABLED: "true"
            SYNC_TOKENSERVER__RUN_MIGRATIONS: "true"
            SYNC_TOKENSERVER__NODE_TYPE: mysql
            SYNC_TOKENSERVER__DATABASE_URL: mysql://${MARIADB_USER}:${MARIADB_PASSWORD}@tokenserver_db:3306/tokenserver
            SYNC_TOKENSERVER__FXA_EMAIL_DOMAIN: api.accounts.firefox.com
            SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL: https://oauth.accounts.firefox.com/v1
            SYNC_TOKENSERVER__FXA_METRICS_HASH_SECRET: ${METRICS_HASH_SECRET}
            SYNC_TOKENSERVER__ADDITIONAL_BLOCKING_THREADS_FOR_FXA_REQUESTS: 2
            RUST_LOG: info
        healthcheck:
            test: ["CMD", "curl", "-f", "http://localhost:8000/__heartbeat__"]
            interval: 30s
            timeout: 10s
            retries: 5
        restart: unless-stopped
        depends_on:
            - syncstorage_db
            - tokenserver_db

    syncstorage_db:
        image: mariadb:10.11
        container_name: firefox_syncstorage_db
        environment:
            MARIADB_RANDOM_ROOT_PASSWORD: true
            MARIADB_DATABASE: syncstorage
            MARIADB_USER: ${MARIADB_USER}
            MARIADB_PASSWORD: ${MARIADB_PASSWORD}
            MARIADB_AUTO_UPGRADE: true
        volumes:
            - syncstorage-db:/var/lib/mysql
        restart: unless-stopped

    tokenserver_db:
        image: mariadb:10.11
        container_name: firefox_tokenserver_db
        environment:
            MARIADB_RANDOM_ROOT_PASSWORD: true
            MARIADB_DATABASE: tokenserver
            MARIADB_USER: ${MARIADB_USER}
            MARIADB_PASSWORD: ${MARIADB_PASSWORD}
            MARIADB_AUTO_UPGRADE: true
        volumes:
            - tokenserver-db:/var/lib/mysql
        restart: unless-stopped

    tokenserver_db_init:
        container_name: firefox_tokenserver_db_init
        image: mariadb:10.11
        depends_on:
            - tokenserver_db
            - syncstorage
        restart: no
        entrypoint:
            bash -c "
            IS_DONE=10;
            while [ $$IS_DONE -gt 0 ]; do
                echo \"INSERT IGNORE INTO services (id, service, pattern) VALUES ('1', 'sync-1.5', '{node}/1.5/{uid}');
                INSERT INTO nodes (id, service, node, available, current_load, capacity, downed, backoff)
                VALUES ('1', '1', '${DOMAIN}', '1', '0', '5', '0', '0') ON DUPLICATE KEY UPDATE node='${DOMAIN}';\"|mysql --host=firefox_tokenserver_db --user=${MARIADB_USER} --password=${MARIADB_PASSWORD} tokenserver;
                RC=$$?;
                echo \"mysql return code was $$RC\";
                if [ $$RC == 0 ] ; then
                IS_DONE=0;
                echo 'Done!';
                exit 0;
                else
                echo 'Waiting for tables...';
                sleep 5;
                ((IS_DONE--));
                fi;
            done;
            echo 'Giving up, sorry';
            exit 42;
            "

volumes:
    syncstorage-db:
    tokenserver-db:

.env :

MARIADB_USER=syncstorage
MARIADB_PASSWORD=syncstorage
SYNC_MASTER_SECRET=<your_master_password>
METRICS_HASH_SECRET=<your_hash_secret>
DOMAIN=https://<example.your.domain>

identity.sync.tokenserver.uri :
https://<example.your.domain>/1.0/sync/1.5

[fwillo]

Hey, small feedback: I could solve my specific issue by the skip-name-resolve flag in MariaDB. I don't know why it is now an issue, however this solves my problem.

[ashald]

What is this FXA thing? Does it mean we still depend on Firefox'es server for something? 🤔

[WAdama]

The new image (0.14.0) seems to crash the container:
https://github.com/mozilla-services/syncstorage-rs/issues/1482

[privacyguy123]

Attaching to firefox_syncstorage, firefox_syncstorage_db, firefox_tokenserver_db, firefox_tokenserver_db_init
firefox_syncstorage_db       | 2023-09-27 18:13:26+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.5+maria~ubu2204 started.
firefox_tokenserver_db       | 2023-09-27 18:13:26+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.5+maria~ubu2204 started.
firefox_syncstorage          | exec /app/bin/syncserver: exec format error
firefox_tokenserver_db       | 2023-09-27 18:13:27+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
firefox_syncstorage_db       | 2023-09-27 18:13:27+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
firefox_tokenserver_db       | 2023-09-27 18:13:28+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.5+maria~ubu2204 started.
firefox_syncstorage_db       | 2023-09-27 18:13:28+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.5+maria~ubu2204 started.
firefox_tokenserver_db_init  | ERROR 2002 (HY000): Can't connect to server on 'firefox_tokenserver_db' (115)
firefox_tokenserver_db_init  | mysql return code was 1
firefox_tokenserver_db_init  | Waiting for tables...
firefox_tokenserver_db       | 2023-09-27 18:13:28+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
firefox_syncstorage_db       | 2023-09-27 18:13:28+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] Starting MariaDB 10.11.5-MariaDB-1:10.11.5+maria~ubu2204 source revision 7875294b6b74b53dd3aaa723e6cc103d2bb47b2c as process 1
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] Starting MariaDB 10.11.5-MariaDB-1:10.11.5+maria~ubu2204 source revision 7875294b6b74b53dd3aaa723e6cc103d2bb47b2c as process 1
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Number of transaction pools: 1
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Using ARMv8 crc32 instructions
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Using liburing
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Completed initialization of buffer pool
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Number of transaction pools: 1
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Using ARMv8 crc32 instructions
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Using liburing
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Completed initialization of buffer pool
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
firefox_syncstorage exited with code 1
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: End of log at LSN=46862
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: End of log at LSN=46862
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: 128 rollback segments are active.
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: log sequence number 46862; transaction id 16
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] Plugin 'FEEDBACK' is disabled.
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Buffer pool(s) load completed at 230927 18:13:29
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] Server socket created on IP: '0.0.0.0'.
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] Server socket created on IP: '::'.
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: 128 rollback segments are active.
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: log sequence number 46862; transaction id 16
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] Plugin 'FEEDBACK' is disabled.
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] InnoDB: Buffer pool(s) load completed at 230927 18:13:29
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] Server socket created on IP: '0.0.0.0'.
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] Server socket created on IP: '::'.
firefox_tokenserver_db       | 2023-09-27 18:13:29 0 [Note] mariadbd: ready for connections.
firefox_tokenserver_db       | Version: '10.11.5-MariaDB-1:10.11.5+maria~ubu2204'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
firefox_syncstorage_db       | 2023-09-27 18:13:29 0 [Note] mariadbd: ready for connections.
firefox_syncstorage_db       | Version: '10.11.5-MariaDB-1:10.11.5+maria~ubu2204'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
firefox_syncstorage exited with code 1
firefox_syncstorage exited with code 1
firefox_syncstorage exited with code 1
firefox_syncstorage exited with code 1
firefox_tokenserver_db_init  | ERROR 1146 (42S02) at line 1: Table 'tokenserver.services' doesn't exist
firefox_tokenserver_db_init  | mysql return code was 1
firefox_tokenserver_db_init  | Waiting for tables...
firefox_syncstorage exited with code 1
firefox_tokenserver_db_init  | ERROR 1146 (42S02) at line 1: Table 'tokenserver.services' doesn't exist
firefox_tokenserver_db_init  | mysql return code was 1
firefox_tokenserver_db_init  | Waiting for tables...
firefox_syncstorage exited with code 1

No idea whats wrong?

EDIT: Nevermind these Docker images aren't built for arm64

[jeena]

I was also successful with deploying the docker-compose from #1428 (comment) thanks for sharing!

Therefor I took it and created a git repo with it and additional files like for systemd and nginx as documentation for future me here: https://github.com/jeena/fxsync-docker but it is getting quite many stars so I guess other people also appreciate the more comprehensive documentation around how to self host it. I think it would be good if we could consolidate it to one mariadb instance because it uses quite many unnecessary resources.

And in long run I would like to see if we could make a PR which would add a docker-compose.yaml to this repo.

[ictabc]

Finally got it working. For some reason, I kept getting login errors.

1696599763276 Sync.Status DEBUG Status.login: success.status_ok => success.login
1696599763276 Sync.Status DEBUG Status.service: error.login.failed => success.status_ok
1696599763276 Sync.SyncAuthManager DEBUG _findCluster returning https://sync.example.com:443/1.5/4/
1696599763276 Sync.SyncAuthManager DEBUG Cluster value = https://sync.example.com:443/1.5/4/
1696599763276 Sync.SyncAuthManager DEBUG Setting cluster to https://sync.example.com:443/1.5/4/
1696599763276 Sync.Service DEBUG Caching URLs under storage user base: https://sync.example.com:443/1.5/4/
1696599763276 FirefoxAccounts TRACE not checking freshness of profile as it remains recent
1696599763276 FirefoxAccounts TRACE not checking freshness of profile as it remains recent
1696599763278 Sync.SyncAuthManager DEBUG unlockAndVerifyAuthState already has (or can fetch) sync keys
1696599763278 Sync.Status DEBUG Status.login: success.login => success.status_ok
1696599763278 Sync.Status DEBUG Status.service: success.status_ok => error.login.failed
1696599763278 Sync.Service DEBUG Fetching unlocked auth state returned success.status_ok
1696599763278 FirefoxAccounts TRACE not checking freshness of profile as it remains recent
1696599763282 Sync.Resource DEBUG GET fail 401 https://sync.example.com/1.5/4/info/collections
1696599763282 Sync.Resource WARN GET request to https://sync.example.com/1.5/4/info/collections failed with status 401
1696599763282 Sync.Service WARN 401: login failed.
1696599763282 Sync.Status DEBUG Status.login: success.status_ok => error.login.reason.network
1696599763283 Sync.Status DEBUG Status.service: error.login.failed => error.login.failed
1696599763283 Sync.ErrorHandler ERROR Sync encountered a login error
1696599763283 Sync.SyncScheduler DEBUG Clearing sync triggers and the global score.
1696599763283 Sync.SyncScheduler DEBUG Next sync in 3600000 ms. (why=schedule)
1696599763284 Sync.Service DEBUG Exception calling WrappedLock: Error: Login failed: error.login.reason.network(resource://services-sync/service.sys.mjs:1038:15) JS Stack trace: onNotify@service.sys.mjs:1038:15
1696599763285 Sync.Service DEBUG Not syncing: login returned false.
1696599763285 FirefoxAccounts TRACE not checking freshness of profile as it remains recent

Only just noticed, that is accessing sync.example.com/1.5/* instead of sync.example.com/1.0/sync/1.5/*

So I added a rewrite rule for 1.5 to 1.0/sync/1.5 in my apache proxy, and I am now able to sync.

        ProxyPreserveHost On
        ProxyPass / http://sync.example.com:8000/
        ProxyPassReverse / http://sync.example.com:8000/
        ProxyPass /1.5/ http://sync.example.com:8000/1.0/sync/1.5/
        ProxyPassReverse /1.5/ http://sync.example.com:8000/1.0/sync/1.5/
        <Proxy *>
           AllowOverride all
           Require all granted
        </Proxy>
        Header set X-Timestamp %t
        Header edit X-Timestamp t= ""

        RequestHeader set X-Forwarded-Proto https
        RequestHeader set X-Forwarded-Ssl on
        RequestHeader set X-Real-IP $remote_addr
        RemoteIPHeader X-Forwarded-For

Now to find out why this is, as far as I can see I have always done everything the same as other people.

[eddieirvine]

Hi there,

I've managed to setup my Firefox Sync server and use it with my Firefox Desktop PCs.
However I can't make it work on my iPhone (newest iOS and Firefox version).

With the old sync server it always worked.
I changed the identity.sync.tokenserver.uri to https://ffsync.example.com/1.0/sync/1.5.
That's correct, right?
Before it was like https://ffsync.example.com/token/1.0/sync/1.5, correct?

My Firefox on iOS shows me 'Sync is offline' and offers me a link to solve the issue via disabeling the bookmark sync to make it work.
I tried that, but it didn't help.

Did anybody here manage to make it run with Firefox on iOS?

[WAdama]

Hi @eddieirvine

That's correct, the "token" has to be gone. I can only speak for Android: Have you logged of on your Firefox before changing the configuration? This is required on the mobile version.

The best way I found is to reset the configuration, changing the sync server and after that log in to my Firefox account.

[eddieirvine]

I had a look at the logs and saw this error:
2023-11-09 07:17:46.274 WARNING [sync] Profile - [RUST][sync15::client::sync_multiple] sync failed: Network error: [no-sentry] Validation error: URL does not use TLS protocol., final status=NetworkError
2023-11-09 07:17:46.285 INFO [sync] Profile - [RUST][sync_manager::manager] Sync finished with status NetworkError

URL is https://ffsync.example.com/1.0/sync/1.5
Internal URL linked via reverse proxy is http://192.168.2.211:8000/

Could it be an issue that one is https and the other http?

[WAdama]

Hmm, I use the same construct over a nginx reverse proxy and it works without any problems.

[ictabc]

@eddieirvine

Did you add the following to your proxy config Apache:

    RequestHeader set X-Forwarded-Proto https
    RequestHeader set X-Forwarded-Ssl on

Or if you use NGINX:

   proxy_set_header X-Forwarded-Proto https;
   proxy_set_header X-Forwarded-Ssl on;

To tell to syncstorage-rs that it is encrypted, only not on the syncstorage-rs docker container.

[eddieirvine]

@ictabc Thanks for your answer.
In the meantime I made it work somehow 😉

I use the reverse proxy of Synology.

[JustSomeHooman]

I am grateful for this project but setting this server up even when using docker is beyond painful. I have tried to do so on synology docker without success, even when using docker-compose code mentioned here and https://github.com/jeena/fxsync-docker repo. It just does not work out of box and documentation is still terrible.

[godfuture]

I am grateful for this project but setting this server up even when using docker is beyond painful. I have tried to do so on synology docker without success, even when using docker-compose code mentioned here and https://github.com/jeena/fxsync-docker repo. It just does not work out of box and documentation is still terrible.

For me also the worst docker project till now...