Failling to sync from desktop to a server

[maniackcrudelis]

Trilium v0.46.6

Hi,

I'm trying to sync my desktop instance to a server instance but the synchronization doesn't work and stay stuck on "Outstanding sync items: 0"

On the desktop side, the log says:

Listening on port 37840
Triggering sync.
Slow 200 GET /setup took 13ms
Nothing to push
sync failed: Request to GET https://domain.tld/trilium//api/sync/changed?lastEntityChangeId=0 failed, error: 401 
stack: Error: Request to GET https://domain.tld/trilium//api/sync/changed?lastEntityChangeId=0 failed, error: 401 
    at generateError (/media/data/Instal/trilium-linux-x64/resources/app.asar/src/services/request.js:191:12)
    at ClientRequest.<anonymous> (/media/data/Instal/trilium-linux-x64/resources/app.asar/src/services/request.js:62:28)
    at ClientRequest.emit (events.js:223:5)
    at SimpleURLLoaderWrapper.<anonymous> (electron/js2c/browser_init.js:93:6152)
    at SimpleURLLoaderWrapper.emit (events.js:223:5)
ERROR: Failed to deserialize sync response: Not authorized
Returning sync stats: {"initialized":false,"outstandingPullCount":0}

On the server side is says only

304 GET /api/setup/status took 2ms
200 POST /api/login/sync took 5ms
GET /api/sync/changed rejected with 401 Not authorized

I use a nginx reverse proxy with a domain name

location /trilium/ {

    if ($scheme = http) {
        rewrite ^ https://$server_name$request_uri? permanent;
    }

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_pass http://127.0.0.1:8080/
    proxy_read_timeout 90;
    proxy_redirect http://127.0.0.1:8080/ https://domain.tld/trilium/;

    proxy_set_header                    Upgrade $http_upgrade;
    proxy_set_header                    Connection 'upgrade';

    client_max_body_size 50M;
}

I tried other settings for the proxy, without any differences...

proxy_redirect http://127.0.0.1:8080/ https://domain.tld;       # Without the path at the end
proxy_cache_bypass                  $http_upgrade;
proxy_set_header                    Host $http_host;
proxy_set_header                    X-Forwarded-Host $server_name;
proxy_set_header                    X-Forwarded-Port $server_port;

On the server side, the notes are just the demo, for the sake of the debugging. On the desktop side, that's a clean install without anything that I try to synchronize at first start.

Because of the configuration of my server, I can't use a simple domain.tld:port config or even a IP:port.

Please note that even if the synchronization doesn't work, the nginx configuration works perfectly for an access from a browser.

Also, I've added

    proxy_set_header                    Upgrade $http_upgrade;
    proxy_set_header                    Connection 'upgrade';

to solve the error Lost websocket connection to the backend. If you keep having this issue repeatedly, you might want to check your reverse proxy (nginx, apache) configuration and allow/unblock WebSocket. into the browser console.

[sigaloid]

Have you tried opening the port and connecting directly? this will narrow it down to issue in reverse proxy or issue in trilium.

[maniackcrudelis]

While I'm used to never use a domain:port, which can't work in my configuration, I forgot that a simple IP:port can be used...

So, it does work with http://IP:port, but not with http://domain.tld/trilium or https://domain.tld/trilium
So clearly, it looks like an issue around my nginx configuration, unfortunately I tried many different configurations without any success. Has to be noted though that I'm not so good with nginx configuration...

[maniackcrudelis]

I have exactly the same error with a sub domain installation with the following nginx config

location / {

    # Force usage of https
    if ($scheme = http) {
        rewrite ^ https://$server_name$request_uri? permanent;
    }

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_pass http://127.0.0.1:8081/;
    proxy_read_timeout 90;
    proxy_redirect http://127.0.0.1:8081 https://sub.domain.tld;

    proxy_set_header                    Upgrade $http_upgrade;
    proxy_set_header                    Connection 'upgrade';

    client_max_body_size 50M;
}

[sigaloid]

Look into the given Nginx proxy: https://github.com/zadam/trilium/wiki/Nginx-proxy-setup

[maniackcrudelis]

That's what I've used to build the nginx config, but it doesn't work for the synchronization in my situation.

I've tried many nginx instructions, in different combinations without any success...

[maniackcrudelis]

Back with a few more tests, still no solution but maybe some infos that may help...

So, I did try the app on android and it works flawlessly with the domain, on Wifi or 4g.
I know the app does not really sync, but yet it worked and the reverse proxy on nginx wasn't a problem.

Then I tried from another computer, outside of my local network but I have the same 401 issue... I don't know what to look at anymore.

For the purpose of the debug, I reduce my nginx configuration to the minimum

location / {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_pass http://127.0.0.1:8081;
    proxy_read_timeout 90;
    proxy_redirect http://127.0.0.1:8081 https://domain.tld;
}

[zadam]

hi, this looks weird.

sync failed: Request to GET https://domain.tld/trilium//api/sync/changed?lastEntityChangeId=0 failed, error: 401 

This failure is already on 2. step of the sync process - meaning that the initial handshake (login) with the server succeeded. Sync process is stateful - login happens in each sync cycle - login establishes a sync session on the server which is then used when pulling/pushing changes. So it looks like it's something with session storage since the login succeeded, but then the next step fails to locate the session. The only similar thing I remember was when there wasn't any free space left on the server so the sessions could not be persisted for later retrieval.

One thing which could cause such problem would be if nginx somehow did not forward the session cookie to the Trilium instance. But I don't know what misconfiguration could be causing that ...

[maniackcrudelis]

Hi @zadam

The only similar thing I remember was when there wasn't any free space left on the server so the sessions could not be persisted for later retrieval.

327G free on the hard drive that have the data for Trilium, 6.4G free on the hard drive where the core of Trilium is.
I guess that the folder sessions which is owned by the same user that run trilium, and contains a lot of json files.

On the client side, the folder is always empty.

I've read a few issues before posting this one, and noticed you said already that it was the second step. Which means that the communication works between the 2 sides, at first at least.

I search on internet for cookie issues with nginx already, and tried a few instructions for the reverse proxy about it (not really knowing what I was doing...), nothing worked.
But if nginx wasn't forwarding the session cookie, wouldn't it be a problem as well for a connection from a browser ?

[zadam]

But if nginx wasn't forwarding the session cookie, wouldn't it be a problem as well for a connection from a browser ?

Yes, that's true. There shouldn't be a difference in session handling between browsing and sync.

It would be interesting to see what data is getting through to the application. You could modify the src/services/auth.js to log more information on stdout:

function checkApiAuth(req, res, next) {
    if (!req.session.loggedIn && !noAuthentication) {
        console.log(req.headers, req.session);

        reject(req, res, "Not authorized");
    }
    else {
        next();
    }
}

After changing the file you need to restart the trilium application.

Here's what I get:

{
 connection: 'upgrade, close',
 host: '[domain]',
 'x-forwarded-for': '[ip address]',
 cookie: 'trilium.sid=[session]; Path=/; Expires=Tue, 06 Apr 2021 19:31:08 GMT; HttpOnly',
 'content-type': 'application/json',
 pagecount: '1',
 pageindex: '0',
 requestid: '4LaYbWvxuG',
 via: 'threatpulse',
 'cache-control': 'max-stale=0',
 pragma: 'no-cache',
} Session {
 cookie: {
   path: '/',
   _expires: 2021-04-06T19:31:10.263Z,
   originalMaxAge: 86400000,
   httpOnly: true
 },
 loggedIn: true,
 __lastAccess: 1617651070264
}

[maniackcrudelis]

Ok, but I'm using the compiled version with chromium on the client side.
Is it ok if I do that only the server side ? (Maybe it was exactly the idea...)

Where should I see that then ? On the backend log ?

[zadam]

Yes, this is intended for server side. Assuming you're using the .tar.gz version and not docker one (there the editing would be complicated).

The debug info will be on standard output, not backend log. So just start ./trilium.sh and watch what's being printed ...

[maniackcrudelis]

We're ok that the modification was to add console.log(req.headers, req.session); at the line 38 ?

I did it, restarted trillium, but I have nothing in both logs, on the client side I got

Slow 200 GET /setup took 18ms
Getting document options FROM sync server.
200 POST /api/setup/sync-from-server took 2ms
Creating database for sync
Schema and not synced options generated.
Triggering sync.
Nothing to push
sync failed: Request to GET https://domain.tld//api/sync/changed?lastEntityChangeId=0 failed, error: 401 
stack: Error: Request to GET https://domain.tld//api/sync/changed?lastEntityChangeId=0 failed, error: 401 
    at generateError (/media/data/Trilium_debug/trilium-linux-x64-debug/resources/app.asar/src/services/request.js:191:12)
    at ClientRequest.<anonymous> (/media/data/Trilium_debug/trilium-linux-x64-debug/resources/app.asar/src/services/request.js:62:28)
    at ClientRequest.emit (events.js:223:5)
    at SimpleURLLoaderWrapper.<anonymous> (electron/js2c/browser_init.js:93:6152)
    at SimpleURLLoaderWrapper.emit (events.js:223:5)
ERROR: Failed to deserialize sync response: Not authorized
Returning sync stats: {"initialized":false,"outstandingPullCount":0}

Which does not seems to be different...

EDIT: I should have missed something... Even using IP:port, I don't have the cookie in my log...

[zadam]

Yes, line 38 was what I intended. If it does not log there, then it probably doesn't fail there which is suspicious.

You can move the logging line to line 37 - in front of the if. That should be logging in all cases ...

[maniackcrudelis]

Ok moved it before the if, just at the beginning of the function.

Still nothing... There's nothing to build nor any cache from nodejs ?

(Kind of off topic, but my systemd should not be correct, the stop kills trillium after a timeout of 90s...)

---

I'm setting up a virtual machine, so I can install anything and torture the server as wished to find the bug. It will be easier. Forgot it doesn't work with self signed certificates...

[zadam]

Still nothing... There's nothing to build nor any cache from nodejs ?

No, just node restart is needed.

So yeah, this is weird. The request must be rejected on some other place.

Forgot it doesn't work with self signed certificates...

self signed certs can be used by disabling the authority check via:

export NODE_TLS_REJECT_UNAUTHORIZED=0

(or windows equivalent)

[maniackcrudelis]

While the trick worked for the certificate, I do not have more logs on the VM. But the same error appears.

At least, we can do whatever we want with that VM, it does never complain...

---

I do not have the cookie in the log either when connecting with IP:PORT, which works.

We're ok, I'm looking at stdout from trilium binary on the client side ?

[zadam]

We're ok, I'm looking at stdout from trilium binary on the client side ?

No, we're interested in the server side - what data the server instance receives in the request.

[maniackcrudelis]

No, we're interested in the server side - what data the server instance receives in the request.

I thought so, it was strange to me to look at the client side. But I don't know anything about JS.

Although, I constantly tail the log on the server side and nothing appeared except

Slow 200 GET /api/setup/status took 10ms
200 POST /api/login/sync took 3ms
GET /api/sync/changed rejected with 401 Not authorized

But the log I tail is the log in trilium-data/log/trilium-2021-04-06.log
It usually shows error from the console, I used it to debug scripts I was trying to play with in the soft.

What's for sure is that it didn't show anything about the cookie so far. And I still have console.log(req.headers, req.session); on the line 37 of trilium/src/services/auth.js

I've notice though that you told me to simply run ./trilium.sh, which led me to think it was the client since I don't have such script on the server side.
And indeed, this script comes with the packaged version, and I did install manually. I don't think that would change anything, but just in case, better to be said.
My systemd runs bin/node src/www, doesn't sounds different.

[maniackcrudelis]

Got the log !
Just hit me that journalctl would give me anything from stdout on the server side.
Strange though that it doesn't go to the log...

200 GET /api/setup/status took 3ms
200 POST /api/login/sync took 1ms
{ host: 'sous.domain.tld',
  'x-real-ip': '192.168.1.49',
  'x-forwarded-for': '192.168.1.49',
  'x-forwarded-proto': 'http',
  connection: 'upgrade',
  'content-type': 'application/json',
  cookie:
   'SSOwAuthRedirect=; Path=/yunohost/sso/; Expires=Thu, 01 Jan 1970 00:00:00 UTC; Secure; HttpOnly; SameSite=Lax,trilium.sid=s%3A42Aqoym3LDwOCVKBoNW8vFomMxnH4Lgt.ygsr6KMnogjkcUYzJ0rsEk%2BIJXGBmdE47tA%2FFotFnHY; Path=/; Expires=Wed, 07 Apr 2021 20:03:17 GMT; HttpOnly',
  pagecount: '1',
  pageindex: '0',
  requestid: 'mcfLjOCKTo',
  'user-agent':
   'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) TriliumNotes/0.46.6 Chrome/83.0.4103.122 Electron/9.4.4 Safari/537.36',
  'accept-encoding': 'gzip, deflate',
  'accept-language': 'en-US' } Session {
  cookie:
   { path: '/',
     _expires: 2021-04-07T20:03:17.678Z,
     originalMaxAge: 86400000,
     httpOnly: true } }
GET /api/sync/changed rejected with 401 Not authorized

[maniackcrudelis]

The first cookie, SSOwAuthRedirect is on my side, if that would be a problem I can try to fix it.
The second one looks like yours, but I don't know how to interpret that result.

[maniackcrudelis]

With a dirty hack in my nginx conf, I got rid of the said part in my previous cookie and got that log

200 GET /api/setup/status took 1ms
Slow 200 POST /api/login/sync took 17ms
{ host: 'domain.tld',
  'x-real-ip': '192.168.1.49',
  'x-forwarded-for': '192.168.1.49',
  'x-forwarded-proto': 'http',
  connection: 'close',
  'content-type': 'application/json',
  cookie:
   'trilium.sid=s%3AN0Mf-AlHph5dYk5HlfoggFvqUhDarkHQ.nb37T5xFyIG260hki00lxpUvPNjkQNV76D%2FqOkEbwD8; Path=/; Expires=Wed, 07 Apr 2021 20:36:44 GMT; HttpOnly',
  pagecount: '1',
  pageindex: '0',
  requestid: 'saoIAnrIXo',
  'user-agent':
   'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) TriliumNotes/0.46.6 Chrome/83.0.4103.122 Electron/9.4.4 Safari/537.36',
  'accept-encoding': 'gzip, deflate',
  'accept-language': 'en-US' } Session {
  cookie:
   { path: '/',
     _expires: 2021-04-07T20:36:44.257Z,
     originalMaxAge: 86400000,
     httpOnly: true },
  loggedIn: true,
  __lastAccess: 1617741404258 }
Returning 248 entity changes in 30ms
Slow 200 GET /api/sync/changed?lastEntityChangeId=0 took 41ms
{ host: 'domain.tld',
  'x-real-ip': '192.168.1.49',
  'x-forwarded-for': '192.168.1.49',
  'x-forwarded-proto': 'http',
  connection: 'close',
  'content-type': 'application/json',
  cookie:
   'trilium.sid=s%3AN0Mf-AlHph5dYk5HlfoggFvqUhDarkHQ.nb37T5xFyIG260hki00lxpUvPNjkQNV76D%2FqOkEbwD8; Path=/; Expires=Wed, 07 Apr 2021 20:36:44 GMT; HttpOnly',
  pagecount: '1',
  pageindex: '0',
  requestid: 'JbBTgaPaVl',
  'user-agent':
   'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) TriliumNotes/0.46.6 Chrome/83.0.4103.122 Electron/9.4.4 Safari/537.36',
  'accept-encoding': 'gzip, deflate',
  'accept-language': 'en-US' } Session {
  cookie:
   { path: '/',
     _expires: 2021-04-07T20:36:44.359Z,
     originalMaxAge: 86400000,
     httpOnly: true },
  loggedIn: true,
  __lastAccess: 1617741404362 }
Returning 465 entity changes in 14ms
Slow 200 GET /api/sync/changed?lastEntityChangeId=254 took 25ms
{ host: 'domain.tld',
  'x-real-ip': '192.168.1.49',
  'x-forwarded-for': '192.168.1.49',
  'x-forwarded-proto': 'http',
  connection: 'close',
  'content-type': 'application/json',
  cookie:
   'trilium.sid=s%3AN0Mf-AlHph5dYk5HlfoggFvqUhDarkHQ.nb37T5xFyIG260hki00lxpUvPNjkQNV76D%2FqOkEbwD8; Path=/; Expires=Wed, 07 Apr 2021 20:36:44 GMT; HttpOnly',
  pagecount: '1',
  pageindex: '0',
  requestid: 'hdZNYlszYL',
  'user-agent':
   'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) TriliumNotes/0.46.6 Chrome/83.0.4103.122 Electron/9.4.4 Safari/537.36',
  'accept-encoding': 'gzip, deflate',
  'accept-language': 'en-US' } Session {
  cookie:
   { path: '/',
     _expires: 2021-04-07T20:36:44.543Z,
     originalMaxAge: 86400000,
     httpOnly: true },
  loggedIn: true,
  __lastAccess: 1617741404546 }
200 GET /api/sync/changed?lastEntityChangeId=758 took 1ms
{ host: 'domain.tld',
  'x-real-ip': '192.168.1.49',
  'x-forwarded-for': '192.168.1.49',
  'x-forwarded-proto': 'http',
  connection: 'close',
  'content-length': '0',
  'content-type': 'application/json',
  cookie:
   'trilium.sid=s%3AN0Mf-AlHph5dYk5HlfoggFvqUhDarkHQ.nb37T5xFyIG260hki00lxpUvPNjkQNV76D%2FqOkEbwD8; Path=/; Expires=Wed, 07 Apr 2021 20:36:44 GMT; HttpOnly',
  pagecount: '1',
  pageindex: '0',
  requestid: 'oCsntbh23G',
  'user-agent':
   'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) TriliumNotes/0.46.6 Chrome/83.0.4103.122 Electron/9.4.4 Safari/537.36',
  'accept-encoding': 'gzip, deflate',
  'accept-language': 'en-US' } Session {
  cookie:
   { path: '/',
     _expires: 2021-04-07T20:36:44.616Z,
     originalMaxAge: 86400000,
     httpOnly: true },
  loggedIn: true,
  __lastAccess: 1617741404617 }
204 POST /api/sync/finished took 2ms
{ host: 'domain.tld',
  'x-real-ip': '192.168.1.49',
  'x-forwarded-for': '192.168.1.49',
  'x-forwarded-proto': 'http',
  connection: 'close',
  'content-type': 'application/json',
  cookie:
   'trilium.sid=s%3AN0Mf-AlHph5dYk5HlfoggFvqUhDarkHQ.nb37T5xFyIG260hki00lxpUvPNjkQNV76D%2FqOkEbwD8; Path=/; Expires=Wed, 07 Apr 2021 20:36:44 GMT; HttpOnly',
  pagecount: '1',
  pageindex: '0',
  requestid: 'VadhEgAC1N',
  'user-agent':
   'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) TriliumNotes/0.46.6 Chrome/83.0.4103.122 Electron/9.4.4 Safari/537.36',
  'accept-encoding': 'gzip, deflate',
  'accept-language': 'en-US' } Session {
  cookie:
   { path: '/',
     _expires: 2021-04-07T20:36:44.643Z,
     originalMaxAge: 86400000,
     httpOnly: true },
  loggedIn: true,
  __lastAccess: 1617741404646 }
Content hash computation took 12ms
Slow 200 GET /api/sync/check took 13ms

And the sync succeed, unfortunately, that was really a dirty hack that I couldn't do on my real server...
After further tests, it appears that indeed that cookie is the cause of the issue. I'm gonna ask how to remove properly that cookie...

[zadam]

Good progress, it seems that a similar issue was already encountered in #1376

[maniackcrudelis]

Indeed, I've read that one, but I wasn't able to fully understand ;)

At least we found where the error lies, thank you for that, I wouldn't have found on my own !

Now I'm trying to get a way to remove that other cookie injected by my nginx configuration.
A way or the other, I guess we're good on that issue since the problem is on my side, not trilium.

If that's ok for you, we can close that issue I think.