State is not persisted over docker restarts

[pierreozoux]

Do you want to request a feature or report a bug?
bug

What is the current behavior?

Let's say I'm editing a document, then, the docker restarts.
When the docker is back online, my changes are lost.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.

• open a document
• make some changes
• click on save button
• wait a bit to be sure that the server had time to eventually save it
• restart the docker container
• reload your page
• your changes are not there anymore

What is the expected behavior?

The changes should be persisted, even in the event of an unwanted restarts.

Did this work in previous versions of DocumentServer?

I don't think so.

DocumentServer Docker tag:
I used the docker-compose file from here, so I compiled it myself then.
I git clone with this commit.

Host Operating System:
Debian.

More info

I'm trying this with Nextcloud 18.0.6 and the onlyoffice application v4.3.0.

I tried to persist the state of the various components with this docker-compose (I redacted some parts):

  onlyoffice-documentserver:
    build:
      context: .
    container_name: onlyoffice-documentserver
    depends_on:
    - onlyoffice-postgresql
    - onlyoffice-rabbitmq
    environment:
    - DB_TYPE=postgres
    - DB_HOST=onlyoffice-postgresql
    - DB_PORT=5432
    - DB_NAME=onlyoffice
    - DB_USER=onlyoffice
    - AMQP_URI=amqp://guest:guest@onlyoffice-rabbitmq
    - VIRTUAL_HOST
    - LETSENCRYPT_HOST
    - JWT_ENABLED=true
    - JWT_SECRET=
    - JWT_HEADER=Authorization
    - JWT_IN_BODY=true
    stdin_open: true
    restart: always
    volumes:
    - ./oo/data:/var/www/onlyoffice/Data
    - ./oo/log:/var/log/onlyoffice
    - ./oo/cache:/var/lib/onlyoffice/documentserver/App_Data/cache/files
    - ./oo/files:/var/www/onlyoffice/documentserver-example/public/files
    - ./fonts:/usr/share/fonts
    - ./oo/all-cache:/var/www/onlyoffice/documentserver/sdkjs/word/sdk-all.cache
    - ./prod.json:/etc/onlyoffice/documentserver/log4js/production.json
    - ./tmp:/tmp
  onlyoffice-rabbitmq:
    container_name: onlyoffice-rabbitmq
    hostname: "rabbitmq"
    image: rabbitmq
    restart: always
    volumes:
    - ./mq/data:/var/lib/rabbitmq
  onlyoffice-postgresql:
    container_name: onlyoffice-postgresql
    image: postgres:9.5
    environment:
    - POSTGRES_DB=onlyoffice
    - POSTGRES_USER=onlyoffice
    - POSTGRES_HOST_AUTH_METHOD=trust
    restart: always
    volumes:
    - ./data:/var/lib/postgresql

But without success.

I did put ALL and DEBUG logs from documentserver, here is the relevant output:

[2020-08-21T10:25:51.136] [INFO] nodeJS - data.type = isSaveLock id = 1264143739
[2020-08-21T10:25:51.136] [DEBUG] nodeJS - isSaveLock: docId = 1264143739; lockRes: true
[2020-08-21T10:25:51.136] [DEBUG] nodeJS - sendData: docId = 1264143739;type = saveLock
[2020-08-21T10:25:51.177] [INFO] nodeJS - data.type = saveChanges id = 1264143739
[2020-08-21T10:25:51.177] [INFO] nodeJS - Start saveChanges docid: 1264143739; reSave: undefined
[2020-08-21T10:25:51.178] [INFO] nodeJS - saveChanges docid: 1264143739 ; deleteIndex: 6 ; startIndex: 6 ; length: 36
[2020-08-21T10:25:51.193] [DEBUG] nodeJS - sendData: docId = 1264143739;type = unSaveLock
[2020-08-21T10:26:00.003] [DEBUG] nodeJS - expireDoc connections.length = 1
[2020-08-21T10:26:00.007] [DEBUG] nodeJS - checkDocumentExpire start
[2020-08-21T10:26:00.007] [DEBUG] nodeJS - checkDocumentExpire end: startSaveCount = 0, removedCount = 0
[2020-08-21T10:26:44.215] [WARN] nodeJS - Express server starting...
[2020-08-21T10:26:44.219] [WARN] nodeJS - Failed to subscribe to plugin folder updates. When changing the list of plugins, you must restart the server. https://nodejs.org/docs/latest/api/fs.html#fs_availability
[2020-08-21T10:26:44.231] [INFO] nodeJS - SockJS v0.3.20 bound to "/doc/[0-9-.a-zA-Z_=]*/c"
[2020-08-21T10:26:44.239] [INFO] nodeJS - End updateLicense
[2020-08-21T10:26:44.240] [INFO] nodeJS - End updateLicense
[2020-08-21T10:26:44.299] [DEBUG] nodeJS - [AMQP] connected
[2020-08-21T10:26:44.367] [DEBUG] nodeJS - [AMQP] connected
[2020-08-21T10:26:44.385] [INFO] nodeJS - Start callbackFunction
[2020-08-21T10:26:44.386] [WARN] nodeJS - Express server listening on port 8000 in production-linux mode
[2020-08-21T10:26:48.572] [INFO] nodeJS - GET /doc/1264143739/c/info 5ms 200
[2020-08-21T10:26:48.692] [INFO] nodeJS - GET /doc/1264143739/c/113/pxs4rjea/websocket 2ms (unfinished)
[2020-08-21T10:26:48.696] [DEBUG] nodeJS - sendData: docId = undefined;type = license
[2020-08-21T10:26:48.735] [INFO] nodeJS - data.type = auth id = undefined
[2020-08-21T10:26:48.736] [DEBUG] nodeJS - checkJwt success: docId = 1264143739 decoded = {"document":{"key":"1264143739","permissions":{"edit":true}},"editorConfig":{"user":{"id":"ocim3ezcla1z_pierre@ozoux.net","name":"Pierre","index":1},"ds_view":false,"ds_isCloseCoAuthoring":false},"iat":1598005474,"exp":1600597474}
[2020-08-21T10:26:48.741] [WARN] nodeJS - error description: docId = 1264143739 errorId = Other error
[2020-08-21T10:26:48.741] [DEBUG] nodeJS - sendData: docId = 1264143739;type = error
[2020-08-21T10:26:48.776] [INFO] nodeJS - data.type = close id = 1264143739
[2020-08-21T10:26:48.776] [INFO] nodeJS - Connection closed or timed out: userId = ocim3ezcla1z_pierre@ozoux.net1 isCloseConnection = false docId = 1264143739
[2020-08-21T10:27:07.852] [WARN] nodeJS - Express server starting...
[2020-08-21T10:27:07.854] [WARN] nodeJS - Failed to subscribe to plugin folder updates. When changing the list of plugins, you must restart the server. https://nodejs.org/docs/latest/api/fs.html#fs_availability
[2020-08-21T10:27:07.855] [INFO] nodeJS - SockJS v0.3.20 bound to "/doc/[0-9-.a-zA-Z_=]*/c"
[2020-08-21T10:27:07.860] [INFO] nodeJS - End updateLicense
[2020-08-21T10:27:07.861] [INFO] nodeJS - End updateLicense
[2020-08-21T10:27:07.907] [DEBUG] nodeJS - [AMQP] connected
[2020-08-21T10:27:07.959] [DEBUG] nodeJS - [AMQP] connected
[2020-08-21T10:27:07.987] [INFO] nodeJS - Start callbackFunction
[2020-08-21T10:27:07.990] [WARN] nodeJS - Express server listening on port 8000 in production-linux mode
[2020-08-21T10:27:10.627] [INFO] nodeJS - GET /doc/1264143739/c/info 2ms 200
[2020-08-21T10:27:11.172] [INFO] nodeJS - GET /doc/1264143739/c/209/k3nxtu4p/websocket 1ms (unfinished)
[2020-08-21T10:27:11.175] [DEBUG] nodeJS - sendData: docId = undefined;type = license
[2020-08-21T10:27:11.426] [INFO] nodeJS - data.type = auth id = undefined

I also tried to run:

find / -mmin -1 -type f -not -path '/proc/**/*' -not -path '/sys/**/*'  -not -path '/proc/*' -exec ls -l {} +

right after the document save, to eventually identify where things could get stored, and no files appeared.

** Interpretation **

With all these information, my guess is that the state is not persisted on disk but the state is in memory. If we restart the process, then this would be the normal behavior to loose all the changes.

If you are alone using Nextcloud and onlyoffice, you have to remember to close the document, but now imagine you are 10 people. How do you make sure that everybody did close the document?

In a distributed environment, especially with docker, processes are more and more restarted, so this behavior makes the usage of onlyoffice complicated currently.
On top, it is regarded as a general good practice to not save state in memory, I recommend to read https://12factor.net/processes for instance.

Would the use of redis help mitigate this behavior?

I tried to run the Nextcloud cron, but this didn't help.

Let me know if I can provide further details.

Thanks for your help!

[ShockwaveNN]

If you are alone using Nextcloud and onlyoffice, you have to remember to close the document, but now imagine you are 10 people. How do you make sure that everybody did close the document?

Every time you shutdown ONLYOFFICE DocumentServer you should call documentserver-prepare4shutdown.sh script as mention here

This script will close all opened documents with message to users

[ShockwaveNN]

If you compiled DocumentServer not sure if that script is include, but it's looking like this:

# cat /usr/bin/documentserver-prepare4shutdown.sh
#!/bin/sh

echo -n Preparing for shutdown, it can take a lot of time, please wait...

curl http://localhost:8000/internal/cluster/inactive -X PUT -s -o /dev/null

echo Done

[pierreozoux]

Hum, I understand, this is already a workaround, I'd say.

But so what you are saying is that if my server crashes, then I lose all documents.

Do you have plan to fix this bug?

Here you discuss about a force save for Nextcloud integration as well. Is it in your pipeline? Is there anything we can do to help move this fix faster?

[ShockwaveNN]

Hum, I understand, this is already a workaround, I'd say.
But so what you are saying is that if my server crashes, then I lose all documents.

I agree with you on that

Do you have plan to fix this bug?

I think it's safe to save we want to fix it, but currently our app architecture do not allow this, so I cannot guarantee it

[pierreozoux]

Ok, I tested your curl, it does actually work, thanks for the tip already :)

Nice to hear that you want to fix it :)

Have a nice day then and thanks for your time!

[ShockwaveNN]

Great to hear, In that case I'll close this issue, but feel free to comment or open new issue if got any more questions

[alexanderonlyoffice]

As for the force save feature within the integration with Nextcloud. It is in our plans and we are working right now on the implementation of this feature. Unfortunately, I cannot specify any exact time frames at the moment. We know that this feature is a long-awaited one and we have just tested it in our DMS (Community Server), so we are doing our best to make it available within the integration in the nearest future.