Crash on startup after upgrade to 0.62.2 #10114

micw · 2018-03-13T05:54:25Z

Description:

I have been trying to upgrade my docker installation from 0.58.4 to 0.62.2. On a direct upgrade, Rocket.Chart crashed shortly after start. So I did a rollback and a step-by-step upgrade from 0.58.4 to 0.59.6 to 0.60.4 to 0.61.2 to 0.62.2.
Up to 0.61.2 all worked and the data was migrated. From 0.61.2 to 0.62.2, the same crash occurs:

Here's the only log:

13.3.2018 06:49:09Will load cache for users
13.3.2018 06:49:0929 records load from users
13.3.2018 06:49:09Will load cache for rocketchat_room
13.3.2018 06:49:09179 records load from rocketchat_room
13.3.2018 06:49:09Will load cache for rocketchat_subscription
13.3.2018 06:49:09435 records load from rocketchat_subscription
13.3.2018 06:49:09Will load cache for rocketchat_settings
13.3.2018 06:49:09614 records load from rocketchat_settings
13.3.2018 06:49:12Updating process.env.MAIL_URL
13.3.2018 06:49:12Starting Email Intercepter...

Server Setup Information:

Version of Rocket.Chat Server: 0.61.2
Operating System: Docker on Debian Stretch
Deployment Method(snap/docker/tar/etc): docker (Official image)
Number of Running Instances: 1
DB Replicaset Oplog: Disabled
Node Version: v8.10.0
mongoDB Version: 3.6

Steps to Reproduce:

See description

Expected behavior:

Upgrade should succeed

Actual behavior:

Rocket.Chat terminates.

Relevant logs:

See above.

The text was updated successfully, but these errors were encountered:

ghost · 2018-03-13T07:41:22Z

Nodejs Version is pretty important right now. "dpkg -l | grep nodejs" will give you that information.

micw · 2018-03-13T08:12:05Z

docker run --rm -it library/rocket.chat:0.62.2 node --version
v8.10.0

(I'm using the official image, added the information above)

ghost · 2018-03-13T08:28:25Z

Well then you are out of luck. You have to downgrade your nodejs version.

nodejs/node#19274

micw · 2018-03-13T09:15:06Z

Thank you a lot. That means the official Rocket.Chat docker image is broken. I'll file a bug report in that repo.

sampaiodiego · 2018-03-13T12:36:37Z

@micw the logs you showed does not contains any errors.. how long did you wait for it to start?

I've tested the official image and seems it is working. here are the full logs I got (notice Node version):

$ docker run --rm -it --link mongo -e MONGO_URL=mongodb://mongo/dockerofficial -e MONGO_OPLOG_URL=mongodb://mongo/local rocket.chat:0.62.2
Will load cache for users
0 records load from users
Will load cache for rocketchat_room
0 records load from rocketchat_room
Will load cache for rocketchat_subscription
0 records load from rocketchat_subscription
Will load cache for rocketchat_settings
0 records load from rocketchat_settings
Updating process.env.MAIL_URL
Starting Email Intercepter...
Will load cache for rocketchat_permissions
0 records load from rocketchat_permissions
Will load cache for rocketchat_roles
0 records load from rocketchat_roles
Warning: connect.session() MemoryStore is not
designed for a production environment, as it will leak
memory, and will not scale past a single process.
Setting default file store to GridFS
LocalStore: store created at
LocalStore: store created at
connect deprecated multipart: use parser (multiparty, busboy, formidable) npm module instead npm/node_modules/connect/lib/middleware/bodyParser.js:56:20
connect deprecated limit: Restrict request size at location of read npm/node_modules/connect/lib/middleware/multipart.js:86:15
Using GridFS for custom sounds storage
Using GridFS for custom emoji storage
ufs: temp directory created at "/tmp/ufs"
Updating process.env.MAIL_URL
➔ System ➔ startup
➔ +----------------------------------------------+
➔ |                SERVER RUNNING                |
➔ +----------------------------------------------+
➔ |                                              |
➔ |  Rocket.Chat Version: 0.62.2                 |
➔ |       NodeJS Version: 8.10.0 - x64           |
➔ |             Platform: linux                  |
➔ |         Process Port: 3000                   |
➔ |             Site URL: http://localhost:3000  |
➔ |     ReplicaSet OpLog: Enabled                |
➔ |          Commit Hash: 07f0e3f77c             |
➔ |        Commit Branch: HEAD                   |
➔ |                                              |
➔ +----------------------------------------------+

micw · 2018-03-13T18:07:21Z

@sampaiodiego The container dies immediately after the log line I copied. Maybe it's an issue with the existing data? As you can see, my installation reads some data while yours reads 0 records.
Any chance to enable some debug log for the docker version?

bbrauns · 2018-03-14T08:42:21Z

hey there, I have a similar issue with docker image version rocket.chat:0.62.2 but not with rocket.chat:0.62.1. I saw that 0.62.1 uses Node v8.9.4 and 0.62.2 uses Node v8.10.0.

Before the container crashes I see the following message in syslog: Mar 14 09:25:45 chat-stage kernel: [372226.247076] traps: node[61380] general protection ip:d8b6b7d5408 sp:7f8878702a18 error:0.

Anyone knows how to further debug this issue?

sampaiodiego · 2018-03-14T12:23:52Z

@bbrauns I've created a pull request RocketChat/Docker.Official.Image#42 so official docker image will use 8.9.4 too..

in the meantime you can use our own image rocketchat/rocket.chat:0.62.2 which is using node 8.9 and is working for @micw

gcommit · 2018-03-14T13:41:58Z

I have the same problem. It repeats over and over again the message

Will load cache for users
2 records load from users
Will load cache for rocketchat_room
1 records load from rocketchat_room
Will load cache for rocketchat_subscription
0 records load from rocketchat_subscription
Will load cache for rocketchat_settings
610 records load from rocketchat_settings
Updating process.env.MAIL_URL
Starting Email Intercepter...

But it does not start at all...

Same issue with 0.62.1, only 0.62.0 is starting up on my side, but there i have this issue:
#10136

sampaiodiego · 2018-03-14T18:58:46Z

do you guys have enabled oplog (have replica set enabled on MongoDB and are setting the env var MONGO_OPLOG_URL to start up rocket.chat)?

I was able to reproduce the error using official Docker image without oplog enabled. it works fine with oplog enabled though (which is the recommended the setup)

micw · 2018-03-14T19:12:02Z

I have no oplog enabled and run a single node mongodb

sampaiodiego · 2018-03-14T19:25:23Z

@micw even on single node mongodb setup using replica set is recommended. you may have issues as your rocket.chat grow up

TomaszDom · 2018-03-15T06:12:06Z

FWIW, I'm running 0.62.2 dual instance with oplog enabled and latest node lts version (8.10.0) with no issues.

bbrauns · 2018-03-15T08:43:12Z

@sampaiodiego oplog has always been activated:

root@chat-stage:~/rocket-chat# cat docker-compose.yml
services:
  app:
    depends_on:
    - mongo
    environment:
    - PORT=3000
    - ROOT_URL=xyz
    - MONGO_URL=mongodb://mongo:27017/rocketchat
    - MONGO_OPLOG_URL=mongodb://mongo:27017/local
    image: rocket.chat:0.62.2
    restart: always
    volumes:
    - /data/upload:/data/upload
  haproxy:
    container_name: haproxy
    hostname: haproxy
    image: haproxy:1.7
    ports:
    - 80:8080
    - 81:8081
    - 82:8082
    - 443:8443
    restart: always
    volumes:
    - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
    - ./chat-combined.pem:/usr/local/etc/haproxy/chat-combined.pem:ro
    - /dev/log:/dev/log:rw
  mongo:
    command: mongod --smallfiles --oplogSize 128 --replSet rs0
    image: mongo:3.4
    logging:
      driver: syslog
      options:
        tag: '{{.Name}}'
    ports:
    - 127.0.0.1:27017:27017
    restart: always
    volumes:
    - /data/db:/data/db
  mongo-init-replica:
    command: 'mongo mongo/rocketchat --eval "rs.initiate({ _id: ''rs0'', members:
      [ { _id: 0, host: ''localhost:27017'' } ]})"'
    depends_on:
    - mongo
    image: mongo:3.4
version: '2'

brtptrs · 2018-03-15T13:43:22Z

We had a similar problem which seems to have disappeared after downgrading to node 8.9.4. We hat the restart problem on a clean install of version 0.62.1

pauloriply · 2018-03-15T14:44:00Z

Same problem with installed rocket.chat via snap... I have no idea what can be done to raise the server
UPD: in my case help dist-upgrade... I do not know what happened, but the chat started normaly
Ubuntu 16.04

AleksanderGrzybowski · 2018-03-15T16:20:38Z

Same here, I wanted to try this out, but it crashes on the first restart, for now I'll pick older docker image, but would really like to see it fixed :)

TwizzyDizzy · 2018-04-24T22:28:53Z

As far as I see it, this (#10331) is the issue we need to keep track of, isn't it? Docker files have been downgraded to my knowledge (and will get upgraded once #10331 is done - which is the case when a newer nodejs version is released which fixes the SEGVs).

So this issue is redundant, isn't it? If not, please let me know :)

Cheers
Thomas

micw · 2018-04-25T07:03:54Z

@TwizzyDizzy I'm not exactly sure if it's the very same issue. While in #10331 users complain about random segfaults, this issue is about a reproducable crash on startup.
Nevertheless, both seems to be related to the nodejs version.

TwizzyDizzy · 2018-04-25T07:38:23Z

Mh. Well then I'm just gonna check back here when the newer nodejs version is released.

Cheers
Thomas

TwizzyDizzy · 2018-05-16T15:32:47Z

New nodejs-Version has been released. Please give me some feedback whether the issue at hand is still relevant. Otherwise I'm gonna close it in the near future.

Cheers
Thomas

TwizzyDizzy · 2018-06-01T09:43:42Z

@rocket-cat close

Closing due to lack of feedback. If this is still an issue, please get back to me and I'll consider reopening based on the given information.

Cheers
Thomas

micw mentioned this issue Mar 13, 2018

0.62.2 broken (crashes on startup) RocketChat/Docker.Official.Image#40

Closed

theorenck added the deploy: docker label Mar 13, 2018

rocket-cat bot closed this as completed Jun 1, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash on startup after upgrade to 0.62.2 #10114

Crash on startup after upgrade to 0.62.2 #10114

micw commented Mar 13, 2018 •

edited

Loading

ghost commented Mar 13, 2018

micw commented Mar 13, 2018 •

edited

Loading

ghost commented Mar 13, 2018

micw commented Mar 13, 2018 •

edited

Loading

sampaiodiego commented Mar 13, 2018

micw commented Mar 13, 2018

bbrauns commented Mar 14, 2018

sampaiodiego commented Mar 14, 2018

gcommit commented Mar 14, 2018 •

edited

Loading

sampaiodiego commented Mar 14, 2018

micw commented Mar 14, 2018

sampaiodiego commented Mar 14, 2018

TomaszDom commented Mar 15, 2018

bbrauns commented Mar 15, 2018

brtptrs commented Mar 15, 2018 •

edited

Loading

pauloriply commented Mar 15, 2018 •

edited

Loading

AleksanderGrzybowski commented Mar 15, 2018

TwizzyDizzy commented Apr 24, 2018

micw commented Apr 25, 2018 •

edited

Loading

TwizzyDizzy commented Apr 25, 2018 •

edited

Loading

TwizzyDizzy commented May 16, 2018

TwizzyDizzy commented Jun 1, 2018

Crash on startup after upgrade to 0.62.2 #10114

Crash on startup after upgrade to 0.62.2 #10114

Comments

micw commented Mar 13, 2018 • edited Loading

Description:

Server Setup Information:

Steps to Reproduce:

Expected behavior:

Actual behavior:

Relevant logs:

ghost commented Mar 13, 2018

micw commented Mar 13, 2018 • edited Loading

ghost commented Mar 13, 2018

micw commented Mar 13, 2018 • edited Loading

sampaiodiego commented Mar 13, 2018

micw commented Mar 13, 2018

bbrauns commented Mar 14, 2018

sampaiodiego commented Mar 14, 2018

gcommit commented Mar 14, 2018 • edited Loading

sampaiodiego commented Mar 14, 2018

micw commented Mar 14, 2018

sampaiodiego commented Mar 14, 2018

TomaszDom commented Mar 15, 2018

bbrauns commented Mar 15, 2018

brtptrs commented Mar 15, 2018 • edited Loading

pauloriply commented Mar 15, 2018 • edited Loading

AleksanderGrzybowski commented Mar 15, 2018

TwizzyDizzy commented Apr 24, 2018

micw commented Apr 25, 2018 • edited Loading

TwizzyDizzy commented Apr 25, 2018 • edited Loading

TwizzyDizzy commented May 16, 2018

TwizzyDizzy commented Jun 1, 2018

micw commented Mar 13, 2018 •

edited

Loading

micw commented Mar 13, 2018 •

edited

Loading

micw commented Mar 13, 2018 •

edited

Loading

gcommit commented Mar 14, 2018 •

edited

Loading

brtptrs commented Mar 15, 2018 •

edited

Loading

pauloriply commented Mar 15, 2018 •

edited

Loading

micw commented Apr 25, 2018 •

edited

Loading

TwizzyDizzy commented Apr 25, 2018 •

edited

Loading