Rocket Chat run out of memory while doing import

[ghost]

I am using latest snap version of Rocket chat.
I have just installed it via snap and tried to import slack dump on it.
The result was that rocket chat instance is not starting anymore.
I have that in logs

May 05 15:01:05 ip-172-31-36-160 systemd[1]: Started Service for snap application rocketchat-server.rocketchat-server. May 05 15:01:05 ip-172-31-36-160 /usr/bin/snap[1741]: cmd.go:114: DEBUG: not restarting into "/snap/core/current/usr/bin/snap" ([VERSION=2.24 2.24]): older than "/usr/bin/snap" (2.24.1) May 05 15:01:05 ip-172-31-36-160 snap[1741]: Checking if oplog has been enabled, and enabling if not May 05 15:01:05 ip-172-31-36-160 snap[1741]: MongoDB shell version: 3.2.7 May 05 15:01:05 ip-172-31-36-160 snap[1741]: connecting to: test May 05 15:01:07 ip-172-31-36-160 snap[1741]: Will load cache for users May 05 15:01:07 ip-172-31-36-160 snap[1741]: 2 records load from users May 05 15:01:07 ip-172-31-36-160 snap[1741]: Will load cache for rocketchat_room May 05 15:01:07 ip-172-31-36-160 snap[1741]: 1 records load from rocketchat_room May 05 15:01:07 ip-172-31-36-160 snap[1741]: Will load cache for rocketchat_subscription May 05 15:01:07 ip-172-31-36-160 snap[1741]: 1 records load from rocketchat_subscription May 05 15:01:07 ip-172-31-36-160 snap[1741]: Will load cache for rocketchat_settings May 05 15:01:07 ip-172-31-36-160 snap[1741]: 480 records load from rocketchat_settings May 05 15:01:08 ip-172-31-36-160 snap[1741]: Updating process.env.MAIL_URL May 05 15:01:08 ip-172-31-36-160 snap[1741]: Will load cache for rocketchat_permissions May 05 15:01:08 ip-172-31-36-160 snap[1741]: 69 records load from rocketchat_permissions May 05 15:01:08 ip-172-31-36-160 snap[1741]: Will load cache for rocketchat_roles May 05 15:01:08 ip-172-31-36-160 snap[1741]: 9 records load from rocketchat_roles May 05 15:01:10 ip-172-31-36-160 snap[1741]: LocalStore: store created at May 05 15:01:11 ip-172-31-36-160 snap[1741]: {"line":"147","file":"rocketchat_migrations.js","message":"Migrations: Not migrating, already at version 91","time":{"$date":1493996471941},"level":"info"} May 05 15:01:12 ip-172-31-36-160 snap[1741]: Using GridFS for custom sounds storage May 05 15:01:12 ip-172-31-36-160 snap[1741]: Using GridFS for custom emoji storage May 05 15:01:12 ip-172-31-36-160 snap[1741]: Updating process.env.MAIL_URL May 05 15:02:50 ip-172-31-36-160 snap[1741]: <--- Last few GCs ---> May 05 15:02:50 ip-172-31-36-160 snap[1741]: 103639 ms: Scavenge 1384.8 (1457.3) -> 1384.8 (1457.3) MB, 0.4 / 0 ms (+ 0.2 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep]. May 05 15:02:50 ip-172-31-36-160 snap[1741]: 104498 ms: Mark-sweep 1384.8 (1457.3) -> 1384.7 (1457.3) MB, 858.8 / 0 ms (+ 1.5 ms in 2 steps since start of marking, biggest step 1.2 ms) [last resort gc]. May 05 15:02:50 ip-172-31-36-160 snap[1741]: 105356 ms: Mark-sweep 1384.7 (1457.3) -> 1384.7 (1457.3) MB, 857.5 / 0 ms [last resort gc]. May 05 15:02:50 ip-172-31-36-160 snap[1741]: <--- JS stacktrace ---> May 05 15:02:50 ip-172-31-36-160 snap[1741]: ==== JS stack trace ========================================= May 05 15:02:50 ip-172-31-36-160 snap[1741]: Security context: 0xd260abb4629 <JS Object> May 05 15:02:50 ip-172-31-36-160 snap[1741]: 1: clone [packages/ejson.js:~482] [pc=0x2e27d402df6e] (this=0x3bf5d51be1f9 <an Object with map 0xe67e822e49>,v=0x4ba63549ff1 <an Object with map 0x35ac59683b91>) May 05 15:02:50 ip-172-31-36-160 snap[1741]: 2: clone [packages/ejson.js:~482] [pc=0x2e27d402dc66] (this=0x3bf5d51be1f9 <an Object with map 0xe67e822e49>,v=0x2583336cd089 <JS Array[1]>) May 05 15:02:50 ip-172-31-36-160 snap[1741]: 3: /* anonymous */ [packages/ejson.js:~520] [pc=0x2e27d133beaf] (this=0xd260abd78e1... May 05 15:02:50 ip-172-31-36-160 snap[1741]: FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory May 05 15:02:51 ip-172-31-36-160 snap[1741]: /snap/rocketchat-server/616/bin/startRocketChat: line 3: 1758 Aborted (core dumped) node $SNAP/main.js May 05 15:02:51 ip-172-31-36-160 systemd[1]: snap.rocketchat-server.rocketchat-server.service: Main process exited, code=exited, status=134/n/a May 05 15:02:51 ip-172-31-36-160 systemd[1]: snap.rocketchat-server.rocketchat-server.service: Unit entered failed state. May 05 15:02:51 ip-172-31-36-160 systemd[1]: snap.rocketchat-server.rocketchat-server.service: Failed with result 'exit-code'. May 05 15:02:51 ip-172-31-36-160 systemd[1]: snap.rocketchat-server.rocketchat-server.service: Service hold-off time over, scheduling restart. May 05 15:02:51 ip-172-31-36-160 systemd[1]: Stopped Service for snap application rocketchat-server.rocketchat-server.
I found a solution to change node command line by adding "--max-old-space-size=4096" but I cant do it as directory /snap/rocketchat-server/616/ mounted read-only.

How can I change that parameter?

[geekgonecrazy]

Please see #6844

Can you give me some stats about your user count / connected when this happens?

[ghost]

@geekgonecrazy there is no connectes users, it is fresh installation

[geekgonecrazy]

@teamfighter how much available ram do you have on the machine? It for sure shouldn't be going anywhere near that limit on a fresh install

[ghost]

@geekgonecrazy i have 8 gb available

[geekgonecrazy]

So this is a brand new install having just run sudo snap install rocketchat-server ?

Can you stop the server via: sudo systemctl stop snap.rocketchat-server.rocketchat-server grab the output of free -mh paste it here then do: sudo systemctl start snap.rocketchat-server.rocketchat-server wait until it's accessible via the UI and then do: free -mh and paste here?

[ghost]

@geekgonecrazy
That issue happened after my try to import 42 mb slack archive. Instance stucked and didnt launched anymore.
I dont have access to server at the moment, but after rocket chat start it tooks about 2gb of ram, so I have about 6 gb free. If i am stopping rocket chat instance i have about 8 gb ram free. In log i have cycling messages that posted at my first post, i.e. instance is failing because of out of memory, trying to start and failing again, etc.

[ghost]

@geekgonecrazy so steps yo reproduce:

1. Install rocketchat via snap
2. Import slack archive (about 42 mb zipped file)
3. Wait for rocket chat hang
4. Restart it.
   Unfortunately, I could not give you that slack archive because of a lot of confidential data there.

[geekgonecrazy]

@teamfighter ah! That's what I was looking for. 😁. Do you have any counts on the number of channels / files / users / messages in the import?

[ghost]

@geekgonecrazy it is more than 150 channels and users in archive, zip file is 42 Mb size.

[ghost]

@geekgonecrazy
well, I have re-installed rocket chat manually and increased available memory for node.js.
Now I have this error during import data from slack:
Exception in defer callback: Error: Notify all in this room not allowed [error-action-not-allowed] at /opt/Rocket.Chat/programs/server/packages/rocketchat_lib.js:12276:10 at /opt/Rocket.Chat/programs/server/packages/rocketchat_lib.js:314:25 at Array.reduce (native) at Object.RocketChat.callbacks.run (/opt/Rocket.Chat/programs/server/packages/rocketchat_lib.js:307:6) at Object.RocketChat.sendMessage (/opt/Rocket.Chat/programs/server/packages/rocketchat_lib.js:3111:34) at Slack.<anonymous> (/opt/Rocket.Chat/programs/server/packages/rocketchat_importer-slack.js:602:38) at [object Object].<anonymous> (/opt/Rocket.Chat/programs/server/packages/rocketchat_importer-slack.js:705:19) at packages/dispatch_run-as-user.js:211:14 at [object Object]._.extend.withValue (packages/meteor.js:1122:17) at Object.Meteor.runAsUser (packages/dispatch_run-as-user.js:210:33) at fn (/opt/Rocket.Chat/programs/server/packages/rocketchat_importer-slack.js:518:25) at /opt/Rocket.Chat/programs/server/packages/rocketchat_importer-slack.js:715:11 at [object Object]._.extend.withValue (packages/meteor.js:1122:17) at packages/meteor.js:445:45 at runWithEnvironment (packages/meteor.js:1176:24)
And import page just hung and shows only:
Importing the messages.
33693 / 451060
Rocket chat instance seems fully functional and even have some imported data.
How do I handle this?

[geekgonecrazy]

@graywolf336 thoughts here on this slack import issue?

[graywolf336]

error-action-not-allowed means what it says, one of the rooms a message was being imported to doesn't allow the usage of @all.

Have you tried to import it again anytime recently? As that's odd it just hangs there...

[ghost]

Still the same issue.
I have increased available memory to 8Gbs by command --max_old_space_size=8192 but I still have such errors in log:

RangeError: Out of memory
at RangeError (native)
at Fibers.run (packages/rocketchat_monitoring.js:3549:22)
at process. (packages/meteor.js:1190:31)
at emitOne (events.js:82:20)
at process.emit (events.js:169:7)
at process._fatalException (node.js:224:26)

What is the best way to import slack archive with size ~42 Mb?

[geekgonecrazy]

if you're still getting that error after having increased the --max_old_space_size to 8GB then it sounds like you don't actually have that much memory available for the nodejs process to use.

Are you running mongo on the same server? If so which version of mongo?

[ghost]

Yes, I am running mongo v.2.6.10 on same server and I have 16 Gb RAM at this node.
By the way, server stops with error above approx. at middle of import process and amount of USED RAM is about 2.7 Gb at all server.
So it looks like node js issue.

[alexweissman]

If you're finding this because you got an "out of memory" error and you're not running a server with 16GB of RAM (e.g., you're using a minimal 1G setup), you might just need to add a swap file: https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-16-04