massive memory load during truffle compile/migrate/build/test

[SCBuergel]

I am using testrpc with truffle on Ubuntu 16.04 x64 (Digital Ocean). I did not make any changes to the default truffle directory that I get with truffle init. I am running my testrpc inside tmux to keep it running in background. This works as expected but at seemingly random times when running truffle compile/migrate/build/test the testrpc is quitting with the following last output:

eth_newBlockFilter
eth_getFilterChanges
eth_getTransactionReceipt
eth_getCode
eth_uninstallFilter
eth_sendTransaction

  Transaction: 0xf02ea86ba3f82894cea4dbdb82ef76bb789d42b1b0dc5be7f1f373c5f2f3771c
  Contract created: 0xd12f38adfe5554bc41e8aec902ac1611d757eb79
  Gas usage: 0x0360b8
  Block Number: 0x04
  Block Time: Tue Jan 17 2017 13:03:19 GMT+0000 (UTC)

eth_newBlockFilter
eth_getFilterChanges
eth_getTransactionReceipt
eth_getCode
eth_uninstallFilter
eth_sendTransaction

  Transaction: 0xf916f212609171587ed6469d3dd3934632366c9b9e1413d90ee13317d6c606fd
  Gas usage: 0x68aa
  Block Number: 0x05
  Block Time: Tue Jan 17 2017 13:03:19 GMT+0000 (UTC)

eth_getTransactionReceipt
Killed

This does not seem to be a truffle issue but rather an issue of the testrpc. I would be happy about any suggestions or help on how to get further information.

These are my install steps from a blank Ubuntu 16.04 DO machine:

curl -sL https://deb.nodesource.com/setup_6.x -o nodesource_setup.sh
bash nodesource_setup.sh
rm nodesource_setup.sh
apt-get install nodejs -y
npm install pm2 -g
apt-get install build-essential
npm install -g truffle
npm install -g ethereumjs-testrpc
tmux

then inside tmux, I am doing a simple testrpc to start it. Next I walked through the documentation of truffle and step by step executed the following order of commands which seem to randomly fail within about 20 minutes:

mkdir mytruffleproject
cd mytruffleproject
truffle init
truffle compile
truffle migrate
truffle build
truffle test

some version info:

root@truffle:~# node --version
v6.9.4
root@truffle:~# npm --version
3.10.10
root@truffle:~# testrpc
EthereumJS TestRPC v3.0.3

[cgewecke]

@SCBuergel Does Digital Ocean cap the RAM size of your process? Recently had similar failures running testrpc / truffle from within a unit test in an Ubuntu container on CircleCI. The problem was caused by hitting a container memory limit. You could try to see if this is the case by logging process.memoryUsage() somewhere (truffle's unit tests?). Because the truffle process was part of a mocha process, it was possible to get around this problem by exposing garbage collection in mocha and explicitly garbage collecting between each truffle invocation.

[johnmcdowall]

So. Any time you see the word 'Killed' its an indication that the Kernel killed your process. The Kernel will kill your process in extreme circumstances, like when you run out of memory. Read more about it here: http://stackoverflow.com/questions/726690/who-killed-my-process-and-why

Immediately I'm thinking that you failed to enable a swap file on your Droplet and the Droplet you are using has an insufficient amount of RAM for what you're trying to use it for.

Here's a guide on enabling swap on Ubuntu (12.04) - I'm sure you can figure out what to do for your version: https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-12-04

[cgewecke]

@johnmcdowall @SCBuergel If this is the agreed cause, it might be helpful to rename the issue so that others can find it or perhaps make a small note in the README about sudden exits in a memory-constrained environment with @johnmcdowall's suggested solutions.

[SCBuergel]

Thanks @cgewecke and @johnmcdowall for the comments. I looked a bit more into this. You are right, the memory usage is crazy, during truffle compile, here the top output while it was compiling the default truffle project:

I am on a DO machine with 2 cores and 2GB RAM, if this is insufficient (it is sufficient even for mining in a private net) then I'd call it a bug and not a system requirement issue. Now tried again with an 8GB RAM machine, and all worked fine.

[SCBuergel]

It would be interesting to explore if this is an issue with truffle or testrpc.

[nogo10]

the problem is with testrpc.. how do I know? because I tried running testrpc on a Cloud9 IDE 1gb instance without truffle installed and got the 'Killed' error...

[tcoulter]

I've heard other reports of memory load in the TestRPC - after all, it is a blockchain in memory. :) This is something we're looking into and will hopefully have more details soon.

[nogo10]

I tried again with a 1gb memory cloud9 instance .. and testrpc works .. So I think 1gb memory is the minimum requirement. thnks

Edit: retried and it fails ..need guidance or settings to limit memory usage

[SCBuergel]

@nogo10 I tried with 2 GB and that also "worked" but we have to be very specific with what "worked" really means: When trying to compile the default truffle project it crashed in about 10% of all attempts. That IMO means it "doesnt work".

[tcoulter]

Compiling is a very different issue that doesn't touch the TestRPC at all. In fact, you can turn off the TestRPC and still compile just fine. This suggests it's the solidity compiler using the memory in that case, which I have little control over. You'll have to file a bug report with them. That said, we're looking into the TestRPC's memory issues now.

…

On Feb 3, 2017 7:48 AM, "Sebastian C. Bürgel" ***@***.***> wrote: @nogo10 <https://github.com/nogo10> I tried with 2 GB and that also "worked" but we have to be very specific with what "worked" really means: When trying to compile the default truffle project it crashed in about 10% of all attempts. That IMO means it "doesnt work". — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#242 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFp1QPKWjXu9Um4v4m5eQTxSTo7uqrHks5rY0xLgaJpZM4Llpq1> .

[SCBuergel]

sorry, that was not precise on my end: while running the tests and deploy it was crashing at some point, so I do think it was a TestRPC issue.

@tcoulter thanks for looking into it - TestRPC is a great tool that many people would benefit from if it becomes even more resilient!

[tcoulter]

This is now fixed on the persistence branch: #275

TL;DR: There's no more memory issues because the data is no longer stored in memory; it's now stored on disk. In my tests with tens of thousands of requests, this branch never got above 250Mb of memory usage on my machine. Compare that to the previous version of the TestRPC (3.0.3), which I was able to make hit 800Mb of memory quite easily.

I'm going to close this ticket for now, although it won't be completely fixed until a new released is pushed. Keep a look out for the next version (4.0.0) which should be out soon.

[andytudhope]

Hi @tcoulter. I just spun up a quick AWS free-tier instance with 2GB RAM and it seems that the latest version of truffle (3.4.6) is back to the same issue above. My guess is that it must be to do with the new solidity compiler, as even without testrpc running in another screen, truffle compile just gets killed no matter what I do.

Reverting to truffle v3.2.2 fixes things.