Bounty for resolving a memory leak issue

[ManfredKarrer]

It seems that there is a memory leak since the trade statistics release in the GUI app.
I could never see a leak before in the Bitsquare GUI app (on OSX), but the seed node app (running under Linux) has a leak since a while. It might be related to open files (network) which is treated differently on OSX and Unix.
I want to give out a bounty for anyone who is able to locate and fix the leak (get in touch to negotiate amount).
To test if the fix was sufficient, I will test both the seed node app and the GUI app to leave it running for min. 1 week to see if memory consumption does not grow.

[ManfredKarrer]

On OSX I could not reproduce a memory leak so far (2 days running), so might be Linux issue only or jsut in certain setups?

[haiqu]

Nothing seen under Windows 7. Must be OS dependent, I get 330k or so memory usage consistently.

[ManfredKarrer]

Seems it happens on Linux. My seednodes have issues as well, they get restarted each day. But had not looked closer into it what causes the issue. Might be a OS config issue (open network connections, files,...)

[haiqu]

Probably some leaky library in the OS. Is it specific to any particular dist?

[ManfredKarrer]

I don't know. I used Ubunty for seednodes. Not sure what the other reporters used.

[ulrichard]

I use it on ubuntu too.
Never noticed considerable increase in RAM use.
But I have enough RAM, and usually don't run the app for longer than 20 hours.

[abelboldu]

I do experience memory problems when running Bitsquare for some time.
I'm using Archlinux with 8GB of RAM, and at some point I have to kill bitsquare because of OOM problems.
I'll try to add more information about it

[abelboldu]

I did some research with the VisualVM tool, looks like the heap size is constantly increasing.
See, the graph looks bullish:
https://imgur.com/a/z1MRh

Also, the application thread list shows that the JavaFX thread is literally eating up the mem:
https://imgur.com/a/7qohG

Triggering the GC had no effect.

I hope it helps...

[abelboldu]

Using java 8 instead of Java 7 seems to solve the memory leak, the graph looks much better now:
https://imgur.com/a/PNBHS

Also the memory usage by the JavaFX is steady at a more reasonable level now:
https://imgur.com/a/P7fVz

I'll leave it running all night to see how it works, but I'm pretty sure it fixes the problem.

[ManfredKarrer]

See #696 (comment)

[lkblkb]

Using Java 8 Update 121 on Win 7, starting memory is around 500mb, hits 600-700mb after a few hours. It was at 1GB (basically all there was available at the time) when I shut it down yesterday after first run. Just updating to 4.9.9.9 and so far it is 600-700mb after a 2-3 hours.

[madorian]

I'm running ubuntu-16.10 and had been running bitsquares for about 2 days, then suddenly my machine froze for 25 minutes. ssh'ing into the box took 15 minutes, then I killed bitsquares and everything was fine. I have 16Gio RAM/4GHz Intel i7

[ManfredKarrer]

could u send me the log file (.local/share)?
that is strange never heard of such.

[madorian]

Here you go.

bitsquare.txt

[ManfredKarrer]

Hm... nothing strange beside high memory usage which is normal if you have enough RAM (Java is greedy and takes what i gets but it runs also with 300mb if there is not more available).

[madorian]

Yeah, I don't really care how much memory it uses, but my whole system froze; that sucks more;) I'll start it again and see if it happens again. Thanks.

[ManfredKarrer]

Yes that is weird. I never saw that before. I have the seeds nodes running 24/7 on linux machines. They have a memory leak so get restarted once a day but never saw a freeze.

[rarigita]

This issue is already open? I'll install the platform tomorrow on my ubuntu 16.04 x64 desktop and I'll keep an eye to memory use along the day.

[cedricwalter]

I found one obvious leak, in TradeStatisticsManager, the TradeStatistics are kept forever in tradeStatisticsSet and can not be GC, Bisq get adding in tradeStatisticsSet with add() but object never get removed. I am
still profiling to find more...

[ManfredKarrer]

Trade statistic items never get removed (as trades can only grow). Longterm when that data set becomes too large we need other models (pruning history,...).

[pmknutsen]

Dropping this here as issue is still open and I am seeing serious memory leaks also in Bisq v0.8.0. On last shutdown usage was 15 gb.

[ghost]

... seems absolutely enormous.
Usually it's around 1GB.
What OS do you have ? How much RAM ?

[pmknutsen]

ubuntu 32gb. See screengrab.

[ghost]

... do you have 2 Bisq instances running ?

[pmknutsen]

No, just one. The second one down is a thread and the first a process if I understand htop coloring correct.

[ghost]

Is Bisq running since a long time ?
(I see 5h58, but process time is not running time).
Anyway, 15GB used on a 32GB machine, something seems not ok.
(On my 16GB 64 bits Debian machine, Bisq consumes 1GB)

If/when you have no offers/trades/disputes running, maybe you could relaunch and see what happens.

[pmknutsen]

Bisq had been running for about a week. Will keep a closer eye. Do have a few offers listed but will spin up a second instance on the LTC markets where I have none, and compare.

[ManfredKarrer]

@pmknutsen Java takes what the system provides, but 16 GB seems a lot....
What do u mean with "LTC markets" - LTC as base currency has been removed since a while due lack of activity.

[pmknutsen]

Yes, LTC as base market is what I meant. Did not know it was shut down.

[ManfredKarrer]

@pmknutsen Can you still run the app to withdraw funds if you have any there? If not send me a PM at the forum or Slack.

[pmknutsen]

I managed to export priv keys @ManfredKarrer Coins out.

[ManfredKarrer]

Might be related to temp files: https://puneeth.wordpress.com/2006/01/23/filedeleteonexit-is-evil/

[dgyg]

The memory consumption (v0.9.5) under Ubuntu 18.04 is sometimes still quite high:

Usually the application uses about 600 MiB. The situation may occur even if the application has been running only for a short time.

[ultimate255]

Very unusual high memory consumption on Manjaro. Occurs shortly after entering the password, the UI freezes up (as the application uses more memory) and eventually loads.

[ghost]

@ultimate255 Which Bisq version do you use ?

[ultimate255]

v0.9.5

[ultimate255]

The high memory usage seems to have stopped after I let it sit and load out, and no longer occurs when I restart bisq.

Had to force kill it when the high memory usage first occurred as it probably hit the swap space and froze my system.

[ManfredKarrer]

Probably it was caused by the blockheader sync of BitcoinJ. You can reproduce that by resyncing the SPV chain file. Depending on the age of the wallet and the number of txs it takes quite a while and consume all available resources.

[robertclarkson]

Just ran both the deb installed and snap installed app and both ate 32GB of RAM and 32GB Swap file until my machine was unresponsive.

[cd2357]

Did you run a recent version? PR #4048 should have limited the mem usage to 4GB.

[robertclarkson]

I downloaded the 1.3.4 deb and tested with that. Same problem.
I know how to code in Java so if you have an idea where the leak might be happening I could run some diagnostics. From the console log it seems to start happening just before or after beginning connecting to the TOR network

[cd2357]

@robertclarkson: Hmm, then 3 questions:

1. You said you tried out the 1.3.4 deb. Could you paste here the contents of /opt/Bisq/app/Bisq.cfg ? More specifically the last 3 sections in that file (JVMOptions, JVMUserOptions and ArgOptions). I'm curious if perhaps due to upgrades from previous versions, somehow an older config file survived with older JVM parameters.

2. When you start Bisq (whichever you prefer, the deb or the snap), the UI will have an indicator bottom left, saying "... Synchronizing DAO" with a sort of moving progressbar. Does that indicator go away after a while (so does "Synchronizing DAO" ever finish)? Or does it keep synchronizing until the 32 GB memory are used up and system hangs?

3. Does this problem with using up all 32 GB of memory + system hanging, does it also happen for a fresh Bisq setup, in other words with a fresh empty data folder? (Backup your existing data directory, then delete the data folder from the original location, and start Bisq. If needed, you can anytime restore your data from this backup.)

[robertclarkson]

[Application]
app.name=Bisq
app.version=1.3.4
app.preferences.id=bisq/desktop/app
app.runtime=$APPDIR/runtime
app.identifier=bisq.desktop.app
app.classpath=
app.application.instance=multiple
app.mainclass=bisq/desktop/app/BisqAppMain
app.mainjar=desktop-1.3.4-all.jar
packager.java.version=10.0.2

[JVMOptions]
-Xss1280k
-XX:MaxRAM=4g
-Djava.net.preferIPv4Stack=true

[JVMUserOptions]

[ArgOptions]

2. Doesn't get that far i dont think, problem happens as soon as it starts connecting to TOR

3. Was a fresh setup from me brand new install on Ubuntu 18.04.4 LTS. Happens on both the snap version and the deb installed version

Here is a video of the issue with the deb version
https://youtu.be/zSWeHzmaOvY

Here is a video of the snap version with debug log
https://youtu.be/xzwOoYCQHR8

[robertclarkson]

I've also boosted the swap file up to 64GB and it still eats it all