Zenbot suddenly stops during longer simulations

[chriscrutt]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): OSX
• Zenbot version (commit ref, or version): 4.1.0
• Zenbot branch: Master
• NodeJS version: v10.16.0
• Exact command to reproduce (include everything): zenbot sim --strategy trend_ema --period_length 2m
• Did I make any changes to conf-sample.js?: c.days = 300 and c.currency_capital = 10000

Describe the problem

During longer simulations (~ more than 57 days for trend_ema) no matter the period, it suddenly stops without giving an output file or anything. It is different lengths depending on the strategy but none of them go ~ more than 100

Source code / Error logs

--debug doesn't give any error logs not does anything else I can find, it just suddenly stops

[hamedheidari]

Also stops during paper trading over HitBTC after a few hours.
(zenbot 4.1.0)

[DJCrashdummy]

unfortunately i have to confirm this behaviour!

---

to me it seems to be partly a RAM (leaking?) problem... so setting c.output.api.on to false helped a lot for the trade command. 🎉

---

for sim IMHO it depends on the complexity of the strategy resp. even more on the number of trades executed. with my setup it is even hard to complete the simulation of some "exciting" resp. volatile months.
sometimes just retrying the simulation a bunch of times until zenbot completes it, works... sometimes restarting mongodb or the whole machine (especially if you have run many sims, long backfills etc. already) helps... especially if the simulation stops not so far away before completing.

i wrote a little script to automatize the retry if the simulations fails, and most time after ~10-100 tries i get a completed simulation. but sometimes it seems impossible and even after ~1000 attempts there is not one completed simulation, so the only way is to split the timespan. 😞

[jorisw]

Having the same problem with zenbot 4.1.0 on Ubuntu 18 and node v10.15.2, but I don't think it's a memory leak. I can see my available memory go down and up when monitoring top.

Unfortunately I can't even get zenbot sim --days 100 --strategy bollinger to complete. No output, no error, just returns to the commandline halfway.

I've tried increasing Node's memory limit:

export NODE_OPTIONS=--max_old_space_size=8192

No difference.

I notice that every time, before zenbot prematurely stops, it freezes for 30 seconds first. Then it exits with code 0. I can't find anything in commands/sim.js that does this.

[jorisw]

I've added some debug to the code and I notice that often, the reason for the premature termination of zenbot seems to be that it has reached the end of the Mongo collection of trades that it is iterating through. Hoping that this may be a hint towards finding the cause. Perhaps it's related to holes in the backfilled history?

[DJCrashdummy]

hmmm... 🤔 but in this case, splitting the time span wouldn't/shouldn't change anything? - but at least at my setup, this always helps to get a complete simulation.

and running backfill for the same time span several times over the last year didn't change anything... it never downloaded any additional data or changed anything at the simulations behaviour. 😐

[mmdiego]

I've never seen this behaviour before, until last week when I updated Node to v10.16 because of some dependency package failing to install.
The same simulations I was running with no problem in an older version of Node, started to failed. I firstly ignored the problem, cause if I tried a couple of times, the simulation completed. But this got annoying so I decided to go back to Node v8.3, rebuild packages (ignored talib compilation errors) and voilá, everything is working ok again.

I used npm "n" package to switch between different node versions. Very simple to use.(https://www.npmjs.com/package/n?activeTab=versions)

[jorisw]

Indeed, switching back to Node v8.3 prolongs operation. Even without reinstalling node_modules. However eventually the bot quits without errors, way before the backfill is exhausted.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jorisw]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs

A very puzzling practice. So existing problems that bear no further discussion, no longer exist?

[jorisw]

FWIW I still have this problem, even with node 8.3, with node modules rebuilt. Just on 8.3 it happens less often than with higher versions of node, both on Mac and Linux.

[jorisw]

The cause lies in the bottom of commands/sim.js, in which it is assumed that the mongo collection's .on('end') comes last, after any .on('data') event. This isn't always the case with MongoDB, therefore the getNext() function sometimes isn't fired. This leads to the Node event queue emptying up, with the bot exiting because it has nothing further to do.

I've fixed this for myself by rewriting this section of commands/sim.js, to count the records coming out of the cursor, and firing getNext() when the end of the cursor is reached by way of trade counts.

I may submit a PR if I think my changes are universally suitable.

In any case the problem has nothing to do with Node.js versions.