Crash when Gekko has full history on second day

[askmike]

it appears we don't load the second day in correctly.

debug log:

2014-01-21 20:29:00 (INFO): Setting up:
2014-01-21 20:29:00 (INFO):    Profit Simulator
2014-01-21 20:29:00 (INFO):    Paper trader that logs fake profits.

2014-01-21 20:29:00 (INFO): Setting up:
2014-01-21 20:29:00 (INFO):    Profit Simulator
2014-01-21 20:29:00 (INFO):    Paper trader that logs fake profits.

2014-01-21 20:29:00 (INFO): Starting to watch the market: BTC-e BTC/USD
2014-01-21 20:29:01 (INFO): We don't have enough history yet to start giving advice, I'll need to gather more data first.
2014-01-21 20:29:01 (INFO): I will start giving advice around 2014-01-22 21:03:00 (local time, that's in a day).
2014-01-22 21:01:21 (INFO): We have all required history, going to calculate advice from this point.

/root/gekko/core/candleManager.js:560
  this.days[mom.dayString].handle.find({
                                  ^
TypeError: Object #<error> has no method 'find'
    at Manager.getCandles (/root/gekko/core/candleManager.js:560:35)
    at bound [as getCandles] (/root/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at iterator (/root/gekko/core/candleManager.js:516:10)
    at bound (/root/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at /root/gekko/node_modules/async/lib/async.js:224:13
    at /root/gekko/node_modules/async/lib/async.js:108:13
    at Array.forEach (native)
    at _each (/root/gekko/node_modules/async/lib/async.js:32:24)
    at async.each (/root/gekko/node_modules/async/lib/async.js:107:9)
    at _asyncMap (/root/gekko/node_modules/async/lib/async.js:223:9)

[l33tconsultant]

Gekko need to be able to retrieve history from a source (like bitcharts). I've been running for 3 days, and can't make a trade b/c I can't be at terminal exactly every 24 hours to catch this error, and restart. A watchdog module would be a hack to this problem. A wait time of 24 is a bit much to ask a user to wait to use the software.

[askmike]

I can't be at terminal exactly every 24 hours to catch this error, and restart.

The bug only happens when the history becomes full and when that happens on the day after startup day. If you have full history this bug won't crash Gekko anymore.

If you want a stable Gekko go for the master branch, this is a new version with known bugs that we are currently in the process of writing. If you don't like the hacky solution please fix the bug yourself or wait until this is stable because I have not been able to solve it yet. The master branch also fetches Bitcoincharts.

So we had some discussions about this before and I really don't like bitcoincharts: Not all their data is calculated 100% correct and the API is not really that stable (see #77 and #24). Here is some more discussion about better solutions. Note that I did already setup Gekko in way that people can just send daily database files to each other (even if they are in different timezones etc, days are UTC based), the only hard thing is trusting the source.

This software is currently not focused or intended as end user software, eg. I'm working on a hosted service that watches all markets and does not require you to save up history or leave something running (platform is built on top of Gekko). In the future we might host pre recorded data so it can fetch on startup.

[boxxa]

I am testing a config option. I am fetching historical data and the gecko being able to pull the pair from the exchange store I built. Still testing but starting to collect data.

[l33tconsultant]

@askmike thanks for the reply. You seemed a little steamed from my hack comment. it shouldn't be me where you place your energy. I'm just a messenger / tester / programmer / user. My comments are not reflective of your application, just your attitude towards, in your words.. "Bugs" - I don't want to flame - I just wanted to share & contribute what I can to gekko.

Thanks boxxa.

[djmuk]

haven't seen this on my setup - is this only an issue if the the 'collecting history' process spans midnight or is >24 hours?
@L33T - perhaps reduce candle size/required history which will reduce the time until gekko is giving advice? It would seem to me that as long as the history >= the long ema period then the figures should be 'close enough', after all the point of the ema is that the influence of the older values gets less as they go back in time.

[PascalPixel]

I'd love for Gekko to have some shared history, perhaps hosted as a simple JSON somewhere to pull from?

[askmike]

@djmuk it only happens sometimes when you require a history which will be complete on the next second day (UTC) since startup.

Instead of reducing the historySize it would be more wise to run Gekko with the advisor disabled for a couple of days and when you have time enable it with a full history and restart.

@Superpencil we have some ideas for this already. The hard thing is trusting the dataprovider.

[djmuk]

@mike would a (temporary) workaround be to collect the history in smaller chunks - say up to 6 hours or so - or collect history up to midnight and then restart?

[askmike]

No the best workaround would be to first run Gekko without a trading advisor to save up history, and after a few days restart it with. This bug can only happen on full history, an event only triggered with a trading advisor enabled and it won't happen on start (since start day == day on full history).

If you want to run it for 6 hours just before midnight UTC you can still encounter this error.

that said I am pretty sure I almost got this bug.

[cummingsrob]

I'm just recently seeing this bug after moving from 'master' to 'candle-rewrite'. 'master' was running Ok for days, but 'candle-rewrite' stops after just an hour or so. I've reduced the history significantly and it crashes with the NaN values.

[askmike]

@cummingsrob the candle-rewrite branch is not finished, it doesn't:

• give trades to the trading methods or plugins (so nothing is working)
• write trades to a database.

That said, are you sure Gekko stops on that branch? Could you update to last commit and check again?

[cummingsrob]

I'm sure it was that branch. I'll update in a bit. I don't know how I got on that branch in the first place. I'm relatively new to Git. I'm not a programmer, but I'll try to help out how ever I can.

[askmike]

fixed!