Unable to get historical data from CEX.io

[askmike]

Since a couple of days the CEX.io API does not give back old trades anymore (using the since parameter). After contacting support they stated:

Hi Mike,

We have limited our trade_history call to 1000 trades now, We will be implementing a daily trade static file in a future update I apologize for the inconvenence and we will try to get this out ASAP.

Which means that until the static file is offered we need to redesign CEX.io implementation to:

• on gekko start, start gathering trades and save them in the database (this functionality is already there). But don't calculate EMA's or give any advice yet.
• Monitor how many trades we have stored in the database, as soon as the oldest data we have is old enough to calculate the start EMA's (config.EMA.candles * config.EMA.interval) start giving advice and trade.

[nicobrion]

looking forward for the CEX.io historical data fix. I truly congratulate you for an outstanding dev, this project is awesome!
Kudos,

Nico

[askmike]

Thanks!

I am working on a new version of Gekko that is able to store history locally, this means it is able to monitor:

• CEX.io
• BTC-e (including altcoin markets)
• Bitstamp (was broken for a long time)
• All other API's that don't offer historical data.

So far I have written the functionality that can monitor the exchange and store candles. If people could test it that would be awesome!

Test new Gekko

• Go to the dev branch and download it.
• Cd to it & npm install
• Set the config to either cexio, bitstamp or btce. (Mt. Gox not implemented yet, just as: real trading / mailer / advice). [edit: and pick a new market here]
• Run it
• History is now saved in the history folder.
• Give me feedback!

[fmjensen]

Is this okay ?
I dont know what to expect, so you just got the output here :

gekko-localDB : node gekko.js 
2013-12-24 10:48:25 (INFO): I'm gonna make you rich, Bud Fox.
2013-12-24 10:48:25 (INFO): Let me show you some Exponential Moving Averages.
2013-12-24 10:48:25 (INFO): Using normal settings to monitor the live market
2013-12-24 10:48:25 (INFO): NOT trading with real money
2013-12-24 10:48:25 (INFO): Starting to watch the market: CEX.io BTC/GHS
2013-12-24 10:48:25 (DEBUG):    Scheduling next fetch: in 1 minutes
2013-12-24 10:48:25 (DEBUG):    Fetched 1001 new trades, from 09:29:18 (UTC) to 09:48:23 (UTC)
2013-12-24 10:48:25 (DEBUG):    Creating a new daily database for day 2013-12-24
2013-12-24 10:48:25 (DEBUG):    inserting candle 569 (09:29:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 570 (09:30:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 571 (09:31:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 572 (09:32:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 573 (09:33:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 574 (09:34:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 575 (09:35:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 576 (09:36:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 577 (09:37:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 578 (09:38:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 579 (09:39:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 580 (09:40:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 581 (09:41:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 582 (09:42:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 583 (09:43:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 584 (09:44:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 585 (09:45:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 586 (09:46:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 587 (09:47:00 UTC)
2013-12-24 10:48:25 (DEBUG):    Leftovers: 588
2013-12-24 10:49:26 (DEBUG):    Scheduling next fetch: in 1 minutes
2013-12-24 10:49:26 (DEBUG):    Fetched 1001 new trades, from 09:29:52 (UTC) to 09:49:25 (UTC)
2013-12-24 10:49:26 (DEBUG):    inserting candle 588 (09:48:00 UTC)
2013-12-24 10:49:26 (DEBUG):    Leftovers: 589
2013-12-24 10:50:27 (DEBUG):    Scheduling next fetch: in 1 minutes
2013-12-24 10:50:27 (DEBUG):    Fetched 1001 new trades, from 09:29:52 (UTC) to 09:50:26 (UTC)
2013-12-24 10:50:27 (DEBUG):    inserting candle 589 (09:49:00 UTC)
2013-12-24 10:50:27 (DEBUG):    Leftovers: 590
2013-12-24 10:51:27 (DEBUG):    Scheduling next fetch: in 1 minutes

[askmike]

Thanks for testing :)

Looks great, I'll explain some things:

• Why so much logging? I am still developing it and I want all feedback I can get.
• You now have a new history folder (inside Gekko) that has files like cexio-BTC-GHS-2013-12-24.db, this is all the trade information Gekko was able to store!
• The times are off: no they are not, they're in in UTC. The reason for this is that everyone from over the world can share historical candles and the timezones doesn't mess everything up.

One small thing I am going to look into though: It shouldn't fetch once every minute (doesn't cause any trouble, but a lot of overhead).

[fmjensen]

I configured it to fetch every minute, and run in debug mode to get some activity on stdout ;-)
The time fits my timezone.
Im running on OSX btw...

[askmike]

Ah, that explains it. And the answers were also for clarifying the new logic for everyone.

---

On CEX.io [and BTC-e] you only get the last 1000 [or 150] trades, whether they happened in a minute or over the last hour. This makes it quite hard to determine when to fetch. But I quite like how the current setup deals with it: after every fetch it calculates a new fetchtime based on trades/min from the previous fetch. It appears to do well but it needs to prove itself over langer periods of time.

Here is a log where you can see this in action:

Fetched 1001 new trades, from 12:06:49 (UTC) to 12:07:19 (UTC)
inserting candle 726 (12:06:00 UTC)
Leftovers: 727
Scheduling next fetch: in 1 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:08:17 (UTC)
inserting candle 727 (12:07:00 UTC)
Leftovers: 728
Scheduling next fetch: in 2 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:09:22 (UTC)
inserting candle 728 (12:08:00 UTC)
Leftovers: 729
Scheduling next fetch: in 3 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:11:00 (UTC)
inserting candle 729 (12:09:00 UTC)
inserting candle 730 (12:10:00 UTC)
Leftovers: 731
Scheduling next fetch: in 5 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:13:52 (UTC) 
inserting candle 731 (12:11:00 UTC)
inserting candle 732 (12:12:00 UTC)
Leftovers: 733
Scheduling next fetch: in 8 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:18:34 (UTC)
inserting candle 733 (12:13:00 UTC)
inserting candle 734 (12:14:00 UTC)
inserting candle 735 (12:15:00 UTC)
inserting candle 736 (12:16:00 UTC)
inserting candle 737 (12:17:00 UTC)
Leftovers: 738
Scheduling next fetch: in 13 minutes

I do want to make it more aggressive, missing trades is not acceptable.

[fmjensen]

I like it !
But... i miss the ADVICE in stdout - and can't figure out if it can be enabled somehow?!

[askmike]

Well this new change means that I need to rewrite how Gekko deals with data under the hood. Both the historicalCandleFetcher.js and the realtimeCandleFetcher.js are going to be obsolete and they provided basically all internal communication (from fetching to passing to trading method, etc).

I am working on it, but it's a lot of work.

[nicobrion]

I downloaded the localDB version and ran some test, but it only seems to monitor candle behaviour, is that right?
Should I send you the logs?
if you need a volunteer to test the trading feature, count me in.

[bppsp1]

Is the test version able to trade yet or is it just at the candle/advice section for now ?

[streetlogics]

Hey - CEX seems to be running fine, but I get this error when I try to launch with Mt. Gox now (this is with debug turned on):

2013-12-26 08:37:43 (INFO): I'm gonna make you rich, Bud Fox.
2013-12-26 08:37:43 (INFO): Let me show you some Exponential Moving Averages.


2013-12-26 08:37:43 (INFO): Using normal settings to monitor the live market
2013-12-26 08:37:43 (INFO): NOT trading with real money
2013-12-26 08:37:43 (INFO): Starting to watch the market: MtGox USD/BTC
either start looping right away (`since`) or first determine starting point dynamically

[askmike]

@bppsp1 nope, not yet. I'll gladly accept pull requests though :)

@streetlogics, the code works in that it takes different paths for different situations, the situations are:

• No historical data available. (BTC-e, bitstamp, CEX.io)
• Historical data available request-able via tid, while we need trades based on a point in time. (BTCChina I think)
• Historical data available request-able via timestamp (MtGox)

I am still finishing up the first scenario. Though because you can always request historical data from Mt. Gox, you can do anything I'm trying to achieve already (except store the historical data) using the master branch.

[streetlogics]

Cool cool - no worries if you're aware of it then, just wanted to let you know just in case. Happy to test again on the branch once it's implemented.

[djmuk]

Mike - looking forward to the trade engine implementation...
Have been running this version and just noticed that it didn't put any data in the new day file - it created them but they just had the "{"$$indexCreated":{"fieldName":"s","unique":true,"sparse":false}}" line in them. Restarting gekko gave me a rash of 'creating new realtime candle' and todays file got populated... Having been running since 26th, I have an empty file for the 27th and todays only got populated when I restarted.

Just another suggestion while looking at candle generation - would it be possible to also store volume weighted price (VWP) as well and also to give the option to use this in calcs rather than the centre price? This would mean iterating the trade data set but this could be calculated when assigning them to candles (and wouldn't it be as easy to pick up the OHCL figures at this point rather than using first/last/min/max later?)

Thanks

[kkwestt]

kwest$ node gekko
2013-12-28 16:37:15 (INFO): I'm gonna make you rich, Bud Fox.
2013-12-28 16:37:15 (INFO): Let me show you some Exponential Moving Averages.

2013-12-28 16:37:15 (INFO): Using normal settings to monitor the live market
2013-12-28 16:37:15 (INFO): NOT trading

/Users/kwest/gekko/gekko.js:60
throw invalid;
^
At this moment Gekko can't monitor CEX.io, find out more info here:

#90

[askmike]

@kkwestt did you read the issue? The rest of the discussion is not about a stable version. Read more here.

@djmuk Thanks. I pushed a new version a couple of hours ago that should fix the daily DB creation. I like the idea for the VWP, will add it to the candles.

[kkwestt]

ok
i run new gekko

172-0-0-7:gekko2 kwest$ node gekko
2013-12-28 17:03:02 (INFO): I'm gonna make you rich, Bud Fox.
2013-12-28 17:03:02 (INFO): Let me show you some Exponential Moving Averages.


2013-12-28 17:03:02 (INFO): Using normal settings to monitor the live market
2013-12-28 17:03:02 (INFO): NOT trading with real money
2013-12-28 17:03:02 (INFO): Starting to watch the market: CEX.io BTC/GHS
2013-12-28 17:03:03 (DEBUG):    Scheduling next fetch: in 5 minutes
2013-12-28 17:03:03 (DEBUG):    Fetched 1001 new trades, from 2013-12-28 12:17:56 (UTC) to 2013-12-28 13:02:45 (UTC)
TRADES 1001
TRADES 12:17:56
TRADES 13:02:45
MINIMUM: 2013-12-28 13:00:00
FILTERED TRADES 16
FILTERED TRADES 2013-12-28 13:00:04
FILTERED TRADES 2013-12-28 13:00:04
FILTERED TRADES 2013-12-28 13:00:05
FILTERED TRADES 2013-12-28 13:00:23
FILTERED TRADES 2013-12-28 13:00:34
FILTERED TRADES 2013-12-28 13:00:34
FILTERED TRADES 2013-12-28 13:00:43
FILTERED TRADES 2013-12-28 13:01:03
FILTERED TRADES 2013-12-28 13:01:04
FILTERED TRADES 2013-12-28 13:01:04
FILTERED TRADES 2013-12-28 13:01:20
FILTERED TRADES 2013-12-28 13:02:03
FILTERED TRADES 2013-12-28 13:02:04
FILTERED TRADES 2013-12-28 13:02:13
FILTERED TRADES 2013-12-28 13:02:45
FILTERED TRADES 2013-12-28 13:02:45
2013-12-28 17:03:03 (DEBUG):    inserting candle 780 (13:00:00 UTC) vol: 3.43184006
2013-12-28 17:03:03 (DEBUG):    inserting candle 781 (13:01:00 UTC) vol: 0.99916237
2013-12-28 17:03:03 (DEBUG):    Leftovers: 782

[askmike]

@kkwestt I consider it stable when it runs a while on busy markets as well as empty ones. Could you let me know if it crashes or anything like that?

I think I got it almost stable now:

• It now also adds empty candles, it does this to differentiate between times when gekko was off and times when there were no trades at the exchange. The result is that all candles generated before (unless there was a trade in every minute) are now considered corrupt.
• fixed lots of bugs around filling gaps with empty candles (between two fetches, around midnight, etc.)
• added VWP
• lots of other bug fixes.

[dasher]

The latest tip of localdb is borking out with JS Allocation failed - process out of memory - trying to track down what & where currently..

[askmike]

@dasher very strange? Could you post (or link to) the output with debug enabled? Haven't seen that behaviour yet. Also: on what market is this happening and do you have a history build up with previous versions?

[GoodDragon]

I started running localDB branch yesterday on two instances. Both of them was crached on last night.

---------------------------------------- First instance ----------------------------------------
Config:
exchange: 'BTCe', // 'MtGox', 'BTCe', 'Bitstamp' or 'cexio'
currency: 'USD',
asset: 'BTC',

Last lines from log:

2013-12-31 02:00:35 (DEBUG):    Creating a new daily database for day 2013-12-31
2013-12-31 02:00:35 (DEBUG):    inserting candle 1439 (2013-12-30 23:59:00 UTC) volume: 7.035317699999999
2013-12-31 02:00:35 (DEBUG):    Leftovers: 0
2013-12-31 02:00:35 (INFO):     Processed trades, sleeping for a minute ...
2013-12-31 02:01:47 (DEBUG):    Requested trade data from BTC-e ...
2013-12-31 02:01:47 (DEBUG):    Fetched 150 new trades, from 2013-12-30 23:55:13 (UTC) to 2013-12-31 00:01:27 (UTC)
2013-12-31 02:01:47 (DEBUG):    minimum trade treshold: 2013-12-31 00:00:28 UTC
2013-12-31 02:01:47 (DEBUG):    processing 44 trade(s)

Weird error

---

When I started it again:

2013-12-31 15:44:00 (INFO):     Using normal settings to monitor the live market
2013-12-31 15:44:00 (INFO):     NOT trading with real money
2013-12-31 15:44:00 (INFO):     Starting to watch the market: BTC-e USD/BTC
2013-12-31 15:44:00 (DEBUG):    Requested trade data from BTC-e ...
2013-12-31 15:44:01 (DEBUG):    Fetched 150 new trades, from 2013-12-31 13:43:41 (UTC) to 2013-12-31 13:43:54 (UTC)
2013-12-31 15:44:01 (WARN):     Found a corrupted database ( 2013-12-31 ), going to clean it up
2013-12-31 15:44:01 (DEBUG):    This should not happen, please post details here: https://github.com/askmike/gekko/issues/90
BAIL
2013-12-31 15:44:01 (INFO):     No history found, starting to build one now

---------------------------------------- Second instance ----------------------------------------

Config:
  exchange: 'BTCe', // 'MtGox', 'BTCe', 'Bitstamp' or 'cexio'
  currency: 'BTC',
  asset: 'LTC',

Last lines from log:

2013-12-31 01:59:22 (DEBUG):    NEW REAL CANDLE
2013-12-31 01:59:22 (DEBUG):    inserting candle 1437 (2013-12-30 23:57:00 UTC) volume: 0
2013-12-31 01:59:22 (DEBUG):    Leftovers: 1438
2013-12-31 01:59:22 (INFO):     Processed trades, sleeping for 2 minutes ...
2013-12-31 02:01:22 (DEBUG):    Requested trade data from BTC-e ...
2013-12-31 02:01:23 (DEBUG):    Fetched 150 new trades, from 2013-12-30 23:30:34 (UTC) to 2013-12-31 00:01:20 (UTC)
2013-12-31 02:01:23 (DEBUG):    minimum trade treshold: 2013-12-30 23:58:12 UTC
2013-12-31 02:01:23 (DEBUG):    processing 6 trade(s)
2013-12-31 02:01:23 (DEBUG):    This batch includes trades for a new day.
2013-12-31 02:01:23 (DEBUG):    Creating a new daily database for day 2013-12-31

TypeError: Cannot set property 's' of undefined.  at Manager.processTrades (/home/user11/gekko/databaseManager.js:372:19)
    at bound (/home/user11/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at EventEmitter.emit (events.js:117:20)
    at bound [as emit] (/home/user11/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at Fetcher.processTrades (/home/user11/gekko/tradeFetcher.js:163:8)
    at bound (/home/user11/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at process (/home/user11/gekko/exchanges/btce.js:135:7)
    at bound (/home/user11/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at Request._callback (/home/user11/gekko/node_modules/btc-e/btc-e.js:76:7)
    at Request.self.callback (/home/user11/gekko/node_modules/btc-e/node_modules/request/index.js:142:22)

Works after restart but probably will crash on next night again.

Anyway. I will now update latest version and try again.

[askmike]

@GoodDragon that should have been resolved by this commit. Let me know what happens next time.

@yin Totally agree, though because of the huge todo I have not written tests yet. Also because of the hard to test edgecases around having parted history / receiving weird fetch data it's hard to write bullet proof testing. Though I'm very interested in building a test suite asap. Pull requests more than welcome!

[yin]

Already working on tests, well one is already passing - Store.read(). I
mean the CSV store from your gist. Correct me, if I am on wrong track.

I used nodeunit, so we can have test framework and examples at hand. Have a
look if you're interested.

https://github.com/yin/gekko/commits/localDB

I'd also copy in my timing code from yesterday as a nice module.

Regards / S pozdravom

Matej Gagyi

On Tue, Dec 31, 2013 at 3:28 PM, Mike van Rossum
notifications@github.comwrote:

@GoodDragon https://github.com/GoodDragon that should have been
resolved by this commithttps://github.com/askmike/gekko/commit/80c3c51e921443925d8c979c6f9d510e5923ec48.
Let me know what happens next time.

@yin https://github.com/yin Totally agree, though because of the huge
today I have not written tests yet. Also because of the hard to test
edgecases around having parted history / receiving weird fetch data it's
hard to write bullet proof testing. Though I'm very interested in building
a test suite asap.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/90#issuecomment-31396383
.

[ArgyleEl1te]

I've got the local database version up and running, but it won't let me change out of BTC/TRC....

// Monitor the live market
config.normal = {
  enabled: true,
  exchange: 'btce', // 'MtGox', 'BTCe', 'Bitstamp' or 'cexio'
  currency: 'BTC',
  asset: 'TRC',
  tradingEnabled: true,

when I change to

  currency: 'USD',
  asset: 'BTC',

I get this:

/Users/dave/Downloads/gekko-localDB/config.js:39
  currency: ‘USD’,
            ^
SyntaxError: Unexpected token ILLEGAL
    at Module._compile (module.js:439:25)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.util.getConfig (/Users/dave/Downloads/gekko-localDB/core/util.js:14:15)
    at Object.<anonymous> (/Users/dave/Downloads/gekko-localDB/core/util.js:105:19)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)

am I missing something obvious? haven't seen anyone else mention this...

[askmike]

@ArgyleEl1te it seems you are using the wrong quotes, they should not be curly but normal ones. (if unsure copy quotes from somewhere else and change the USD inside).

Note that Gekko watches per market, so you do need to rebuild a history for this other one.

[ArgyleEl1te]

@askmike wow... not sure how that happened. Even copied and pasted out of the old config.js from the previous version... but yes, you are right. Wiped out databases, it fired up, excited to see how it goes. Thanks for the 5 minute help turnaround!

[ArgyleEl1te]

Ran for about 4 hours, then threw this error and crashed:

2013-12-31 17:59:53 (INFO): Processed trades, sleeping for 2 minutes ...
2013-12-31 18:01:28 (DEBUG): Requested trade data from BTC-e ...
2013-12-31 18:01:28 (DEBUG): Fetched 150 new trades, from 2013-12-31 23:57:32 (UTC) to 2014-01-01 00:01:18 (UTC)
2013-12-31 18:01:28 (DEBUG): minimum trade treshold: 2013-12-31 23:59:51 UTC
2013-12-31 18:01:28 (DEBUG): processing 47 trade(s)
2013-12-31 18:01:28 (DEBUG): This batch includes trades for a new day.
2013-12-31 18:01:28 (DEBUG): Creating a new daily database for day 2014-01-01

TypeError: Cannot set property 's' of undefined
at Manager.processTrades (/Users/dave/Downloads/gekko-localDB/core/databaseManager.js:373:19)
at bound (/Users/dave/Downloads/gekko-localDB/node_modules/lodash/dist/lodash.js:729:21)
at EventEmitter.emit (events.js:117:20)
at bound as emit
at Fetcher.processTrades (/Users/dave/Downloads/gekko-localDB/core/tradeFetcher.js:163:8)
at bound (/Users/dave/Downloads/gekko-localDB/node_modules/lodash/dist/lodash.js:729:21)
at process (/Users/dave/Downloads/gekko-localDB/exchanges/btce.js:135:7)
at bound (/Users/dave/Downloads/gekko-localDB/node_modules/lodash/dist/lodash.js:729:21)
at Request._callback (/Users/dave/Downloads/gekko-localDB/node_modules/btc-e/btc-e.js:76:7)
at Request.self.callback (/Users/dave/Downloads/gekko-localDB/node_modules/btc-e/node_modules/request/index.js:142:22)

Loaded it back up and it gave me this:
2013-12-31 19:16:42 (INFO): I'm gonna make you rich, Bud Fox.

2013-12-31 19:16:43 (INFO): Starting to watch the market: BTC-e USD/BTC
2013-12-31 19:16:43 (DEBUG): Requested trade data from BTC-e ...

2013-12-31 19:16:43 (INFO): Seting up a new price actor:
2013-12-31 19:16:43 (INFO): Trade advisor
2013-12-31 19:16:43 (INFO): Calculate trading advice based on the Exponential Moving Average Crossovers

2013-12-31 19:16:48 (DEBUG): Fetched 150 new trades, from 2014-01-01 00:48:42 (UTC) to 2014-01-01 01:16:24 (UTC)
2013-12-31 19:16:48 (WARN): Found a corrupted database ( 2014-01-01 ), going to clean it up
2013-12-31 19:16:48 (DEBUG): This should not happen, please post details here: #90
2013-12-31 19:16:48 (INFO): No history found, starting to build one now
2013-12-31 19:16:48 (INFO): Expected to start giving advice in 3 hours (2014-01-01 04:36:48 UTC)
2013-12-31 19:16:48 (DEBUG): Creating a new daily database for day 2014-01-01
2013-12-31 19:16:48 (DEBUG): minimum trade treshold: 2014-01-01 00:00:00 UTC
2013-12-31 19:16:48 (DEBUG): processing 150 trade(s)
2013-12-31 19:16:48 (DEBUG): inserting candle 48 (2014-01-01 00:48:00 UTC) volume: 0.207574
2013-12-31 19:16:48 (DEBUG): inserting candle 49 (2014-01-01 00:49:00 UTC) volume: 0.128544
2013-12-31 19:16:48 (DEBUG): inserting candle 50 (2014-01-01 00:50:00 UTC) volume: 0
2013-12-31 19:16:48 (DEBUG): inserting candle 51 (2014-01-01 00:51:00 UTC) volume: 1.4048787200000001
2013-12-31 19:16:48 (DEBUG): inserting candle 52 (2014-01-01 00:52:00 UTC) volume: 0
2013-12-31 19:16:48 (DEBUG): inserting candle 53 (2014-01-01 00:53:00 UTC) volume: 0

So, corrupted database somehow.... FYI...

[askmike]

@ArgyleEl1te hmm the first error is weird, let me know if it still happens...

For the second one I am going to write a document explaining the issue, but it boils down to: it's not really corrupt but just incomplete:

You started it 75 minutes after the crash, but BTC-e most likely won't tell you what happened before a couple of minutes.

you should think of it like this: you are trying to count the cars on the road for 24 hours
if you are watching for 23 hours and counted how many cars passed by you know a lot. But if you take a break for the last hour you don't know the total
and you can never make assumptions about what happened during breaks if you want to make financial decisions.
(or give advice)

---

So I created a new issue that hosts the TODO before I consider all the localDB stuff stable (and will merge this into master):

#114

[kingcrunch]

Hi. Didn't follow up the whole discussion, but the app told me, I should contact you 😕

If you see this please let me know the following:

~Sat Jan 04 2014 18:38:56 GMT+0000
~Sat Jan 04 2014 19:08:00 GMT+0000

Btw: Local time was 20:08 (MEZ/GMT+1)

[askmike]

Hi. Didn't follow up the whole discussion, but the app told me, I should contact you

Awesome thanks! So my question is: did it crash it did it work?

[kingcrunch]

@askmike Actually it looks like it works. At least no crash. 😉

[trader-cheat-codes]

May I recommend removing the "If you see this please let me know the following:"
I've been running this in two different instances (mtgox btc-e)since the midnight commit. No issues yet and the history looks good. Gotta say I'm impressed at this from looking over localDB and look forward to working on this project in the future.

[askmike]

@trader-cheat-codes I'm still working on that code and there still is one situation where it crashes after that. Gonna be out soon!

[ejko]

i have this error :

user@host:/gekko$ node gekko
2014-01-18 03:39:06 (INFO): I'm gonna make you rich, Bud Fox.
2014-01-18 03:39:06 (INFO): Let me show you some Exponential Moving Averages.
2014-01-18 03:39:06 (INFO): Using normal settings to monitor the live market
2014-01-18 03:39:06 (INFO): NOT trading with real money
/home/user/gekko/gekko.js:60
throw invalid;
^
At this moment Gekko can't monitor CEX.io, find out more info here:
#90
user@host:/gekko$

do I have a mistake or a problem with the source?

[askmike]

@ejko it depends a little on what you want:

• There is a stable Gekko (the one you downloaded), this one does not have support for CEX.io but can trade based on EMA on Mt. Gox or BTC-e.
• There is an unstable Gekko which does have support for CEX.io and advice using EMA, MACD or PPO. However you can't trade with this version yet and it is not stable.

[w1ckd]

Hey,

Ran the Gekko-LocalDB for 48 hours to monitor LTC-USD with no isseus.
A few Trade propositions were made.

Not sure if you need a log from a working one?

Cant wait to test the Auto Trade future for Altcoin! :)

With friendly Regards,
R

[ejko]

I apologize again, I'm new in this system.
how to install unstable Gekko ?

[askmike]

Because the unstable and unverified state of the new version I've disabled real trading for now. All code is located in another branch (like a nightly build), to get it download and install git and run:

git clone -b localDB --single-branch git://github.com/askmike/gekko.git
cd gekko
npm install

and follow instructions in the readme.

[gr225607]

hope this helps

2014-01-27 04:53:45 (INFO): I'm gonna make you rich, Bud Fox.

2014-01-27 04:53:45 (INFO): Setting up Gekko in realtime mode

2014-01-27 04:53:45 (INFO): Setting up:
2014-01-27 04:53:45 (INFO): Trading Advisor
2014-01-27 04:53:45 (INFO): Calculate trading advice
2014-01-27 04:53:45 (INFO): Using the trading method: MACD

2014-01-27 04:53:45 (INFO): Setting up:
2014-01-27 04:53:45 (INFO): Profit Simulator
2014-01-27 04:53:45 (INFO): Paper trader that logs fake profits.

2014-01-27 04:53:46 (INFO): Setting up:
2014-01-27 04:53:46 (INFO): Profit Simulator
2014-01-27 04:53:46 (INFO): Paper trader that logs fake profits.

2014-01-27 04:53:48 (INFO): Starting to watch the market: BTC-e BTC/USD
2014-01-27 04:53:49 (INFO): We don't have enough history yet to start giving advice, I'll need to gather more data first.
2014-01-27 04:53:49 (INFO): I will start giving advice around 2014-02-19 17:54:00 (local time, that's in 24 days).
2014-01-27 20:21:10 (WARN): DOUBLE UNIQUE INSERT, this should never happen. Please post details here: #90 2014-01-27 undefined { [Error: ENOSPC, write] errno: 54, code: 'ENOSPC' }

/home/pi/gekko-localDB/core/candleManager.js:898
throw err;
^
Error: ENOSPC, write

[askmike]

Excited to announce that the new localDB branch is out. Trading is CEX.io is working and stable.