dynamic/configurable exchange polling interval

[valeriominetti]

While gekko is scanning the exchange it sometimes gets this kind of errors. Is it possible to define how often to poll exchange for new trade data? this would help me to pin if it's some kind of flood protection on cex or some firewall dropping connections.

2016-06-21 07:53:26 (DEBUG): Processing 3 new trades.
2016-06-21 07:53:26 (DEBUG): From 2016-06-21 07:53:03 UTC to 2016-06-21 07:53:03 UTC. (a few seconds)
2016-06-21 07:53:35 (DEBUG): Requested / trade data from CEX.io ...
2016-06-21 07:53:40 (DEBUG): cex.io returned an error, retrying.. [SyntaxError: Unexpected end of input] waiting for 10000 ms
2016-06-21 07:53:56 (DEBUG): Requested / trade data from CEX.io ...

[askmike]

This used to be configurable, currently it is not (This could be added again, it should probably be a property on each exchange object in exchange.js). If you want to play with it yourself, you can do that here for now.

[askmike]

After looking at more code I see some problems with the current setup:

The flow was meant to be:

• (sec 0) fetch trades
• (sec 1) if fetch fails: keep retrying every second until it works (wait 1 seond per try, for a maximum of 20 seconds after "fetch trades")
• (sec 2) retry fetched good data, pass on
• (sec 20) repeat fetch trades

All this code is there however it is not hooked up correctly:

The getTrades function from each exchange (in the exchanges folder) is meant to return an error when it fails. However most exchanges are currently retrying themselves (indefinitely). So the retry mechanism that handles the above flow (in marketFetcher) is never used.

A problem with this is that there is now a new piece of code that gets initial market data to see if Gekko can sync up trade data from local database to the the exchange, and this code cannot handle retries (I think we should replace the descending parameter in getTries with a retry flag, indicating who should handle retries.

[s4p0]

Hey, hi!
What about using sockets? Bitstamp has web sockets notifying you about trades, this is faster than pooling it out. I've managed to "integrate" it on my really special way (not beautiful, but it worked).
Probably would be a good idea to structure this kind of use.

[askmike]

Yes, definitely a good idea! We also need websockets for other exchangs (like OKcoin, because of the amount of trades). Gekko works with a concept of markets, so we just need to add a "websocket" market type. Not a lot of functionality needs to be written for that (we can just pipe them straight into the tradeBatcher or candleCreator.

The bad thing about websockets is that you don't have access to historical data, also the situation that if you lose connection for a minute you have incomplete data.

[STRML]

@askmike That depends on the exchange. For instance, on BitMEX, you are automatically pushed historical data as soon as you subscribe to any public or private stream.

[greenbigfrog]

@askmike this can be closed.
The using WS to receive data should be opened in a extra issue (in case there isn't any open one. Tracking 200 issues is kinda hard)

[askmike]

@greenbigfrog is right, this can indeed be closed. This is now handled by Gekko Broker!