OkCoin China API Wrapper

README.md

@@ -17,6 +17,13 @@ Gekko is a Bitcoin trading bot and backtesting platform that connects to popular
 * Backtester (for TA strategies)
 * Tool for systematic trading
 * Low level market library
+ * Monitor the live market
+ * Import historical market data
+ * Broadcast market data over pubsub messaging systems
+
+## TA strategies
+
+Gekko comes with some [basic strategies](https://github.com/askmike/gekko/blob/stable/docs/Trading_methods.md) (which implement a single indicator). But with some basic javascript you can [create your own strategies](https://github.com/askmike/gekko/blob/stable/docs/internals/trading_methods.md). You can use over 130 indicators to create your perfect prediction model. *Why don't you combine Bollinger Bands, CCI and MACD with a STOCHRSI indicator?*
 
 ## Automated Trading platform
 
@@ -32,13 +39,29 @@ Gekko is not built for HFT or anything related to being the fastest (like arbitr
 - An exchange.
 - An arbitrage bot.
 
-## Market interface
-
-Gekko has a plugin system that can do certain things whenever something happens or let Gekko communicate through external platforms. Check [all currently existing plugins](https://github.com/askmike/gekko/blob/stable/docs/Plugins.md) or [add your own](https://github.com/askmike/gekko/blob/stable/docs/internal/plugins.md).
-
 ## Supported exchanges
 
-Gekko supports multiple cryptocurreny exchanges, see [here](https://github.com/askmike/gekko/blob/stable/docs/supported_exchanges.md).
+| Exchange        | Monitoring | [Trading](https://github.com/askmike/gekko/blob/stable/docs/Plugins.md#trader) | [Importing](https://github.com/askmike/gekko/blob/stable/docs/Importing.md) | Notes |
+| --------------- |:----------:|:-------:|:---------:|-------|
+| [Poloniex](https://poloniex.com/)      | ✓ | ✓ | ✓ | |
+| [BTCC](https://btcc.com/)      | ✓ | ✓ | ✓ | is BTCChina |
+| [Bitstamp](https://bitstamp.com/)      | ✓ | ✓ | ✗ | |
+| [Kraken](https://kraken.com/)      | ✓ | ✓ | ✗ | |
+| [Bitfinex](https://bitfinex.com/)      | ✓ | ✓ | ✗ | |
+| [BTC-e](https://btc-e.com/)      | ✓ | ✓ | ✗ | |
+| [Cex.io](https://bitstamp.com/)      | ✓ | ✗ | ✗ | |
+| [bitX](https://www.bitx.co/)      | ✓ | ✗ | ✗ | |
+| [lakeBTC](https://lakebtc.com/)      | ✓ | ✗ | ✗ | |
+| [meXBT](https://mexbt.com/)      | ✓ | ✗ | ✗ | see [here](https://github.com/askmike/gekko/issues/288#issuecomment-223810974). |
+| [zaif](https://zaif.jp/trade_btc_jpy)      | ✓ | ✗ | ✗ | |
+| [lakeBTC](https://lakebtc.com/)      | ✓ | ✗ | ✗ | |
+| [bx.in.th](https://bx.in.th/)      | ✓ | ✗ | ✗ | |
+
+Monitoring means that Gekko is able to watch the realtime market and thus also:
+
+- Run trading strategies against the data (in semi-realtime)
+- Simulate trading profits (paper trader)
+- Store all data (to backtest in the future)
 
 ## Installing Gekko
 
@@ -78,9 +101,7 @@ If you installed Gekko via git you can easily fetch the latest updates by runnin
 
 ## How does Gekko work?
 
-![Gekko 0.1.0 architecture](http://data.wizb.it/misc/gekko-0.1.0-architecture.jpg)
-
-If you want to contribute or are interested in how Gekko works:
+![Gekko architecture](https://wizb.it/gekko/static/architecture.jpg)
 
 - Read about [Gekko's overall architecture](https://github.com/askmike/gekko/tree/stable/docs/internals/architecture.md).
 - Read on how to add [a new exchange to Gekko](https://github.com/askmike/gekko/tree/stable/docs/internals/exchanges.md).
@@ -89,10 +110,20 @@ If you want to contribute or are interested in how Gekko works:
 
 ## TODO
 
+* Stabilize importing API.
 * More tests
-* More exchanges
-* More indicators
-* Webbased interface (?)
+* Better documentation for TA-lib indicators.
+* More indicators (maybe use [this native js lib](https://github.com/anandanand84/technicalindicators)?)
+* Webbased interface ([first step](https://github.com/askmike/gekko/issues/338#issuecomment-228368499))?
+
+*Better exchange support:*
+
+- add GDAX exchange (supports [importing](https://docs.gdax.com/#get-historic-rates))
+- support importing at bitfinex ([here](http://docs.bitfinex.com/#trades)).
+- add okcoin China
+- add okcoin
+- add bitmex
+- fix cryptsy integration..
 
 ## Credits
 

core/baseTradingMethod.js

@@ -196,15 +196,20 @@ Base.prototype.addIndicator = function(name, type, parameters) {
 
   // some indicators need a price stream, others need full candles
   this.indicators[name].input = Indicators[type].input;
-} 
+}
 
 Base.prototype.advice = function(newPosition) {
-  if(!newPosition)
-    return this.emit('soft advice');
+  // Possible values are long and short. Long will trigger a buy method
+  // while short will trigger a sell method
+  var advice = 'soft';
+  if(newPosition) {
+    advice = newPosition;
+  }
 
   this.emit('advice', {
-    recommandation: newPosition,
-    portfolio: 1
+    recommandation: advice,
+    portfolio: 1,
+    moment: this.candle.start
   });
 }
 

core/budfox/marketFetcher.js

@@ -11,6 +11,7 @@ var moment = require('moment');
 var utc = moment.utc;
 var util = require(__dirname + '/../util');
 
+var config = util.getConfig();
 var log = require(util.dirs().core + 'log');
 var exchangeChecker = require(util.dirs().core + 'exchangeChecker');
 

core/budfox/tradeBatcher.js

@@ -47,6 +47,7 @@ var TradeBatcher = function(tid) {
 util.makeEventEmitter(TradeBatcher);
 
 TradeBatcher.prototype.write = function(batch) {
+
   if(!_.isArray(batch))
     throw 'batch is not an array';
 
@@ -64,8 +65,8 @@ TradeBatcher.prototype.write = function(batch) {
   var last = _.last(momentBatch);
   var first = _.first(momentBatch);
 
-  log.debug('Processing', amount, 'new trades.');
   log.debug(
+    'Processing', amount, 'new trades.',
     'From',
     first.date.format('YYYY-MM-DD HH:mm:ss'),
     'UTC to',

core/exchangeChecker.js

@@ -44,6 +44,21 @@ Checker.prototype.cantMonitor = function(conf) {
   return false;
 }
 
+// check if the exchange is configured correctly for fetching
+// full history
+Checker.prototype.cantFetchFullHistory = function(conf) {
+  var slug = conf.exchange.toLowerCase();
+  var exchange = _.find(exchanges, function(e) { return e.slug === slug });
+
+  if(this.cantMonitor(conf))
+    return this.cantMonitor(conf);
+
+  var name = exchange.name;
+
+  if(!exchange.providesFullHistory)
+    return 'The exchange ' + name + ' does not provide full history (or Gekko doesn\'t support importing it)';
+}
+
 // check if the exchange if configured correctly for real trading
 Checker.prototype.cantTrade = function(conf) {
   var cantMonitor = this.cantMonitor(conf);

core/backtestMarket.js → core/markets/backtest.js

@@ -1,8 +1,8 @@
 var _ = require('lodash');
-var util = require('./util');
+var util = require('../util');
 var config = util.getConfig();
 var dirs = util.dirs();
-var log = require('./log');
+var log = require(dirs.core + 'log');
 var moment = require('moment');
 
 var adapter = config.adapters[config.backtest.adapter];
@@ -37,7 +37,7 @@ Market.prototype = Object.create(Readable.prototype, {
   constructor: { value: Market }
 });
 
-Market.prototype._read = function noop() {
+Market.prototype._read = function() {
   if(this.pushing)
     return;
 
@@ -61,13 +61,16 @@ Market.prototype.processCandles = function(candles) {
   this.pushing = true;
   var amount = _.size(candles);
 
-  if(!this.ended && amount <= this.batchSize) {
+  if(amount === 0)
+    util.die('Query returned no candles (do you have local data for the specified range?)');
+
+  if(!this.ended && amount < this.batchSize) {
     var d = function(ts) {
       return moment.unix(ts).utc().format('YYYY-MM-DD HH:mm:ss');
     }
     var from = d(_.first(candles).start);
     var to = d(_.last(candles).start);
-    log.warn(`Simulation based on incomplete market data (missing between ${from} and ${to}).`);
+    log.warn(`Simulation based on incomplete market data (${this.batchSize - amount} missing between ${from} and ${to}).`);
   }
 
   _.each(candles, function(c, i) {

core/markets/importer.js

@@ -0,0 +1,94 @@
+var _ = require('lodash');
+var util = require('../util');
+var config = util.getConfig();
+var dirs = util.dirs();
+var log = require(dirs.core + 'log');
+var moment = require('moment');
+
+var adapter = config.adapters[config.importer.adapter];
+var daterange = config.importer.daterange;
+
+var TradeBatcher = require(dirs.budfox + 'tradeBatcher');
+var CandleManager = require(dirs.budfox + 'candleManager');
+var exchangeChecker = require(dirs.core + 'exchangeChecker');
+
+var error = exchangeChecker.cantFetchFullHistory(config.watch);
+if(error)
+  util.die(error, true);
+
+var fetcher = require(dirs.importers + config.watch.exchange);
+
+if(!daterange.to) {
+  var now = moment();
+  daterange.to = now;
+  log.debug(
+    'No end date specified for importing, setting to',
+    now.format('YYYY-MM-DD HH:mm:ss')
+  );
+}
+
+if(daterange.to <= daterange.from)
+  util.die('This daterange does not make sense.')
+
+var Market = function() {
+  _.bindAll(this);
+  this.exchangeSettings = exchangeChecker.settings(config.watch);
+
+  this.tradeBatcher = new TradeBatcher(this.exchangeSettings.tid);
+  this.candleManager = new CandleManager;
+  this.fetcher = fetcher(daterange);
+
+  this.done = false;
+
+  this.fetcher.bus.on(
+    'trades',
+    this.processTrades
+  );
+
+  this.fetcher.bus.on(
+    'done',
+    function() {
+      this.done = true;
+    }.bind(this)
+  )
+
+  this.tradeBatcher.on(
+    'new batch',
+    this.candleManager.processTrades
+  );
+
+  this.candleManager.on(
+    'candles',
+    this.pushCandles
+  );  
+
+  Readable.call(this, {objectMode: true});
+
+  this.get();
+}
+
+var Readable = require('stream').Readable;
+Market.prototype = Object.create(Readable.prototype, {
+  constructor: { value: Market }
+});
+
+Market.prototype._read = _.noop;
+
+Market.prototype.pushCandles = function(candles) {
+  _.each(candles, this.push);
+}
+
+Market.prototype.get = function() {
+  this.fetcher.fetch();
+}
+
+Market.prototype.processTrades = function(trades) {
+  this.tradeBatcher.write(trades);
+
+  if(this.done)
+    return log.info('Done importing!');
+
+  setTimeout(this.get, 1000);
+}
+
+module.exports = Market;

core/realtimeMarket.js → core/markets/realtime.js

@@ -1,6 +1,6 @@
 var _ = require('lodash');
 
-var util = require('./util');
+var util = require('../util');
 var dirs = util.dirs();
 
 var config = util.getConfig();

core/util.js

@@ -113,9 +113,12 @@ var util = {
     return {
       gekko: ROOT,
       core: ROOT + 'core/',
+      markets: ROOT + 'core/markets/',
+      exchanges: ROOT + 'exchanges/',
       plugins: ROOT + 'plugins/',
       methods: ROOT + 'methods/',
-      budfox: ROOT + 'core/budfox/'
+      budfox: ROOT + 'core/budfox/',
+      importers: ROOT + 'importers/exchanges/'
     }
   },
   inherit: function(dest, source) {
@@ -127,10 +130,11 @@ var util = {
   makeEventEmitter: function(dest) {
     util.inherit(dest, require('events').EventEmitter);
   },
-  // TODO:
   gekkoMode: function() {
-    if(program.backtest)
-      return 'backtest'
+    if(program['import'])
+      return 'importer';
+    else if(program.backtest)
+      return 'backtest';
     else
       return 'realtime';
   }
@@ -139,7 +143,8 @@ var util = {
 program
   .version(util.logVersion())
   .option('-c, --config <file>', 'Config file')
-  .option('-b, --backtest', 'backtest')
+  .option('-b, --backtest', 'backtesting mode')
+  .option('-i, --import', 'importer mode')
   .parse(process.argv);
 
 var config = util.getConfig();

docs/Backtesting.md

@@ -13,7 +13,7 @@ Besides that, make sure to configure `config.watch`.
 
 ## Historical data
 
-Gekko requires historical data to backtest strategies against. The easiest way to get this is to run Gekko on real markets with the plugin sqliteWriter enabled (this will cause Gekko to store realtime data on disk).
+Gekko requires historical data to backtest strategies against. The easiest way to get this is to let Gekko import historical data, however this is not supported by a lot of exchanges (see [here](https://github.com/askmike/gekko#supported-exchanges)). The second easiest and most universal way is to run Gekko on real markets with the plugin sqliteWriter enabled (this will cause Gekko to store realtime data on disk).
 
 ## Configure
 

docs/Importing.md

@@ -0,0 +1,39 @@
+# Importing
+
+If you want to use Gekko to [backtest against historical data](./Backtesting.md), you most likely need some historical data to test against. Gekko comes with the functionality to automatically import historical data from some exchanges. However, only a few exchanges support this. You can find out with which exchanges Gekko is able to do this [here](https://github.com/askmike/gekko#supported-exchanges).
+
+## Setup
+
+For importing you should [enable and configure](./Plugins.md) the following plugin:
+
+ - candleWriter (to store the imported data in a database)
+
+Besides that, make sure to configure `config.watch` properly.
+
+## Configure
+
+In your config set the `importer.daterange` properties to the daterange you would like to import.
+
+## Run
+
+    node gekko --import
+
+The result will be something like this:
+
+    2016-06-26 09:12:16 (INFO): Gekko v0.2.2 started
+    2016-06-26 09:12:16 (INFO): I'm gonna make you rich, Bud Fox. 
+
+    2016-06-26 09:12:17 (INFO): Setting up Gekko in importer mode
+    2016-06-26 09:12:17 (INFO): 
+    2016-06-26 09:12:17 (INFO): Setting up:
+    2016-06-26 09:12:17 (INFO):      Candle writer
+    2016-06-26 09:12:17 (INFO):      Store candles in a database
+    2016-06-26 09:12:17 (INFO): 
+
+    2016-06-26 09:12:17 (WARN): The plugin Trading Advisor does not support the mode importer. It has been disabled.
+    2016-06-26 09:12:17 (WARN): The plugin Advice logger does not support the mode importer. It has been disabled.
+    2016-06-26 09:12:17 (WARN): The plugin Profit Simulator does not support the mode importer. It has been disabled.
+    2016-06-26 09:12:18 (DEBUG):    Processing 798 new trades.
+    2016-06-26 09:12:18 (DEBUG):    From 2015-09-09 12:00:04 UTC to 2015-09-09 13:58:55 UTC. (2 hours)
+    2016-06-26 09:12:20 (DEBUG):    Processing 211 new trades.
+    (...)