CCXT Exchanges: Binance, KuCoin, Cryptopia

[avn3r]

Dear Catalyst Team,

Environment

• Operating System: Linux Conda environment with python 2.7, pip install enigma-catalyst

Description of Issue:

1. Current Volume is displaying the volume in the market currency instead of the base currency. For example, eth_btc reports current_volume= data.current(coin, 'volume') in eth volume instead of BTC volume. I believe same error occurs in data.history. The need for base currency value is in order to properly scale and compare values.

   • What did you expect to happen? current daily/24h volume in BTC
   • What happened instead? got current daily/24h volume in ETH (market currency)

2. CCXT Kucoin: Many request error when getting the history of the pairs.
   Error: ccxt.base.errors.RequestTimeout: kucoin GET https://api.kucoin.com/v1/open/chart/history?from=1516062779&symbol=CAT-BTC&type=30&resolution=30&to=1516667579 request timeout

3. CCXT Cryptopia: Cannot get all BTC pairs. Mainly because some pair symbols are all numbers and it it converting the symbol into an int instead of a string.
   Error: AttributeError: 'int' object has no attribute 'lower'
   catalyst/exchange/ccxt/ccxt_exchange.py", line 534, Line 534 creates the error.

• What did you expect to happen?
• What happened instead?

Reproduction Steps

1. Run code using: python <filename.py> -x <exchange_name>

from datetime import timedelta

import numpy as np
import pandas as pd
import argparse

from catalyst import run_algorithm
from catalyst.exchange.utils.exchange_utils import get_exchange_symbols
from catalyst.api import (symbols, )


def initialize(context):
    context.i = -1  # minute counter
    context.exchange = context.exchanges.values()[0].name.lower()
    context.base_currency = context.exchanges.values()[0].base_currency.lower()
    # print(context.coins)


def handle_data(context, data):
    context.i += 1
    lookback_days = 7  # 7 days

    # current date & time in each iteration formatted into a string
    now = pd.to_datetime(data.current_dt, utc=True)
    date, time = now.strftime('%Y-%m-%d %H:%M:%S').split(' ')
    lookback_date = now - timedelta(days=lookback_days)

    one_day_in_minutes = 1440  # 60 * 24 assumes data_frequency='minute'
    # update universe everyday at midnight
    if not context.i % one_day_in_minutes:
        context.universe = universe(context, lookback_date, now)

    # get data every 30 minutes
    minutes = 30
    # get lookback_days of history data: that is 'lookback' number of bins
    lookback = one_day_in_minutes / minutes * lookback_days
    if not context.i % minutes and context.universe:
        # we iterate for every pair in the current universe
        for coin in context.coins:
            pair = str(coin.symbol)

            # Get 30 minute interval OHLCV data. This is the standard data
            # required for candlestick or indicators/signals. Return Pandas
            # DataFrames. 30T means 30-minute re-sampling of one minute data.
            # Adjust it to your desired time interval as needed.
            current_volume = data.current(coin, 'volume')

            opened = fill(data.history(coin, 'open',
                                  bar_count=lookback, frequency='30T')).values
            high = fill(data.history(coin, 'high',
                                  bar_count=lookback, frequency='30T')).values
            low = fill(data.history(coin, 'low',
                                  bar_count=lookback, frequency='30T')).values
            close = fill(data.history(coin, 'price',
                                  bar_count=lookback, frequency='30T')).values
            volume = fill(data.history(coin, 'volume',
                                  bar_count=lookback, frequency='30T')).values

            # close[-1] is the last value in the set, which is the equivalent
            # to current price (as in the most recent value)
            # displays the minute price for each pair every 30 minutes
            print('{now}: {pair} -\tO:{o},\tH:{h},\tL:{c},\tC:{c},\tV:{v},\tCV:{v2}'.format(
                    now=now,
                    pair=pair,
                    o=opened[-1],
                    h=high[-1],
                    l=low[-1],
                    c=close[-1],
                    v=volume[-1],
                    v2=current_volume,
                 ))

            # -------------------------------------------------------------
            # --------------- Insert Your Strategy Here -------------------
            # -------------------------------------------------------------


def analyze(context=None, results=None):
    pass


# Get the universe for a given exchange and a given base_currency market
# Example: Poloniex BTC Market
def universe(context, lookback_date, current_date):
    # get all the pairs for the given exchange and given market
    context.coins = context.exchanges[context.exchange].assets
    context.coins = [c for c in context.coins if c.quote_currency == context.base_currency]

    return context.coins


# Replace all NA, NAN or infinite values with its nearest value
def fill(series):
    if isinstance(series, pd.Series):
        return series.replace([np.inf, -np.inf], np.nan).ffill().bfill()
    elif isinstance(series, np.ndarray):
        return pd.Series(series).replace(
                     [np.inf, -np.inf], np.nan
                    ).ffill().bfill().values
    else:
        return series


if __name__ == '__main__':
    parser = argparse.ArgumentParser()

    # Directory Parameters:
    parser.add_argument('--exchange', '-x', type=str, default='poloniex',
                        help='Specify exchange')

    start_date = pd.to_datetime('2018-01-01', utc=True)
    end_date = pd.to_datetime('2018-01-22', utc=True)

    args = parser.parse_args()
    performance = run_algorithm(# start=start_date, end=end_date,
                                capital_base=1.0,  # amount of base_currency
                                initialize=initialize,
                                handle_data=handle_data,
                                analyze=analyze,
                                exchange_name=args.exchange,
                                data_frequency='minute',
                                base_currency='btc',
                                live_graph=False,
                                live=True,
                                simulate_orders=True,
                                algo_namespace='simple_universe')

Anything else?

Trying to get all of these 3 exchanges to run paper trading. The above script is similar to simple_universe.py and will serve to test other exchange data are retrieved properly and compatible with Catalyst. Currently, only Binance (out of binance, kucoin, cryptopia) are working out of the box, in terms of providing me with history data.

Sincerely,
avn3r

[fredfortier]

Thanks for that. I'm aware of the issue with Cryptopia. We're working on releasing something currently on a separate branch for later this week. This should solve this problem as well as offer more historical data. We'll validate through your script before releasing.

…

On Mon, Jan 22, 2018 at 7:43 PM Abner Ayala-Acevedo < ***@***.***> wrote: Dear Catalyst Team, Environment - Operating System: Linux Conda environment with python 2.7, pip install enigma-catalyst Description of Issue: 1. Current Volume is displaying the volume in the market currency instead of the base currency. For example, eth_btc reports current_volume= data.current(coin, 'volume') in eth volume instead of BTC volume. Inconsistent with data.history API which uses base currency for all values. - What did you expect to happen? current daily/24h volume in BTC - What happened instead? got current daily/24h volume in ETH (market currency) 2. CCXT Kucoin: Many request error when getting the history of the pairs. Error: ccxt.base.errors.RequestTimeout: kucoin GET https://api.kucoin.com/v1/open/chart/history?from=1516062779&symbol=CAT-BTC&type=30&resolution=30&to=1516667579 request timeout 3. CCXT Cryptopia: Cannot get all BTC pairs. Mainly because some pair symbols are all numbers and it it converting the symbol into an int instead of a string. Error: AttributeError: 'int' object has no attribute 'lower' catalyst/exchange/ccxt/ccxt_exchange.py", line 534, Line 534 creates the error. - What did you expect to happen? - What happened instead? Reproduction Steps 1. Run code using: python <filename.py> -x <exchange_name> from datetime import timedelta import numpy as np import pandas as pd import argparse from catalyst import run_algorithm from catalyst.exchange.utils.exchange_utils import get_exchange_symbols from catalyst.api import (symbols, ) def initialize(context): context.i = -1 # minute counter context.exchange = context.exchanges.values()[0].name.lower() context.base_currency = context.exchanges.values()[0].base_currency.lower() # print(context.coins) def handle_data(context, data): context.i += 1 lookback_days = 7 # 7 days # current date & time in each iteration formatted into a string now = pd.to_datetime(data.current_dt, utc=True) date, time = now.strftime('%Y-%m-%d %H:%M:%S').split(' ') lookback_date = now - timedelta(days=lookback_days) one_day_in_minutes = 1440 # 60 * 24 assumes data_frequency='minute' # update universe everyday at midnight if not context.i % one_day_in_minutes: context.universe = universe(context, lookback_date, now) # get data every 30 minutes minutes = 30 # get lookback_days of history data: that is 'lookback' number of bins lookback = one_day_in_minutes / minutes * lookback_days if not context.i % minutes and context.universe: # we iterate for every pair in the current universe for coin in context.coins: pair = str(coin.symbol) # Get 30 minute interval OHLCV data. This is the standard data # required for candlestick or indicators/signals. Return Pandas # DataFrames. 30T means 30-minute re-sampling of one minute data. # Adjust it to your desired time interval as needed. current_volume = data.current(coin, 'volume') opened = fill(data.history(coin, 'open', bar_count=lookback, frequency='30T')).values high = fill(data.history(coin, 'high', bar_count=lookback, frequency='30T')).values low = fill(data.history(coin, 'low', bar_count=lookback, frequency='30T')).values close = fill(data.history(coin, 'price', bar_count=lookback, frequency='30T')).values volume = fill(data.history(coin, 'volume', bar_count=lookback, frequency='30T')).values # close[-1] is the last value in the set, which is the equivalent # to current price (as in the most recent value) # displays the minute price for each pair every 30 minutes print('{now}: {pair} -\tO:{o},\tH:{h},\tL:{c},\tC:{c},\tV:{v},\tCV:{v2}'.format( now=now, pair=pair, o=opened[-1], h=high[-1], l=low[-1], c=close[-1], v=volume[-1], v2=current_volume, )) # ------------------------------------------------------------- # --------------- Insert Your Strategy Here ------------------- # ------------------------------------------------------------- def analyze(context=None, results=None): pass # Get the universe for a given exchange and a given base_currency market # Example: Poloniex BTC Market def universe(context, lookback_date, current_date): # get all the pairs for the given exchange and given market context.coins = context.exchanges[context.exchange].assets context.coins = [c for c in context.coins if c.quote_currency == context.base_currency] return context.coins # Replace all NA, NAN or infinite values with its nearest value def fill(series): if isinstance(series, pd.Series): return series.replace([np.inf, -np.inf], np.nan).ffill().bfill() elif isinstance(series, np.ndarray): return pd.Series(series).replace( [np.inf, -np.inf], np.nan ).ffill().bfill().values else: return series if __name__ == '__main__': parser = argparse.ArgumentParser() # Directory Parameters: parser.add_argument('--exchange', '-x', type=str, default='poloniex', help='Specify exchange') start_date = pd.to_datetime('2018-01-01', utc=True) end_date = pd.to_datetime('2018-01-22', utc=True) args = parser.parse_args() performance = run_algorithm(# start=start_date, end=end_date, capital_base=1.0, # amount of base_currency initialize=initialize, handle_data=handle_data, analyze=analyze, exchange_name=args.exchange, data_frequency='minute', base_currency='btc', live_graph=False, live=True, simulate_orders=True, algo_namespace='simple_universe') Anything else? Trying to get all of these 3 exchanges to run paper trading. The above script is similar to simple_universe.py and will serve to test other exchange data are retrieved properly and compatible with Catalyst. Currently, only Binance (out of binance, kucoin, cryptopia) are working out of the box, in terms of providing me with history data. Sincerely, avn3r — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#182>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABZ-QnRjSQTpHoR_prWL02gQ-bFIaF9eks5tNSs-gaJpZM4Ro4nf> .

[avn3r]

Any Updates on this bug? Still unable to get Cryptopia to paper trade.

[fredfortier]

Expect something this week for sure

…

On Fri, Feb 23, 2018 at 4:50 PM Abner Ayala-Acevedo < ***@***.***> wrote: Any Updates on this bug? Still unable to get Cryptopia to paper trade. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#182 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABZ-QqO9CXHXr79zCh-eTWZnIPp5mzoaks5tX1zFgaJpZM4Ro4nf> .

[fredfortier]

Actively testing this at the moment. This took longer than expected due to complexity of the data. Using CCXT as a common interface help, but we had quite a few edge cases to deal with.

[gitlines]

Since you you use ccxt (which is not documented anywhere) should it be possible to ingest data from a ccxt exchange besides beside polo, bitfinex and bittrex? Trying for example:
catalyst ingest-exchange -x binance -i btc_usdt
does not work. Thank you

[lenak25]

Hi @gitlines, catalyst is indeed using CCXT which is also mentioned in our documentation. We are still actively working to add more historical data (like binance) to our servers, to enable users to ingest it and backtest their strategies with. Unfortunately it takes much longer than anticipated but hopefully we will be able to release it soon.