Skip to content
This repository has been archived by the owner on Nov 26, 2022. It is now read-only.

Crash when a timeout occurs when ordering #356

Closed
sam31415 opened this issue Jun 2, 2018 · 4 comments
Closed

Crash when a timeout occurs when ordering #356

sam31415 opened this issue Jun 2, 2018 · 4 comments
Assignees
@AvishaiW
Labels
Bug Res - Fixed

Comments

@sam31415
Copy link

sam31415 commented Jun 2, 2018 •
edited by catalyst-support

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

  • Operating System: Linux 4.13.0-39-generic Ingest Data on v0.3.1 and Data History #44-Ubuntu SMP Thu Apr 5 14:25:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  • Python Version: 3.6.4
  • Python Bitness: 64
  • How did you install Catalyst: conda
  • Python packages:
    aiodns 1.1.1
    aiohttp 3.1.3
    alembic 0.9.9
    async-timeout 2.0.1
    attrdict 2.0.0
    attrs 17.4.0
    bcolz 0.12.1
    bcolz 1.1.2 py36h00f5784_0
    bokeh 0.12.14 py36_0
    boto3 1.7.4
    botocore 1.10.4
    Bottleneck 1.2.1
    bzip2 1.0.6 h9a117a8_4
    ca-certificates 2017.08.26 h1d4fec5_0
    cchardet 2.1.1
    ccxt 1.13.1
    certifi 2018.4.16
    certifi 2018.1.18 py36_0
    chardet 3.0.4
    click 6.7 py36h5253387_0
    cloudpickle 0.5.2 py36_1
    configparser 3.5.0
    contextlib2 0.5.5
    cycler 0.10.0
    cyordereddict 1.0.0
    cython 0.27.3 py36h1860423_0
    Cython 0.28.2
    cytoolz 0.9.0.1 py36h14c3975_0
    dask 0.17.2 py36_0
    dask-core 0.17.2 py36_0
    decorator 4.3.0
    distributed 1.21.4 py36_0
    docutils 0.14
    empyrical 0.2.1
    enigma-catalyst 0.5.12
    eth-abi 1.0.0
    eth-account 0.1.0a2
    eth-hash 0.1.2
    eth-keyfile 0.5.1
    eth-keys 0.2.0b3
    eth-rlp 0.1.0
    eth-utils 1.0.2
    hdf5 1.10.1 h9caa474_1
    heapdict 1.0.0 py36_2
    hexbytes 0.1.0
    idna 2.6
    idna-ssl 1.0.1
    intel-openmp 2018.0.0 8
    intervaltree 2.1.0
    jinja2 2.10 py36ha16c418_0
    jmespath 0.9.3
    kiwisolver 1.0.1
    libedit 3.1 heed3624_0
    libffi 3.2.1 hd88cf55_4
    libgcc-ng 7.2.0 hdf63c60_3
    libgfortran-ng 7.2.0 hdf63c60_3
    libstdcxx-ng 7.2.0 hdf63c60_3
    locket 0.2.0 py36h787c0ad_1
    Logbook 1.3.3
    lru-dict 1.1.6
    lxml 4.2.1
    lzo 2.10 h49e0be7_2
    Mako 1.0.7
    markupsafe 1.0 py36hd9260cd_1
    matplotlib 2.2.2
    mkl 2018.0.2 1
    mpld3 0.3
    msgpack-python 0.5.5 py36h6bb024c_0
    multidict 4.2.0
    multipledispatch 0.5.0
    ncurses 6.0 h9df7e31_2
    networkx 2.1
    numexpr 2.6.4 py36hc4a3f9a_0
    numpy 1.14.2
    numpy 1.14.2 py36hdbf6ddf_0
    openssl 1.0.2n hb7f436b_0
    packaging 17.1 py36_0
    pandas 0.22.0 py36hf484d3e_0
    pandas 0.19.2
    pandas-datareader 0.6.0
    partd 0.3.8 py36h36fd896_0
    patsy 0.5.0 py36_0
    patsy 0.5.0
    pip 10.0.1
    pip 9.0.1 py36_5
    psutil 5.4.3 py36h14c3975_0
    pycares 2.3.0
    pycryptodome 3.6.1
    pyparsing 2.2.0 py36hee85983_1
    pytables 3.4.2 py36h3b5282a_2
    python 3.6.4 hc3d631a_3
    python-dateutil 2.7.2
    python-dateutil 2.7.0 py36_0
    python-editor 1.0.3
    pytz 2018.3 py36_0
    pytz 2018.4
    pyyaml 3.12 py36hafb9ca4_1
    readline 7.0 ha6073c6_4
    redo 1.6
    requests 2.18.4
    requests-file 1.4.3
    requests-ftp 0.3.1
    requests-toolbelt 0.8.0
    rlp 0.6.0
    s3transfer 0.1.13
    scipy 1.0.0 py36hbf646e7_0
    scipy 1.0.1
    setuptools 39.0.1
    setuptools 38.5.1 py36_0
    six 1.11.0 py36h372c433_1
    six 1.10.0
    sortedcontainers 1.5.9 py36_0
    SQLAlchemy 1.2.6
    sqlite 3.22.0 h1bed415_0
    statsmodels 0.8.0 py36h8533d0b_0
    statsmodels 0.8.0
    tabulate 0.8.2
    tblib 1.3.2 py36h34cf8b6_0
    tk 8.6.7 hc745277_3
    toolz 0.9.0 py36_0
    tornado 5.0 py36_0
    urllib3 1.22
    web3 4.1.0
    websockets 4.0.1
    wheel 0.30.0 py36hfd4bba0_1
    wrapt 1.10.11
    xz 5.2.3 h55aa19d_2
    yaml 0.1.7 had09818_2
    yarl 1.1.0
    zict 0.1.3 py36h3a3bf81_0
    zlib 1.2.11 ha838bed_2

Now that you know a little about me, let me tell you about the issue I am
having:

Description of Issue

A timeout occured when ordering on Poloniex and the algorithm crashed.

[2018-06-02 20:20:21.158924] INFO: CCXT: received a RequestTimeout exception while creating an order on poloniex /
 ETC/BTC                         
 Checking if an order was filled during the timeout 

[2018-06-02 20:20:24.488976] ERROR: algo:   File "CryptoTrendIII.py", line 1068, in <module>
    simulate_orders=PAPER_TRADE,                                                                                  
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/run_algo.py", lin
e 601, in run_algorithm                           
    stats_output=stats_output                                                                                    
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/run_algo.py", lin
e 363, in _run         
    overwrite_sim_params=False,                                                                                   
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algor
ithm.py", line 1050, in run                                     
    data, overwrite_sim_params                                       
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algor
ithm.py", line 358, in run
    data, overwrite_sim_params
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/algorithm.py", line 724
, in run
    for perf in self.get_generator():
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/gens/tradesimulation.py
", line 224, in transform
    for capital_change_packet in every_bar(dt):
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/gens/tradesimulation.py
", line 137, in every_bar
    handle_data(algo, current_data, dt_to_use)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/events.py", line
216, in handle_data
    dt,
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/events.py", line
235, in handle_data
    self.callback(context, data)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algor
ithm.py", line 904, in handle_data
    self._handle_data(self, data)
  File "CryptoTrendIII.py", line 611, in handle_data
    MAX_POS_ENTRY*0.99)
  File "CryptoTrendIII.py", line 739, in market_order_simple
    price_data.loc[pair], curr_pos_amount[pair], "Buying")
  File "CryptoTrendIII.py", line 802, in order_target_fraction
    error = ''.join(traceback.format_stack())
    
    Traceback (most recent call last):                                                                                
  File "CryptoTrendIII.py", line 1068, in <module>
    simulate_orders=PAPER_TRADE,                                                                                 
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/run_algo.py", lin
e 601, in run_algorithm
    stats_output=stats_output                                                                                     
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/run_algo.py", lin
e 363, in _run                                                  
    overwrite_sim_params=False,                                      
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algor
ithm.py", line 1050, in run
    data, overwrite_sim_params
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algor
ithm.py", line 358, in run
    data, overwrite_sim_params       
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/algorithm.py", line 724
, in run                 
    for perf in self.get_generator():          
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/gens/tradesimulation.py
", line 224, in transform
    for capital_change_packet in every_bar(dt):
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/gens/tradesimulation.py
", line 137, in every_bar
    handle_data(algo, current_data, dt_to_use)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/events.py", line
216, in handle_data
    dt,                         
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/events.py", line 
235, in handle_data               
    self.callback(context, data) 
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algor
ithm.py", line 894, in handle_data
    cleanup=lambda: log.warn('Syncing portfolio again.')    
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/redo/__init__.py", line 162, in
retry                                                         
    return action(*args, **kwargs) 
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algor
ithm.py", line 717, in synchronize_portfolio
    cash=required_cash,
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange.py",
line 755, in sync_positions
    cash=cash,
catalyst.exchange.exchange_errors.NotEnoughCashError: Total btc amount on poloniex is lower than the cash reserved
 for this algo: 0.02328973 < 0.0246560084032739. While trades can be made on the exchange accounts outside of the
algo, exchange must have enough free btc to cover the algo cash.

The buy order was placed with the order function with a limit price well above the current price. On the exchange, the order has been filled against three limit orders.

ETC/BTC | Buy | Exchange | 0.00209413 | 0.05308416 | 0.00010617 ETC (0.2%) | 0.00011116 BTC | 2018-06-02 20:20:23
-- | -- | -- | -- | -- | -- | -- | --
  | ETC/BTC | Buy | Exchange | 0.00209442 | 0.11908249 | 0.00023816 ETC (0.2%) | 0.00024940 BTC | 2018-06-02 20:20:23
  | ETC/BTC | Buy | Exchange | 0.00209442 | 1.10197813 | 0.00220396 ETC (0.2%) | 0.00230800 BTC | 2018-06-02 20:20:23
  • What did you expect to happen?
    I expect the algorithm to keep connecting to the exchange until it figures out whether the order has filled or not.
  • What happened instead?
    The algorithm considered the order as not filled, detected a lower balance in cash (because the order WAS filled), and then crashed.

This issue seems closely related to #317. At the time, the algorithm was ordering multiple times. Now it crashes.

Side remark: Please, a trading algorithm should ideally never crash. This is terrible. I do my best to handle any error I can within the algorithm, but it's not possible to do so with all of them. In the present case, if a lower balance is detected, at the very least the algo could just send a warning, update its cash variable and continue trading with less money. Why make it crash?

Here is how you can reproduce this issue on your machine:

Reproduction Steps

  1. Arrange to have a timeout error just after ordering.
    ...

Sincerely,
Samuel
@lenak25
Copy link
Contributor

lenak25 commented Jun 3, 2018

@sam31415 thanks for reporting, we will investigate this.

@lenak25 lenak25 added the Bug label Jun 3, 2018
@sam31415
Copy link
Author

sam31415 commented Jun 4, 2018 •
edited 

[2018-06-03 23:50:20.182772] INFO: CCXT: received a RequestTimeout exception while creating an order on poloniex / XRP/BTC                                     
 Checking if an order was filled during the timeout                              
[2018-06-03 23:50:20.558704] ERROR: algo:   File "algo.py", line 1068, in <module>                                                    
    simulate_orders=PAPER_TRADE,       
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/run_algo.py", line 601, in run_algorithm                      
    stats_output=stats_output          
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/run_algo.py", line 363, in _run                               
    overwrite_sim_params=False,                                                
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py", line 1050, in run                  
    data, overwrite_sim_params                                                 
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py", line 358, in run                   
    data, overwrite_sim_params         
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/algorithm.py", line 724, in run                                     
    for perf in self.get_generator():  
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/gens/tradesimulation.py", line 224, in transform                    
    for capital_change_packet in every_bar(dt):                                
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/gens/tradesimulation.py", line 137, in every_bar                    
    handle_data(algo, current_data, dt_to_use)                                 
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/events.py", line 216, in handle_data                          
    dt,                                
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/events.py", line 235, in handle_data                          
    self.callback(context, data)       
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py", line 904, in handle_data           
    self._handle_data(self, data)                                                                                                                              
  File "algo.py", line 611, in handle_data
      MAX_POS_ENTRY*0.99)                
  File "algo.py", line 739, in market_order_simple                                                                                                   
    price_data.loc[pair], curr_pos_amount[pair], "Buying")                     
  File "algo.py", line 802, in order_target_fraction                 
    error = ''.join(traceback.format_stack())                                  
                                                                               
Traceback (most recent call last):     
  File "algo.py", line 1068, in <module>                             
    simulate_orders=PAPER_TRADE,       
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/run_algo.py", line 601, in run_algorithm                      
    stats_output=stats_output                                                  
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/run_algo.py", line 363, in _run                               
    overwrite_sim_params=False,        
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py", line 1050, in run                  
    data, overwrite_sim_params         
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py", line 358, in run                   
    data, overwrite_sim_params         
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/algorithm.py", line 724, in run                                     
    for perf in self.get_generator():  
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/gens/tradesimulation.py", line 224, in transform                    
    for capital_change_packet in every_bar(dt):                                
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/gens/tradesimulation.py", line 137, in every_bar                    
    handle_data(algo, current_data, dt_to_use)                                 
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/events.py", line 216, in handle_data                          
    dt,                                
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/events.py", line 235, in handle_data                          
    self.callback(context, data)       
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py", line 894, in handle_data           
    cleanup=lambda: log.warn('Syncing portfolio again.')                       
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/redo/__init__.py", line 162, in retry                                        
    return action(*args, **kwargs)                                             
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py", line 717, in synchronize_portfolio 
    cash=required_cash,                
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange.py", line 755, in sync_positions                  
    cash=cash,                         
catalyst.exchange.exchange_errors.NotEnoughCashError: Total btc amount on poloniex is lower than the cash reserved for this algo: 0.02850145 < 0.03. While trad
es can be made on the exchange accounts outside of the algo, exchange must have enough free btc to cover the algo cash.

Again, the order was executed on the exchange, but not detected by Catalyst, who only saw the btc balance go down.

AvishaiW added a commit that referenced this issue Jun 10, 2018
@AvishaiW
BUG: #350 #356 Added retry to creating an order and fixed symbol issu…
ca01441 
…es during RequestTimeout
@AvishaiW
Copy link
Contributor

Added a retry method and fixed bugs on the missing order process. Thanks for reporting this issue.

@AvishaiW
Copy link
Contributor

added to release 0.5.14

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@AvishaiW AvishaiW

Labels
Bug Res - Fixed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lenak25 @sam31415 @AvishaiW