New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NoDataAvailableOnExchange for bitfinex daily #359

Closed
sawyercade opened this Issue Jun 5, 2018 · 9 comments

Comments

Projects
None yet
6 participants
@sawyercade
Copy link

sawyercade commented Jun 5, 2018

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

  • Operating System: Darwin Kernel Version 17.3.0
  • Python Version: Python 3.6.4 :: Anaconda, Inc.
  • Python Bitness: 64
  • How did you install Catalyst: conda
  • Python packages: # packages in environment at /Users/sawyeranderson/anaconda3/envs/catalyst:

Name Version Build Channel

aiodns 1.1.1
aiohttp 3.0.1
alembic 0.9.7
appnope 0.1.0 py36hf537a9a_0
appnope 0.1.0
async-timeout 2.0.0
attrdict 2.0.0
attrs 17.4.0
backcall 0.1.0 py36_0
backcall 0.1.0
backports 1.0 py36ha3c1827_1
backports.functools_lru_cache 1.5 py36_0 conda-forge
bcolz 0.12.1
bleach 2.1.3 py36_0
bleach 2.1.3
boto3 1.5.27
botocore 1.8.41
Bottleneck 1.2.1
ca-certificates 2017.08.26 ha1e5d58_0
cchardet 2.1.1
ccxt 1.12.131
certifi 2018.1.18 py36_0
chardet 3.0.4
click 6.7
contextlib2 0.5.5
cycler 0.10.0 py36hfc81398_0
cyordereddict 1.0.0
Cython 0.27.3
cytoolz 0.9.0
dbus 1.13.2 h760590f_1
decorator 4.3.0 py36_0
decorator 4.2.1
docutils 0.14
empyrical 0.2.1
enigma-catalyst 0.5.12
entrypoints 0.2.3 py36hd81d71f_2
entrypoints 0.2.3
eth-abi 1.0.0b0
eth-account 0.1.0a2
eth-keyfile 0.5.1
eth-keys 0.2.0b1
eth-rlp 0.1.0a2
eth-utils 1.0.0b1
expat 2.2.5 hb8e80ba_0
freetype 2.8 h12048fb_1
gettext 0.19.8.1 h15daf44_3
glib 2.56.1 h35bc53a_0
hexbytes 0.1.0b0
html5lib 1.0.1
html5lib 1.0.1 py36h2f9c1c0_0
icu 58.2 h4b95b61_1
idna 2.6
idna-ssl 1.0.0
intel-openmp 2018.0.0 8
intervaltree 2.1.0
ipykernel 4.8.2 py36_0
ipykernel 4.8.2
ipython 6.4.0 py36_0
ipython-genutils 0.2.0
ipython_genutils 0.2.0 py36h241746c_0
ipywidgets 7.2.1 py36_0
ipywidgets 7.2.1
jedi 0.12.0 py36_1
jedi 0.12.0
jinja2 2.10 py36hd36f9c5_0
Jinja2 2.10
jmespath 0.9.3
jpeg 9b he5867d9_2
jsonschema 2.6.0
jsonschema 2.6.0 py36hb385e00_0
jupyter 1.0.0
jupyter 1.0.0 py36_4
jupyter-client 5.2.3
jupyter-console 5.2.0
jupyter-core 4.4.0
jupyter_client 5.2.3 py36_0
jupyter_console 5.2.0 py36hccf5b1c_1
jupyter_core 4.4.0 py36h79cf704_0
kiwisolver 1.0.1 py36h792292d_0
libcxx 4.0.1 h579ed51_0
libcxxabi 4.0.1 hebd6815_0
libedit 3.1.20170329 hb402a30_2
libffi 3.2.1 h475c297_4
libgfortran 3.0.1 h93005f0_2
libiconv 1.15 hdd342a3_7
libpng 1.6.34 he12f830_0
libsodium 1.0.16 h3efe00b_0
Logbook 1.2.1
lru-dict 1.1.6
lxml 4.1.1
Mako 1.0.7
markupsafe 1.0 py36h3a1e703_1
MarkupSafe 1.0
matplotlib 2.1.2 py36_0 conda-forge
mistune 0.8.3
mistune 0.8.3 py36h1de35cc_1
mkl 2018.0.1 hfbd8650_4
multidict 4.1.0
multipledispatch 0.4.9
nbconvert 5.3.1 py36h810822e_0
nbconvert 5.3.1
nbformat 4.4.0
nbformat 4.4.0 py36h827af21_0
ncurses 6.1 h0a44026_0
networkx 2.1
notebook 5.5.0 py36_0
numexpr 2.6.4
numpy 1.14.0 py36ha726252_2
openssl 1.0.2n hdbc3d79_0
pandas 0.19.2
pandas-datareader 0.6.0
pandoc 1.19.2.1 ha5e8f32_1
pandocfilters 1.4.2 py36h3b0b094_1
pandocfilters 1.4.2
parso 0.2.1
parso 0.2.0 py36_0
patsy 0.5.0
pcre 8.42 h378b8a2_0
pexpect 4.5.0 py36_0
pexpect 4.6.0
pickleshare 0.7.4 py36hf512f8e_0
pickleshare 0.7.4
pip 9.0.1 py36_5
prompt-toolkit 1.0.15
prompt_toolkit 1.0.15 py36haeda067_0
ptyprocess 0.5.2 py36he6521c3_0
ptyprocess 0.5.2
pycares 2.3.0
pycryptodome 3.4.11
pygments 2.2.0 py36h240cd3f_0
Pygments 2.2.0
pyparsing 2.2.0 py36hb281f35_0
pyqt 5.9.2 py36h11d3b92_0
pysha3 1.0.2
python 3.6.4 hc167b69_3
python-dateutil 2.6.1
python-dateutil 2.7.3 py36_0
python-editor 1.0.3
pytz 2018.4 py36_0
pytz 2018.3
pyzmq 17.0.0
pyzmq 17.0.0 py36h1de35cc_1
qt 5.9.5 h02808f3_0
qtconsole 4.3.1
qtconsole 4.3.1 py36hd96c0ff_0
readline 7.0 hc1231fa_4
redo 1.6
requests 2.18.4
requests-file 1.4.3
requests-ftp 0.3.1
requests-toolbelt 0.8.0
rlp 0.6.0
s3transfer 0.1.12
scipy 1.0.0 py36h1de22e9_0
Send2Trash 1.5.0
send2trash 1.5.0 py36_0
setuptools 38.4.0 py36_0
setuptools 38.5.1
simplegeneric 0.8.1 py36_2
simplegeneric 0.8.1
sip 4.19.8 py36h0a44026_0
six 1.11.0 py36h0e22d5e_1
sortedcontainers 1.5.9
SQLAlchemy 1.2.2
sqlite 3.22.0 h3efe00b_0
statsmodels 0.8.0
tables 3.4.2
terminado 0.8.1 py36_1
terminado 0.8.1
testpath 0.3.1
testpath 0.3.1 py36h625a49b_0
tk 8.6.7 h35a86e2_3
toolz 0.9.0
tornado 5.0.2 py36_0
traitlets 4.3.2 py36h65bd3ce_0
traitlets 4.3.2
urllib3 1.22
virtualenv 16.0.0
wcwidth 0.1.7
wcwidth 0.1.7 py36h8c6ec74_0
web3 4.0.0b9
webencodings 0.5.1 py36h3b9701d_1
webencodings 0.5.1
wheel 0.30.0 py36h5eb2c71_1
widgetsnbextension 3.2.1 py36_0
widgetsnbextension 3.2.1
wrapt 1.10.11
xz 5.2.3 h727817e_4
yarl 1.1.0
zeromq 4.2.5 h378b8a2_0
zlib 1.2.11 hf3cbc9b_2

Now that you know a little about me, let me tell you about the issue I am
having:

Description of Issue

I expected to run a backtest on bitfinex daily eth_usd data. When running the backtest (one that I had successfully run before on the same machine), I got a NoDataAvailableOnExchange error.

I'd ingested bitfinex eth_usd daily and minute data previously, and had been running backtests against it in a Jupyter notebook. Running the notebook this morning produced the following error:

NoDataAvailableOnExchange: Requested data for trading pair ETH/USD is not available on exchange bitfinex in `daily` frequency at this time. Check `http://enigma.co/catalyst/status` for market coverage.

Here is how you can reproduce this issue on your machine:

Reproduction Steps

I'm not certain that the issue isn't local to my environment, but the following may/may not repro in your environment:

  1. catalyst ingest-exchange -x bitfinex -f daily -i eth_usd (does this produce a progress bar and download for you?)
  2. Run this:
import pandas as pd
from catalyst.utils.run_algo import run_algorithm

def initialize(context):
   context.ASSET_NAME = 'eth_usd'
   context.asset = symbol(context.ASSET_NAME)

def handle_data(context, data):
   price = data.current(context.asset, 'price')

def analyze(context=None, results=None):
   import matplotlib.pyplot as plt
   return plt

if __name__ == '__main__':
   
   run_algorithm(
       capital_base=1000,
       data_frequency='daily',
       initialize=initialize,
       handle_data=handle_data,
       analyze=analyze,
       exchange_name='bitfinex',
       algo_namespace='namespace,
       quote_currency='usd',
       start=pd.to_datetime('2018-1-12', utc=True),
       end=pd.to_datetime('2018-5-22', utc=True),
   )

What steps have you taken to resolve this already?

I attempted to re-ingest the data with:

catalyst clean-exchange -x bitfinex
catalyst ingest-exchange -x bitfinex -f daily -i eth_usd

ingest-exchange doesn't appear to actually download anything. Normally there is a progress bar displayed, but this simple shows Trying to ingest exchange bundle bitfinex... and returns.

I also tried rm -rf ~/.catalyst/data/exchanges/bitfinex and re-ingesting, with the same result.

Anything else?

$ catalyst --version
catalyst, version 0.5.12

Sincerely,
Sawyer

@jeanfredericplante

This comment has been minimized.

Copy link

jeanfredericplante commented Jun 6, 2018

+1, and when you check on https://enigma.co/catalyst/status/ for bitfinex, it lists 3 currency pairs available including c_ode and er_ror :).

@AvishaiW

This comment has been minimized.

Copy link
Collaborator

AvishaiW commented Jun 6, 2018

Thanks for noting this out. We are looking in to this.

@lacabra

This comment has been minimized.

Copy link
Member

lacabra commented Jun 6, 2018

This issue has been resolved, and data is available again for ingestion. We are still investigating the root cause that leads to the unavailability of data to prevent this from happening again in the future, but at least the data is already available again.

Right now, there might be a stale config file in your local system, which you can remove and clean by running catalyst clean-exchange -x bitfinex, which has the side effect that also wipes out all historical data that you have already donwloaded for this exchange (if you have ample bandwidth this shouldn't be a problem). Alternatively, you can manually remove the problematic file by deleting this file: ~/.catalyst/data/exchanges/bitfinex/symbols.json, or wait up to 24h, when it will be re-downloaded automatically.

@jeanfredericplante

This comment has been minimized.

Copy link

jeanfredericplante commented Jun 7, 2018

thanks @lacabra and @AvishaiW!

@lenak25

This comment has been minimized.

Copy link
Collaborator

lenak25 commented Jun 10, 2018

Closing this, @sawyercade and @jeanfredericplante, feel free to reopen if the problem occurs again.

Thanks!

@lenak25 lenak25 closed this Jun 10, 2018

@burak-yenigun

This comment has been minimized.

Copy link

burak-yenigun commented Jun 20, 2018

I'm having this exact same issue. I'm working based on a fresh installation of catalyst. And I tried the above steps to fix the problem, but I'm having exactly same issue mentioned above, with no progress bar showing up.

I attempted to re-ingest the data with:

catalyst clean-exchange -x bitfinex
catalyst ingest-exchange -x bitfinex -f daily -i eth_usd

ingest-exchange doesn't appear to actually download anything. Normally there is a progress bar displayed, but this simple shows Trying to ingest exchange bundle bitfinex... and returns.

I also tried rm -rf ~/.catalyst/data/exchanges/bitfinex and re-ingesting, with the same result.

@lenak25

This comment has been minimized.

Copy link
Collaborator

lenak25 commented Jun 20, 2018

Hi @burak-yenigun , the original issue preventing from ingesting bitfinex data was resolved already.
Have you checked that the data was not ingested?

@burak-yenigun

This comment has been minimized.

Copy link

burak-yenigun commented Jun 20, 2018

Yes, I see that the issue is closed, and problem was resolved for sawyercade. However I'm getting this exact same problem, despite applying the recommendations in this thread.

@lenak25

This comment has been minimized.

Copy link
Collaborator

lenak25 commented Jun 21, 2018

I meant that I tried to reproduce what you have experienced without any luck.
Have you checked that the data you requested was not ingested? Is this phenomena repeated on other trading pairs as well / on other exchanges?
Thanks,
Lena

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment