Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kraken: KeyError: 400? #347

Closed
yohplala opened this issue Dec 5, 2020 · 17 comments
Closed

Kraken: KeyError: 400? #347

yohplala opened this issue Dec 5, 2020 · 17 comments
Labels
bug

Comments

@yohplala
Copy link
Contributor

yohplala commented Dec 5, 2020

Describe the bug
From time to time, I get this error message.

2020-12-05 20:48:42,032 : ERROR : KRAKEN: encountered an exception, reconnecting
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/cryptofeed-1.6.2-py3.8.egg/cryptofeed/feedhandler.py", line 271, in _connect
    await self._handler(websocket, feed.message_handler, feed.uuid)
  File "/usr/local/lib/python3.8/dist-packages/cryptofeed-1.6.2-py3.8.egg/cryptofeed/feedhandler.py", line 300, in _handler
    await handler(message, self.last_msg[feed_id])
  File "/usr/local/lib/python3.8/dist-packages/cryptofeed-1.6.2-py3.8.egg/cryptofeed/exchange/kraken.py", line 150, in message_handler
    if self.channel_map[msg[0]][0] == 'trade':
KeyError: 400

To Reproduce
Extract of the config file.

    KRAKEN:
        channel_timeouts:
            trades: 90
            l2_book: 90
        retries: -1
        trades: [BCH-BTC,BTC-USD,LTC-BTC]
        l2_book:
            symbols: [BCH-BTC,BTC-USD,LTC-BTC]
            book_delta: true
            book_interval: 100000

storage_interval: 2M

Expected behavior
I have no idea what a KeyError: 400 refers to, but it seems that if a key is not existing, it is masking another trouble that might be worth to catch.

Operating System:
Ubuntu 18.04

Cryptofeed Version
1.6.2
@yohplala yohplala added the bug label Dec 5, 2020
@yohplala
Copy link
Contributor Author

yohplala commented Dec 5, 2020

Checking another server, I can see this time KeyError is sometime with 400, sometime with 1200:

2020-12-05 19:11:39,140 : ERROR : KRAKEN: encountered an exception, reconnecting
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/cryptofeed-1.6.2-py3.8.egg/cryptofeed/feedhandler.py", line 271, in _connect
    await self._handler(websocket, feed.message_handler, feed.uuid)
  File "/usr/local/lib/python3.8/dist-packages/cryptofeed-1.6.2-py3.8.egg/cryptofeed/feedhandler.py", line 300, in _handler
    await handler(message, self.last_msg[feed_id])
  File "/usr/local/lib/python3.8/dist-packages/cryptofeed-1.6.2-py3.8.egg/cryptofeed/exchange/kraken.py", line 150, in message_handler
    if self.channel_map[msg[0]][0] == 'trade':
KeyError: 1200

@bmoscon
Copy link
Owner

bmoscon commented Dec 5, 2020

were there other connection resets for kraken before these messages ? when you connect to kraken it gives you an id for pair/channel updates, thats the key this is referring to. So you're getting data for a pair/channel combination thats not in the mapping. the channel map should only be reset on subscribe, so if you're disconnecting for some reason and reconnecting, you may be still getting old updates for the old channel ids that no longer exist in the mapping table.

@bmoscon
Copy link
Owner

bmoscon commented Dec 5, 2020

I put in some code so that this wont cause an error / disconnect. It should just log some information. If this is happening frequently I can take the issue up with kraken (they have had issues with their websocket data before). Once you move to my new code, keep an eye out for data loss after you see the new log lines. That would indicate an error on Kraken's end

@yohplala
Copy link
Contributor Author

yohplala commented Dec 5, 2020

Thanks Bryant,
Ok, to be honest, I am not yet at being able to say if there is data loss or not, even if this is something I would like to be able to do.
However, it will not be for the short term :)
Thanks again Bryant!

@bmoscon
Copy link
Owner

bmoscon commented Dec 5, 2020

I just mean once you update to the new code, if you see this message and you notice you are no longer getting data for a pair, that would be something I could raise with Kraken

@yohplala
Copy link
Contributor Author

yohplala commented Dec 5, 2020

Ok, good to know, thanks Bryant.
I will probably update to the new code next week-end :)
I will let you know.
Thanks again!

@yohplala
Copy link
Contributor Author

Hi @bmoscon,
I rolled-out last cryptofeed version yesterday evening and am coming back on this topic.
In the past 8 hours, I did receive 6 warnings (so not a big deal) as follow:

2020-12-14 04:12:41,851 : WARNING : KRAKEN: Invalid channel id recevied 1200
2020-12-14 04:12:41,851 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-14 04:21:42,074 : WARNING : KRAKEN: Invalid channel id recevied 400
2020-12-14 04:21:42,074 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-14 05:43:32,330 : WARNING : KRAKEN: Invalid channel id recevied 400
2020-12-14 05:43:32,330 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}

Within this 8 hours, I do have a few corresponding parquet files, so my understanding is that the stream is still open, despite the warnings.
Do you think this is related to previous error?

@yohplala
Copy link
Contributor Author

(small typo noticed with 'recevied')

@bmoscon
Copy link
Owner

bmoscon commented Dec 14, 2020

it looks like you're subscribed to USDC, USD, USDT and DAI trading pairs. Are you still getting data for all 4? Were those the only 4 you subscribed to on kraken?

@yohplala
Copy link
Contributor Author

yohplala commented Dec 15, 2020 •
edited

Hi @bmoscon
I have weird things indeed. I started the server at approximately 9:45 pm (13th of Dec.)

  • For USDT pair, over 24 hours, when I record parquet file every 2 hours, I only have 5 of them:

    • first at ~3:45am (14th of Dec.)
    • 2nd at ~9:15am
    • 3rd at ~2pm
    • 4th at ~6pm
    • 5th at ~10:30pm
      Parquet appending does not ensure fix time writing, only how many appendings there are, but it seems to me that BTC-USDT pair is a traded pair, so this should be every 2 hours. (I do have the 12 l2_book files)

  • For DAI pair, there are simply 0 files. (here also I do have the 12 l2-book)

  • For USDC, I only have 2 files. (here also I do have the 12 l2-book)

So yes, weird results. I am not expecting many trades with DAI, but I am definitely surprise about this with USDT.

  • It seems I have no missing file with USD:
    • 1st at ~11:45pm (13th of Dec.)
    • 2nd at ~1:45am (14th of Dec.)
    • 3rd at ~3:45am
    • 4th at ~5:45am
    • 5th at ~7:45am
    • 6th at ~9:50am
    • 7th at ~11:55am
    • 8th at ~13:55am
    • 9th at ~15:55am
    • 10th at ~17:55am
    • 11th at ~19:55am
    • 12th at ~21:55am

@bmoscon
Copy link
Owner

bmoscon commented Dec 15, 2020

your timestamp from the logs correspond with a kraken maintenance window. It makes me think that after a maintenance window connections may need to be reestablished. https://status.kraken.com/incidents/2fr2t4gcchdc

@bmoscon
Copy link
Owner

bmoscon commented Dec 15, 2020

do the warnings go on forever or do they stop at some point?

@yohplala
Copy link
Contributor Author

yohplala commented Dec 15, 2020 •
edited

Here are they all snce the start of the server till now. It is not 'forever', but they do be some.

root@cs13sl1:~# cat yohplala/feedhandler.log | grep 'WARNING : KRAKEN'
2020-12-14 04:12:41,851 : WARNING : KRAKEN: Invalid channel id recevied 1200
2020-12-14 04:12:41,851 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-14 04:21:42,074 : WARNING : KRAKEN: Invalid channel id recevied 400
2020-12-14 04:21:42,074 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-14 05:43:32,330 : WARNING : KRAKEN: Invalid channel id recevied 400
2020-12-14 05:43:32,330 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-14 11:46:54,250 : WARNING : KRAKEN: Invalid channel id recevied 1200
2020-12-14 11:46:54,250 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-14 14:00:34,961 : WARNING : KRAKEN: Invalid channel id recevied 400
2020-12-14 14:00:34,962 : WARNING : KRAKEN: channel map: {2384: ('book', 'BTC-USDC'), 320: ('book', 'BTC-USD'), 2304: ('book', 'BTC-USDT'), 2336: ('book', 'BTC-DAI')}
2020-12-14 14:46:24,396 : WARNING : KRAKEN: encountered connection issue code = 1006 (connection closed abnormally [internal]), no reason - reconnecting...
2020-12-14 17:52:06,120 : WARNING : KRAKEN: Invalid channel id recevied 1440
2020-12-14 17:52:06,121 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-14 17:52:26,125 : WARNING : KRAKEN: Invalid channel id recevied 400
2020-12-14 17:52:26,125 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-14 18:41:47,526 : WARNING : KRAKEN: Invalid channel id recevied 400
2020-12-14 18:41:47,526 : WARNING : KRAKEN: channel map: {2384: ('book', 'BTC-USDC'), 320: ('book', 'BTC-USD'), 2304: ('book', 'BTC-USDT'), 2336: ('book', 'BTC-DAI')}
2020-12-14 22:46:33,922 : WARNING : KRAKEN: encountered connection issue code = 1006 (connection closed abnormally [internal]), no reason - reconnecting...
2020-12-14 22:49:14,038 : WARNING : KRAKEN: encountered connection issue code = 1006 (connection closed abnormally [internal]), no reason - reconnecting...
2020-12-14 22:50:57,062 : WARNING : KRAKEN: Invalid channel id recevied 1200
2020-12-14 22:50:57,062 : WARNING : KRAKEN: channel map: {2384: ('book', 'BTC-USDC'), 320: ('book', 'BTC-USD'), 2304: ('book', 'BTC-USDT'), 2336: ('book', 'BTC-DAI')}
2020-12-14 22:52:06,903 : WARNING : KRAKEN: Invalid channel id recevied 400
2020-12-14 22:52:06,904 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-15 01:10:57,477 : WARNING : KRAKEN: Invalid channel id recevied 1200
2020-12-15 01:10:57,478 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-15 02:42:58,100 : WARNING : KRAKEN: Invalid channel id recevied 400
2020-12-15 02:42:58,100 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-15 02:55:27,829 : WARNING : KRAKEN: Invalid channel id recevied 4368
2020-12-15 02:55:27,829 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-15 02:55:28,006 : WARNING : KRAKEN: Invalid channel id recevied 3216
2020-12-15 02:55:28,006 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-15 02:55:28,006 : WARNING : KRAKEN: Invalid channel id recevied 2784
2020-12-15 02:55:28,006 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-15 02:55:28,007 : WARNING : KRAKEN: Invalid channel id recevied 4384
2020-12-15 02:55:28,007 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-15 04:11:38,003 : WARNING : KRAKEN: encountered connection issue code = 1006 (connection closed abnormally [internal]), no reason - reconnecting...
2020-12-15 04:11:38,298 : WARNING : KRAKEN: encountered connection issue code = 1006 (connection closed abnormally [internal]), no reason - reconnecting...
2020-12-15 06:20:20,619 : WARNING : KRAKEN: Invalid channel id recevied 400
2020-12-15 06:20:20,619 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-15 07:49:49,770 : WARNING : KRAKEN: encountered connection issue code = 1006 (connection closed abnormally [internal]), no reason - reconnecting...
2020-12-15 07:49:50,379 : WARNING : KRAKEN: encountered connection issue code = 1006 (connection closed abnormally [internal]), no reason - reconnecting...
2020-12-15 08:33:22,514 : WARNING : KRAKEN: Invalid channel id recevied 1200
2020-12-15 08:33:22,514 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-15 09:00:22,660 : WARNING : KRAKEN: Invalid channel id recevied 1200
2020-12-15 09:00:22,660 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-15 11:36:00,332 : WARNING : KRAKEN: encountered connection issue code = 1006 (connection closed abnormally [internal]), no reason - reconnecting...
2020-12-15 13:03:13,351 : WARNING : KRAKEN: Invalid channel id recevied 1200
2020-12-15 13:03:13,351 : WARNING : KRAKEN: channel map: {2384: ('book', 'BTC-USDC'), 320: ('book', 'BTC-USD'), 2304: ('book', 'BTC-USDT'), 2336: ('book', 'BTC-DAI')}
2020-12-15 15:41:43,535 : WARNING : KRAKEN: Invalid channel id recevied 2736
2020-12-15 15:41:43,535 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}
2020-12-15 15:41:43,535 : WARNING : KRAKEN: Invalid channel id recevied 1632
2020-12-15 15:41:43,535 : WARNING : KRAKEN: channel map: {2385: ('trade', 'BTC-USDC'), 321: ('trade', 'BTC-USD'), 2305: ('trade', 'BTC-USDT'), 2337: ('trade', 'BTC-DAI')}

@bmoscon
Copy link
Owner

bmoscon commented Feb 17, 2021

I updated the logging here. Didn't realize we werent actually logging the message. the fact that its always channel id 400 makes me think it some other informational message that is being misinterpreted. If you see it again (and you're using the updated code that logs the message), please post it here @yohplala

@yohplala
Copy link
Contributor Author

@bmoscon thanks, i will.
Yes, not using the updated code yet.
'lazily waiting' for @olibre proposal regarding whale alert.
(much busied on other stuff any way, but yes, as soon as we have something for whale alert, i will give it a test)

@bmoscon
Copy link
Owner

bmoscon commented May 14, 2021

I'm closing this. If it comes up again, let me know and we can try and diagnose it further. I havent seen it happen in a while so closing

@bmoscon bmoscon closed this as completed May 14, 2021
@bmoscon
Copy link
Owner

bmoscon commented May 26, 2021

this is definitely fixed now - kraken deprecated the channel mapping, and I removed it entirely from the kraken code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bmoscon @yohplala