kraken fetch_trades pagination broken

[xmatthias]

• OS:
• Programming Language version: python
• CCXT version: 2.1.96

based on the following article, pagination in kraken's fetch_trades endpoint works "id based" (in an odd way though).

• use the parameter "since" to get trades from a starting date (this is clear).
• use the parameter "since" with the "last" id from the prior trade, which is provided as 'last' in the response.

This used to work in older versions of ccxt (i can't really pin it down when it stopped working, but i did try kraken not too long ago and it seemed to work fine).

import ccxt
exchange = ccxt.kraken()
exchange.verbose = True
# Intentionally circumvent ccxt's "since" conversion, otherwise it'll break the trade-id
res = exchange.fetch_trades('XRP/USD', params={'since': '1495136532757496916'})
print("last trade: ", res[-1])

1495136532757496916 is a "last" id i got from a direct call.

fetch Response: kraken GET https://api.kraken.com/0/public/Trades?pair=XXRPZUSD&since=1495136532757496916 200 ResponseHeaders: {'Date': 'Thu, 24 Nov 2022 19:37:49 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'cache-control': 'max-age=0, public', 'content-encoding': 'gzip', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'referrer-policy': 'strict-origin-when-cross-origin', 'Last-Modified': 'Thu, 24 Nov 2022 19:32:42 GMT', 'CF-Cache-Status': 'EXPIRED', 'Set-Cookie': '__cf_bm=_5nP7NXI1ShVP3fSqRa9_w3H9Ler855JM9J4qnmeSe0-1669318669-0-AdODTxQL+YorN7kYj4TeiMXpYog4tQy7vcMsWJUCpqymbfRiWtggRGcTIU6nXuGgafJI0ZyrBNPETQS+FRApXRk=; path=/; expires=Thu, 24-Nov-22 20:07:49 GMT; domain=.kraken.com; HttpOnly; Secure; SameSite=None, __cfruid=d8878ddfa28d5998c95a0aa263e058cfe410cee8-1669318669; path=/; domain=.kraken.com; HttpOnly; Secure; SameSite=None', 'Vary': 'Accept-Encoding', 'X-Content-Type-Options': 'nosniff', 'Server': 'cloudflare', 'CF-RAY': '76f499f40b37b3a7-MUC'} ResponseBody: {"error":[],"result":{"XXRPZUSD":[["0.36000000","1119.80434694",1495136549.6543894,"s","l","",341],["0.36000000","985.21940000",1495136549.7110112,"s","l","",342],

....
// Many trades later ...

["0.34999600","7.07700000",1495155045.2222428,"b","l","",1337],["0.35000000","2278.70937225",1495155045.2327409,"b","l","",1338],["0.35000000","326.20030000",1495155047.4302456,"b","l","",1339],["0.35000000","1511.48170000",1495155050.3065326,"b","l","",1340]],"last":"1495155050306532562"}}
last trade: {'id': '1340', 'order': None, 'info': ['0.35000000', '1511.48170000', '1495155050.3065326', 'b', 'l', '', '1340', '1495155050306532562'], 'timestamp': 1495155050306, 'datetime': '2017-05-19T00:50:50.306Z', 'symbol': 'XRP/USD', 'type': 'limit', 'side': 'buy', 'takerOrMaker': None, 'price': 0.35, 'amount': 1511.4817, 'cost': 529.018595, 'fee': None, 'fees': []}

If we look at the verbose output - i get a new "last" id in 1495155050306532562.
This "id" is however not properly passed into the "id" field - which oddly is now 1340.

ccxt even has some logic for this in place here:

ccxt/js/kraken.js

Lines 1182 to 1185 in 5d4e51f

	const lastTrade = trades[length - 1];
	const lastTradeId = this.safeString (result, 'last');
	lastTrade.push (lastTradeId);
	return this.parseTrades (trades, market, since, limit);

But it seems to no longer properly work (for no apparent reason).

It's very well possible that this is caused by a change in the kraken API (now returning one more column) - but that's a suspicion, nothing more.
most likely this:

Nov 2022 - Added tick_size and status parameters to AssetPairs, status and collateral_value to Assets, and trade_id to public Trades.

source

[sc0Vu]

Hi @xmatthias

Looks like kraken added and returned trade_id 1340. We probably can overwrite the last id. How do you think @carlosmiei ?

[carlosmiei]

@sc0Vu as I said not sure what is the best approach here, but overriding the last trade id seems a bit tricky. This trade's id would have a different format from the others.

[xmatthias]

i'm not quite sure what the new "tradeid" column is supposed to represent.

at first glance, it looks somehow like a running number for each pair - but is clearly not valid as pagination "since" - which is (according to the linked article above) also a trade-id (and more aligned with actual id's executed trades get assigned).

For my usecase (download trade history), i can also piece the pagination ID out of the info object with something around trades[-1]['info'][-1] (untested, and probably slightly off) - but it seems important to have pagination working without workarounds or "info" for ccxt in general.