kraken convert-trade-data --pairs option doesn't work

[chitown2016]

Describe your environment

• Operating system: windows 10 Pro
• Python Version: 3.11.7 (inside a freqtrade docker container)
• CCXT version: 4.2.39
• Freqtrade Version: docker-2024.2-dev-5bb4fe16

Note: All issues other than enhancement requests will be closed without further comment if the above template is deleted or not filled out.

Describe the problem:

I've encountered two problems:

convert-trade-data is breaking down in the middle of execution and leaving many files unconverted.

Secondly, I figured that this is because maybe some of the very illiquid coins have bad files so I wanted to restrict the convert-trade-data to a specific number of pairs:

docker compose run --rm freqtrade convert-trade-data --exchange kraken --pairs BTC/USD ETH/USD XRP/USD DOGE/USD ADA/USD SOL/USD LTC/USD DOT/USD BCH/USD LINK/USD --format-from kraken_csv --format-to feather

This also didn't change anything as the function starts to convert all the files in the folder and not just the requested pairs.

Steps to reproduce:

1. Downloaded Kraken_full_history, Kraken_Trading_History_Q3_2023 and Kraken_Trading_History_Q4_2023 from Kraken webpage as suggested in freqtrade documentation
2. Ran the following command: docker compose run --rm freqtrade convert-trade-data --exchange kraken --format-from kraken_csv --format-to feather
3. Also ran the following command: docker compose run --rm freqtrade convert-trade-data --exchange kraken --pairs BTC/USD ETH/USD XRP/USD DOGE/USD ADA/USD SOL/USD LTC/USD DOT/USD BCH/USD LINK/USD --format-from kraken_csv --format-to feather

Observed Results:

Command 2 converts many files but then breaks in the middle and stops.
Command 3 does the same exact thing even though I requested only a small number of coins

Relevant code exceptions or logs

The following command broke down in the middle of execution:

docker compose run --rm freqtrade convert-trade-data --exchange kraken --format-from kraken_csv --format-to feather
With the following error message:

2024-02-13 12:58:05,108 - freqtrade.data.converter.trade_converter_kraken - INFO - C98/EUR: 5721 trades, from 2022-09-29 15:14:14 to 2023-12-31 23:46:46
2024-02-13 12:58:05,215 - freqtrade.data.converter.trade_converter_kraken - INFO - ARPA/EUR: 10829 trades, from 2022-07-28 14:36:17 to 2023-12-31 23:00:24
2024-02-13 12:58:05,278 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
File "/freqtrade/freqtrade/main.py", line 42, in main
  return_code = args['func'](args)
                ^^^^^^^^^^^^^^^^^^
File "/freqtrade/freqtrade/commands/data_commands.py", line 87, in start_convert_data
  convert_trades_format(config,
File "/freqtrade/freqtrade/data/converter/trade_converter.py", line 137, in convert_trades_format
  import_kraken_trades_from_csv(config, convert_to)
File "/freqtrade/freqtrade/data/converter/trade_converter_kraken.py", line 66, in import_kraken_trades_from_csv
  logger.info(f"{pair}: {len(trades_df)} trades, from "
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "nattype.pyx", line 58, in pandas._libs.tslibs.nattype._make_error_func.f
ValueError: NaTType does not support strftime

Also for the following command, the convert-trade-data seems to ignore when I added --pairs option

docker compose run --rm freqtrade convert-trade-data --exchange kraken --pairs BTC/USD ETH/USD XRP/USD DOGE/USD ADA/USD SOL/USD LTC/USD DOT/USD BCH/USD LINK/USD --format-from kraken_csv --format-to feather

In the logs its says:
2024-02-13 13:57:32,686 - freqtrade.configuration.configuration - INFO - Using pairs ['BTC/USD', 'ETH/USD', 'XRP/USD', 'DOGE/USD', 'ADA/USD', 'SOL/USD', 'LTC/USD', 'DOT/USD', 'BCH/USD', 'LINK/USD']

However afterwards it goes ahead and starts to convert all of the csv trade files located in trades_csv

[xmatthias]

support for --pairs in the kraken data conversion has been added in f89147f (roughly one day newer than your version - which is rather unfortunate).

Please update to the latest develop version - where it'll be available / working.

---

The error does indeed look like a wrong / broken file. would you mind sharing the actual pair you suspect as the culprit? (Including the files - as i don't intend to download then full 10G that kraken is giving you again ... and i don't think i have the "last" ones available anymore).

[chitown2016]

The issue is I am not getting many hints about the order of processing of the files. From the log you can see the last file that went through was ARPA/EUR. But they aren't seem to be in alphabetical order. Also they aren't processed in the same order that's printed in the beginning which says "Found csv files for ..." Or is the ARPA/EUR file the problem?

[xmatthias]

processing should happen in the sequence of the output of "Converting pairs:" - which should be shown right before the actual conversion loop happens.

I've just added a new debug statement which will show you the pair that's about to fail (see linked commit) - as you're on docker, please make sure to give ci time to complete before pulling the new version.
To show the new debug message, please append one single -v to the command you're using.

[chitown2016]

Would this be freqtrade:latest? How long do I need to wait? A day or do you think the image is already ready?

[chitown2016]

Ok, I think Ive found the problem files. With the latest image, the error was as follows:

2024-02-13 18:02:16,697 - freqtrade.data.converter.trade_converter_kraken - INFO - USDC/EUR: 5265057 trades, from 2020-01-08 15:17:49 to 2023-12-31 23:59:58
2024-02-13 18:02:19,618 - freqtrade - ERROR - Fatal exception!

And in the above list we have ...,USDC/EUR, BTC/AED,...

I am attaching the BTCAED files that showed up in Kraken_Trading_History_Q3_2023 and Kraken_Trading_History_Q4_2023.
XBTAED.csv
XBTAED.csv

These files seem empty..

[xmatthias]

hm yeah, seems like that's an edge-case i didn't test so far

should be fixed with the commit closing this issue.