[WIP] Memory Leak when using MarketCapPairList

[devilsinthedetail]

Describe your environment

Operating system: Ubuntu
Python Version: Python 3.9.16 (python -V)
CCXT version: ccxt==4.3.11 (pip freeze | grep ccxt)
Freqtrade Version: freqtrade 2024.5-dev-e2a9bc9c6 (freqtrade -V or docker compose run --rm freqtrade -V for Freqtrade running in docker)

Describe the problem:

Bot during Live keeps crashing as its running out of memory (runs for roughly 2 weeks until it hits 4Gb)

I profiled it a bit but have not much time to go into more detail (thats why its tagged worked in progress). The strategy file + config works in older versions (<2024.2) as its not using the MarketCapPairList but a remote pairlist. Now when using MarketCapPairList and a version >2024.2 it keeps accumulating memory until it crashes (roughly 4Gb in two weeks but also noticeable after some hours running when logging output of htop).

It could be any other addition introduced after 2024.2 independent of the pairlist but unsure how to profile this. Any hints are appreciated. I dont think its the strategy as it did not change and its a very simple one (its also not using any custom logic where this might play a role: #10166).

Steps to reproduce:

No steps yet, hopefully have more time to check soon

[stash86]

using the pairlist myself, never get such error.

[xmatthias]

you'll need to be more precise on that for sure
MarketCapPairList does support 2 modes (filter, generator) depending on it's position in the pairlist - and depending on which way it's used, there's (slightly) different code-paths it can take.

Both a look through the of that pairlist, as well as testing it under extreme conditions (very low refresh_interval) don't show any increase in memory usage - or reasons for that.

Unless we get some more info, i'll be going to close this - as it's not providing anything useful at this point - and it doesn't seem to reproduce, either.

[devilsinthedetail]

Hey @xmatthias, thanks for answering. I debugged it a bit more and found a configuration which should be able to reproduce the problem. Its now independent of the strategy and uses the following config (also independent of MarketCapPairList, sorry about the confusion).

The config I use:

{
    "max_open_trades": 121,
    "stake_currency": "BTC",
    "stake_amount": "unlimited",
    "tradable_balance_ratio": 0.99,
    "fiat_display_currency": "USD",
    "dry_run": true,
    "dry_run_wallet": 1000,
    "cancel_open_orders_on_exit": false,
    "trading_mode": "spot",
    "margin_mode": "",
    "unfilledtimeout": {
        "entry": 10,
        "exit": 10,
        "exit_timeout_count": 0,
        "unit": "minutes"
    },
    "entry_pricing": {
        "price_side": "same",
        "use_order_book": true,
        "order_book_top": 1,
        "price_last_balance": 0.0,
        "check_depth_of_market": {
            "enabled": false,
            "bids_to_ask_delta": 1
        }
    },
    "exit_pricing":{
        "price_side": "same",
        "use_order_book": true,
        "order_book_top": 1
    },
    "exchange": {
        "name": "binance",
        "key": "",
        "secret": "",
        "ccxt_config": {},
        "ccxt_async_config": {},
        "pair_whitelist": [
        ],
        "pair_blacklist": [
            "BNB/.*"
        ]
    },
    "pairlists": [
        {
            "method": "VolumePairList",
            "number_assets": 221,
            "sort_key": "quoteVolume",
            "min_value": 0,
            "refresh_period": 1800
        }
    ],
    "telegram": {
        "enabled": false,
        "token": "",
        "chat_id": ""
    },
    "api_server": {
        "enabled": false,
        "listen_ip_address": "127.0.0.1",
        "listen_port": 8080,
        "verbosity": "error",
        "enable_openapi": false,
        "jwt_secret_key": "b76bd81e929027a5fac344d1032f124d9828728fe8b6bf96ce913cd390853383",
        "ws_token": "tBye0H-LlOZU24XNmo60cf97I4j8G-Ohbw",
        "CORS_origins": [],
        "username": "",
        "password": ""
    },
    "bot_name": "freqtrade",
    "initial_state": "running",
    "force_entry_enable": false,
    "internals": {
        "process_throttle_secs": 5
    }
}

The strategy I tested it with to make sure its not mine is Strategy001 and I run it with
freqtrade trade --db-url=sqlite:///btc.sqlite --config=user_data/config.json --strategy=Strategy001 --dry-run

When decreasing number of pairs the increase of memory slows down but does not go away. I run it on a raspberry pi 5, but confirmed its also present on a raspberry pi 4 on which I had freqtrade running for a long time without problems, thats why I assume its something which is related to a change in the codebase. Last test was on freqtrade 2024.5-dev-2c740059d

[xmatthias]

quite honestly - that config uses 121 (!) max open trades.

If all of these trade-slots are filled (which WILL happen over time) - then i'm not entirely sure what the memory requirements will come out as, but (in combination with the 200+ pairs) - it could be non-neglectible.

With that config - in what time would you expect memory to raise? Does it also happen if you use a sane number of open trades?

[devilsinthedetail]

Yes but as I said it will be much slower. I think I tried with 3 and checked for a while but it only increased by like 20Mb over two days, but with 121 it goes up way faster (1Gb in two days). Also it goes up without ever filling a trade, so the leak has to have something to do with the pairlist?

[xmatthias]

well what you're saying makes no sense

if using max_open_trades=3 vs. 121 makes a difference - also without filling a trade - then that's something that's simply impossible - as this is one open_trades < max_open_trades) comparison - where the actual number you compare against won't matter.
if python would leak here depending if the comparison is against 3 or 121, then i think we'd all know about.

So either, trade entries / fills do make a difference (hence, keeping more trades in memory while they're being processed) - or it doesn't - so i can use a strategy without any signals to trigger this - which however also means that max_open_trades is irrelevant (which above you said - isn't the case).

Now i'm running bots - sometimes for months at a time - without noticing any memory going up (it's not immediately at it's height - but it's not leaking, either) - so in a random configuration / combination (with uptodate dependencies) - is something i can pretty much exclude (independent of fills - as my bots DO actually trade).

It quite honestly sounds like "i'm fishing in the dark, pointing fingers at everything and everyone" (we've been at pairlist, now at max_open_trades, what's next?) - all the while changing random things and taking more guesses.
I'll treat this as a question for now - as in reality - there's nothing to actually look at other than random guesses.

[xmatthias]

Closing this as we had no further feedback, and it's not something i can reproduce (nor something i've heared from others having the same problem).