Memory management help

[DanielFigueiredo1992]

Hello.
I have a very big dataset that i want to use on pycaret on a ubuntu remote server. But Ubuntu server are always killing my python script because out of memory. Im working on time series module Thats my script

import resource
from pycaret.time_series import setup, compare_models, predict_model, plot_model, save_model
import dask.dataframe as dd




def main():
    # Carregar o dataset usando Dask
    dtypes = {
        'open_time': 'int64',
        'open': 'float32',
        'high': 'float32',
        'low': 'float32',
        'close': 'float32',
        'volume': 'float32',
        'close_time': 'int32',
        'quote_volume': 'float32',
        'count': 'int32',
        'taker_buy_volume': 'float32',
        'taker_buy_quote_volume': 'float32',
        'ignore': 'float64'
    }

    usecols = ['close_time', 'close', 'high', 'volume', 'quote_volume','count','taker_buy_volume', 'taker_buy_quote_volume']  # Carregar apenas as colunas necessárias para a análise

    # Carregar dataset usando Dask com dtypes e usecols especificados
    print("Leitura de datasets iniciada")
    df = dd.read_csv('merged.csv', dtype=dtypes, usecols=usecols)
    df = df.persist()
    df = df.compute()

    # Supondo que sua variável alvo (se houver) esteja armazenada em uma coluna chamada 'close'
    y = df['close']

    # Exibir a forma do dataset original
    print("Original dataset shape:", df.shape)
    df.reset_index(drop=True, inplace=True)

    # PyCaret setup and model comparison
    s = setup(data=df, target='close', session_id=123, fh=3, use_gpu=False)  # Usar 3 núcleos de CPU
    print("Comparação iniciada")
    del df

# Force garbage collection to free up memory
    import gc
    gc.collect()

    best = compare_models(turbo=True)  # Pode retornar 'killed' aqui se o limite de memória for atingido

    print("Previsão")
    predict_model(best)
    plot_model(best, 'forecast')
    save_model(best, 'caminhoparasalvaromodelo')
main()

[BradKML]

Seconding this, but this is even the care for classification and regression. What is the RAM usage of the DataFrame?
See also this major concept #1297 (reply in thread)