Double Ensemble Model neads more than 100GB memory. How to reduce it?

[Wendroff]

🌟 Feature Description

How to reduce the memory cost of double ensemble model.

Motivation

I tried to use DoubleEnsembleModel which seems to be the best over Alpha158 dataset. However, I failed with error:

numpy.core._exceptions._ArrayMemoryError: Unable to allocate 119. GiB for an array with shape (8174296, 1960) and data type float64

8174296 = sample_number * factor_number
1960 = traing steps

Can we reduce the memory cost of the training process?

By the way, it's not parallel runing when selected features and calculated the sample weights.

[chuhuan88]

Hello, can you share the downloaded dataset with me, I can't get on Yahoo, thank you very much.

[Fivele-Li]

I run
qrun workflow_config_doubleensemble_Alpha158.yaml
but can't reproduce the problem.
Could you please send us more details about your error(e.g the backpack information, your yaml file)?