How to use LAMMPS to train a larger system?

[stargolike]

I used a system of 200 atoms for training, and selected hidden irreps: '64x0e+64x1o'.
But when I want to use LAMMPS for MD simulation, I can only run a system with 1k atoms. If I use a larger system, there will be a memory out error, such as

RuntimeError: CUDA out of memory. Tried to allocate 2.36 GiB. GPU 0 has a total capacity of 47.45 GiB of which 1.43 GiB is free. Including non-PyTorch memory, this process has 46.01 GiB memory in use. Of the allocated memory 37.87 GiB is allocated by PyTorch, and 7.86 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

The graphics card I am currently using is the RTX8000, which can be used for training.
I would like to ask what this memory is related to and how can I have a larger system for MD

[zeayng]

I encountered the same problem. I tried using smaller hidden layers and r_max to fine-tune the model again, but the results remained unchanged.

[dominicvarghese]

I used V100 GPU (32 GB) to run for simulations between 3500 and 4500 atoms (mainly geometry optimization tasks).