Does MatterSim support efficient multi-core CPU parallel training/inference?

[sjli-gravity]

Hi MatterSim developers,

I would like to ask about the CPU parallel performance of MatterSim for training/inference workloads.

Currently, I am running training using:

torchrun --nproc_per_node=1 finetune_train.py
--load_model_path 30float.pth
--train_data_path ./train.extxyz
--epochs 1
--device cpu

The training dataset contains about 3600 structures, and I benchmarked the performance for one training epoch.

The observed performance is:

• 160-core CPU node: ~40 minutes
• Single NVIDIA H200 GPU: ~4 minutes

A ~10x slowdown compared to one GPU seems much larger than I expected, especially considering the large number of CPU cores available.

During the CPU run, top shows:

• only one Python process,
• but extremely high CPU utilization (~10000% CPU),
• while the machine still has many idle CPU resources.

For example:

%Cpu(s): 50.4 us, 12.5 sy, 36.5 id
python3.12 ... %CPU 10032

I would like to ask:

1. Is this behavior expected for MatterSim/PyTorch CPU training?
2. Does MatterSim currently support efficient CPU scaling across many cores?
3. Are there recommended settings for:
   • OpenMP,
   • MKL,
   • PyTorch threading,
   • NUMA binding,
   • distributed CPU training,
   • or data parallelism?
4. Is CPU-only large-scale training considered practical for MatterSim?
5. Are there known bottlenecks limiting CPU scaling efficiency?

More generally, my goal is to understand whether large-scale CPU parallelism can realistically compensate for the performance gap between CPUs and GPUs for MatterSim workloads.

Thank you very much!