Distributed training

[vict0rsch]

Enables distributed training:

• 1 node
• N GPUs
• N SLURM tasks

(DistributedDataParallel is more efficient than DataParallel so that's why there's N 1-GPU tasks instead of 1 N-GPUs task)

[vict0rsch]

• [wandb] 4 GPUs New ForceNet
  

• [wandb] 4 GPUs New SchNet
  

  • not great but at least all gpus are used ; may be due to lack of metadata.npz

[vict0rsch]

2022-06-06 18:08:06 (WARNING): No metadata file found at '/network/projects/_groups/ocp/oc20/is2re/all/train/metadata.npz'. Batches will not be balanced, which can incur significant overhead!

Following the instructions in TRAIN.md I ran the following commands to create those files

python scripts/make_lmdb_sizes.py --num-workers 6 --data-path /network/projects/ocp/oc20/is2re/all/test_id/data.lmdb
python scripts/make_lmdb_sizes.py --num-workers 6 --data-path /network/projects/ocp/oc20/is2re/all/test_ood_both/data.lmdb
python scripts/make_lmdb_sizes.py --num-workers 6 --data-path /network/projects/ocp/oc20/is2re/all/train/data.lmdb
python scripts/make_lmdb_sizes.py --num-workers 6 --data-path /network/projects/ocp/oc20/is2re/all/val_ood_ads/data.lmdb
python scripts/make_lmdb_sizes.py --num-workers 6 --data-path /network/projects/ocp/oc20/is2re/all/val_ood_cat/data.lmdb
python scripts/make_lmdb_sizes.py --num-workers 6 --data-path /network/projects/ocp/oc20/is2re/all/test_ood_ads/data.lmdb
python scripts/make_lmdb_sizes.py --num-workers 6 --data-path /network/projects/ocp/oc20/is2re/all/test_ood_cat/data.lmdb
python scripts/make_lmdb_sizes.py --num-workers 6 --data-path /network/projects/ocp/oc20/is2re/all/val_id/data.lmdb
python scripts/make_lmdb_sizes.py --num-workers 6 --data-path /network/projects/ocp/oc20/is2re/all/val_ood_both/data.lmdb
python scripts/make_lmdb_sizes.py --num-workers 6 --data-path /network/projects/ocp/oc20/is2re/100k/train/data.lmdb
python scripts/make_lmdb_sizes.py --num-workers 6 --data-path /network/projects/ocp/oc20/is2re/10k/train/data.lmdb