Fix DDP final eval

[vict0rsch]

• distutils.synchronize() after trainer.save() to prevent checkpoint corruption
• add git_checkout command-line arg to sbatch.py to ensure enqueued jobs have the appropriate code state
  • defaults to None meaning the training will use the code as it is when the job starts (not when it is queued)
  • writes git checkout {git_checkout} to the sbatch file so use as:
    • python sbatch.py git_checkout=your-branch
    • python sbatch.py git_checkout=somecommithash
• removes the error.txt files, instead use a per-task output file output-%t.txt where %t is the task id in a single job (%j) multi-task (%t) setting such as ours

[vict0rsch]

Waiting for a full training to complete before merging

python sbatch.py py_args="--mode train --config-yml configs/is2re/10k/schnet/new_schnet.yml" note="Distributed training test" git_checkout=fix-ddp mem=96GB