Which GPU did you use?

[nessessence]

Sorry, there is training-time show in your experiment. I wonder which GPU did you use, and how many of them?

[GalSang17]

I am also using this implementation, is it impossible to call distributed training?

[jeonsworld]

Transfer learning was performed using v100. Check the relative time of the tensorboard for the learning time.

[jeonsworld]

Distributed training on a single node can be executed as follows.

python3 -m torch.distributed.launch --nproc_per_node=NUM_OF_GPU train.py --train_batch_size BATCH_SIZE_PER_GPU --name cifar10-100_500 --dataset cifar10 --model_type ViT-B_16 --pretrained_dir checkpoint/ViT-B_16.npz 

[TitaniumOne]

@jeonsworld Hello, how can I use multi-GPUs?

[jeonsworld]

@jeonsworld Hello, how can I use multi-GPUs?

There are DataParallel and Distributed ways to use multi-gpu in pytorch.
The current code supports distributed learning and uses the following command.

python3 -m torch.distributed.launch --nproc_per_node=NUM_OF_GPU train.py --train_batch_size BATCH_SIZE_PER_GPU --name cifar10-100_500 --dataset cifar10 --model_type ViT-B_16 --pretrained_dir checkpoint/ViT-B_16.npz