Different recognition results on different GPU

[RudikAleksandr]

I did a training for the same images with the same parameters for image size, number of epochs on different devices, but the recognition result for different devices is different. Why is this happening?

First device configuration:

• CPU: Intel Core i5-8400 2.80GHz
• GPU: GeForce GTX 1080/PCIe/SSE2
• OS: Ubuntu

Recognition result after training:

Second device configuration:

• CPU: Intel Core i5-8250U 1.60GHz
• GPU: GeForce MX150
• OS: Windows 10

Recognition result after training:

[glenn-jocher]

👋 Hello! Thanks for asking about training reproducibility. YOLOv5 🚀 uses a single training seed which is set here using the init_seeds() function. CPU and Single-GPU trainings should be fully reproducible with torch>=1.12.0. Multi-GPU DDP trainings are still not reproducible unfortunately. This is an open issue for us and we could use any help in tracking down this problem.

yolov5/train.py

Line 104 in 7215a0f

init_seeds(opt.seed + 1 + RANK, deterministic=True)

This function sets python, numpy and torch seeds and updates cudnn settings:

yolov5/utils/general.py

Lines 198 to 214 in 7215a0f

	def init_seeds(seed=0, deterministic=False):
	# Initialize random number generator (RNG) seeds https://pytorch.org/docs/stable/notes/randomness.html
	# cudnn seed 0 settings are slower and more reproducible, else faster and less reproducible
	import torch.backends.cudnn as cudnn
	if deterministic and check_version(torch.__version__, '1.12.0'): # https://github.com/ultralytics/yolov5/pull/8213
	torch.use_deterministic_algorithms(True)
	os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8'
	os.environ['PYTHONHASHSEED'] = str(seed)
	random.seed(seed)
	np.random.seed(seed)
	torch.manual_seed(seed)
	cudnn.benchmark, cudnn.deterministic = (False, True) if seed == 0 else (True, False)
	torch.cuda.manual_seed(seed)
	torch.cuda.manual_seed_all(seed) # for Multi-GPU, exception safe

To set a new training seed for example:

python train.py --seed 3  # default seed=0

Note that even when using the same seed trainings may produce different results, especially when using CUDA backends. See https://pytorch.org/docs/stable/notes/randomness.html for details on factors affecting PyTorch training reproducibility and sources of randomness.

Good luck 🍀 and let us know if you have any other questions!

[RudikAleksandr]

Hello, @glenn-jocher!

Thanks for your detailed answer.

Could you also explain how to train the model on a local device so that the result tends to be the same for different developers with the same input?

It is also necessary to understand that the model will recognize things in production, as we expect this on a local device.

[glenn-jocher]

@RudikAleksandr in general any differences should be relatively small. For torch 1.12 trainings should be fully reproducible on the same hardware. Even when not fully reproducible, i.e. torch 1.11 trainings should still be similar. On COCO for example the mAP std between trainings might be 1% of the total, or about 0.005 mAP.

Deployment should produce identical metrics on different hardware though no matter where they were trained.