Support 30 GPU and bug of random seed

[Senwang98]

Hi, @xinzhuma
Q1: It seems that code can't run on 30x0 GPU. I guess your code can't support cuda11 now.

Q2:It can't set random seed to do further experiment.
Since you have set fixed random seed now, but it is not really work as expected. Each time you train one model, the final results are different. Do you have any suggestions about it? thanks very much! (This is strange, because we always use this method to make model reproducted)

[alfinnurhalim]

I used cuda 11 and its works

[Senwang98]

I used cuda 11 and its works

Thanks for replying, which type of GPU do you use for traing now?

[Senwang98]

Q1:
The training processing is stucked in

monodle/lib/helpers/trainer_helper.py

Line 54 in e426aa6

self.model = torch.nn.DataParallel(model, device_ids=self.gpu_ids).to(self.device)

[xinzhuma]

@Senwang98

1. the information is limited, maybe you need to check the device_id in the config file is consistent with your machine
2. it works well in my local environment, please ensure you use the same environments when you try different seeds.

[Senwang98]

@xinzhuma
Hi,
Q1: It's my fault!!! The codes is work well now, sorry. I make a mistake, 30 GPU can't support pytorch version < 1.7. pytorch>=1.7 work well except some warning.
Q2: I didn't change my random seed, I keep seed=444,but two training results are different (I use the orginal code of your repo).
I will try again, thanks for your reply.

[xinzhuma]

okay, please tell me if Q2 is confirmed.

[Senwang98]

@xinzhuma
I further test three times training without any change of config file.
Now, the training is stable and the results are reproducted. The best results is 14.16 now.
I will close this issue and sorry for disturb.