accuracy regression for resnet50 in test_train_imagenet.py

[zcain117]

I've tried 2 ways, both using the real imagenet dataset. I have not submitted #1012 with the resnet50 learning rate scheduler:

1. Brand new compute VM, using pytorch-nightly conda env
2. Brand new compute VM, pull from master on Github and run build_torch_wheels.sh

Both versions reach ~12% accuracy around epoch 2 or 3 and can't make it any higher no matter how many more epochs they train.

I have a different compute VM made on August 28 that uses pytorch-nightly conda env and on that compute VM I am able to reach 60% accuracy with no changes and 76% accuracy when I implement a learning rate schedule.

[zcain117]

Ran a diff of /usr/share/torch-xla-nightly/pytorch/vision/torchvision/models/resnet.py on the old VM that works and the new VM that does not and found no difference

[jysohn23]

Can we try bisect a little more with different Compute VM image dates? You can bisect using the Compute VM image dates using the, for example, --image=debian-9-torch-xla-v20190911 --image-project=ml-images flags when creating with gcloud CLI.

Also, could you try updating the torch_xla wheel in the August 28th VM to just verify it's not us that caused this regression?

[zcain117]

confirmed that test_train_cifar.py passes using the in-line version of resnet18 but fails to reach 80% accuracy using the torchvision version of resnet18

edit:
this doesn't seem like a useful distinction since the same things happens on my compute VM where test_train_imagenet.py has high accuracy

[taylanbil]

did we rule out the lr_scheduler PR? IIRC, high accuracy run is w/o that pr, and the low accuracy is with, is that correct?

[zcain117]

did we rule out the lr_scheduler PR? IIRC, high accuracy run is w/o that pr, and the low accuracy is with, is that correct?

No, I've been using master version of pytorch/xla and haven't checked in that PR

[dlibenzi]

So it seems like a torchvision change?

[zcain117]

So it seems like a torchvision change?

there was no difference in /usr/share/torch-xla-nightly/pytorch/vision/torchvision/models/resnet.py between the older, higher accuracy compute VM and the new compute VMs with bad accuracy

[zcain117]

I've run a lot of experiments and here are the 3 that I think are most enlightening:

1. Copied the image from the high-performing compute VM originally created August 28. Built new torch and torch_xla wheels and torchivision using update_nightly_torch_wheels.sh. Ran against a new TPU VM. This made it to ~45% accuracy on epoch 7, i.e. aligned with the trajectory that gets us to 76% final accuracy.
2. New compute VM built using build_torch_wheels.sh and a new TPU VM. Whether I run on master branch or on the branch that pulls demonstration of scheduler-style learning rate #1012, accuracy can't make it past ~12-15%
3. New compute VM using pytorch-nightly conda env and a new TPU VM made it to ~43% by epoch 11. Looks promising

From this, it seems building from head using build_torch_wheels.sh results in test_train_imagenet.py not working for some reason.

I'll keep investigating

[ailzhang]

@zcain117 I'm a little confused. I thought we use build_torch_wheels.sh to build nightly conda env, no?
fyi https://github.com/pytorch/xla/blob/master/scripts/build_torch_wheels.sh#L98 pins version to torchvision 0.3.0, maybe check whether nightly has master or 0.3.0 to see if that makes a difference?

[zcain117]

@zcain117 I'm a little confused. I thought we use build_torch_wheels.sh to build nightly conda env, no?
fyi https://github.com/pytorch/xla/blob/master/scripts/build_torch_wheels.sh#L98 pins version to torchvision 0.3.0, maybe check whether nightly has master or 0.3.0 to see if that makes a difference?

That might be true, but @jysohn23 pointed out that we override the local source for pytorch/xla and pytorch(?). I think the pytorch/xla override is here but not sure about pytorch.

I just checked and I see 0.3.0 in both the newer conda env that has bad accuracy and the older conda env that has good accuracy.

I think next it's best to see differences in python source code between the high-accuracy and low-accuracy compute VMs

[dlibenzi]

Did we rule out this is not a pytorch or pytorch/xla commit breakage?
IOW, does it exist a valid VM configuration where building from HEAD works?

[zcain117]

Did we rule out this is not a pytorch or pytorch/xla commit breakage?
IOW, does it exist a valid VM configuration where building from HEAD works?

I have not been able to get >20% accuracy when building from head

[zcain117]

I'm preparing 3 compute VMs using different pytorch/xla commits to hopefully see whether or not this problem came from a pytorch/xla change