AttributeError: Can't get attribute '_rebuild_tensor_v2'

[roy7]

This error happens when trying to read in the weights file using an older version of Pytorch. I assume this is why you say Pytorch needs to be built from source. However I've tried that all evening and I can't find a way to navigate all of the nvcc/gcc/cuda incompatibilities to get it to compile. Many errors, all of which are common when I google, with lots of workarounds, but all of them only partially work. Fundamentally it seems like some sort of std::tuple issue with CUDA/nvcc which Nvidia acknowledges but say they won't fix until the next CUDA release.

Is there any chance you could cave your weight file out into an older PyTorch format? Then I could just install python-pytorch-cuda-0.3.1-2 for my version of linux and be up and running in moments. ELF itself compiled fine, and it runs with python-pytorch-cuda-0.3.1-2... it just can't read the weights file. pytorch/pytorch#5729 states it's because of the newer file format.

Thanks!

[roy7]

I did try going to https://pytorch.org/ and choosing Linux / pip / Python 3.6 / CUDA 9.2 which gives me these commands:

pip3 install http://download.pytorch.org/whl/cu91/torch-0.4.0-cp36-cp36m-linux_x86_64.whl
pip3 install torchvision

Which doesn't give me the error above, but I do instead get:

RuntimeError: Error(s) in loading state_dict for Model_PolicyValue:
Unexpected key(s) in state_dict: "init_conv.1.num_batches_tracked", "pi_final_conv.1.num_batches_tracked", "value_final_conv.1.num_batches_tracked", "resnet.resnet.0.conv_lower.1.num_batches_tracked", "resnet.resnet.0.conv_upper.1.num_batches_tracked", "resnet.resnet.1.conv_lower.1.num_batches_tracked", "resnet.resnet.1.conv_upper.1.num_batches_tracked", "resnet.resnet.2.conv_lower.1.num_batches_tracked", "resnet.resnet.2.conv_upper.1.num_batches_tracked", "resnet.resnet.3.conv_lower.1.num_batches_tracked", "resnet.resnet.3.conv_upper.1.num_batches_tracked", "resnet.resnet.4.conv_lower.1.num_batches_tracked", "resnet.resnet.4.conv_upper.1.num_batches_tracked", "resnet.resnet.5.conv_lower.1.num_batches_tracked", "resnet.resnet.5.conv_upper.1.num_batches_tracked", "resnet.resnet.6.conv_lower.1.num_batches_tracked", "resnet.resnet.6.conv_upper.1.num_batches_tracked", "resnet.resnet.7.conv_lower.1.num_batches_tracked", "resnet.resnet.7.conv_upper.1.num_batches_tracked", "resnet.resnet.8.conv_lower.1.num_batches_tracked", "resnet.resnet.8.conv_upper.1.num_batches_tracked", "resnet.resnet.9.conv_lower.1.num_batches_tracked", "resnet.resnet.9.conv_upper.1.num_batches_tracked", "resnet.resnet.10.conv_lower.1.num_batches_tracked", "resnet.resnet.10.conv_upper.1.num_batches_tracked", "resnet.resnet.11.conv_lower.1.num_batches_tracked", "resnet.resnet.11.conv_upper.1.num_batches_tracked", "resnet.resnet.12.conv_lower.1.num_batches_tracked", "resnet.resnet.12.conv_upper.1.num_batches_tracked", "resnet.resnet.13.conv_lower.1.num_batches_tracked", "resnet.resnet.13.conv_upper.1.num_batches_tracked", "resnet.resnet.14.conv_lower.1.num_batches_tracked", "resnet.resnet.14.conv_upper.1.num_batches_tracked", "resnet.resnet.15.conv_lower.1.num_batches_tracked", "resnet.resnet.15.conv_upper.1.num_batches_tracked", "resnet.resnet.16.conv_lower.1.num_batches_tracked", "resnet.resnet.16.conv_upper.1.num_batches_tracked", "resnet.resnet.17.conv_lower.1.num_batches_tracked", "resnet.resnet.17.conv_upper.1.num_batches_tracked", "resnet.resnet.18.conv_lower.1.num_batches_tracked", "resnet.resnet.18.conv_upper.1.num_batches_tracked", "resnet.resnet.19.conv_lower.1.num_batches_tracked", "resnet.resnet.19.conv_upper.1.num_batches_tracked".

[bochen2027]

Hey guys, can you provide a binary instead? failing that, how about a VM/ova with one click compile? or at least some step by step instructions

@yuandong-tian @qucheng @jma127 @shubho

[ghost]

https://github.com/pytorch/ELF#dependencies

"You also need to install PyTorch from source"

https://github.com/pytorch/pytorch#from-source

[roy7]

I know. The problem is after a few hours of trying I gave up on compiling Pytorch from source. I tried all the tricks I could find online, using various gcc versions, making changes to cuda include files, etc etc. Part of it is a bug in nvcc that Nvidia won't fix until next CUDA release, but I just couldn't find the right combinations of tricks to make it compile.

There's no way I'll be alone in this. :)

[ghost]

If you require tricks your environment is set up wrong or perhaps your hardware is old.

I have been using a simple Ubuntu 16.04 environment for the last year and building pytorch from source automatically every single day. 1080 x3 setup.

[bochen2027]

I dont think he is using ubuntu. Recommend using clonezilla to dd harddrive, wipe, install Ubuntu version that ELF used, and then try again.

[roy7]

Correct, I'm on Manjaro / Arch. For others having the same issues I am, here were some of the relevant issue posts that gave some ideas to try. Maybe you'll have better luck than I.

https://devtalk.nvidia.com/default/topic/1028112/cuda-setup-and-installation/nvcc-bug-related-to-gcc-6-lt-tuple-gt-header-/

https://www.bountysource.com/issues/56264805-build-fails-on-gentoo-with-cuda-9-1-gcc-6-4-python-3-5

tensorflow/tensorflow#16165
facebookarchive/caffe2#1898
fireice-uk/xmr-stak#733
pytorch/pytorch#3807
pytorch/pytorch#3828
pytorch/pytorch#3963
pytorch/pytorch#5136
pytorch/pytorch#5729
pytorch/pytorch#5831

[zchen0211]

@roy7 set track_running_stats=True in the operators may help.

[soumith]

hey everyone, i'll get you a pytorch-nightly binary by tomorrow, so that you dont have to go through the trouble of installing from source.

[roy7]

@soumith Thanks! :)

The python-pytorch-cuda maintainer has been working on getting a newer package out but has trouble getting it to compile as well. I realize 0.4.0 might also be too old and only the master branch is compatible. Down the road of course I'm sure a 0.4.1 will come out, and packages will get made. Just takes time.

[jma127]

@roy7 many thanks for braving other distros with our project, and thanks @soumith for preparing the binary!

I'll keep this issue open as I'm very curious to hear about how it goes after the PyTorch issue is resolved.

[soumith]

if you use anaconda python, you can now use:

conda install -c pytorch pytorch-nightly

[soumith]

if you are using volta GPU, use conda install -c pytorch pytorch-nightly cuda90 (updated readme now)

[14327319]

Could you please make saved models backward compatible with older versions of pytorch?

[jma127]

Backwards compatibility with old PyTorch versions is not currently on our radar.