Error while running max_unpool2d with xla: RuntimeError: torch_xla/csrc/tensor.cpp:817 : Check failed: xla_data != nullptr

[Asar5]

❓ Questions and Help

Hi, i'm extremely new to this platform and I'm trying. my hand at using google colab's TPU.
I'm trying to train a SegNet Model using TPU and this is the error i get:

2020-06-03 03:25:28.796417: E tensorflow/compiler/xla/xla_client/tf_logging.cc:11] Check failed: xla_data != nullptr 
*** Begin stack trace ***
	tensorflow::CurrentStackTrace[abi:cxx11]()
	torch_xla::XLATensor::GatherTensorsXlaData(std::vector<torch_xla::XLATensor, std::allocator<torch_xla::XLATensor> > const&, absl::Span<unsigned long const>, absl::Span<std::shared_ptr<xla::ComputationClient::Data> const>)
	torch_xla::XLATensor::GetTensorsFused(std::vector<torch_xla::XLATensor, std::allocator<torch_xla::XLATensor> >*)
	torch_xla::XLATensor::GetTensors(std::vector<torch_xla::XLATensor, std::allocator<torch_xla::XLATensor> >*)
	torch_xla::bridge::XlaCreateTensorList(c10::ArrayRef<at::Tensor> const&)
	torch_xla::AtenXlaTypeDefault::max_unpool2d(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>)
	c10::detail::wrap_kernel_functor_unboxed_<c10::detail::WrapRuntimeKernelFunctor_<at::Tensor (*)(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>), at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, c10::ArrayRef<long> > >, at::Tensor (at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>)>::call(c10::OperatorKernel*, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>)
	
	
	
	
	
	
	
	_PyEval_EvalFrameDefault
	
	
	
	_PyEval_EvalFrameDefault
	_PyFunction_FastCallDict
	
	PyObject_Call
	_PyEval_EvalFrameDefault
	
	_PyFunction_FastCallDict
	
	
	_PyObject_FastCallKeywords
	
	_PyEval_EvalFrameDefault
	
	
	_PyEval_EvalFrameDefault
	
	PyEval_EvalCode
	
	PyRun_FileExFlags
	PyRun_SimpleFileExFlags
	Py_Main
	main
	__libc_start_main
	_start
*** End stack trace ***

Traceback (most recent call last):
  File "train.py", line 128, in <module>
    train()
  File "train.py", line 57, in train
    predicted_tensor, softmaxed_tensor = model(input_tensor)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/drive/My Drive/Bonus/PySegNet/model.py", line 278, in forward
    **x_4d = F.max_unpool2d(x_4, indices_4, kernel_size=2, stride=2, output_size=dim_4_try)**
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 671, in max_unpool2d
    return torch._C._nn.max_unpool2d(input, indices, output_size)
RuntimeError: torch_xla/csrc/tensor.cpp:817 : Check failed: xla_data != nullptr 
*** Begin stack trace ***
	tensorflow::CurrentStackTrace[abi:cxx11]()
	torch_xla::XLATensor::GatherTensorsXlaData(std::vector<torch_xla::XLATensor, std::allocator<torch_xla::XLATensor> > const&, absl::Span<unsigned long const>, absl::Span<std::shared_ptr<xla::ComputationClient::Data> const>)
	torch_xla::XLATensor::GetTensorsFused(std::vector<torch_xla::XLATensor, std::allocator<torch_xla::XLATensor> >*)
	torch_xla::XLATensor::GetTensors(std::vector<torch_xla::XLATensor, std::allocator<torch_xla::XLATensor> >*)
	torch_xla::bridge::XlaCreateTensorList(c10::ArrayRef<at::Tensor> const&)
	torch_xla::AtenXlaTypeDefault::max_unpool2d(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>)
	c10::detail::wrap_kernel_functor_unboxed_<c10::detail::WrapRuntimeKernelFunctor_<at::Tensor (*)(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>), at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, c10::ArrayRef<long> > >, at::Tensor (at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>)>::call(c10::OperatorKernel*, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>)
	
	
	
	
	
	
	
	_PyEval_EvalFrameDefault
	
	
	
	_PyEval_EvalFrameDefault
	_PyFunction_FastCallDict
	
	PyObject_Call
	_PyEval_EvalFrameDefault
	
	_PyFunction_FastCallDict
	
	
	_PyObject_FastCallKeywords
	
	_PyEval_EvalFrameDefault
	
	
	_PyEval_EvalFrameDefault
	
	PyEval_EvalCode
	
	PyRun_FileExFlags
	PyRun_SimpleFileExFlags
	Py_Main
	main
	__libc_start_main
	_start
*** End stack trace ***

Here's a snippet of the code in model.py where the error is showing up.

        # Encoder Stage - 5
        dim_4 = x_3.size()
        x_40 = F.relu(self.encoder_conv_40(x_3))
        x_41 = F.relu(self.encoder_conv_41(x_40))
        x_42 = F.relu(self.encoder_conv_42(x_41))
        dim_4_try = x_42.size()
        x_4, indices_4 = F.max_pool2d(x_42, kernel_size=2, stride=2, return_indices=True)

        # Decoder
        # print("22===> input of decode = ", x_4.shape)
        dim_d = x_4.size()

        # Decoder Stage - 5
        **x_4d = F.max_unpool2d(x_4, indices_4, kernel_size=2, stride=2, output_size=dim_4_try)**
        x_42d = F.relu(self.decoder_convtr_42(x_4d))
        x_41d = F.relu(self.decoder_convtr_41(x_42d))
        x_40d = F.relu(self.decoder_convtr_40(x_41d))
        dim_4d = x_40d.size()

Can someone help me. understand how to go about debugging this. Thanks!

[Asar5]

FYI, the code works fine on GPU.

[dlibenzi]

The little snippet provides no help unfortunately.
Can you create a Colab which reproduces the issue following this template?

https://github.com/pytorch/xla/blob/master/contrib/colab/issue-report.ipynb

[dlibenzi]

So, max_unpool2d is currently not supported by XLA. We will look into it.
Nevertheless, it is important for us to get a repro of your issue, because the error you get, should not be happening.

[dlibenzi]

This is the first step: #2175

[Asar5]

Hi, thanks for the response. I will create a repro of the issue early next week using the template and get back to you

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Dapeng960208]

hello ，I also encountered the same problem. How did you solve it?

[Asar5]

hello, this is the response i got: So, max_unpool2d is currently not supported by XLA. We will look into it. So I didn't use tpu for this. Regards, Anjali Asar M.Sc. Robotics Oregon State University

…

On Tue, Dec 1, 2020 at 6:47 PM Dapeng960208 ***@***.***> wrote: [This email originated from outside of OSU. Use caution with links and attachments.] hello ，I also encountered the same problem. How did you solve it? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2162 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AN7OBVJQYB34RXWHCIDRJMLSSWTCHANCNFSM4NRJEA4A> .

[JackCaoG]

fyi, max_unpool2d lowering is added

[Asar5]

thats great! thanks! Regards, Anjali Asar M.Sc. Robotics Oregon State University

…

On Tue, Dec 1, 2020 at 7:05 PM JackCaoG ***@***.***> wrote: [This email originated from outside of OSU. Use caution with links and attachments.] fyi, max_unpool2d lowering is added — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2162 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AN7OBVOPQPPAE6J4UVO3YY3SSWVIRANCNFSM4NRJEA4A> .