Error while Running monodepth2 on Jetson Nano

[arjunkeerthi]

Hi, thanks a lot for this project. I am trying to use your work for a summer research project at my university and want to run monodepth2 on the Nvidia Jetson Nano. However, when I try to run the example command to test out the depth prediction for a single image, I get this error:

Weirdly, the file in question (SpatialUpSamplingBilinear.cu) doesn't exist anywhere on my machine. I installed pytorch using this guide: https://devtalk.nvidia.com/default/topic/1049071/jetson-nano/pytorch-for-jetson-nano/post/5324123/#5324123
(This might be a pytorch specific issue, in which case I'd be happy to close this)

Thank you!

[mdfirman]

Thanks for your interest in this repo, and thanks for the interesting issue!

We have never seen this problem on our machines here, though we are using more 'traditional' GPUs.

There is an issue here with pytorch which seems to be similar. There they are using the Nvidia TX-2, which seems to be similar to the Jetson Nano.Unfortunately the solutions proposed there seem to involve modifying the pytorch source code, which is a bit of a pain.

[mrharicot]

To add to michael's answer, you might want to consider converting the model to TensorRT.
https://github.com/onnx/onnx-tensorrt

It will likely fail due to the use of an upsample bilinear, but you could replace them with nearest neighbor upsamples and fine tune the models a bit.

[arjunkeerthi]

Thank you for the suggestions!

@mdfirman I had already seen that issue, but the problem with that is the file in question doesn't seem to be on my machine anywhere (I was running into separate errors with building pytorch from source, so I used the prebuilt wheels provided at the link).

@mrharicot I'll be sure to look into that.

[mdfirman]

@arjunkeerthi thanks, please do report back here so that other people using this repo on a Jetson don't run into the same problem.

[arjunkeerthi]

@mdfirman Hi, I'm still working on the suggestion @mrharicot provided. I'm pretty new to this stuff so it may take me awhile, but feel free to close this and I'll get back when I gotten more done. Thanks.

[arjunkeerthi]

Hi, sorry for the super late reply. I did try @mrharicot 's suggestion, and was able to successfully convert the encoder over to TensorRT. But reflection padding is not supported in TensorRT (only zero-padding is included), so I was unable to convert the decoder. Since I'm busy with other components of our project, I didn't pursue this, but I suppose it wouldn't be overly difficult to add support for this in TensorRT and be able to convert the decoder model.

But as quick fix, changing the mode from "bilinear" to "nearest" in the call to torch.nn.functional.interpolate() in test_simple.py (line 127-128) fixed it on the Nano:

disp_resized = torch.nn.functional.interpolate(disp, (original_height, original_width), mode="nearest")

I didn't notice any glaring differences between the disparity images produced from these modes (I tested with "bilinear" on my personal laptop), but my testing was not very extensive. Using pytorch version 1.1.0 and torchvision version 0.3.0, the Nano took about 0.5 seconds per image. When running on a webcam, we were able to achieve roughly 4fps.

Hopefully this helps someone avoid a lot of headache in trying to get this running on the Jetson Nano.

[mdfirman]

Super, thank you for the update!

[remindchobits]

Hello!
I have the same problem, I got a jetson nano as well and I cannnot find the
SpatialUpSamplingBilinear.cu as well, is there any solution for it? thx!

[mrharicot]

Hi,
It seems that using nearest neighbor upsampling is the best way to go here.

But as quick fix, changing the mode from "bilinear" to "nearest" in the call to torch.nn.functional.interpolate() in test_simple.py (line 127-128) fixed it on the Nano.

[remindchobits]

Hi,
thank you for the advice, but could you please show your torch version pls?
when I switch 'bilinear' to 'nearest' I got such an error:

ValueError: align_corners option can only be set with the interpolating modes: linear | bilinear | bicubic | trilinear

thanks!

[arjunkeerthi]

Hi @remindchobits, sorry I forgot to add in my comment above that when switching to nearest neighbor, you need to take out the last argument (align_corners=True) since it can only be used with the modes listed in the error message given.
Edit: I added the full line to my previous comment.

[remindchobits]

Hi, @arjunkeerthi
Thank you! it works :)