Import Error when calling the propagator

[erellaz]

Thanks for writing Deepwave and generously putting it on Github.

Each time I try to use the propagator I have the following import error. I have tried with various versions of pytorch (18 and 17.1), as well as with various examples (the main example and the seam example provided on kolab) and am not sure which direction to go from here. Any pointer would be helpful. Thanks!

Traceback (most recent call last):
File "deepwave_demo.py", line 26, in
receiver_amplitudes = prop(source_amplitudes, x_s, x_r, dt)
File "/home/gu/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/gu/.local/lib/python3.8/site-packages/deepwave/base/propagator.py", line 144, in forward
return self.propfunc.apply(source_amplitudes, source_locations,
File "/home/gu/.local/lib/python3.8/site-packages/deepwave/scalar/scalar.py", line 90, in forward
scalar_wrapper = _select_propagator(model.ndim, vp.dtype, vp.is_cuda)
File "/home/gu/.local/lib/python3.8/site-packages/deepwave/scalar/scalar.py", line 320, in _select_propagator
import scalar2d_cpu_iso_4_float as scalar_wrapper
ImportError: /home/gu/.local/lib/python3.8/site-packages/scalar2d_cpu_iso_4_float.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe28TypeMeta21_typeMetaDataInstanceISt7complexIdEEEPKNS_6detail12TypeMetaDataEv

[ar4]

Thank you for trying Deepwave, especially when it unfortunately didn't work. I have not encountered that error before. I wonder if it might be caused by some mismatch between the compilation and execution environments, or between versions of installed packages. Can you tell me how you installed Python, PyTorch, and Deepwave? Have you tried creating a virtual environment to keep your Python package setup clean? If it is possible to do your work on Colaboratory rather than on your local computer, you can run it on there instead and I am quite sure that you will not encounter such problems.

…

On Tue, 30 Mar 2021 at 20:08, erellaz ***@***.***> wrote: Thanks for writing Deepwave and generously putting it on Github. Each time I try to use the propagator I have the following import error. I have tried with various versions of pytorch (18 and 17.1), as well as with various examples (the main example and the seam example provided on kolab) and am not sure which direction to go from here. Any pointer would be helpful. Thanks! Traceback (most recent call last): File "deepwave_demo.py", line 26, in receiver_amplitudes = prop(source_amplitudes, x_s, x_r, dt) File "/home/gu/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/gu/.local/lib/python3.8/site-packages/deepwave/base/propagator.py", line 144, in forward return self.propfunc.apply(source_amplitudes, source_locations, File "/home/gu/.local/lib/python3.8/site-packages/deepwave/scalar/scalar.py", line 90, in forward scalar_wrapper = _select_propagator(model.ndim, vp.dtype, vp.is_cuda) File "/home/gu/.local/lib/python3.8/site-packages/deepwave/scalar/scalar.py", line 320, in _select_propagator import scalar2d_cpu_iso_4_float as scalar_wrapper ImportError: /home/gu/.local/lib/python3.8/site-packages/ scalar2d_cpu_iso_4_float.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe28TypeMeta21_typeMetaDataInstanceISt7complexIdEEEPKNS_6detail12TypeMetaDataEv — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#27>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACMJSKUCHE57JG6I4ZKG6XLTGIOR7ANCNFSM42CX4AFQ> .

[erellaz]

Thanks for your very prompt response and your time. Really appreciated.

I just did an install of Ubuntu 20.10, which comes with Nvidia GPU drivers for my card. Then I installed cuda (sudo apt install nvidia-cuda-toolkit), then Anaconda, then pytorch (conda install pytorch torchvision cudatoolkit=10.1 -c pytorch) then deepwave. I tried both versions of deepwave (stable and dev) with (pip3 install deepwave). This installed pytorch 1.8. As it was not working I tried pytorch 1.7.1 (pip3 install "torch==1.7.1") since it seems to be the version preferred by deepwave. The message when calling the propagator remains the same.
The Python environment should be clean, as Deepwave and its dependency are the only things I have installed since installing Linux.
I will try to repeat the procedure, but without using Anaconda in a new virtual environment.

[ar4]

I'm sorry that it is causing such trouble for you. That does sound like the setup should have been very clean. When you changed to PyTorch 1.7.1, did you reinstall Deepwave? I don't know whether pip will do that automatically, but, if not, then I imagine that you might need to do it manually since I think the compiled Deepwave objects will depend on the PyTorch libraries used during compilation. The fact that you got the same error message with both 1.8.1 and 1.7.1 is not what I would have expected if this is the problem, though. I see that the error message you posted was for the CPU version of the propagator. Do you get a similar message when you try to run on the GPU? I have PyTorch 1.7.1 on my computer, and Colaboratory uses 1.8.1, and I have no problem with Deepwave on either, so I do not think the PyTorch version is the problem. Interestingly, the symbols in the compiled objects that I have, and that I get when I install Deepwave on Colaboratory, are different than the one that your compiled object wants. The symbols present in my scalar2d_cpu_iso_4_float.cpython-38-x86_64-linux-gnu.so are of the form _ZZN8pybind1112cpp_..., and they are the same on Colaboratory. None of the symbols have "caffe" in the name, but it seems that your one does. I don't know why that might be. Did you get the same result when you tried the installation again?

[erellaz]

Thanks again for your continued support!

When changing torch versions I removed deepwave first, then reinstalled deepwave after the other version of torch is set up:
pip3 uninstall deepwave
pip3 uninstall torch
pip3 install "torch==1.7.1"
pip3 install deepwave

Both CPU and GPU give similar error, with caffe in the symbol of the compiled object.

Caffe2 comes with the Cuda toolkit, so I tested removing the Cuda toolkit:

sudo apt-get remove nvidia-cuda-toolkits
sudo apt-get autoremove nvidia-cuda-toolkit
sudo apt-get remove --purge nvidia*

and re installed torch & deepwave without Cuda toolkit, but the error persists. However, I am not 100% sure that I removed all the dependencies coming with the first install of the Cuda toolkit. At this point I am testing various combinations of Cuda & pytorch versions to see if the error goes away. I will also test various Nvidia drivers. If that fails I will reinstall Ubuntu, which may be the quickest way to get rid of all cuda dependencies with certainty.

[ar4]

You definitely seem to have done everything right, so I must have made a mistake somewhere. I checked that the GPU version also runs on Colaboratory, so it's not as obvious as it being caused by nvidia-cuda-toolkit. Tomorrow morning I will try installing on Ubuntu 20.10 to see if that might be the cause of the problem (I've only tried it on 20.04 before).

[ar4]

Just one thing to check, though: It appears (from https://packages.ubuntu.com/groovy/nvidia-cuda-toolkit) that the CUDA toolkit version in Ubuntu 20.10 is 11, but in your PyTorch install command ("conda install pytorch torchvision cudatoolkit=10.1 -c pytorch") it looks like you specified toolkit version 10.1.

[erellaz]

Just tried Cuda toolkit 11. Unfortunately it did not help.

[ar4]

I tried on Ubuntu 20.10 and didn't receive any errors. The steps I followed after a fresh install of Ubuntu 20.10: sudo apt update sudo apt install python3-pip wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh chmod +x Miniconda3-latest-Linux-x86_64.sh ./Miniconda3-latest-Linux-x86_64.sh [allow Miniconda to add setup to .bashrc, then restart terminal] conda install pytorch torchvision torchaudio cpuonly -c pytorch conda install scipy pip install deepwave python3 test.py where test.py contains the "full example" from forward modelling in the Deepwave README.md. For this test I didn't try with a GPU. Do you have time to try again to check if this also works for you? If it does, then I will try with a GPU to see if the problem occurs in that case.

[erellaz]

I tried the above procedure on a new install of Ubuntu 20.10, and it works perfectly. No more error message. So I will stay on CPUs for the time being and enjoy the possibilities of Deepwave.

Thanks again.

[ar4]

That's wonderful. Thank you for persevering (I know that reinstalling is not fun), and I hope that it is worth it. It's so much better when you can use the GPU, though, so I'm not satisfied with you not being able to use that. Tomorrow morning I will try 20.10 with a GPU and see if I get the error.

[ar4]

I used the CUDA version of PyTorch on Ubuntu 20.10 and still didn't receive any error messages. I followed the same steps as above on a fresh install, with the only difference being that I used "conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge" to install PyTorch. I did not install Ubuntu's nvidia-cuda-toolkit package as I believe the PyTorch installation includes everything it needs.

[erellaz]

Thanks again!
I tried the above method to install Pytorch with cudatoolkit 11.1. When installing Deepwave it gives an error (CUDA_HOME variable not set) and indeed Cuda is not installed (at least not in /usr/local/cuda). I did not investigate too deeply, but instead reversed to CPUs only, which should be fine for my current application.
Thanks a lot for all the help and the time spent on it. At this point I am very happy to have a functional version of deepwave and will focus on my application.

[ar4]

Ah, it is a pity that it did not work. Thank you for trying. I am glad that the CPU version will be sufficient for your needs for now, but please let me know if you even want to try again on the GPU.

[Shipeng-Xiao]

HI ,Thanks for writing Deepwave!
I ran into the same problem as erellaz, and I can't even implement it on the CPU. I installed deepwave on the computers of friends around me, but we encountered many warnings during the installation process.
Thanks for your sharing. If you have a good way, please let me know.

[ar4]

Hi Shipeng,

Thank you for your comment. I created a new issue (#38) for this in case it turns out to be a separate problem to that encountered by @erellaz .