GPU examples undefined symbol

[nedo99]

Hi,

I wanted to test GPU example but got into the issue:

python3.9/site-packages/intel_extension_for_pytorch/lib/libintel-ext-pt-gpu.so: undefined symbol: _ZNK4sycl3_V16detail12buffer_plain13handleReleaseEv

intel-extension-for-pytorch version: 2.0.110+xpu
torch version: 2.0.1a0+cxx11.abi

I have oneAPI installed and sourced. clinfo shows both iGPU and my arc A770M graphics.

I followed the instructions from the link mentioned above.

Similar issue was mentioned here intel/intel-extension-for-pytorch#317 but with different versions.

What am I missing?

Regards,
Nedim

[jason-dai]

@yangw1234 please take a look

[yangw1234]

The symbol is defined oneapi sycl library.

Would you mind trying run ldd python3.9/site-packages/intel_extension_for_pytorch/lib/libintel-ext-pt-gpu.so after sourcing oneapi setvars.sh and share the output?

[nedo99]

hmm libtorch is missing.

linux-vdso.so.1 (0x00007ffdbb1e0000)
	libtorch.so => not found
	libtorch_cpu.so => not found
	libc10.so => not found
	libmkl_intel_lp64.so.2 => /opt/intel/oneapi/mkl/2023.0.0/lib/intel64/libmkl_intel_lp64.so.2 (0x00007faef1265000)
	libmkl_core.so.2 => /opt/intel/oneapi/mkl/2023.0.0/lib/intel64/libmkl_core.so.2 (0x00007faeece83000)
	libmkl_gnu_thread.so.2 => /opt/intel/oneapi/mkl/2023.0.0/lib/intel64/libmkl_gnu_thread.so.2 (0x00007faeeb2d4000)
	libmkl_sycl.so.3 => /opt/intel/oneapi/mkl/2023.0.0/lib/intel64/libmkl_sycl.so.3 (0x00007faec18e3000)
	libOpenCL.so.1 => /opt/intel/oneapi/compiler/2023.0.0/linux/lib/libOpenCL.so.1 (0x00007faec18d4000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007faec18cd000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007faec18c8000)
	libze_loader.so.1 => /lib/x86_64-linux-gnu/libze_loader.so.1 (0x00007faec1884000)
	libsvml.so => /opt/intel/oneapi/compiler/2023.0.0/linux/compiler/lib/intel64_lin/libsvml.so (0x00007faec0278000)
	libirng.so => /opt/intel/oneapi/compiler/2023.0.0/linux/compiler/lib/intel64_lin/libirng.so (0x00007faebfe00000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007faebfbd6000)
	libimf.so => /opt/intel/oneapi/compiler/2023.0.0/linux/compiler/lib/intel64_lin/libimf.so (0x00007faebf7f1000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007faec018f000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007faec016f000)
	libintlc.so.5 => /opt/intel/oneapi/compiler/2023.0.0/linux/compiler/lib/intel64_lin/libintlc.so.5 (0x00007faebf77e000)
	libsycl.so.6 => /opt/intel/oneapi/compiler/2023.0.0/linux/lib/libsycl.so.6 (0x00007faebf499000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007faebf271000)
	/lib64/ld-linux-x86-64.so.2 (0x00007faf7c08d000)

[yangw1234]

hmm libtorch is missing.

linux-vdso.so.1 (0x00007ffdbb1e0000)
	libtorch.so => not found
	libtorch_cpu.so => not found
	libc10.so => not found
	libmkl_intel_lp64.so.2 => /opt/intel/oneapi/mkl/2023.0.0/lib/intel64/libmkl_intel_lp64.so.2 (0x00007faef1265000)
	libmkl_core.so.2 => /opt/intel/oneapi/mkl/2023.0.0/lib/intel64/libmkl_core.so.2 (0x00007faeece83000)
	libmkl_gnu_thread.so.2 => /opt/intel/oneapi/mkl/2023.0.0/lib/intel64/libmkl_gnu_thread.so.2 (0x00007faeeb2d4000)
	libmkl_sycl.so.3 => /opt/intel/oneapi/mkl/2023.0.0/lib/intel64/libmkl_sycl.so.3 (0x00007faec18e3000)
	libOpenCL.so.1 => /opt/intel/oneapi/compiler/2023.0.0/linux/lib/libOpenCL.so.1 (0x00007faec18d4000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007faec18cd000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007faec18c8000)
	libze_loader.so.1 => /lib/x86_64-linux-gnu/libze_loader.so.1 (0x00007faec1884000)
	libsvml.so => /opt/intel/oneapi/compiler/2023.0.0/linux/compiler/lib/intel64_lin/libsvml.so (0x00007faec0278000)
	libirng.so => /opt/intel/oneapi/compiler/2023.0.0/linux/compiler/lib/intel64_lin/libirng.so (0x00007faebfe00000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007faebfbd6000)
	libimf.so => /opt/intel/oneapi/compiler/2023.0.0/linux/compiler/lib/intel64_lin/libimf.so (0x00007faebf7f1000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007faec018f000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007faec016f000)
	libintlc.so.5 => /opt/intel/oneapi/compiler/2023.0.0/linux/compiler/lib/intel64_lin/libintlc.so.5 (0x00007faebf77e000)
	libsycl.so.6 => /opt/intel/oneapi/compiler/2023.0.0/linux/lib/libsycl.so.6 (0x00007faebf499000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007faebf271000)
	/lib64/ld-linux-x86-64.so.2 (0x00007faf7c08d000)

I think the problem is the oneapi version. IPEX xpu 2.0.100 requires oneapi 2023.2.0. See https://intel.github.io/intel-extension-for-pytorch/xpu/2.0.110+xpu/tutorials/installations/linux.html.

I guess libtorch will be loaded when imporing torch, so that should not be an issue.

[nedo99]

Yes, that was the issue. I would still suggest to add a note in the readme requiring minimum oneAPI version.

[nedo99]

Actually, not solved totally. I have the following issue now:

python3.9/site-packages/transformers/convert_slow_tokenizer.py", line 40, in import_protobuf
    return sentencepiece_model_pb2
UnboundLocalError: local variable 'sentencepiece_model_pb2' referenced before assignment

[rnwang04]

This error is caused by transformer version, and we have fixed it.
You can try pip install --pre --upgrade bigdl-llm[xpu] -f https://developer.intel.com/ipex-whl-stable-xpu again or directly pip install transformers==4.31.0 .

[nedo99]

Tha solved that issue, but we have another one:

python3.9/site-packages/torch/_tensor.py", line 997, in __contains__
    return (element == self).any().item()  # type: ignore[union-attr]
RuntimeError: The program was built for 1 devices
Build program log for 'Intel(R) Graphics [0x5690]':
 -42 (PI_ERROR_INVALID_BINARY)

[rnwang04]

I haven't seen this error before, but I think it's maybe a problem of driver version.
ipex 2.0 requires Intel GPU Drivers Stable 647.21.
and this is my sycl-ls output:

[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device 1.2 [2023.16.7.0.21_160000]
[opencl:cpu:1] Intel(R) OpenCL, 12th Gen Intel(R) Core(TM) i9-12900K 3.0 [2023.16.7.0.21_160000]
[opencl:gpu:2] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) A770 Graphics 3.0 [23.17.26241.33]
[opencl:gpu:3] Intel(R) OpenCL Graphics, Intel(R) UHD Graphics 770 3.0 [23.17.26241.33]
[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Arc(TM) A770 Graphics 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:1] Intel(R) Level-Zero, Intel(R) UHD Graphics 770 1.3 [1.3.26241]

[nedo99]

Yes, thank! I update the drivers and it works with these versions:

[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device 1.2 [2023.16.6.0.22_223734]
[opencl:cpu:1] Intel(R) OpenCL, 12th Gen Intel(R) Core(TM) i7-12700H 3.0 [2023.16.6.0.22_223734]
[opencl:gpu:2] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) A770M Graphics 3.0 [23.17.26241.33]
[opencl:gpu:3] Intel(R) OpenCL Graphics, Intel(R) Graphics [0x46a6] 3.0 [23.17.26241.33]
[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Arc(TM) A770M Graphics 1.3 [1.3.26241]
[ext_oneapi_level_zero:gpu:1] Intel(R) Level-Zero, Intel(R) Graphics [0x46a6] 1.3 [1.3.26241]

Thanks!