ImportError undefined symbol: iJIT_NotifyEvent encountered when MKL 2024.1 is installed.

[LiutongZhou]

The bug

Importing torch raises undefined symbol: iJIT_NotifyEvent from torch/lib/libtorch_cpu.so: when pytorch and MKL 2024.1.0 are installed together. Downgrading MKL to 2024.0.0 resolves it.

import torch
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
----> 1 import torch

File ~/.../lib/python3.10/site-packages/torch/__init__.py:237
    235     if USE_GLOBAL_DEPS:
    236         _load_global_deps()
--> 237     from torch._C import *  # noqa: F403
    239 # Appease the type checker; ordinarily this binding is inserted by the
    240 # torch._C module initialization code in C
    241 if TYPE_CHECKING:

ImportError: /.../lib/python3.10/site-packages/torch/lib/libtorch_cpu.so: undefined symbol: iJIT_NotifyEvent

To Reproduce

mamba create -y -n test_pytorch_mkl python=3.10 pytorch=2.2 pytorch-cuda=12.1 mkl=2024.1 -c pytorch  -c nvidia -c intel
mamba activate test_pytorch_mkl
python -c "import torch"

Versions

PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.10.13 (tags/v3.10.13-25-g07fbd8e9251-dirty:07fbd8e9251, Sep 27 2023, 23:32:09) [GCC 13.2.0] (64-bit runtime)
Python platform: Linux-5.4.0-174-generic-x86_64-with-glibc2.31
Is CUDA available: N/A
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: GPU 0: Tesla V100-SXM2-32GB
Nvidia driver version: 525.60.13
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.6.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.6.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      46 bits physical, 48 bits virtual
CPU(s):                             32
On-line CPU(s) list:                0-31
Thread(s) per core:                 1
Core(s) per socket:                 32
Socket(s):                          1
NUMA node(s):                       1
Vendor ID:                          GenuineIntel
CPU family:                         6
Model:                              85
Model name:                         Intel Xeon Processor (Skylake)
Stepping:                           4
CPU MHz:                            2394.374
BogoMIPS:                           4788.74
Virtualization:                     VT-x
Hypervisor vendor:                  KVM
Virtualization type:                full
L1d cache:                          1 MiB
L1i cache:                          1 MiB
L2 cache:                           128 MiB
L3 cache:                           16 MiB
NUMA node0 CPU(s):                  0-31
Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Mitigation; PTE Inversion; VMX flush not necessary, SMT disabled
Vulnerability Mds:                  Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown:             Mitigation; PTI
Vulnerability Mmio stale data:      Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed:             Vulnerable
Vulnerability Spec store bypass:    Vulnerable
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat umip pku ospke arch_capabilities

Versions of relevant libraries:
[pip3] torch==2.2.2
[pip3] triton==2.2.0
[conda] blas                      1.0                         mkl    intel
[conda] mkl                       2024.1.0              intel_691    intel
[conda] pytorch                   2.2.2           py3.10_cuda12.1_cudnn8.9.2_0    pytorch
[conda] pytorch-cuda              12.1                 ha16c6d3_5    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchtriton               2.2.0                     py310    pytorch

cc @seemethere @malfet @osalpekar @atalman

[min-jean-cho]

cc. @CuiYifeng

[walidabualafia]

Hi all,

I am currently hitting this bug as well. It breaks the installation of the latest tbepler/topaz. Is there an ETA for a fix?

Thank you! :)

[ElHouas]

I am experiencing this issue as well, when installing in a docker container using conda. Here is my conda env.yaml:

name: ml_env
channels:

• pytorch
• nvidia
• conda-forge
• nodefaults
  dependencies:
• python=3.10.7
• mamba
• pip
• poetry=1.6.1
• pytorch::pytorch=2.0.1
• pytorch::torchaudio=2.0.2
• pytorch::torchvision=0.15.2
• pytorch::pytorch-cuda=11.8
  platforms:
• linux-64

Any advice?

Thanks! :)

[StefanGitHuber]

I can reproduce this issue as well:

1. First I had the issue with
   ImportError: intel_extension_for_pytorch xpu libintel-ext-pt-gpu.so: undefined symbol for _ZNK5torch8autograd4Node4nameB5cxx11Ev
   Following this thread GPU examples undefined symbol intel-analytics/ipex-llm#8803 I ran
   ldd /home/suhu/.local/lib/python3.10/site-packages/intel_extension_for_pytorch/lib/libintel-ext-pt-gpu.so
   linux-vdso.so.1 (0x00007ffe60f8a000)
   libtorch.so => not found
   libtorch_cpu.so => not found
   libc10.so => not found
   libxetla_kernels.so => /home/suhu/.local/lib/python3.10/site-packages/intel_extension_for_pytorch/lib/libxetla_kernels.so (0x0000739e4f600000)
   libmkl_intel_lp64.so.2 => /opt/intel/oneapi/mkl/2024.1/lib/libmkl_intel_lp64.so.2 (0x0000739e4e000000)
   libmkl_core.so.2 => /opt/intel/oneapi/mkl/2024.1/lib/libmkl_core.so.2 (0x0000739e49e00000)
   libmkl_gnu_thread.so.2 => /opt/intel/oneapi/mkl/2024.1/lib/libmkl_gnu_thread.so.2 (0x0000739e48400000)
   libmkl_sycl_blas.so.4 => /opt/intel/oneapi/mkl/2024.1/lib/libmkl_sycl_blas.so.4 (0x0000739e42c00000)
   libmkl_sycl_lapack.so.4 => /opt/intel/oneapi/mkl/2024.1/lib/libmkl_sycl_lapack.so.4 (0x0000739e40400000)
   libmkl_sycl_sparse.so.4 => /opt/intel/oneapi/mkl/2024.1/lib/libmkl_sycl_sparse.so.4 (0x0000739e39e00000)
   libmkl_sycl_dft.so.4 => /opt/intel/oneapi/mkl/2024.1/lib/libmkl_sycl_dft.so.4 (0x0000739e36e00000)
   libmkl_sycl_vm.so.4 => /opt/intel/oneapi/mkl/2024.1/lib/libmkl_sycl_vm.so.4 (0x0000739e2e000000)
   libmkl_sycl_rng.so.4 => /opt/intel/oneapi/mkl/2024.1/lib/libmkl_sycl_rng.so.4 (0x0000739e26200000)
   libmkl_sycl_stats.so.4 => /opt/intel/oneapi/mkl/2024.1/lib/libmkl_sycl_stats.so.4 (0x0000739e24200000)
   libmkl_sycl_data_fitting.so.4 => /opt/intel/oneapi/mkl/2024.1/lib/libmkl_sycl_data_fitting.so.4 (0x0000739e23800000)
   libze_loader.so.1 => /lib/x86_64-linux-gnu/libze_loader.so.1 (0x0000739ee9ac8000)
   libOpenCL.so.1 => /opt/intel/oneapi/compiler/2024.1/opt/oclfpga/host/linux64/lib/libOpenCL.so.1 (0x0000739e23400000)
   libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x0000739ee9ac3000)
   libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x0000739ee9abe000)
   libsvml.so => /opt/intel/oneapi/compiler/2024.1/lib/libsvml.so (0x0000739e21c00000)
   libirng.so => /opt/intel/oneapi/compiler/2024.1/lib/libirng.so (0x0000739e49d07000)
   libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x0000739e21800000)
   libimf.so => /opt/intel/oneapi/compiler/2024.1/lib/libimf.so (0x0000739e21200000)
   libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000739e52f19000)
   libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x0000739e4f5e0000)
   libintlc.so.5 => /opt/intel/oneapi/compiler/2024.1/lib/libintlc.so.5 (0x0000739e4df9e000)
   libsycl.so.7 => /opt/intel/oneapi/compiler/2024.1/lib/libsycl.so.7 (0x0000739e20e00000)
   libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000739e20a00000)
   /lib64/ld-linux-x86-64.so.2 (0x0000739ee9b45000)

pip install transformers==4.31.0
pip install --pre --upgrade bigdl-llm[xpu] -f https://developer.intel.com/ipex-whl-stable-xpu

2. Now right at the beginning with
   import torch
   even before importing impex
   import intel_extension_for_pytorch as ipex
   I receive

File "~/.local/lib/python3.10/site-packages/torch/init.py", line 235, in
from torch._C import * # noqa: F403
ImportError: ~/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so: undefined symbol: iJIT_NotifyEvent

Here my output from collect_env.py:

PyTorch version: N/A
PyTorch CXX11 ABI: N/A
IPEX version: N/A
IPEX commit: N/A
Build type: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: N/A
IGC version: 2024.1.0 (2024.1.0.20240308)
CMake version: version 3.26.4
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.5.0-26-generic-x86_64-with-glibc2.35
Is XPU available: N/A
DPCPP runtime version: 2024.1
MKL version: 2024.1
GPU models and configuration:
N/A
Intel OpenCL ICD version: 23.52.28202.39-82122.04
Level Zero version: 1.3.28202.39-82122.04

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 39 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 20
On-line CPU(s) list: 0-19
Vendor ID: GenuineIntel
Model name: 12th Gen Intel(R) Core(TM) i7-12700H
CPU family: 6
Model: 154
Thread(s) per core: 2
Core(s) per socket: 14
Socket(s): 1
Stepping: 3
CPU max MHz: 4700.0000
CPU min MHz: 400.0000
BogoMIPS: 5376.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities
Virtualization: VT-x
L1d cache: 544 KiB (14 instances)
L1i cache: 704 KiB (14 instances)
L2 cache: 11.5 MiB (8 instances)
L3 cache: 24 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-19
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] intel-extension-for-pytorch==2.1.10+xpu
[pip3] numpy==1.26.4
[pip3] pytorch-lightning==1.9.4
[pip3] pytorch-metric-learning==1.7.3
[pip3] torch==2.1.0a0+cxx11.abi
[pip3] torch-audiomentations==0.11.0
[pip3] torch-pitch-shift==1.2.4
[pip3] torch-stft==0.1.4
[pip3] torchaudio==2.1.0.post0+cxx11.abi
[pip3] torchmetrics==0.11.4
[pip3] torchvision==0.16.0a0+cxx11.abi
[conda] N/A

[syedazi]

Is there a resolution to this issue? I faced the same issue using conda.

conda create -n fsdp python=3.10
conda activate fsdp

# Install pytorch and other dependencies
conda install -y pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia

[titeup]

While not having the exact explanation of the missing symbol, here is what worked for me, if it can help others.

I had the same error after installing ipex_llm.
Just painfully found my way out. Python was mixing packages from intel python in oneAPI 2024.1 and my local cache.

after loading intel env I did:
python -m pip install torch==2.1.0.post0 torchvision==0.16.0.post0 torchaudio==2.1.0.post0 intel-extension-for-pytorch==2.1.20+xpu oneccl_bind_pt==2.1.200+xpu --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/

This magic combination comes from this Doc

And then (needed in my case)
pip install transformers==4.36.2

I suppose for those who want to use other python, you can load only the env variable of the MKL and use the pytorch install that worked for you before. (haven't tried it myself)

Then everything worked fine.

[jingxu10]

The reason is that PyTorch was built against an old version of MKL distribution which contains this symbol. However, this symbol got removed in MKL 2024.1.
The PyTorch binary released via conda channel was linked to MKL dynamically, so you got this error.
The PyTorch binary released via pip (pip install) was linked to MKL statically. You can switch to the pip install one to get rid of this error with MKL 2024.1.

[StefanGitHuber]

Hi Jing Xu,
I never used conda but only pip to install PyTorch. I want to use the XPU of my Intel Arc A730M, running on Ubuntu 22.04.4 LTS on my Notebook.

Here again my steps:

1. Install oneAPI

bash ./intelpython3-2024.1.0_814-Linux-x86_64.sh -b -u -p ~/intel/oneapi/intelpython
source ~/intel/oneapi/intelpython/env/vars.sh

Opens environment on bash:
(oneapi-intelpython)

2. Install PyTorch extension

github.com/intel/intel-extension-for-pytorch/tree/xpu-main
python -m pip install torch==2.1.0a0 torchvision==0.16.0a0 torchaudio==2.1.0a0 intel-extension-for-pytorch==2.1.10+xpu --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/

import torch
import intel_extension_for_pytorch as ipex
ImportError: ~/intel/oneapi/intelpython/lib/python3.9/site-packages/intel_extension_for_pytorch/lib/libintel-ext-pt-gpu.so: undefined symbol: _ZNK5torch8autograd4Node4nameB5cxx11Ev

3. Solve _ZNK5torch8autograd4Node4nameB5cxx11Ev

https://community.intel.com/t5/Intel-Developer-Cloud/ImportError-libintel-ext-pt-gpu-so-undefined-symbol/m-p/1561667
pip install --pre --upgrade bigdl-llm[xpu_2.1] -f https://developer.intel.com/ipex-whl-stable-xpu

4. Solve iJIT_NotifyEvent

import torch
ImportError: ~/intel/oneapi/intelpython/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so: undefined symbol: iJIT_NotifyEvent

Possibly I shouldn't perform step 3 and "upgrade" via installing the bigdl-llm[xpu_2.1] which leads me to this ImportError of iJIT_NotifyEvent. Only showing the diff from collect_env.py, it actually downgrades from before

PyTorch version: 2.2.2+cu121
PyTorch CXX11 ABI: No
[pip3] intel-extension-for-pytorch==2.1.20+xpu
[pip3] torch==2.2.2
[pip3] torchvision==0.16.0.post0+cxx11.abi
[conda] intel-extension-for-pytorch 2.1.20+xpu pypi_0 pypi
[conda] torch 2.2.2 pypi_0 pypi
[conda] torchvision 0.16.0.post0+cxx11.abi pypi_0 pypi

to

PyTorch version: N/A
PyTorch CXX11 ABI: N/A
[pip3] intel-extension-for-pytorch==2.1.10+xpu
[pip3] numpy==1.26.4
[pip3] torch==2.1.0a0+cxx11.abi
[pip3] torchaudio==2.1.0.post0+cxx11.abi
[pip3] torchvision==0.16.0a0+cxx11.abi
[conda] intel-extension-for-pytorch 2.1.10+xpu pypi_0 pypi
[conda] torch 2.1.0a0+cxx11.abi pypi_0 pypi
[conda] torchvision 0.16.0a0+cxx11.abi pypi_0 pypi

Question:
Apparently there are two torch versions, one installed via pip3 and one installed via conda. How to switch to the one installed via pip, please? I have to set an environment variable pointing to it? How exactly, please?

Thanks in advance ...

[yanbing-j]

Hi @LiutongZhou , according to https://github.com/pytorch/builder/blob/main/conda/pytorch-nightly/meta.yaml#L18-L45, PyTorch conda wheel is using dynamic linked MKL, which version should be MKL 2023.x. And it is compatible up to MKL 2024.0. Please downgrade MKL <= 2024.0, or do not specify MKL version.

Meanwhile, PyTorch conda wheel and pip wheel are using different MKL version, conda wheel will dynamic link MKL 2023, and pip wheel will static link MKL 2022. Hi @malfet , could you please clarify why conda/pip wheels use different channels to build? Does https://github.com/pytorch/builder/blob/main/conda/pytorch-nightly/meta.yaml#L18-L45 control MKL version in conda wheel? Does https://github.com/pytorch/builder/blob/main/common/install_mkl.sh control MKl version in pip wheel? Thanks!

[moi90]

Please downgrade MKL <= 2024.0

Thanks, @yanbing-j! this fixed the problem for me.

[Tianci-Wen]

I solved this problem by adding - mkl==2024.0 to the environment.yml.

name: scaffold_gs
channels:
  - pytorch
  - pyg
  - conda-forge
  - defaults
dependencies:
  - cudatoolkit=11.6
  - plyfile=0.8.1
  - python=3.7.13
  - pip=22.3.1
  - pytorch=1.12.1
  - torchaudio=0.12.1
  - torchvision=0.13.1
  - pytorch-scatter
  - tqdm
  - mkl==2024.0
  - pip:
    - einops
    - wandb
    - lpips
    - laspy
    - submodules/diff-gaussian-rasterization
    - submodules/simple-knn

[avivko]

Seems to be mkl indeed. All it takes is (here with mamba, but conda would work too) this and then you can import torch successfully:
mamba install mkl==2024.0