openvino with UHD Graphics 600 error "GPU program build failed!"

[telent]

Describe the bug

Using a Celeron J4125 I am trying to run OpenVINO, but get

[Step 7/11] Loading the model to the device
[ ERROR ] Check 'false' failed at src/inference/src/core.cpp:114:
Check 'false' failed at src/plugins/intel_gpu/src/plugin/program.cpp:384:
GPU program build failed!
[GPU] clWaitForEvents, error code: -14

Steps To Reproduce

Steps to reproduce the behavior:

1. nix-shell -p 'python310.withPackages(ps: with ps; [ openvino opencv4])' opencv2 pciutils
2. git clone https://github.com/openvinotoolkit/openvino/ && cd openvino
3. python tools/benchmark_tool/benchmark_app.py -m rc/bindings/python/tests/test_utils/utils/test_model_fp32.xml -d GPU

(It also fails using ssdlite_mobilenet_v2.xml from the frigate NVR package which is the one I really want to use, so I don't think this is a model-specific problem. I'm no expert though)

Expected behavior

Good question. I expected it not to fail with an error message, but I don't know what correct behaviour looks like. I'd like frigate to work, but it fails with the same error messages, so I hope that this is a smaller test case

Additional context

configuration.nix has

  hardware.opengl = {
    # needed for frigate to use hw accel in ffmpeg
    enable = true; driSupport = true;
    extraPackages = with pkgs; [
      intel-media-driver
      intel-compute-runtime
      ocl-icd
    ];
  };

clinfo works:

NULL platform behavior
  clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...)  Intel(R) OpenCL Graphics
  clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...)   Success [INTEL]
  clCreateContext(NULL, ...) [default]            Success [INTEL]
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT)  Success (1)
    Platform Name                                 Intel(R) OpenCL Graphics
    Device Name                                   Intel(R) UHD Graphics 600
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU)  Success (1)
    Platform Name                                 Intel(R) OpenCL Graphics
    Device Name                                   Intel(R) UHD Graphics 600
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL)  Success (1)
    Platform Name                                 Intel(R) OpenCL Graphics
    Device Name                                   Intel(R) UHD Graphics 600

ICD loader properties
  ICD loader Name                                 OpenCL ICD Loader
  ICD loader Vendor                               OCL Icd free software
  ICD loader Version                              2.3.2
  ICD loader Profile                              OpenCL 3.0

If I set LD_DEBUG=libs I can see quite voluminous output which includes several fatal lines. I don't know if these are relevant ...

[nix-shell:~/src/openvino]$ LD_DEBUG=libs python tools/benchmark_tool/benchmark_app.py -m  /etc/nixos/openvino/ssdlite_mobilenet_v2.xml  -d GPU  2>&1 |grep fatal
     51110:     /nix/store/9c03r86hcdn43dm3hsgjirifvyzfkhwh-python3-3.10.12/bin/python: error: symbol lookup error: undefined symbol: kmp_set_thread_affinity_mask_initial (fatal)
     51110:     /nix/store/syx6gbglc3s41nw08g0g72233h6aima4-openvino-2023.0.0/runtime/lib/intel64/libopenvino_intel_gpu_plugin.so: error: symbol lookup error: undefined symbol: CreateExtensionShared (fatal)
     51110:     /nix/store/syx6gbglc3s41nw08g0g72233h6aima4-openvino-2023.0.0/runtime/lib/intel64/libopenvino_intel_gpu_plugin.so: error: symbol lookup error: undefined symbol: CreateExtension (fatal)
     51110:     /nix/store/syx6gbglc3s41nw08g0g72233h6aima4-openvino-2023.0.0/runtime/lib/intel64/libopenvino_auto_batch_plugin.so: error: symbol lookup error: undefined symbol: CreateExtensionShared (fatal)
     51110:     /nix/store/syx6gbglc3s41nw08g0g72233h6aima4-openvino-2023.0.0/runtime/lib/intel64/libopenvino_auto_batch_plugin.so: error: symbol lookup error: undefined symbol: CreateExtension (fatal)
     51110:     /nix/store/9c03r86hcdn43dm3hsgjirifvyzfkhwh-python3-3.10.12/bin/python: error: symbol lookup error: undefined symbol: scalable_malloc (fatal)

Notify maintainers

@tfmoraes

From git history: @mweinelt @superherointj

Metadata

[nix-shell:~/src/openvino]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.4.4, NixOS, 23.11 (Tapir), 23.11pre506474.12303c652b8`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.15.1`
 - channels(root): `"nixos"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`
 

[mweinelt]

Works with the discrete Intel GPU I have access to:

03:00.0 VGA compatible controller: Intel Corporation DG2 [Arc A380] (rev 05)

$ python tools/benchmark_tool/benchmark_app.py -m src/bindings/python/tests/test_utils/u
tils/test_model_fp32.xml -d GPU
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.0.0-000--
[ INFO ] 
[ INFO ] Device info:
[ INFO ] GPU
[ INFO ] Build ................................. 2023.0.0-000--
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(GPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 2.42 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ]     data (node: data) : f32 / [...] / [1,3,32,32]
[ INFO ] Model outputs:
[ INFO ]     fc_out (node: fc_out) : f32 / [...] / [1,10]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ]     data (node: data) : u8 / [N,C,H,W] / [1,3,32,32]
[ INFO ] Model outputs:
[ INFO ]     fc_out (node: fc_out) : f32 / [...] / [1,10]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 1355.77 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1024
[ INFO ]   NETWORK_NAME: test_model
[ INFO ]   EXECUTION_DEVICES: ['GPU.0']
[ INFO ]   AUTO_BATCH_TIMEOUT: 1000
[ INFO ]   LOADED_FROM_CACHE: False
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'data'!. This input will be filled with random values!
[ INFO ] Fill input 'data' with random values 
[Step 10/11] Measuring performance (Start inference asynchronously, 1024 inference requests, limits: 60000 ms duration)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 614.32 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['GPU.0']
[ INFO ] Count:            7646208 iterations
[ INFO ] Duration:         60012.08 ms
[ INFO ] Latency:
[ INFO ]    Median:        7.30 ms
[ INFO ]    Average:       6.93 ms
[ INFO ]    Min:           3.02 ms
[ INFO ]    Max:           14.47 ms
[ INFO ] Throughput:   127411.15 FPS

[andrey-skat]

I have J4125 too with the same problem

[Step 7/11] Loading the model to the device
[ ERROR ] Exception from src/inference/src/core.cpp:99:
[ GENERAL_ERROR ] Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:179:
[GPU] ProgramBuilder build failed!
[GPU] clWaitForEvents, error code: -14

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/openvino/tools/benchmark/main.py", line 408, in main
    compiled_model = benchmark.core.compile_model(model, benchmark.device, device_config)
  File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 547, in compile_model
    super().compile_model(model, device_name, {} if config is None else config),
RuntimeError: Exception from src/inference/src/core.cpp:99:
[ GENERAL_ERROR ] Check 'false' failed at src/plugins/intel_gpu/src/plugin/program_builder.cpp:179:
[GPU] ProgramBuilder build failed!
[GPU] clWaitForEvents, error code: -14

[mweinelt]

I wouldn't be surprised if this was a limitation on GPU inference on that model.

Even though Intel claims to support basically everying from a 6th gen CPU. https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/system-requirements.html#tab-blade-1-1

[wb666greene]

I'm having the same "[GPU] clWaitForEvents, error code: -14" issue using the GPU on an N4020 with UHD graphics running openvino 2024.2 on Ubuntu 22.04.

Funny thing is, this system was running Ubuntu 20.04 and openvino 2021.3 and the GPU worked fine.

[mweinelt]

Funny thing is, this system was running Ubuntu 20.04 and openvino 2021.3 and the GPU worked fine.

That sounds like Intel dropped support in a newer release. Nothing I can for you I'm afraid.

That being said, I am now running Frigate on a relatively cheap N100 platform and I would recommend upgrading to that. It also comes with AVX2 support, which is required since we switched to the binary tensorflow distribution.