Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Tensorflow Lite GPU Delegates for the desktop #28830

Closed
fire opened this issue May 18, 2019 · 10 comments
Closed

Add Tensorflow Lite GPU Delegates for the desktop #28830

fire opened this issue May 18, 2019 · 10 comments
Assignees
@impjdi
Labels
comp:lite TF Lite related issues stat:awaiting tensorflower Status - Awaiting response from tensorflower TF 2.0 Issues relating to TensorFlow 2.0 type:feature Feature requests

Comments

@fire
Copy link

fire commented May 18, 2019 •
edited

System information

  • TensorFlow version (you are using): Tensorflow2.0alpha
  • Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state.

Tensorflow lite only has GPU delegates for iOS and Android devices.

In theory, Tensorflow lite should be able to use desktop Opengles apis through GLAD.

Will this change the current api? How?

This will use the existing interfaces for gpu delegates.

Who will benefit with this feature?

  1. Tensorflow lite will be able to be gpu accelerated on desktops
  2. Users of Tensorflow lite can have the same acceleration on the desktop and the mobile devices for ease of development.
  3. Nvidia Tegra kits are a desktop-like platform that isn't an Android or iOS device but is able to run compute shaders.
  4. Inference on the GPU may now run fast enough and now become suitable for real-time applications compared to the CPU.
@achandraa achandraa self-assigned this May 20, 2019
@achandraa achandraa added 2.0.0-alpha0 comp:lite TF Lite related issues type:feature Feature requests labels May 20, 2019
@achandraa achandraa assigned ymodak and unassigned achandraa May 20, 2019
@ymodak ymodak assigned impjdi and unassigned ymodak May 20, 2019
@ymodak ymodak added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label May 20, 2019
@impjdi
Copy link
Contributor

impjdi commented Jun 7, 2019

@fire

Oh my apologies. This issue somehow dropped from my radar, and I haven't responded for very long.

Yes, we have had successful running of the TFLite GPU on desktop with mesa. It wasn't robust, e.g. I couldn't make it to run, but some of my colleagues were able to.

Our team is pretty short staffed, and have all our forces on supporting mobile devices. And even so, we have trouble fixing Mali performance (which is going on for over a month now). I'm sure we can somehow make it work for desktop, but that goes with supporting the community, and we just don't have the resources to do so.

For now, I will close this issue as infeasible.

@impjdi impjdi closed this as completed Jun 7, 2019
@kingofthebongo2008
Copy link
Contributor

I have here fixed gpu delegate to run on windows with angle library.

@fire
Copy link
Author

fire commented Aug 18, 2019

Is it testable? Can you link to a git branch? I am interested.

@wolterlw
Copy link

wolterlw commented Dec 5, 2019

Can anyone give a short summary of how tflite runs on GPU using mesa as used in Mediapipe project?

@fire
Copy link
Author

fire commented Apr 24, 2020

image

@impjdi
Copy link
Contributor

impjdi commented Apr 24, 2020

Haha good one. MediaPipe uses

--copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11

to bypass the X11 Status issue.

@lvenugopalan lvenugopalan added the TF 2.0 Issues relating to TensorFlow 2.0 label Apr 29, 2020
@emepetres emepetres mentioned this issue Jun 9, 2020
Closed
@zldrobit zldrobit mentioned this issue Jun 11, 2020
Closed
@fire fire mentioned this issue Jun 25, 2020
Closed
@u0251077 u0251077 mentioned this issue Jul 27, 2020
Open
@magnusg-sa
Copy link

I have here fixed gpu delegate to run on windows with angle library.

@kingofthebongo2008 is there any information on how to get this to work?

It feels like I am close to the solution but I must be missing something fundamental. I am quite new to most of the tools and concepts involved, but I managed to stitch together the following procedure:

  1. Installed ANGLE according to https://chromium.googlesource.com/angle/angle/+/main/doc/DevSetup.md (COMMIT=596c2acfa5071eb643eaffe855a5ccb51c345427)

  2. Built ANGLE libraries with following args (args.gn):

is_component_build = false
target_cpu = "x64"
is_debug = false
angle_assert_always_on = false

  1. Added ANGLE build output directory (containing libEGL.dll, libEGL.dll.lib, libGLESv2.dll, libGLESv2.dll.lib, d3dcompiler_47.dll) to PATH environment variable.

  2. Added ANGLE library dependency in bazel:

    1. In WORKSPACE file:

      new_local_repository(
    name = "angle",
    path = "../../../angle/angle/include",
    build_file_content = """
package(default_visibility = ["//visibility:public"])
cc_library(
    name = "headers",
    srcs = glob(["**/*.h"])
)
"""
)

    2. Added necessary dependencies (copts = ["-Iexternal/angle"] and "@angle//:headers") to targets in bazel BUILD files (all targets that failed compilation due to missing OpenGL headers):

      Example:

      cc_library(
    name = "gl_errors",
    srcs = ["gl_errors.cc"],
    hdrs = ["gl_errors.h"],
    copts = ["-Iexternal/angle"],             <--- ADDED THIS
    deps = [
        "@angle//:headers",                   <--- ADDED THIS
        ":portable",
        "//tensorflow/lite/delegates/gpu/common:status",
        "@com_google_absl//absl/strings",
    ],
)

    3. Updated .bazel to use '/std:c++latest'

      ---build:windows --cxxopt=/std:c++17
---build:windows --host_cxxopt=/std:c++17
+++build:windows --cxxopt=/std:c++latest
+++build:windows --host_cxxopt=/std:c++latest

  3. Build TensorFlow Lite GPU delegate (v2.10.0) //tensorflow/lite/delegates/gpu:libtensorflowlite_gpu_delegate.so

bazel2 build --local_cpu_resources=3 --features=windows_export_all_symbols -s -c opt --copt="-DMESA_EGL_NO_X11_HEADERS" --copt="-DEGL_NO_X11" //tensorflow/lite/delegates/gpu:libtensorflowlite_gpu_delegate.so

But I get errors that I do not know how to mitigate.

ERROR: C:/users/magnusg/src/tensorflow_lite/tflite_2.10.0_gpu_delegate_build_bazel/tensorflow_src/tensorflow/lite/delegates/gpu/cl/kernels/BUILD:129:11: Compiling tensorflow/lite/delegates/gpu/cl/kernels/converter.cc failed: (Exit 2): cl.exe failed: error executing command
  cd /d C:/users/magnusg/_bazel_magnusg/4du655w4/execroot/org_tensorflow
  SET INCLUDE=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\include;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\ucrt;C:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\shared;C:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\um;C:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\winrt;C:\Program Files (x86)\Windows Kits\10\include\10.0.20348.0\cppwinrt
    SET PATH=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\bin\HostX64\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\VC\VCPackages;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Current\bin\Roslyn;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools\x64\;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\Tools\devinit;C:\Program Files (x86)\Windows Kits\10\bin\10.0.20348.0\x64;C:\Program Files (x86)\Windows Kits\10\bin\x64;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\\MSBuild\Current\Bin;c:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\Tools\;;C:\Windows\system32;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja
    SET PWD=/proc/self/cwd
    SET PYTHON_BIN_PATH=C:/Users/magnusg/anaconda3/envs/sa/python.exe
    SET PYTHON_LIB_PATH=C:/Users/magnusg/anaconda3/envs/sa/lib/site-packages
    SET RUNFILES_MANIFEST_ONLY=1
    SET TEMP=C:\Users\magnusg\AppData\Local\Temp
    SET TF2_BEHAVIOR=1
    SET TMP=C:\Users\magnusg\AppData\Local\Temp
    SET avariable=3
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\bin\HostX64\x64\cl.exe @bazel-out/x64_windows-opt/bin/tensorflow/lite/delegates/gpu/cl/kernels/_objs/converter/converter.obj.params
# Configuration: 4692499b740a707abb55d09dbd7c915fbefe7b433a9f783356c086c94c980334
# Execution platform: @local_execution_config_platform//:platform
cl : Command line warning D9035 : option 'experimental:preprocessor' has been deprecated and will be removed in a future release
cl : Command line warning D9036 : use 'Zc:preprocessor' instead of 'experimental:preprocessor'
cl_version.h: CL_TARGET_OPENCL_VERSION is not defined. Defaulting to 300 (OpenCL 3.0)
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(526): error C2059: syntax error: '__cdecl'
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(532): error C2065: 'cl_command_buffer_khr': undeclared identifier
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(532): error C2165: 'left-side modifier': cannot modify pointers to data
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(532): error C2513: 'cl_int *': no variable declared before '='
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(535): error C2065: 'cl_command_buffer_khr': undeclared identifier
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(535): error C2165: 'left-side modifier': cannot modify pointers to data
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(535): error C2513: 'cl_int *': no variable declared before '='
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(538): error C2065: 'cl_command_buffer_khr': undeclared identifier
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(538): error C2165: 'left-side modifier': cannot modify pointers to data
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(538): error C2513: 'cl_int *': no variable declared before '='
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(542): error C2061: syntax error: identifier 'cl_command_buffer_khr'
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(547): error C2065: 'cl_command_buffer_khr': undeclared identifier
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(548): error C2275: 'cl_command_queue': illegal use of this type as an expression
external/angle\CL/cl.h(32): note: see declaration of 'cl_command_queue'
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(549): error C2059: syntax error: 'const'
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(559): error C2065: 'cl_command_buffer_khr': undeclared identifier
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(560): error C2065: 'cl_command_buffer_info_khr': undeclared identifier
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(560): error C2275: 'size_t': illegal use of this type as an expression
tensorflow/lite/delegates/gpu/cl/kernels/converter.cc: note: see declaration of 'size_t'
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(561): error C2062: type 'void' unexpected
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(682): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(682): error C2146: syntax error: missing ';' before identifier 'clCreateCommandBufferKHR'
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(687): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(687): error C2146: syntax error: missing ';' before identifier 'clCommandNDRangeKernelKHR'
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(688): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
.\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h(688): error C2146: syntax error: missing ';' before identifier 'clGetCommandBufferInfoKHR'
Target //tensorflow/lite/delegates/gpu:libtensorflowlite_gpu_delegate.so failed to build
INFO: Elapsed time: 375.856s, Critical Path: 62.04s
INFO: 686 processes: 15 internal, 671 local.
FAILED: Build did NOT complete successfully

I do not know if I am on the right track and simply missing a minor detail or if I am missing the target completely. Any feedback would be helpful.

@kingofthebongo2008
Copy link
Contributor

kingofthebongo2008 commented Nov 4, 2022 via email

@kingofthebongo2008
Copy link
Contributor

kingofthebongo2008 commented Nov 4, 2022 via email

@magnusg-sa
Copy link

magnusg-sa commented Nov 7, 2022 •
edited

Run build from this command, cl.exe will be in the path ERROR: C:/users/magnusg/src/tensorflow_lite/tflite_2.10.0_gpu_delegate_build_bazel/tensorflow_src/tensorflow/lite/delegates/gpu/cl/kernels/BUILD:129:11: Compiling tensorflow/lite/delegates/gpu/cl/kernels/converter.cc failed: (Exit 2): cl.exe failed: error executing command [image: image.png]

@kingofthebongo2008 I am not sure what you mean by "Run build from this command"? Can you elaborate on that? And did you provide some image image.png that is not displayed in your post?

I have tried re-running the build multiple times from the tensorflow source root directory. It compiles most of the ~1,111 files properly but it always fails on this cl.exe error for the file .\tensorflow/lite/delegates/gpu/cl/opencl_wrapper.h when there are ~30 files left to compile. It seems like it finds cl.exe at the path C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29910\bin\HostX64\x64\cl.exe but that cl.exe has trouble compiling opencl_wrapper.h.

Even though the error specifically says syntax error: '__cdecl' I cannot find any instances of __cdecl in the code.

I have seen that others get cl.exe errors when compiling tensorflow for Windows (e.g., failed: (Exit 2): cl.exe failed: error executing command，Bazel=5.0.0, tf=2.8.0,MSVC2019) but since I could compile TensorFlow Lite on Windows without a problem I thought my environment was set up correctly.

I have fixed the experimental:preprocessor warning but the problem remains.

Could it have something to do with CL_TARGET_OPENCL_VERSION defaulting to 300 (OpenCL 3.0)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@impjdi impjdi

Labels
comp:lite TF Lite related issues stat:awaiting tensorflower Status - Awaiting response from tensorflower TF 2.0 Issues relating to TensorFlow 2.0 type:feature Feature requests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@fire @kingofthebongo2008 @impjdi @wolterlw @ymodak @achandraa @lvenugopalan @magnusg-sa