Python Dependency Management

[alexk101]

Issue

This issue relates to how dependencies in the python portion of this project's build system are managed. This project makes the assumption that as long as there is an available python interpreter, that it is able to manage its dependencies through pip. This, however, isn't true on a number of linux distributions, namely arch linux and NixOS, which manage dependencies centrally through their os package manager. As a result, when attempting to build DiligentTools, the process fails since it is unable to successfully install libclang. Looking at the code causing this problem

DiligentTools/RenderStateNotation/CMakeLists.txt

Line 29 in bb61367

set(LIBCLANG_INSTALL_CMD ${Python3_EXECUTABLE} -m pip install libclang==16.0.6)

we can see that pip is run on every build. Therefore, even if I manually install libclang through my distribution's package manager, it will still fail. I don't really expect platform specific code to accommodate such cases, though that would be nice, but doing a check if the python package even needs to be installed seems pretty sensible, not only in a usability case for such systems, but also for the sake of efficiency. This would at least allow me to manually install the package and continue the build process.

My personal recommendation would be something like this...

execute_process(
    COMMAND ${Python3_EXECUTABLE} -c "import clang"
    RESULT_VARIABLE EXIT_CODE
    OUTPUT_QUIET
)

execute_process(
    COMMAND ${Python3_EXECUTABLE} -c "import jinja2"
    RESULT_VARIABLE EXIT_CODE
    OUTPUT_QUIET
)

I include jinja as well, since it is required, though my build fails on libclang before even getting to jinja. Doing this and then running pip only if these check fail would allow for manual intervention of the user, whereas now, there is simply no way for me to build.

Debug Output

Click me

-- Fetching DiligentCore ...
Target processor: x86_64
CMake generator: Ninja
Target platform: Linux 64

D3D11_SUPPORTED:  FALSE
D3D12_SUPPORTED:  FALSE
GL_SUPPORTED:     TRUE
GLES_SUPPORTED:   FALSE
VULKAN_SUPPORTED: TRUE
METAL_SUPPORTED:  FALSE

-- optimizer enabled
CMake Deprecation Warning at build/debug/_deps/diligentcore-src/ThirdParty/SPIRV-Cross/CMakeLists.txt:22 (cmake_minimum_required):
Compatibility with CMake < 3.5 will be removed from a future version of
CMake.

Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.


-- SPIRV-Cross: Finding Git version for SPIRV-Cross.
-- SPIRV-Cross: Git hash: 42aac916
-- xxHash build type: Debug
-- Architecture: x86_64
-- DiligentCore commit hash: b0e438cf
CMake Warning (dev) at /usr/share/cmake/Modules/FindOpenGL.cmake:381 (message):
Policy CMP0072 is not set: FindOpenGL prefers GLVND by default when
available.  Run "cmake --help-policy CMP0072" for policy details.  Use the
cmake_policy command to set the policy and suppress this warning.

FindOpenGL found both a legacy GL library:

  OPENGL_gl_LIBRARY: /usr/lib/libGL.so

and GLVND libraries for OpenGL and GLX:

  OPENGL_opengl_LIBRARY: /usr/lib/libOpenGL.so
  OPENGL_glx_LIBRARY: /usr/lib/libGLX.so

OpenGL_GL_PREFERENCE has not been set to "GLVND" or "LEGACY", so for
compatibility with CMake 3.10 and below the legacy GL library will be used.
Call Stack (most recent call first):
build/debug/_deps/diligentcore-src/Graphics/GraphicsEngineOpenGL/CMakeLists.txt:202 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at /usr/share/cmake/Modules/FindOpenGL.cmake:381 (message):
Policy CMP0072 is not set: FindOpenGL prefers GLVND by default when
available.  Run "cmake --help-policy CMP0072" for policy details.  Use the
cmake_policy command to set the policy and suppress this warning.

FindOpenGL found both a legacy GL library:

  OPENGL_gl_LIBRARY: /usr/lib/libGL.so

and GLVND libraries for OpenGL and GLX:

  OPENGL_opengl_LIBRARY: /usr/lib/libOpenGL.so
  OPENGL_glx_LIBRARY: /usr/lib/libGLX.so

OpenGL_GL_PREFERENCE has not been set to "GLVND" or "LEGACY", so for
compatibility with CMake 3.10 and below the legacy GL library will be used.
Call Stack (most recent call first):
build/debug/_deps/diligentcore-src/Graphics/Archiver/CMakeLists.txt:136 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Fetching DiligentTools ...
CMake Deprecation Warning at build/debug/_deps/diligenttools-src/ThirdParty/libpng/CMakeLists.txt:82 (message):
The option PNG_EXECUTABLES has been deprecated in favour of PNG_TOOLS


CMake Warning (dev) at build/debug/_deps/diligenttools-src/ThirdParty/libpng/CMakeLists.txt:84 (message):
Setting PNG_TOOLS to OFF, to stay compatible with PNG_EXECUTABLES
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Symbol prefix: 
CMake Warning (dev) at /usr/share/cmake/Modules/FindOpenGL.cmake:381 (message):
Policy CMP0072 is not set: FindOpenGL prefers GLVND by default when
available.  Run "cmake --help-policy CMP0072" for policy details.  Use the
cmake_policy command to set the policy and suppress this warning.

FindOpenGL found both a legacy GL library:

  OPENGL_gl_LIBRARY: /usr/lib/libGL.so

and GLVND libraries for OpenGL and GLX:

  OPENGL_opengl_LIBRARY: /usr/lib/libOpenGL.so
  OPENGL_glx_LIBRARY: /usr/lib/libGLX.so

OpenGL_GL_PREFERENCE has not been set to "GLVND" or "LEGACY", so for
compatibility with CMake 3.10 and below the legacy GL library will be used.
Call Stack (most recent call first):
build/debug/_deps/diligenttools-src/NativeApp/CMakeLists.txt:472 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try 'pacman -S
  python-xyz', where xyz is the package you are trying to
  install.
  
  If you wish to install a non-Arch-packaged Python package,
  create a virtual environment using 'python -m venv path/to/venv'.
  Then use path/to/venv/bin/python and path/to/venv/bin/pip.
  
  If you wish to install a non-Arch packaged Python application,
  it may be easiest to use 'pipx install xyz', which will manage a
  virtual environment for you. Make sure you have python-pipx
  installed via pacman.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
CMake Error at build/debug/_deps/diligenttools-src/RenderStateNotation/CMakeLists.txt:40 (message):
Command '/bin/python3.11;-m;pip;install;libclang==16.0.6' failed with error
code 1
}

Additional Info

• OS: linux;EndeavourOS
• Build Tools: clang version 16.0.6 + cmake 3.28.3
• GPU: Driver Version: 550.54.14 + CUDA Version: 12.4