Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Vulkan, SPIR-V, glslang and shaderc #94049

Merged
merged 9 commits into from Oct 15, 2020
Merged

Conversation

@CajuM
Copy link

@CajuM CajuM commented Jul 28, 2020

Motivation for this change

Keeping master up to date with recent developments in upstream projects

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Ralith
Ralith approved these changes Jul 28, 2020
Copy link
Contributor

@Ralith Ralith left a comment

Looks good, thanks! Note that this subsumes #93604.

@jonringer
Copy link
Contributor

@jonringer jonringer commented Aug 2, 2020

closes #94247

@jonringer
Copy link
Contributor

@jonringer jonringer commented Aug 3, 2020

the shaderc bump is causing some regressions, I now get:

builder for '/nix/store/86lrf5m4vy9r4hbwm6r95ljv6w0pwi4d-libplacebo-2.72.0.drv' failed with exit code 1; last 10 log lines:
  build flags: -j128 -l128
  [42/44] Compiling C++ object 'src/25a6634@@placebo@sha/glsl_glslang.cc.o's_gen.c.o'
  FAILED: src/25a6634@@placebo@sha/glsl_glslang.cc.o
  g++ -Isrc/25a6634@@placebo@sha -Isrc -I../src -I../src/include -I../src/include/dummy -I../subprojects/xtalloc/include -I../subprojects/bstr/include -I/nix/store/81iy20n5rhvcm18ha289anygpgq5qyjc-lcms2-2.11-dev/include -I/nix/store/riw48ysh0rhkf2fck7m1iiany4rif2mv-vulkan-headers-1.2.141.0/include -I/nix/store/psda85jw38cxzqp6fa59vrdipz67h6bk-epoxy-1.5.4-dev/include -I/usr/include/glslang -I/nix/store/my4bdsgi308bq216gkhfwnwnk35yhszn-libplacebo-2.72.0/include/glslang -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -std=c++11 -Wundef -Wshadow -Wparentheses -Wpointer-arith -fvisibility=hidden -fPIC -pthread -MD -MQ 'src/25a6634@@placebo@sha/glsl_glslang.cc.o' -MF 'src/25a6634@@placebo@sha/glsl_glslang.cc.o.d' -o 'src/25a6634@@placebo@sha/glsl_glslang.cc.o' -c ../src/glsl/glslang.cc
  ../src/glsl/glslang.cc:229:1: error: cannot convert '<brace-enclosed initializer list>' to 'int' in initialization
    229 | };
        | ^

@jonringer
Copy link
Contributor

@jonringer jonringer commented Aug 3, 2020

@CajuM
Copy link
Author

@CajuM CajuM commented Aug 3, 2020

I see two solutions:

  1. we drop the https://code.videolan.org/videolan/libplacebo/-/commit/82e3be1839379791b58e98eb049415b42888d5b0.patch from libplacebo
  2. we update shaderc or glslang to newer versions: the header which is causing us trouble is in glslang, not shaderc. But for some reason #93604 broke on shaderc, maybe because libplacebo got the header from shaderc's glslang dependency?

@CajuM
Copy link
Author

@CajuM CajuM commented Aug 3, 2020

I have also noticed that the shaderc-dev derivation has bundled the glslang headers, shouldn't they be in a separate derivation?
Also, do we have a requirement for overriding dependencies for glslang and vulkan-validation-layers? When I made this PR I just copied this pattern. But now I'm beginning to wonder if we can just use the same versions everywhere.

@jonringer
Copy link
Contributor

@jonringer jonringer commented Aug 3, 2020

@CajuM shaderc is re-exporting all of the glslang headers it was built with. Easiest solution I found is just make shaderc the last dependency in buildInputs for libplacebo

$ git diff pkgs/
diff --git a/pkgs/development/libraries/libplacebo/default.nix b/pkgs/development/libraries/libplacebo/default.nix
index dc3e5e30c74..15b52c400df 100644
--- a/pkgs/development/libraries/libplacebo/default.nix
+++ b/pkgs/development/libraries/libplacebo/default.nix
@@ -47,10 +47,10 @@ stdenv.mkDerivation rec {
   buildInputs = [
     vulkan-headers
     vulkan-loader
-    shaderc
     glslang
     lcms2
     epoxy
+    shaderc
   ];

   mesonFlags = [

@jonringer
Copy link
Contributor

@jonringer jonringer commented Aug 3, 2020

Not sure if this is intended behavior, and probably exists in the current package:

/nix/store/5prb4s19gv9l4xm71zwkad8yh6x5rz4v-shaderc-2020.2-dev/include/glslang/
├── build_info.h
├── HLSL
│   ├── hlslAttributes.h
│   ├── hlslGrammar.h
│   ├── hlslOpMap.h
│   ├── hlslParseables.h
│   ├── hlslParseHelper.h
│   ├── hlslScanContext.h
│   ├── hlslTokens.h
│   └── hlslTokenStream.h
├── Include
│   ├── arrays.h
│   ├── BaseTypes.h
│   ├── Common.h
│   ├── ConstantUnion.h
│   ├── glslang_c_interface.h
│   ├── glslang_c_shader_types.h
│   ├── InfoSink.h
│   ├── InitializeGlobals.h
│   ├── intermediate.h
│   ├── PoolAlloc.h
│   ├── ResourceLimits.h
│   ├── ShHandle.h
│   └── Types.h
├── MachineIndependent
│   ├── attribute.h
│   ├── glslang_tab.cpp.h
│   ├── gl_types.h
│   ├── Initialize.h
│   ├── iomapper.h
│   ├── LiveTraverser.h
│   ├── localintermediate.h
│   ├── ParseHelper.h
│   ├── parseVersions.h
│   ├── preprocessor
│   │   ├── PpContext.h
│   │   └── PpTokens.h
│   ├── propagateNoContraction.h
│   ├── reflection.h
│   ├── RemoveTree.h
│   ├── ScanContext.h
│   ├── Scan.h
│   ├── SymbolTable.h
│   └── Versions.h
├── Public
│   └── ShaderLang.h
└── SPIRV
    ├── bitutils.h
    ├── disassemble.h
    ├── doc.h
    ├── GlslangToSpv.h
    ├── GLSL.ext.AMD.h
    ├── GLSL.ext.EXT.h
    ├── GLSL.ext.KHR.h
    ├── GLSL.ext.NV.h
    ├── GLSL.std.450.h
    ├── hex_float.h
    ├── Logger.h
    ├── NonSemanticDebugPrintf.h
    ├── spirv.hpp
    ├── SpvBuilder.h
    ├── spvIR.h
    ├── SPVRemapper.h
    └── SpvTools.h

6 directories, 58 files

@Ralith
Copy link
Contributor

@Ralith Ralith commented Aug 5, 2020

The situation here's a bit weird because various vulkan tooling packages all have different hardcoded revision dependencies on glslang, none of which are stable releases. If a package depends on both shaderc and glslang (why?) then it should probably try to use the same version of glslang that shaderc does. Maybe the explicit glslang dep should be removed from libplacebo entirely?

@CajuM
Copy link
Author

@CajuM CajuM commented Sep 3, 2020

I made one more patch, on top of this PR, but I'm not sure if it's ok because I removed the custom versions for glslang, everything works now. I also updated vulkan tooling to newer versions, except shaderc which requires an unreleased version of glslang. I also had some problems with vkd3d and vkquake, but managed to fix them. Also about glslang, there are several workarounds available made by openSUSE, Fedora and Gentoo. Gentoo patched shaderc to support the older version of glslang. Fedora and openSUSE used unreleased versions in rawhide and factory
https://gist.github.com/CajuM/86b867aa73eb85ab997cca73c8b9416b

@ofborg ofborg bot requested a review from Ralith Sep 24, 2020
@CajuM CajuM requested a review from doronbehar Sep 24, 2020
@@ -15824,24 +15824,24 @@ in
src = fetchFromGitHub {
Copy link
Contributor

@doronbehar doronbehar Sep 24, 2020

Again, I know you are merely attempting to update a few packages, but I think it ought to be taken care of while at it:

I understand why Google needs overrides as explained here. But could you explain what's with the overrides here? Ideally, these overrides should be in a separate file, and automatically generated via an passthru.updateScript. If that's not possible, it would be nice to get these details out of all-packages.nix.

Copy link
Author

@CajuM CajuM Sep 25, 2020

there's an easier way to do it, the patch I posted on gist removes the custom buildInputs versions all-together. So we'd be using one stable version for each of the updated packages in the entire repo. I didn't push it to this branch because I wasn't sure if it would be accepted. One problem I see is that we'd have to wait a bit to update the whole set of packages due to incompatible versions. Another might be that we'd be introducing regressions by not using the provided dependency versions for glslang and shaderc.

Copy link
Contributor

@Ralith Ralith Sep 27, 2020

The general issue is that upstream only advertises support for specific revisions, and every package that depends on glslang/spirv-tools seems to have its own unique vetted revision. Using a different revision than the one supported by upstream seems like asking for trouble. No opinion on the proper place/mechanism to encode the information, though; I can't keep up with the latest nixpkgs tricks.

Copy link
Contributor

@doronbehar doronbehar Sep 27, 2020

This is what I was thinking of:

diff --git i/pkgs/development/compilers/glslang/default.nix w/pkgs/development/compilers/glslang/default.nix
index dee3f488ecc..135eb94dd70 100644
--- i/pkgs/development/compilers/glslang/default.nix
+++ w/pkgs/development/compilers/glslang/default.nix
@@ -3,9 +3,27 @@
 , cmake
 , jq
 , python3
-, spirv-headers
-, spirv-tools
+, spirv-headers'
+, spirv-tools'
 }:
+let
+  spirv-tools = spirv-tools'.overrideAttrs (_: {
+    src = fetchFromGitHub {
+      owner = "KhronosGroup";
+      repo = "SPIRV-Tools";
+      rev = "fd8e130510a6b002b28eee5885a9505040a9bdc9";
+      sha256 = "00b7xgyrcb2qq63pp3cnw5q1xqx2d9rfn65lai6n6r89s1vh3vg6";
+    };
+  });
+  spirv-headers = spirv-headers'.overrideAttrs (_: {
+    src = fetchFromGitHub {
+      owner = "KhronosGroup";
+      repo = "SPIRV-Headers";
+      rev = "f8bf11a0253a32375c32cad92c841237b96696c0";
+      sha256 = "1znwjy02dl9rshqzl87rqsv9mfczw7gvwfhcirbl81idahgp4p6l";
+    };
+  });
+in
 
 stdenv.mkDerivation rec {
   pname = "glslang";
diff --git i/pkgs/development/tools/vulkan-validation-layers/default.nix w/pkgs/development/tools/vulkan-validation-layers/default.nix
index 0d3c227291f..0bd9dd0a662 100644
--- i/pkgs/development/tools/vulkan-validation-layers/default.nix
+++ w/pkgs/development/tools/vulkan-validation-layers/default.nix
@@ -1,6 +1,46 @@
-{ stdenv, fetchFromGitHub, cmake, writeText, python3
-, vulkan-headers, vulkan-loader, glslang, spirv-headers
-, pkgconfig, xlibsWrapper, libxcb, libXrandr, wayland }:
+{ stdenv
+, fetchFromGitHub
+, cmake
+, writeText
+, python3
+, spirv-headers
+, spirv-tools
+, vulkan-headers
+, vulkan-loader
+, glslang'
+, pkgconfig
+, xlibsWrapper
+, libxcb
+, libXrandr
+, wayland
+}:
+let
+  glslang = (glslang'.override {
+    spirv-tools' = spirv-tools.overrideAttrs (_: {
+      src = fetchFromGitHub {
+        owner = "KhronosGroup";
+        repo = "SPIRV-Tools";
+        rev = "e128ab0d624ce7beb08eb9656bb260c597a46d0a";
+        sha256 = "0jj8zrl3dh9fq71jc8msx3f3ifb2vjcb37nl0w4sa8sdhfff74pv";
+      };
+    });
+    spirv-headers' = spirv-headers.overrideAttrs (_: {
+      src = fetchFromGitHub {
+        owner = "KhronosGroup";
+        repo = "SPIRV-Headers";
+        rev = "ac638f1815425403e946d0ab78bac71d2bdbf3be";
+        sha256 = "1lkhs7pxcrfkmiizcxl0w5ajx6swwjv7w3iq586ipgh571fc75gx";
+      };
+    });
+  }).overrideAttrs (_: {
+    src = fetchFromGitHub {
+      owner = "KhronosGroup";
+      repo = "glslang";
+      rev = "e00d27c6d65b7d3e72506a311d7f053da4051295";
+      sha256 = "00lzvzk613gpm1vsdxffmx52z3c52ijwvzk4sfhh95p71kdydhgv";
+    };
+  });
+in
 
 stdenv.mkDerivation rec {
   pname = "vulkan-validation-layers";
@@ -21,7 +61,7 @@ stdenv.mkDerivation rec {
 
   buildInputs = [
     glslang
-    spirv-headers
+    glslang.spirv-headers
     vulkan-headers
     vulkan-loader
     libxcb
diff --git i/pkgs/top-level/all-packages.nix w/pkgs/top-level/all-packages.nix
index f4b0ae7dc31..fbf6f625e0c 100644
--- i/pkgs/top-level/all-packages.nix
+++ w/pkgs/top-level/all-packages.nix
@@ -9095,22 +9095,8 @@ in
   dotnetPackages = recurseIntoAttrs (callPackage ./dotnet-packages.nix {});
 
   glslang = callPackage ../development/compilers/glslang {
-    spirv-tools = spirv-tools.overrideAttrs (_: {
-      src = fetchFromGitHub {
-        owner = "KhronosGroup";
-        repo = "SPIRV-Tools";
-        rev = "fd8e130510a6b002b28eee5885a9505040a9bdc9";
-        sha256 = "00b7xgyrcb2qq63pp3cnw5q1xqx2d9rfn65lai6n6r89s1vh3vg6";
-      };
-    });
-    spirv-headers = spirv-headers.overrideAttrs (_: {
-      src = fetchFromGitHub {
-        owner = "KhronosGroup";
-        repo = "SPIRV-Headers";
-        rev = "f8bf11a0253a32375c32cad92c841237b96696c0";
-        sha256 = "1znwjy02dl9rshqzl87rqsv9mfczw7gvwfhcirbl81idahgp4p6l";
-      };
-    });
+    spirv-headers' = spirv-headers;
+    spirv-tools' = spirv-tools;
   };
 
   go_bootstrap = if stdenv.isAarch64 then
@@ -15819,31 +15805,7 @@ in
   vulkan-loader = callPackage ../development/libraries/vulkan-loader { };
   vulkan-tools = callPackage ../tools/graphics/vulkan-tools { };
   vulkan-validation-layers = callPackage ../development/tools/vulkan-validation-layers {
-    glslang = (glslang.override {
-      spirv-tools = spirv-tools.overrideAttrs (_: {
-        src = fetchFromGitHub {
-          owner = "KhronosGroup";
-          repo = "SPIRV-Tools";
-          rev = "e128ab0d624ce7beb08eb9656bb260c597a46d0a";
-          sha256 = "0jj8zrl3dh9fq71jc8msx3f3ifb2vjcb37nl0w4sa8sdhfff74pv";
-        };
-      });
-      spirv-headers = spirv-tools.overrideAttrs (_: {
-        src = fetchFromGitHub {
-          owner = "KhronosGroup";
-          repo = "SPIRV-Headers";
-          rev = "ac638f1815425403e946d0ab78bac71d2bdbf3be";
-          sha256 = "1lkhs7pxcrfkmiizcxl0w5ajx6swwjv7w3iq586ipgh571fc75gx";
-        };
-      });
-    }).overrideAttrs (_: {
-      src = fetchFromGitHub {
-        owner = "KhronosGroup";
-        repo = "glslang";
-        rev = "e00d27c6d65b7d3e72506a311d7f053da4051295";
-        sha256 = "00lzvzk613gpm1vsdxffmx52z3c52ijwvzk4sfhh95p71kdydhgv";
-      };
-    });
+    glslang' = glslang;
   };
 
   vtkWithQt5 = vtk.override { qtLib = qt514; };

The diff doesn't change the inputs of glslang at all, from the version of the current PR, but it does change them for vulkan-validation-layers. I think it's unavoidable if the typo is to be fixed.

Copy link
Author

@CajuM CajuM Sep 27, 2020

how about?

diff --git a/pkgs/development/compilers/glslang/default.nix b/pkgs/development/compilers/glslang/default.nix
index dee3f488ecc..183d0f4df02 100644
--- a/pkgs/development/compilers/glslang/default.nix
+++ b/pkgs/development/compilers/glslang/default.nix
@@ -5,7 +5,28 @@
 , python3
 , spirv-headers
 , spirv-tools
+, argSpirv-tools ? null
+, argSpirv-headers ? null
 }:
+let
+  localSpirv-tools = ({ argSpirv-tools ? spirv-tools.overrideAttrs (_: {
+    src = fetchFromGitHub {
+      owner = "KhronosGroup";
+      repo = "SPIRV-Tools";
+      rev = "fd8e130510a6b002b28eee5885a9505040a9bdc9";
+      sha256 = "00b7xgyrcb2qq63pp3cnw5q1xqx2d9rfn65lai6n6r89s1vh3vg6";
+    };
+  }) }: argSpirv-tools) { argSpirv-tools = argSpirv-tools; };
+
+  localSpirv-headers = ({ argSpirv-headers ? spirv-headers.overrideAttrs (_: {
+    src = fetchFromGitHub {
+      owner = "KhronosGroup";
+      repo = "SPIRV-Headers";
+      rev = "f8bf11a0253a32375c32cad92c841237b96696c0";
+      sha256 = "1znwjy02dl9rshqzl87rqsv9mfczw7gvwfhcirbl81idahgp4p6l";
+    };
+  }) }: argSpirv-headers) { argSpirv-headers = argSpirv-headers; };
+in
 
 stdenv.mkDerivation rec {
   pname = "glslang";
@@ -20,22 +41,22 @@ stdenv.mkDerivation rec {
 
   # These get set at all-packages, keep onto them for child drvs
   passthru = {
-    inherit spirv-tools spirv-headers;
+    inherit localSpirv-tools localSpirv-headers;
   };
 
   nativeBuildInputs = [ cmake python3 bison jq ];
   enableParallelBuilding = true;
 
   postPatch = ''
-    cp --no-preserve=mode -r "${spirv-tools.src}" External/spirv-tools
-    ln -s "${spirv-headers.src}" External/spirv-tools/external/spirv-headers
+    cp --no-preserve=mode -r "${localSpirv-tools.src}" External/spirv-tools
+    ln -s "${localSpirv-headers.src}" External/spirv-tools/external/spirv-headers
   '';
 
   # Ensure spirv-headers and spirv-tools match exactly to what is expected
   preConfigure = ''
     HEADERS_COMMIT=$(jq -r < known_good.json '.commits|map(select(.name=="spirv-tools/external/spirv-headers"))[0].commit')
     TOOLS_COMMIT=$(jq -r < known_good.json '.commits|map(select(.name=="spirv-tools"))[0].commit')
-    if [ "$HEADERS_COMMIT" != "${spirv-headers.src.rev}" ] || [ "$TOOLS_COMMIT" != "${spirv-tools.src.rev}" ]; then
+    if [ "$HEADERS_COMMIT" != "${localSpirv-headers.src.rev}" ] || [ "$TOOLS_COMMIT" != "${localSpirv-tools.src.rev}" ]; then
       echo "ERROR: spirv-tools commits do not match expected versions: expected tools at $TOOLS_COMMIT, headers at $HEADERS_COMMIT";
       exit 1;
     fi
diff --git a/pkgs/development/tools/vulkan-validation-layers/default.nix b/pkgs/development/tools/vulkan-validation-layers/default.nix
index 0d3c227291f..9c8490f7277 100644
--- a/pkgs/development/tools/vulkan-validation-layers/default.nix
+++ b/pkgs/development/tools/vulkan-validation-layers/default.nix
@@ -1,6 +1,46 @@
-{ stdenv, fetchFromGitHub, cmake, writeText, python3
-, vulkan-headers, vulkan-loader, glslang, spirv-headers
-, pkgconfig, xlibsWrapper, libxcb, libXrandr, wayland }:
+{ stdenv
+, fetchFromGitHub
+, cmake
+, writeText
+, python3
+, spirv-headers
+, spirv-tools
+, vulkan-headers
+, vulkan-loader
+, glslang
+, pkgconfig
+, xlibsWrapper
+, libxcb
+, libXrandr
+, wayland
+}:
+let
+  localGlslang = (glslang.override {
+    argSpirv-tools = spirv-tools.overrideAttrs (_: {
+      src = fetchFromGitHub {
+        owner = "KhronosGroup";
+        repo = "SPIRV-Tools";
+        rev = "e128ab0d624ce7beb08eb9656bb260c597a46d0a";
+        sha256 = "0jj8zrl3dh9fq71jc8msx3f3ifb2vjcb37nl0w4sa8sdhfff74pv";
+      };
+    });
+    argSpirv-headers = spirv-headers.overrideAttrs (_: {
+      src = fetchFromGitHub {
+        owner = "KhronosGroup";
+        repo = "SPIRV-Headers";
+        rev = "ac638f1815425403e946d0ab78bac71d2bdbf3be";
+        sha256 = "1lkhs7pxcrfkmiizcxl0w5ajx6swwjv7w3iq586ipgh571fc75gx";
+      };
+    });
+  }).overrideAttrs (_: {
+    src = fetchFromGitHub {
+      owner = "KhronosGroup";
+      repo = "glslang";
+      rev = "e00d27c6d65b7d3e72506a311d7f053da4051295";
+      sha256 = "00lzvzk613gpm1vsdxffmx52z3c52ijwvzk4sfhh95p71kdydhgv";
+    };
+  });
+in
 
 stdenv.mkDerivation rec {
   pname = "vulkan-validation-layers";
@@ -20,8 +60,8 @@ stdenv.mkDerivation rec {
   ];
 
   buildInputs = [
-    glslang
-    spirv-headers
+    localGlslang
+    localGlslang.localSpirv-headers
     vulkan-headers
     vulkan-loader
     libxcb
@@ -32,7 +72,7 @@ stdenv.mkDerivation rec {
   enableParallelBuilding = true;
 
   cmakeFlags = [
-    "-DGLSLANG_INSTALL_DIR=${glslang}"
+    "-DGLSLANG_INSTALL_DIR=${localGlslang}"
   ];
 
   # Help vulkan-loader find the validation layers
diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index f4b0ae7dc31..7ee745abe9f 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -9094,24 +9094,7 @@ in
 
   dotnetPackages = recurseIntoAttrs (callPackage ./dotnet-packages.nix {});
 
-  glslang = callPackage ../development/compilers/glslang {
-    spirv-tools = spirv-tools.overrideAttrs (_: {
-      src = fetchFromGitHub {
-        owner = "KhronosGroup";
-        repo = "SPIRV-Tools";
-        rev = "fd8e130510a6b002b28eee5885a9505040a9bdc9";
-        sha256 = "00b7xgyrcb2qq63pp3cnw5q1xqx2d9rfn65lai6n6r89s1vh3vg6";
-      };
-    });
-    spirv-headers = spirv-headers.overrideAttrs (_: {
-      src = fetchFromGitHub {
-        owner = "KhronosGroup";
-        repo = "SPIRV-Headers";
-        rev = "f8bf11a0253a32375c32cad92c841237b96696c0";
-        sha256 = "1znwjy02dl9rshqzl87rqsv9mfczw7gvwfhcirbl81idahgp4p6l";
-      };
-    });
-  };
+  glslang = callPackage ../development/compilers/glslang { };
 
   go_bootstrap = if stdenv.isAarch64 then
     srcOnly {
@@ -15818,33 +15801,7 @@ in
   vulkan-headers = callPackage ../development/libraries/vulkan-headers { };
   vulkan-loader = callPackage ../development/libraries/vulkan-loader { };
   vulkan-tools = callPackage ../tools/graphics/vulkan-tools { };
-  vulkan-validation-layers = callPackage ../development/tools/vulkan-validation-layers {
-    glslang = (glslang.override {
-      spirv-tools = spirv-tools.overrideAttrs (_: {
-        src = fetchFromGitHub {
-          owner = "KhronosGroup";
-          repo = "SPIRV-Tools";
-          rev = "e128ab0d624ce7beb08eb9656bb260c597a46d0a";
-          sha256 = "0jj8zrl3dh9fq71jc8msx3f3ifb2vjcb37nl0w4sa8sdhfff74pv";
-        };
-      });
-      spirv-headers = spirv-tools.overrideAttrs (_: {
-        src = fetchFromGitHub {
-          owner = "KhronosGroup";
-          repo = "SPIRV-Headers";
-          rev = "ac638f1815425403e946d0ab78bac71d2bdbf3be";
-          sha256 = "1lkhs7pxcrfkmiizcxl0w5ajx6swwjv7w3iq586ipgh571fc75gx";
-        };
-      });
-    }).overrideAttrs (_: {
-      src = fetchFromGitHub {
-        owner = "KhronosGroup";
-        repo = "glslang";
-        rev = "e00d27c6d65b7d3e72506a311d7f053da4051295";
-        sha256 = "00lzvzk613gpm1vsdxffmx52z3c52ijwvzk4sfhh95p71kdydhgv";
-      };
-    });
-  };
+  vulkan-validation-layers = callPackage ../development/tools/vulkan-validation-layers { };
 
   vtkWithQt5 = vtk.override { qtLib = qt514; };
 

Copy link
Contributor

@doronbehar doronbehar Oct 13, 2020

@CajuM sorry for the long wait. This diff is just as good. Please commit it, and please mention near every override why it is needed.

pkgs/top-level/all-packages.nix Outdated Show resolved Hide resolved
@jonringer
Copy link
Contributor

@jonringer jonringer commented Oct 13, 2020

Please follow CONTRIBUTING.md and manual#submitting-changes-making-patches and squash the fix-up commits.

git rebase -i makes this easy todo. I created a small video demonstrating it's use here. A more indepth text tutorial can be found here

@CajuM CajuM requested a review from doronbehar Oct 14, 2020
Copy link
Contributor

@doronbehar doronbehar left a comment

Otherwise looks good to me.

@@ -20,22 +49,22 @@ stdenv.mkDerivation rec {

# These get set at all-packages, keep onto them for child drvs
passthru = {
inherit spirv-tools spirv-headers;
inherit localSpirv-tools localSpirv-headers;
Copy link
Contributor

@doronbehar doronbehar Oct 14, 2020

Small nitpick - this is a change in the "API" of it - use spirv-tools = localSpirv-tools, etc.

# vulkan-validation-layers requires a custom glslang version, while glslang requires
# custom versions for spirv-tools and spirv-headers
# The vulkan-validation-layers dependency is taken from:
# https://github.com/KhronosGroup/Vulkan-ValidationLayers/blob/master/scripts/known_good.json
# While those for glslang are taken from:
# https://github.com/KhronosGroup/Vulkan-ValidationLayers/blob/master/scripts/known_good.json
Copy link
Contributor

@doronbehar doronbehar Oct 14, 2020

Suggested change
# vulkan-validation-layers requires a custom glslang version, while glslang requires
# custom versions for spirv-tools and spirv-headers
# The vulkan-validation-layers dependency is taken from:
# https://github.com/KhronosGroup/Vulkan-ValidationLayers/blob/master/scripts/known_good.json
# While those for glslang are taken from:
# https://github.com/KhronosGroup/Vulkan-ValidationLayers/blob/master/scripts/known_good.json
# vulkan-validation-layers requires a custom glslang version, while glslang requires
# custom versions for spirv-tools and spirv-headers. The git hashes required for all
# of these deps is documented upstream here:
# https://github.com/KhronosGroup/Vulkan-ValidationLayers/blob/master/scripts/known_good.json

glslang
spirv-headers
localGlslang
localGlslang.localSpirv-headers
Copy link
Contributor

@doronbehar doronbehar Oct 14, 2020

If you change back the API, as I think you should, don't forget this place. Note it shouldn't trigger rebuilds, but please check it all.

@doronbehar doronbehar merged commit 0826f2e into NixOS:master Oct 15, 2020
19 checks passed
@CajuM CajuM deleted the vulkan-up branch Oct 15, 2020
@SuperSandro2000
Copy link
Member

@SuperSandro2000 SuperSandro2000 commented Oct 17, 2020

the shaderc bump is causing some regressions, I now get:

builder for '/nix/store/86lrf5m4vy9r4hbwm6r95ljv6w0pwi4d-libplacebo-2.72.0.drv' failed with exit code 1; last 10 log lines:
  build flags: -j128 -l128
  [42/44] Compiling C++ object 'src/25a6634@@placebo@sha/glsl_glslang.cc.o's_gen.c.o'
  FAILED: src/25a6634@@placebo@sha/glsl_glslang.cc.o
  g++ -Isrc/25a6634@@placebo@sha -Isrc -I../src -I../src/include -I../src/include/dummy -I../subprojects/xtalloc/include -I../subprojects/bstr/include -I/nix/store/81iy20n5rhvcm18ha289anygpgq5qyjc-lcms2-2.11-dev/include -I/nix/store/riw48ysh0rhkf2fck7m1iiany4rif2mv-vulkan-headers-1.2.141.0/include -I/nix/store/psda85jw38cxzqp6fa59vrdipz67h6bk-epoxy-1.5.4-dev/include -I/usr/include/glslang -I/nix/store/my4bdsgi308bq216gkhfwnwnk35yhszn-libplacebo-2.72.0/include/glslang -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -std=c++11 -Wundef -Wshadow -Wparentheses -Wpointer-arith -fvisibility=hidden -fPIC -pthread -MD -MQ 'src/25a6634@@placebo@sha/glsl_glslang.cc.o' -MF 'src/25a6634@@placebo@sha/glsl_glslang.cc.o.d' -o 'src/25a6634@@placebo@sha/glsl_glslang.cc.o' -c ../src/glsl/glslang.cc
  ../src/glsl/glslang.cc:229:1: error: cannot convert '<brace-enclosed initializer list>' to 'int' in initialization
    229 | };
        | ^

I still get this error. Reverting the shaderc bump fixes it for me.

@SuperSandro2000 SuperSandro2000 mentioned this pull request Oct 17, 2020
10 tasks
@doronbehar
Copy link
Contributor

@doronbehar doronbehar commented Oct 17, 2020

😬 Then this means vlc and mpv are broken now. I wish all of maintainers of those packages would have used actual releases.. Using latest libplacebo from master doesn't fix this issue.. I'm working on a local override for libplacebo to use the version(s) that were previously used before this PR.

@CajuM
Copy link
Author

@CajuM CajuM commented Oct 17, 2020

grimacing Then this means vlc and mpv are broken now. I wish all of maintainers of those packages would have used actual releases.. Using latest libplacebo from master doesn't fix this issue.. I'm working on a local override for libplacebo to use the version(s) that were previously used before this PR.

It would seem that removing the https://code.videolan.org/videolan/libplacebo/-/commit/523056828ab86c2f17ea65f432424d48b6fdd389.patch patch fixes things

@doronbehar
Copy link
Contributor

@doronbehar doronbehar commented Oct 17, 2020

All of these packages are a mess (I'm not sure who is to blame). How come Arch Linux doesn't need such overrides? Perhaps they do need and use

Maybe we shouldn't give to much attention to the known_good.json file - just because they have verified these versions work together, doesn't mean that any other combination of versions won't work. For instance Arch Linux even disables shaderc vendoring, somewhat similarly to us, but with the same glslang and spirv-tools versions as the normal packages:

https://github.com/archlinux/svntogit-packages/blob/2900077c8c054d2c9604780d3f189e889271ec6d/trunk/PKGBUILD#L18-L29

Since there are no shaderc maintainers, I'm reluctant into doing this.

@SuperSandro2000
Copy link
Member

@SuperSandro2000 SuperSandro2000 commented Oct 17, 2020

grimacing Then this means vlc and mpv are broken now. I wish all of maintainers of those packages would have used actual releases.. Using latest libplacebo from master doesn't fix this issue.. I'm working on a local override for libplacebo to use the version(s) that were previously used before this PR.

It would seem that removing the code.videolan.org/videolan/libplacebo/-/commit/523056828ab86c2f17ea65f432424d48b6fdd389.patch patch fixes things

I removed https://code.videolan.org/videolan/libplacebo/-/commit/82e3be1839379791b58e98eb049415b42888d5b0.patch and it builds for me. Reverting shaderc also works #100818. Not sure what the right way is. Open for suggestions.

@doronbehar
Copy link
Contributor

@doronbehar doronbehar commented Oct 17, 2020

It would seem that removing the code.videolan.org/videolan/libplacebo/-/commit/523056828ab86c2f17ea65f432424d48b6fdd389.patch patch fixes things

I removed code.videolan.org/videolan/libplacebo/-/commit/82e3be1839379791b58e98eb049415b42888d5b0.patch and it builds for me.

I don't know what you did, but this sounds like a reasonable idea in comparison to #100818. I get:

  patching file src/glsl/glslang.cc
  Hunk #1 FAILED at 211.
  1 out of 1 hunk FAILED -- saving rejects to file src/glsl/glslang.cc.rej

With this change:

diff --git i/pkgs/development/libraries/libplacebo/default.nix w/pkgs/development/libraries/libplacebo/default.nix
index dc3e5e30c74..91de1f70fa7 100644
--- i/pkgs/development/libraries/libplacebo/default.nix
+++ w/pkgs/development/libraries/libplacebo/default.nix
@@ -27,10 +27,6 @@ stdenv.mkDerivation rec {
 
   patches = [
     # to work with latest glslang, remove on release >2.72.0
-    (fetchpatch {
-      url = "https://code.videolan.org/videolan/libplacebo/-/commit/523056828ab86c2f17ea65f432424d48b6fdd389.patch";
-      sha256 = "051vhd0l3yad1fzn5zayi08kqs9an9j8p7m63kgqyfv1ksnydpcs";
-    })
     (fetchpatch {
       url = "https://code.videolan.org/videolan/libplacebo/-/commit/82e3be1839379791b58e98eb049415b42888d5b0.patch";
       sha256 = "0nklj9gfiwkbbj6wfm1ck33hphaxrvzb84c4h2nfj88bapnlm90l";

@CajuM
Copy link
Author

@CajuM CajuM commented Oct 17, 2020

My bad, it was 82e3be1839379791b58e98eb049415b42888d5b0.patch

doronbehar added a commit to doronbehar/nixpkgs that referenced this issue Oct 17, 2020
From some reason, if we take this patch out, libplacebo builds fine, and
mpv is again usable.
@doronbehar
Copy link
Contributor

@doronbehar doronbehar commented Oct 17, 2020

Taking that patch out makes mpv and vlc build and work. Hence I prefer #100825 over #100818 .

@doronbehar
Copy link
Contributor

@doronbehar doronbehar commented Oct 17, 2020

Out of scope for the issue with libplacebo: All projects of KhronosGroup should not have local overrides, as according to my tests they are compatible with each other, as you'd expect since it's the same group that develops all of these tools - (glslang has local overrides).

To improve the situation of all ugly overrides, I think it'd be best to introduce an attribute set such as KhronosGroupPackages where it'd have it's own callPackage that will insure compatible versions are used in scope.

It should be also worth nagging upstream that their build process is not ideal for downstream package maintainers, considering the fact spirv-tools for instance needs to vendor it's libraries, and is uncapable of using distribution's builds directly.

dtzWill added a commit to dtzWill/nixpkgs that referenced this issue Nov 6, 2020
dtzWill added a commit to dtzWill/nixpkgs that referenced this issue Nov 6, 2020
@acowley
Copy link
Contributor

@acowley acowley commented Feb 10, 2021

Does anyone have any new thoughts on how to handle these upgrades? I packaged the tint shader compiler, but it needed newer spirv-tools. I'd be happy to PR tint to nixpkgs, but I don't want to break glslang dependents.

@SuperSandro2000
Copy link
Member

@SuperSandro2000 SuperSandro2000 commented Feb 11, 2021

We can either add multiple versions of glslang (maybe 11.1 or 8.X?) or you add a local overwrite in your package if only the version changes and nothing else.

@doronbehar
Copy link
Contributor

@doronbehar doronbehar commented Feb 15, 2021

We can either add multiple versions of glslang (maybe 11.1 or 8.X?)

That sounds great to me - to remove all overrides inside packages' expressions, and have as many variants as needed of the same packages all declared in all-packages.nix.

@Ralith
Copy link
Contributor

@Ralith Ralith commented Feb 15, 2021

Prominent consumers use distinct, specific, non-release revisions that must be updated in lockstep with the consumers. Moving that into a separate file sounds painful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants