Won't build on Linux

[Kr3m]

In function ‘void* memcpy(void*, const void*, size_t)’,
    inlined from ‘bool idDataQueue<maxItems, maxBuffer>::Append(int, const byte*, int, const byte*, int) [with int maxItems = 64; int maxBuffer = 8000]’ at idlib/../idlib/DataQueue.h:112:8,
    inlined from ‘void idPacketProcessor::VerifyEmptyReliableQueue(byte, byte)’ at /home/kevin/GitHub/RBDOOM-3-BFG/neo/sys/PacketProcessor.cpp:660:16:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:33: error: argument 2 null where non-null expected [-Werror=nonnull]
   29 |   return __builtin___memcpy_chk (__dest, __src, __len,
      |          ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
   30 |                                  __glibc_objsize0 (__dest));
      |                                  ~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:33: note: in a call to built-in function ‘void* __builtin___memcpy_chk(void*, const void*, long unsigned int, long unsigned int)’
cc1plus: some warnings being treated as errors
make[2]: *** [CMakeFiles/RBDoom3BFG.dir/build.make:7678: CMakeFiles/RBDoom3BFG.dir/sys/PacketProcessor.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:195: CMakeFiles/RBDoom3BFG.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

[Kr3m]

With more lax error checking, it will build, but it will not run.

----- Initializing Render Shaders -----
Sys_Error: Couldn't read the binary file for shader renderprogs2/spirv/builtin/gui.vs.bin

[RobertBeckebans]

The README should be updated with instructions that align more with the full Windows builds. Right now it is just how you get the minimal version of RBDoom running.

This is usually what I do under Linux:

1. Run in a terminal $ git clone --recursive https://github.com/RobertBeckebans/RBDOOM-3-BFG.git DoomCode
2. Run cmake-linux-release.sh in DoomCode/neo/
3. Run make -j<number of your cores> in DoomCode/build/
4. Copy DoomCode/build/RBDoom3BFG to DoomCode/ or symlink it
5. Copy the base folder of your Steam\steamapps\common\DOOM 3 BFG Edition over to DoomCode/base/
6. [OPTIONAL] Download https://www.moddb.com/mods/rbdoom-3-bfg/downloads/rbdoom-3-bfg-130 and unpack it over your DoomCode/ folder and then run git checkout . in DoomCode/ so the local git files are newer for the files that have the same names
7. Run the engine in DoomCode/

[Kr3m]

The README should be updated with instructions that align more with the full Windows builds. Right now it is just how you get the minimal version of RBDoom running.

This is usually what I do under Linux:

1. Run in a terminal $ git clone --recursive https://github.com/RobertBeckebans/RBDOOM-3-BFG.git DoomCode
2. Run cmake-linux-release.sh in DoomCode/neo/
3. Run make -j<number of your cores> in DoomCode/build/
4. Copy DoomCode/build/RBDoom3BFG to DoomCode/ or symlink it
5. Copy the base folder of your Steam\steamapps\common\DOOM 3 BFG Edition over to DoomCode/base/
6. [OPTIONAL] Download https://www.moddb.com/mods/rbdoom-3-bfg/downloads/rbdoom-3-bfg-130 and unpack it over your DoomCode/ folder and then run git checkout . in DoomCode/ so the local git files are newer for the files that have the same names
7. Run the engine in DoomCode/

Only way I can seem to get it to build at all is two comment out these two lines in CMakeLists.txt

#add_definitions(-Werror=format-security)
#add_definitions(-Werror=format)

I'll try it again using your method in combination with this, and see what happens.

[Kr3m]

Still having issues even after that build.

Sys_Error: Couldn't find the required shader permutation in the blob, or the blob is corrupted.
Required permutation key: 
USE_GPU_SKINNING=0;
Permutations available in the blob:
USE_GPU_SKINNING=0 
USE_GPU_SKINNING=1 

[RobertBeckebans]

Which Linux system are you using?

[Kr3m]

Which Linux system are you using?

I'm on Ubuntu 24.04 at the moment.

[naggie]

I have the same runtime error, after building master (33b74a7) instead of using 1.5.1 to work around #787 on NixOS 23.11

[Kr3m]

I have the same runtime error, after building master (33b74a7) instead of using 1.5.1 to work around #787 on NixOS 23.11

Definitely odd. I remember building this a few years ago with no issues at all. Ran the script, ran make, and was off to the races without using error suppression or anything...

[RobertBeckebans]

This is the latest official build for the Discord people for testing. Could you copy the base/renderprogs2/ folder from there into your local setup and run the engine? It's maybe a problem with the DXC compiler build for Linux.

[RobertBeckebans]

Here is the link, sorry: https://drive.google.com/file/d/1M0HhGfkK1viQz3HktusFjaqNS8lALxDO/view?usp=drive_link

[Kr3m]

This is the latest official build for the Discord people for testing. Could you copy the base/renderprogs2/ folder from there into your local setup and run the engine? It's maybe a problem with the DXC compiler build for Linux.

That works great. Sadly, it is only good for testing or playing the game. With the DXC compiler busted and the binary only building with error suppression on, I won't be able to use this for development much outside of Windows. I guess the error suppression issue isn't that big of a deal. Unfortunately, the only compiler I've fixed for Linux is the q3lcc compiler for Quake 3 and that was just one line of code lol. I'm not even sure where I'd start with the DXC compiler, but I'll take a look at it when I have some time.

[RobertBeckebans]

Can you try these DXC compiler binaries from October? https://drive.google.com/file/d/1LdQxsXicIp_W5Y5GsNWgzB7MGwCxzypp/view?usp=drive_link

If that works then I will link it to the README.

[naggie]

I can also confirm that the 1.6.0.6 build shaders work perfectly. Thanks @RobertBeckebans (and for the whole project, it's incredible and fantastic) -- I will wait for the main release on github.

[RobertBeckebans]

I can also confirm that the 1.6.0.6 build shaders work perfectly. Thanks @RobertBeckebans (and for the whole project, it's incredible and fantastic) -- I will wait for the main release on github.

Thanks man

[SRSaunders]

What version of dxc is failing? I can confirm that version 1.7 (dev;4006-69e54e29) still works fine on linux. On macOS I am using version 1.8 (dev;4508-263a7733) without any issues.

[Kr3m]

What version of dxc is failing? I can confirm that version 1.7 (dev;4006-69e54e29) still works fine on linux. On macOS I am using version 1.8 (dev;4508-263a7733) without any issues.

I was using 1.8 on Linux.

[legluondunet]

Can we hope that you will provide a Linux binary for the next release? Linux users are like Windows users, they appreciate being able to download and launch their applications directly without having to compile them first. Thank you for your attention to my request.

[hemebond]

I've tried compiling the latest RBDoom3BFG with these versions of DXC:

• linux_dxc_2023_08_14.x86_64
• linux_dxc_2023_10_07.x86_64
• linux_dxc_2024_05_24.x86_64

And all gave me the blob error:

RBDOOM 3 BFG 1.6.0 preview.1403 linux-x86_64 Jun  6 2024 19:00:20
...
----- Initializing Render Shaders -----
Loading shader program builtin/gui
Sys_Error: Couldn't find the required shader permutation in the blob, or the blob is corrupted.
Required permutation key: 
USE_GPU_SKINNING=0;
Permutations available in the blob:
USE_GPU_SKINNING=0 
USE_GPU_SKINNING=1 

[legluondunet]

@hemebond which Linux distribution are you using?

[hemebond]

@hemebond which Linux distribution are you using?

Debian Linux x86_64

 $ uname -a
Linux excession 6.7.12-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.7.12-1 (2024-04-24) x86_64 GNU/Linux
 $ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux trixie/sid
Release:	n/a
Codename:	trixie

[legluondunet]

I just compiled RBDoom3BFG with success, did you follow all theses steps?

1. download DXC package (I used linux_dxc_2024_03_29.x86_64.tar.gz) and extract it in ~/.local
2. make ~/. local/bin/dxc as executable
3. git clone --recursive https://github.com/RobertBeckebans/RBDOOM-3-BFG.git
4. cd RBDOOM-3-BFG
5. git submodule init
6. git submodule update --recursive
7. cd neo
8. ./cmake-linux-release.sh
9. cd ../build/
10. make

The folder where you stored git code should not contains space or special characters.

Did you obtain errors messages at step 8? If so copy the log here.
What is your Vulkan library headers version? Mine is 1.3.279
What is your MESA library headers version? Mine is 24.0.6

My config:
Manjaro 24.0.0 Wynsdey
Kernel: x86_64 Linux 6.6.30-2-MANJARO
Proc: AMD Ryzen 9 5900HX
GPU: AMD Radeon RX 6600M
Drivers: Mesa 24.0.6
Resolution: 1920x1080
32 Go RAM
DE: Xfce4

[hemebond]

@legluondunet Sorry, the compilation is fine. The error I get is the same runtime error others are getting:

Sys_Error: Couldn't find the required shader permutation in the blob, or the blob is corrupted.

libvulkan1/testing,now 1.3.280.0-1 amd64 [installed]
mesa-common-dev/testing,now 24.0.7-1 amd64 [installed]

I just did #884 (comment) and the game runs now.

Would the tools mentioned in #887 be an alternative method of compiling the shaders?

[ericwomer]

I still get

Sys_Error: Couldn't find the required shader permutation in the blob, or the blob is corrupted.

as an error message when running RBDoom3BFG

[BielBdeLuna]

I'm getting the initial error in this issue, how did you guys pass further than the error?

I've added this definitions in cmake:

	add_definitions(-Wno-format-zero-length)
	add_definitions(-Wno-nonnull)

[DistantThunder]

The problem appears to be lying elsewhere. I used several builds of DXC and was always met with the same problem.
Oldest I used was "linux_dxc_2023_08_14.x86_64" and even used the "special" artifact mentioned in one of the comment. Still getting the same:

RBDoom3BFG[825278]: ----- Initializing Render Shaders -----
RBDoom3BFG[825278]: Loading shader program builtin/gui
RBDoom3BFG[825278]: Sys_Error: Couldn't find the required shader permutation in the blob, or the blob is corrupted.
RBDoom3BFG[825278]: Required permutation key:
RBDoom3BFG[825278]: USE_GPU_SKINNING=0;
RBDoom3BFG[825278]: Permutations available in the blob:
RBDoom3BFG[825278]: USE_GPU_SKINNING=0
RBDoom3BFG[825278]: USE_GPU_SKINNING=1

[SRSaunders]

This issue seems to have arisen following the incorporation of ShaderMake into the code base. I am wondering whether there are subtle differences in shader blob construction from old to new. I would like to ask @DistantThunder to perform a very simple test if you have not already done so: do a make clean and then rebuild from scratch (dxc version should not matter, but I guess it is possible). This will force the new ShaderMake code to run and rebuild the shader blob. Please let me know if this helps.

FYI - there is a known make dependency issue where the blob will not be rebuilt even with changes in dxc version as well as other changes. For instance, when building initially off of master, then switching branches and building incrementally, the blob will not be rebuilt even though in some cases it should be (e.g. with #855).

[DistantThunder]

@SRSaunders My rebuild command across DXC versions was:

rm -rf build ; pushd neo/ && env CC=/usr/lib/ccache/bin/clang CXX=/usr/lib/ccache/bin/clang++ ./cmake-linux-release.sh -DDXC_SPIRV_PATH=/linux_dxc_2024_05_24.x86_64/bin/dxc && popd && pushd build && make -j8 ; popd

I cloned the repo for the first time just yesterday.
I understand the compiled shaders would be in the build directory wouldn't they? So I was always purging that dir anyway.

[SRSaunders]

Thanks @DistantThunder for the extra info.

-DDXC_SPIRV_PATH=/linux_dxc_2024_05_24.x86_64/bin/dxc

Should actually be

-DDXC_CUSTOM_PATH=/linux_dxc_2024_05_24.x86_64/bin/dxc

DXC_SPIRV_PATH is an output variable of the build system. The input variable for setting the dxc path is DXC_CUSTOM_PATH. See docs at https://github.com/RobertBeckebans/RBDOOM-3-BFG?tab=readme-ov-file#compile_linux. Please try this and see if it makes a difference for you.

I am beginning to regret reporting that internal error message in compileshaders.cmake about setting DXC_SPIRV_PATH. I put it there primarily as a development aid when incorporating ShaderMake and to be consistent with the other error messages. However it is misleading since DXC_SPIRV_PATH is set by ShaderMake based on platform search paths, and as a last resort sets the value to DXC_CUSTOM_PATH. i.e. DXC_SPIRV_PATH is not meant to be a user input setting, but one that is set by ShaderMake and used by compileshaders.cmake.

[DistantThunder]

@SRSaunders I finally understood my mistake: I was using master against latest stable v1.5.1 which is more than 1 year old. So I surmise there was some mismatch between the compiled shader binaries in v1.5.1 vs the binary build from master.

• I checked out out v1.5.1 tag
• Adapted the DXC path var for CMAKE indeed
• Cherry picked a commit for an SSAO_COMPUTE shader that was incompatible with recent DXC builds: git cherry-pick 8f84e54bb053cad9599e443675761087ce884563
• Compiled and finally I could get on main menu

TLDR: RBDOOM binary from master is not necessarily compatible with current stable patched game content.

[SRSaunders]

@DistantThunder thanks for coming back with this info.

TLDR: RBDOOM binary from master is not necessarily compatible with current stable patched game content.

You can't mix RBDOOM-3-BFG binaries from one version with shader binaries from another. Either use the v1.5.1 tag or the current master branch (1.6 preview), but don't mix the two.

That's also why I asked you to rebuild from scratch with a consistent version of dxc, i.e. to make sure the shader binaries match the version of the code you are running.

[DistantThunder]

@DistantThunder thanks for coming back with this info.

TLDR: RBDOOM binary from master is not necessarily compatible with current stable patched game content.

You can't mix RBDOOM-3-BFG binaries from one version with shader binaries from another. Either use the v1.5.1 tag or the current master branch (1.6 preview), but don't mix the two.

That's also why I asked you to rebuild from scratch with a consistent version of dxc, i.e. to make sure the shader binaries match the version of the code you are running.

Yes but since I wasn't running v1.5.1 anyway this was never going to work. I don't know how to produce the shader binaries with the build. I don't see any ".bin" files produced when running make or even make Shaders so I wouldn't know how to produce both game binary & shader binaries from the source.

[SRSaunders]

I don't know how to produce the shader binaries with the build. I don't see any ".bin" files produced when running make or even make Shaders so I wouldn't know how to produce both game binary & shader binaries from the source.

The shaders are compiled automatically as part of the build (that's what dxc is for). There is no separate action required on your part. The compiled shader .bin files are located in <yourhomedir>/RBDOOM-3-BFG/base/renderprogs2/spirv/...

FYI:

1. For 1.5.X builds you can set the dxc path using -DDXC_SPIRV_EXECUTABLE=<...> on the cmake command line
2. For 1.6 (master) builds you can set the dxc path using -DDXC_CUSTOM_PATH=<...> on the cmake command line

I am glad you have a solution working now. However I believe there is a use case that causes failures, and I suspect that is when you compile the code base (or use a precompiled version) and then checkout a new branch and incrementally build without doing a build clean. This can happen if you are trying to checkout master against an old 1.5.1 build, or any other scenario where there are shader updates. The old shader binaries will then mismatch the new application binary and things will not run. The make dependencies for shader builds should instead detect and handle this case, and force a rebuild of the shaders without a manual build clean by the user.

[DistantThunder]

The compiled shader .bin files are located in /RBDOOM-3-BFG/base/renderprogs2/spirv/...

Ah! So they're not under the <path>/RBDOOM-3-BFG/build> tree. Well that's what I was missing then. I only ever copied over the binary from build/rbdoom3.bin and considered those dirs to be just source code.

Since it's customary to have all the build artifacts of the whole runtime in the same build dir, I didn't suspect I had to copy over the base directory.

I compiled directly master and only copied over the produced binary in build dir. I didn't check to v1.5.1 before I understood what was going on. The shaders are systematically compiled, I don't believe they are ever cached.

So if there's anything worth correcting with the build system, I'd say is to put all of the built runtime in the same directory (both RBDOOM binary and the "base" content matching the binary) or update the doc?

[SRSaunders]

So if there's anything worth correcting ... update the doc?

Yes, the docs should probably be updated to reflect that built shaders are written to the <path>/RBDOOM-3-BFG/base directory within the tree containing the build directory. Note that in most cases you don't need to move the binary from the build directory. The search paths within the app allow you to leave it as is and the game will find the assets and built shaders in base and run. The default cmake build scripts assume you execute the script within <path>/RBDOOM-3-BFG/neo and the build directory is placed one level up at the top level.

FYI, the game generates other assets into the base directory as you play - i.e. baked lighting data (into base/generated) and environment probes (into base/env). So if you have multiple installations or build trees, you need to be aware that the game data is not static, but can grow as you step into new map locations and play.

[ericwomer]

I was able to get RBDoom3BFG to compile and run with the latest release version of DXC.

RBDoom3BFG branch: master
RBDoom3BFG Commit #: a3fb4a0
dxc --version: libdxcompiler.so: 1.8(dev;4597-fd7e54bc); libdxil.so: 1.8
dxc package name: linux_dxc_2024_05_24.x86_64.tar.gz
dxc download link: https://github.com/microsoft/DirectXShaderCompiler/releases

Note: Added the following lines to the neo/CMakeLists.txt file

        add_definitions(-Wno-format-zero-length)
	add_definitions(-Wno-nonnull)

after the line add_definitions(-Werror=format) following the instructions in @BielBdeLuna 's comment #884 (comment)

[SRSaunders]

@ericwomer what compiler are you using? gcc or clang and which versions? Thanks.

[ericwomer]

@RobertBeckebans

Both with clang and gcc.

GCC:

gcc --version
gcc (Ubuntu 13.2.0-23ubuntu4) 13.2.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

CLang/CLang++:

clang --version
Ubuntu clang version 18.1.8 (++20240701074744+3b5b5c1ec4a3-1~exp1~20240701194905.84)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

clang++ --version
Ubuntu clang version 18.1.8 (++20240701074744+3b5b5c1ec4a3-1~exp1~20240701194905.84)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

[RobertBeckebans]

I just followed the new Linux instructions in the README on a fresh Fedora 40 installation and everything works out of the box. Therefore I'm closing this.