MAINT/BUILD: Clean up scripts directory

[Krzmbrzl]

Inside scripts there were several entries that appear to have no use anymore or scripts that have been written in Pearl, which is not really usable in a cross-platform environment.

This PR attempts to radically clean up the mess inside the scripts directory by deleting unneeded entries and by rewriting several scripts in Python.

Furthermore, all scripts that are used to generate source files are now directly embedded into the build process in order to generate the source files on-the-fly instead of generating them once and then keeping them as part of the source tree.

Finally, non-script entries where moved to a new auxiliary_files directory. These files mostly got converted to make use of cmake's configure_file mechanism to replace certain things (as e.g. a binary's name) with the appropriate value. This has to be seen as a first step of resolving #5436.

Checks

• My commits follow the commit guidelines

[Krzmbrzl]

@davidebeatrici could you please check that the files that I deleted are indeed not needed anymore? I probably did not think of all use-cases 👀

[Krzmbrzl]

@dvzrv I think I have addressed all of your concerns now. Mind having another look at this PR to check whether I missed anything?

[dvzrv]

Okay, here are the results of building the client:

cmake -D CMAKE_INSTALL_PREFIX=/usr -D CMAKE_BUILD_TYPE=None -D warnings-as-errors=OFF -D bundled-opus=OFF -D bundled-speex=OFF -D update=OFF -D server=OFF -D translations=OFF  -W no-dev -S mumble -B build-client
cmake --build build-client
DESTDIR=install_mumble-client cmake --install build-client

I noticed a bunch of files that are not part of mumble and should not be installed (MS GSL? Tracy?). Not sure if that's due to this PR though:

$ tree -n install_mumble-client

install_mumble-client
└── usr
    ├── bin
    │   ├── mumble
    │   └── mumble-overlay
    ├── include
    │   ├── Tracy.hpp
    │   ├── TracyC.h
    │   ├── TracyD3D11.hpp
    │   ├── TracyD3D12.hpp
    │   ├── TracyLua.hpp
    │   ├── TracyOpenCL.hpp
    │   ├── TracyOpenGL.hpp
    │   ├── TracyVulkan.hpp
    │   ├── client
    │   │   ├── TracyArmCpuTable.hpp
    │   │   ├── TracyCallstack.h
    │   │   ├── TracyCallstack.hpp
    │   │   ├── TracyDebug.hpp
    │   │   ├── TracyDxt1.hpp
    │   │   ├── TracyFastVector.hpp
    │   │   ├── TracyLock.hpp
    │   │   ├── TracyProfiler.hpp
    │   │   ├── TracyRingBuffer.hpp
    │   │   ├── TracyScoped.hpp
    │   │   ├── TracyStringHelpers.hpp
    │   │   ├── TracySysTime.hpp
    │   │   ├── TracySysTrace.hpp
    │   │   ├── TracyThread.hpp
    │   │   ├── tracy_SPSCQueue.h
    │   │   ├── tracy_concurrentqueue.h
    │   │   └── tracy_rpmalloc.hpp
    │   ├── common
    │   │   ├── TracyAlign.hpp
    │   │   ├── TracyAlloc.hpp
    │   │   ├── TracyApi.h
    │   │   ├── TracyColor.hpp
    │   │   ├── TracyForceInline.hpp
    │   │   ├── TracyMutex.hpp
    │   │   ├── TracyProtocol.hpp
    │   │   ├── TracyQueue.hpp
    │   │   ├── TracySocket.hpp
    │   │   ├── TracyStackFrames.hpp
    │   │   ├── TracySystem.hpp
    │   │   ├── TracyYield.hpp
    │   │   ├── tracy_lz4.hpp
    │   │   └── tracy_lz4hc.hpp
    │   └── gsl
    │       ├── gsl
    │       ├── gsl_algorithm
    │       ├── gsl_assert
    │       ├── gsl_byte
    │       ├── gsl_util
    │       ├── multi_span
    │       ├── pointers
    │       ├── span
    │       ├── span_ext
    │       └── string_span
    ├── lib
    │   ├── libTracyClient.a
    │   └── mumble
    │       ├── libcelt0.so -> libcelt0.so.0.7.0
    │       ├── libcelt0.so.0.7.0
    │       ├── libmumbleoverlay.x86.so -> libmumbleoverlay.x86.so.1.5.0
    │       ├── libmumbleoverlay.x86.so.1.5.0
    │       ├── libmumbleoverlay.x86_64.so -> libmumbleoverlay.x86_64.so.1.5.0
    │       ├── libmumbleoverlay.x86_64.so.1.5.0
    │       └── plugins
    │           ├── libamongus.so
    │           ├── libaoc.so
    │           ├── libarma2.so
    │           ├── libbf1.so
    │           ├── libbf1942.so
    │           ├── libbf2.so
    │           ├── libbf2142.so
    │           ├── libbf3.so
    │           ├── libbf4.so
    │           ├── libbf4_x86.so
    │           ├── libbfbc2.so
    │           ├── libbfheroes.so
    │           ├── libblacklight.so
    │           ├── libborderlands.so
    │           ├── libborderlands2.so
    │           ├── libbreach.so
    │           ├── libcod2.so
    │           ├── libcod4.so
    │           ├── libcod5.so
    │           ├── libcodmw2.so
    │           ├── libcodmw2so.so
    │           ├── libcs.so
    │           ├── libdys.so
    │           ├── libetqw.so
    │           ├── libffxiv.so
    │           ├── libffxiv_x64.so
    │           ├── libgmod.so
    │           ├── libgtaiv.so
    │           ├── libgtasa.so
    │           ├── libgtav.so
    │           ├── libgw.so
    │           ├── libinsurgency.so
    │           ├── libjc2.so
    │           ├── liblink.so
    │           ├── liblol.so
    │           ├── liblotro.so
    │           ├── libql.so
    │           ├── librl.so
    │           ├── libse.so
    │           ├── libsr.so
    │           ├── libut2004.so
    │           ├── libut3.so
    │           ├── libut99.so
    │           ├── libwolfet.so
    │           ├── libwow.so
    │           └── libwow_x64.so
    └── share
        ├── Tracy
        │   ├── TracyConfig-none.cmake
        │   └── TracyConfig.cmake
        ├── applications
        │   └── info.mumble.Mumble.desktop
        ├── cmake
        │   └── Microsoft.GSL
        │       ├── Microsoft.GSLConfig.cmake
        │       └── Microsoft.GSLConfigVersion.cmake
        ├── icons
        │   └── hicolor
        │       ├── 256x256
        │       │   └── apps
        │       │       └── mumble.png
        │       └── scalable
        │           └── apps
        │               └── mumble.svg
        ├── man
        │   └── man1
        │       ├── mumble-overlay.1
        │       └── mumble.1
        └── metainfo
            └── info.mumble.Mumble.appdata.xml

23 directories, 114 files

Apart from the unrelated files this seems alright. Will have a look at the server next

[Krzmbrzl]

I noticed a bunch of files that are not part of mumble and should not be installed (MS GSL? Tracy?).

Yeah, that's unrelated from this PR. I noticed these issues in #5006 as well. Probably cmake automatically includes the install targets of linked-against targets as well 🤔

[dvzrv]

Hm, with the server I'm running into some issues:

cmake -D CMAKE_INSTALL_PREFIX=/usr -D CMAKE_BUILD_TYPE=None -D warnings-as-errors=OFF -D bundled-opus=OFF -D bundled-speex=OFF -D update=OFF -D grpc=ON -D client=OFF -D translations=OFF  -W no-dev -S mumble -B build-server
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python interpreter version 3.10.6
-- Performing Test COMPILER_HAS_MBIG_OBJ
-- Performing Test COMPILER_HAS_MBIG_OBJ - Failed
-- ##################################################
-- Mumble version:          1.5.0
-- Architecture:            x64
-- Build type:              None
-- Using LTO:               ON
-- ##################################################
-- Qt5 component found: Core | Version: 5.15.5
-- Qt5 component found: Network | Version: 5.15.5
-- Qt5 component found: Xml | Version: 5.15.5
-- OpenSSL component found: Crypto | Version: 1.1.1q
-- OpenSSL component found: SSL | Version: 1.1.1q
-- Found ZLIB: /usr/lib/libz.so (found version "1.2.12")
-- Protobuf found | Version: 3.21.5.0
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- TRACY_ENABLE: OFF
-- TRACY_ON_DEMAND: ON
-- TRACY_CALLSTACK: OFF
-- TRACY_ONLY_LOCALHOST: OFF
-- TRACY_NO_BROADCAST: OFF
-- TRACY_NO_CODE_TRANSFER: OFF
-- TRACY_NO_CONTEXT_SWITCH: OFF
-- TRACY_NO_EXIT: OFF
-- TRACY_NO_FRAME_IMAGE: OFF
-- TRACY_NO_SAMPLING: OFF
-- TRACY_NO_VERIFY: OFF
-- TRACY_NO_VSYNC_CAPTURE: OFF
-- Disabling warnings for targets: TracyClient
-- Tracy: OFF
-- Found Git: /usr/bin/git (found version "2.37.3")
-- Qt5 component found: Sql | Version: 5.15.5
-- File: /home/dave/Downloads/build-server/src/murmur/murmur_autogen
-- Qt5 component found: DBus | Version: 5.15.5
-- Ice component found: Ice | Version: 3.7.8
-- Ice component found: IceSSL | Version: 3.7.8
-- Ice optional component found: IceDiscovery | Version: 3.7.8
-- Ice optional component found: IceLocatorDiscovery | Version: 3.7.8
CMake Error at cmake/delayed_configure_files.cmake:39 (message):
  delayed_configure_files: Provided target "overlay_gl" is not actually a
  (valid) target
Call Stack (most recent call first):
  auxiliary_files/CMakeLists.txt:91 (delayed_configure_files)


-- Configuring incomplete, errors occurred!
See also "/home/dave/Downloads/build-server/CMakeFiles/CMakeOutput.log".
See also "/home/dave/Downloads/build-server/CMakeFiles/CMakeError.log".

[Krzmbrzl]

Ah yes, the problem seems to be that the internal overlay option is enabled, even if the client isn't built. This should be fixed with my latest commit

[Krzmbrzl]

I now also fixed the additional, unrelated files getting installed, when installing Mumble.

[dvzrv]

Alright, here are the results for building the server-side:

cmake -D CMAKE_INSTALL_PREFIX=/usr -D CMAKE_BUILD_TYPE=None -D warnings-as-errors=OFF -D bundled-opus=OFF -D bundled-speex=OFF -D update=OFF -D grpc=ON -D client=OFF -D translations=OFF  -W no-dev -S mumble -B build-server
cmake --build build-server
DESTDIR=install_mumble-server cmake --install build-server

Here we can also see that additional, unwanted files are being installed, but I guess that's something for another PR?

tree -n install_mumble-server

install_mumble-server
└── usr
    ├── bin
    │   ├── mumble-server
    │   └── mumble-server-user-wrapper
    ├── etc
    │   ├── mumble
    │   │   └── mumble-server.ini
    │   └── sysconfig.d
    │       └── mumble-server.conf
    ├── include
    │   ├── Tracy.hpp
    │   ├── TracyC.h
    │   ├── TracyD3D11.hpp
    │   ├── TracyD3D12.hpp
    │   ├── TracyLua.hpp
    │   ├── TracyOpenCL.hpp
    │   ├── TracyOpenGL.hpp
    │   ├── TracyVulkan.hpp
    │   ├── client
    │   │   ├── TracyArmCpuTable.hpp
    │   │   ├── TracyCallstack.h
    │   │   ├── TracyCallstack.hpp
    │   │   ├── TracyDebug.hpp
    │   │   ├── TracyDxt1.hpp
    │   │   ├── TracyFastVector.hpp
    │   │   ├── TracyLock.hpp
    │   │   ├── TracyProfiler.hpp
    │   │   ├── TracyRingBuffer.hpp
    │   │   ├── TracyScoped.hpp
    │   │   ├── TracyStringHelpers.hpp
    │   │   ├── TracySysTime.hpp
    │   │   ├── TracySysTrace.hpp
    │   │   ├── TracyThread.hpp
    │   │   ├── tracy_SPSCQueue.h
    │   │   ├── tracy_concurrentqueue.h
    │   │   └── tracy_rpmalloc.hpp
    │   ├── common
    │   │   ├── TracyAlign.hpp
    │   │   ├── TracyAlloc.hpp
    │   │   ├── TracyApi.h
    │   │   ├── TracyColor.hpp
    │   │   ├── TracyForceInline.hpp
    │   │   ├── TracyMutex.hpp
    │   │   ├── TracyProtocol.hpp
    │   │   ├── TracyQueue.hpp
    │   │   ├── TracySocket.hpp
    │   │   ├── TracyStackFrames.hpp
    │   │   ├── TracySystem.hpp
    │   │   ├── TracyYield.hpp
    │   │   ├── tracy_lz4.hpp
    │   │   └── tracy_lz4hc.hpp
    │   └── gsl
    │       ├── gsl
    │       ├── gsl_algorithm
    │       ├── gsl_assert
    │       ├── gsl_byte
    │       ├── gsl_util
    │       ├── multi_span
    │       ├── pointers
    │       ├── span
    │       ├── span_ext
    │       └── string_span
    ├── lib
    │   ├── libTracyClient.a
    │   └── systemd
    │       └── system
    │           └── mumble-server.service
    └── share
        ├── Tracy
        │   ├── TracyConfig-none.cmake
        │   └── TracyConfig.cmake
        ├── cmake
        │   └── Microsoft.GSL
        │       ├── Microsoft.GSLConfig.cmake
        │       └── Microsoft.GSLConfigVersion.cmake
        └── man
            └── man1
                ├── mumble-server-user-wrapper.1
                └── mumble-server.1

18 directories, 61 files

All looks mainly fine so far, but the systemd service file points at a wrong location for the config and that the config is also located at the wrong location (we don't need the prefix there):

cat install_mumble-server/usr/lib/systemd/system/mumble-server.service
[Unit]
Description=Mumble server
After=network.target
Wants=network-online.target

[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
ExecStart=/usr/bin/mumble-server -ini /usr/etc/mumble/mumble-server.ini -fg
Group=_mumble-server
LockPersonality=yes
MemoryDenyWriteExecute=yes
NoNewPrivileges=yes
PrivateDevices=true
PrivateTmp=true
ProtectClock=yes
ProtectControlGroups=yes
ProtectHome=true
ProtectHostname=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectSystem=full
RestrictAddressFamilies=~AF_PACKET AF_NETLINK
RestrictNamespaces=yes
RestrictSUIDSGID=yes
RestrictRealtime=yes
Restart=always
SystemCallArchitectures=native
SystemCallFilter=@system-service
Type=simple
User=_mumble-server

[Install]
WantedBy=multi-user.target

The sysusers.d file is also in the wrong location below /usr/etc/sysconfig.d/. It needs to go to /usr/lib/sysconfig.d/ (on Arch).
The correct location, distribution independent, can be gotten from pkg-config --variable=sysusersdir systemd.

It seems that the dbus file is still missing (on Arch it is currently located in /usr/share/dbus-1/system.d/murmur.conf). This and also the Murmur.ice file (but renamed) should be retained, as they are (at least according to the config file) available methods for doing RPC.
Do you know if grpc is still supported? The server does not seem to link against it anymore, but there's MurmurRPC.proto.

[Krzmbrzl]

Here we can also see that additional, unwanted files are being installed, but I guess that's something for another PR?

See my latest commit - that should have fixed it

Do you know if grpc is still supported?

No, it's not. It has been removed due to our implementation being broken and unmaintained

but the systemd service file points at a wrong location for the config and that the config is also located at the wrong location (we don't need the prefix there)

By config file, I suppose you are referring to mumble-server.ini?
The service file should point exactly to where the file is being installed to (/etc/mumble/mumble-server.ini). That seems to have worked correctly.

What is the rule as to when to use a prefix when writing files to /etc? To me it seemed to make sense to use a prefix, in case we ever wanted to install more than one file there (in which case we can use the prefix to nicely group those files together).

The sysusers.d file is also in the wrong location below /usr/etc/sysconfig.d/. It needs to go to /usr/lib/sysconfig.d/ (on Arch).
The correct location, distribution independent, can be gotten from pkg-config --variable=sysusersdir systemd.

I'll fix that.

It seems that the dbus file is still missing (on Arch it is currently located in /usr/share/dbus-1/system.d/murmur.conf). This and also the Murmur.ice file (but renamed) should be retained, as they are (at least according to the config file) available methods for doing RPC.

Ah yes indeed, I forgot about the DBus file. Is there also some pkgconf magic that can be used to fetch the correct DBus directory?
And where should the ice file be installed to? Also /etc? 🤔

[Krzmbrzl]

I'll merge this as-is now. If any of the install paths require further tuning, that can be done in a separate PR.