Add 'secret-opts' option to manifest

[tytan652]

Meant to replace #406

Some applications have security tokens passed by their build system. In these cases, this option allow their use without distributing them in the bundle.

This PR add the new option secret-opts and secret-env.
The first is meant to be used with build system like cmake and meson, the last is meant to be used with 'build-command' and 'post-install'.

The builder check for secret-opts with a $+"env var name" showing which host env var shall replace it. The builder will replace them on the fly.
If the env var doesn't exist or the $ is missing, the option will be ignored.

With this way, the bundle still contain the manifest and even show that this bundle have secrets.

It's mainly meant to be used with CI like github actions or gitlab CI/CD secret's.

This is the first time I manipulate C code with Glib function, so feedback are welcomed.
Same for the documentation, I feel I didn't done great job.

Here, I try this new option with a modified obs-studio manifest with setting -DCEF_ROOT_DIR as a secret option.

{
      "name": "obs",
      "buildsystem": "cmake-ninja",
      "builddir": true,
      "config-opts": [
        "-DCMAKE_BUILD_TYPE=Release",
        "-DENABLE_WAYLAND=ON",
        "-DBUILD_BROWSER=ON",
        "-DUNIX_STRUCTURE=ON",
        "-DUSE_XDG=ON",
        "-DDISABLE_ALSA=ON",
        "-DENABLE_PULSEAUDIO=ON",
        "-DWITH_RTMPS=ON"
      ],
      "secret-opts": [
      	"-DCEF_ROOT_DIR=$CEF"
      ],
      "sources": [
        {
          "type": "dir",
          "path": "../../"
        }
      ]
    }

And put this command before building:

export CEF=/app/cef

Since those secrets leak in the verbose output with the print of the used command, it now print a version of this command with unresolved arguments if there is secrets.
And so the verbose log now looks like that:

FB: Running: flatpak build --die-with-parent --env=FLATPAK_BUILDER_BUILDDIR=/run/build/obs --nofilesystem=host --filesystem=/home/tytan652/Programming/Flatpak/obs-studio/.flatpak-builder/build/obs-2 --bind-mount=/run/build/obs=/home/tytan652/Programming/Flatpak/obs-studio/.flatpak-builder/build/obs-2 --build-dir=/run/build/obs/_flatpak_build --bind-mount=/run/ccache=/home/tytan652/Programming/Flatpak/obs-studio/.flatpak-builder/ccache --env=SOURCE_DATE_EPOCH=1628833344 '--env=CFLAGS=-O2 -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection ' '--env=CXXFLAGS=-O2 -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection ' '--env=LDFLAGS=-L/app/lib -Wl,-z,relro,-z,now -Wl,--as-needed ' --env=CCACHE_DIR=/run/ccache/disabled --env=PATH=/app/bin:/usr/bin --env=LD_LIBRARY_PATH=/app/lib --env=PKG_CONFIG_PATH=/app/lib/pkgconfig:/app/share/pkgconfig:/usr/lib/pkgconfig:/usr/share/pkgconfig --env=FLATPAK_BUILDER_N_JOBS=4 /home/tytan652/Programming/Flatpak/obs-studio/.flatpak-builder/rofiles/rofiles-ET7d3q cmake '-DCMAKE_INSTALL_PREFIX:PATH='\''/app'\''' -G Ninja .. -DCMAKE_BUILD_TYPE=Release -DENABLE_WAYLAND=ON -DBUILD_BROWSER=ON -DUNIX_STRUCTURE=ON -DUSE_XDG=ON -DDISABLE_ALSA=ON -DENABLE_PULSEAUDIO=ON -DWITH_RTMPS=ON '-DCEF_ROOT_DIR=$host:CEF'

[tytan652]

You prefer what for showing unresolved argument (I use OBS Studio CEF as an example):

• '-DCEF_ROOT_DIR=$host:CEF'
• '-DCEF_ROOT_DIR=$CEF'
• '-DCEF_ROOT_DIR=$host_env:CEF'
• -DCEF_ROOT_DIR=UNRESOLVED
• -DCEF_ROOT_DIR=CONCEALED

Note: the quote are added by flatpak-builder

[TingPing]

One use case we need to think about is how to handle projects that use buildsystem: simple as they don't use config-opts at all but want a way of passing secrets.

[tytan652]

I added secret-env which enable secrets for 'build-commands' and 'post-install' as transferred env var. Which should enable secrets for simple buildsystem.

Example:

{
      "name": "obs",
      "buildsystem": "cmake-ninja",
      "builddir": true,
      "build-commands": [
        "echo $CEF"
      ],
      "secret-env": [
      	"CEF"
      ],
      "config-opts": [
        "-DCMAKE_BUILD_TYPE=Release",
        "-DENABLE_WAYLAND=ON",
        "-DBUILD_BROWSER=ON",
        "-DUNIX_STRUCTURE=ON",
        "-DUSE_XDG=ON",
        "-DDISABLE_ALSA=ON",
        "-DENABLE_PULSEAUDIO=ON",
        "-DWITH_RTMPS=ON"
      ],
      "secret-opts": [
        "-DCEF_ROOT_DIR=$CEF"
      ],
      "sources": [
        {
          "type": "dir",
          "path": "../../"
        }
      ]
    }

[TingPing]

After somebody does some hands on testing I think this is good.

[GeorgesStavracas]

I've tested this pull request with OBS Studio (which is where this use case steamed from) by modifying the manifest to this:

        {
            "name": "obs",
            "buildsystem": "cmake-ninja",
            "builddir": true,
            "secret-env": [
                "SECRET_TWITCH_CLIENTID",
                "SECRET_TWITCH_HASH"
            ],
            "config-opts": [
                "-DCMAKE_BUILD_TYPE=Release",
                "-DENABLE_WAYLAND=ON",
                "-DBUILD_BROWSER=ON",
                "-DCEF_ROOT_DIR=/app/cef",
                "-DUNIX_STRUCTURE=ON",
                "-DUSE_XDG=ON",
                "-DDISABLE_ALSA=ON",
                "-DENABLE_PULSEAUDIO=ON",
                "-DENABLE_PIPEWIRE=ON",
                "-DWITH_RTMPS=ON",
                "-DRELEASE_CANDIDATE=27.1.0-rc2"
            ],
            "secret-opts": [
                "-DTWITCH_CLIENTID=$SECRET_TWITCH_CLIENTID",
                "-DTWITCH_HASH=$SECRET_TWITCH_HASH"
            ],
            "post-install": [
                "install -d /app/plugins",
                "install -d /app/extensions/Plugins",
                "install -d /app/lib/blackmagic /app/lib/ndi /app/lib/v4l2sink",
                "ln -s /app/lib/ndi/obs-ndi.so /app/lib/obs-plugins/obs-ndi.so",
                "mkdir -p /app/share/obs/obs-plugins/obs-ndi",
                "ln -s /app/lib/ndi/locale /app/share/obs/obs-plugins/obs-ndi/locale",
                "ln -s /app/lib/v4l2sink/v4l2sink.so /app/lib/obs-plugins/obs-v4l2sink.so"
            ],
            "sources": [
                {
                    "type": "git",
                    "url": "https://github.com/obsproject/obs-studio.git",
                    "tag": "27.1.0-rc2",
                    "commit": "e2b7597fb0190c3b2de53201fcd246885a96e9a6"
                }
            ]
        }

To buid, I first set some environment variables in my host system:

$ export SECRET_TWITCH_CLIENTID=0xbadc0ffee
$ export SECRET_TWITCH_HASH=0xdeadbeef

(I purposefully set these env vars to something that would break the build later on, so that we can be sure it's being propagated properly)

Then I ran patches flatpak-builder:

$ flatpak-builder --user --ccache --force-clean --verbose __build com.obsproject.Studio.json 

The secret variables were not leaked by the build command:

flatpak build --die-with-parent --env=FLATPAK_BUILDER_BUILDDIR=/run/build/obs --nofilesystem=host --filesystem=<build directory>/.flatpak-builder/build/obs-2 --bind-mount=/run/build/obs=<build directory>/.flatpak-builder/build/obs-2 --build-dir=/run/build/obs/_flatpak_build --bind-mount=/run/ccache=<build directory>/.flatpak-builder/ccache --env=SOURCE_DATE_EPOCH=1631792851 '--env=CFLAGS=-O2 -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection ' '--env=CXXFLAGS=-O2 -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection ' '--env=LDFLAGS=-L/app/lib -Wl,-z,relro,-z,now -Wl,--as-needed ' --env=CCACHE_DIR=/run/ccache --env=PATH=/run/ccache/bin:/app/bin:/usr/bin --env=LD_LIBRARY_PATH=/app/lib --env=PKG_CONFIG_PATH=/app/lib/pkgconfig:/app/share/pkgconfig:/usr/lib/pkgconfig:/usr/share/pkgconfig --env=FLATPAK_BUILDER_N_JOBS=4 <build directory>/.flatpak-builder/rofiles/rofiles-NU02Pz cmake '-DCMAKE_INSTALL_PREFIX:PATH='\''/app'\''' -G Ninja .. -DCMAKE_BUILD_TYPE=Release -DENABLE_WAYLAND=ON -DBUILD_BROWSER=ON -DCEF_ROOT_DIR=/app/cef -DUNIX_STRUCTURE=ON -DUSE_XDG=ON -DDISABLE_ALSA=ON -DENABLE_PULSEAUDIO=ON -DENABLE_PIPEWIRE=ON -DWITH_RTMPS=ON -DRELEASE_CANDIDATE=27.1.0-rc2 '-DTWITCH_CLIENTID=$host:SECRET_TWITCH_CLIENTID' '-DTWITCH_HASH=$host:SECRET_TWITCH_HASH'

The build went fine, and as expected it failed with the values I set to the environment variables:

UI/ui-config.h:21:25: error: unable to find numeric literal operator ‘operator""xdeadbeef’
   21 | #define TWITCH_HASH     0x0xdeadbeef

So there you go, the evidence that this pull request is working 🙂