nixos/gitlab-runner: support multiple services

[misuzu]

Motivation for this change

A more declarative module for gitlab-runner with support for multiple services.
Inspired by @arianvp's module.

Main changes:

1. Multiple runner services can be registered.
2. Automatic registration via registration token so same config can be used on many machines.
3. Runner services is registered/unregistered only when needed (idempotent).
4. Dead runner services are removed from config on start/reload.
5. DynamicUser is used so no need for static user and group.
6. Support for running nix in docker via host nix-daemon.

What is missing:

1. Global configuration could only be changed by editing /var/lib/gitlab-runner/.gitlab-runner/config.toml. Fixed.

Please comment if you think that something else is missing or would be great to have.

Example config with four runner services:

{ config, pkgs, lib, ...}:
with lib;
{
  services.gitlab-runner = {
    enable = true;
    services = {
      # runner for building via nix in docker
      nix = {
        registrationConfigFile = pkgs.writeText "gitlab-runner-nix-registration" ''
          CI_SERVER_URL=https://gitlab.com/
          REGISTRATION_TOKEN=<token1>
        '';
        dockerImage = "alpine";
        dockerVolumes = [
          "/nix/store:/nix/store:ro"
          "/nix/var/nix/db:/nix/var/nix/db:ro"
          "/nix/var/nix/daemon-socket:/nix/var/nix/daemon-socket:ro"
        ];
        dockerDisableCache = true;
        preBuildScript = pkgs.writeScript "setup-container" ''
          mkdir -p -m 0755 /nix/var/log/nix/drvs
          mkdir -p -m 0755 /nix/var/nix/gcroots
          mkdir -p -m 0755 /nix/var/nix/profiles
          mkdir -p -m 0755 /nix/var/nix/temproots
          mkdir -p -m 0755 /nix/var/nix/userpool
          mkdir -p -m 1777 /nix/var/nix/gcroots/per-user
          mkdir -p -m 1777 /nix/var/nix/profiles/per-user
          mkdir -p -m 0755 /nix/var/nix/profiles/per-user/root
          mkdir -p -m 0700 "$HOME/.nix-defexpr"

          . ${pkgs.nix}/etc/profile.d/nix.sh

          ${pkgs.nix}/bin/nix-env -i ${concatStringsSep " " (with pkgs; [ nix cacert git openssh ])}

          ${pkgs.nix}/bin/nix-channel --add https://nixos.org/channels/nixpkgs-unstable
          ${pkgs.nix}/bin/nix-channel --update nixpkgs
        '';
        environmentVariables = {
          ENV = "/etc/profile";
          USER = "root";
          NIX_REMOTE = "daemon";
          PATH = "/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin:/bin:/sbin:/usr/bin:/usr/sbin";
          NIX_SSL_CERT_FILE = "/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt";
        };
        tagList = [ "nix" ];
      };
      # runner for building docker images
      docker-images = {
        registrationConfigFile = pkgs.writeText "gitlab-runner-docker-images-registration" ''
          CI_SERVER_URL=https://gitlab.com/
          REGISTRATION_TOKEN=<token2>
        '';
        dockerImage = "docker:stable";
        dockerVolumes = [
          "/var/run/docker.sock:/var/run/docker.sock"
        ];
        tagList = [ "docker-images" ];
      };
      # runner for executing stuff on host system
      # make sure to add required packages (including git!)
      # to `environment.systemPackages`
      shell = {
        registrationConfigFile = pkgs.writeText "gitlab-runner-shell-registration" ''
          CI_SERVER_URL=https://gitlab.com/
          REGISTRATION_TOKEN=<token3>
        '';
        executor = "shell";
        tagList = [ "shell" ];
      };
      # runner for everything else
      default = {
        registrationConfigFile = pkgs.writeText "gitlab-runner-default-registration" ''
          CI_SERVER_URL=https://gitlab.com/
          REGISTRATION_TOKEN=<token4>
        '';
        dockerImage = "debian:stable";
      };
    };
  };
  environment.systemPackages = with pkgs; [ git ];
}

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[rissson]

You probably want to document the method to use the host's nix-daemon in the runner somewhere

[misuzu]

You probably want to document the method to use the host's nix-daemon in the runner somewhere

Maybe adding a module for this would be even better?

[rissson]

Maybe adding a module for this would be even better?

You could add an option for this, with the proper warning that it exposes the host's store to anything running on that runner

EDIT: actually, as it is, it might cause problems for people with a custom store path

[misuzu]

I've added examples for services option, built manual with nix-build nixos/release.nix -A manualHTML.x86_64-linux and checked that there is no display issues.

[flokli]

@arianvp, could you review this?

[misuzu]

Global check_interval and concurrent is now configurable.

[bachp]

@misuzu Can't we combine this with the existing gitlab-runner module.

Both modules have some features the other one is missing. Here is a short list of what I think those are.

gitlab-runner-multi

• multiple services
• declartive
• automatic registration 🍾

gitlab-runner

• runner executable can be selected via services.gitlab-runner.package attribute
• graceful termination where it can configured how long gitlab-runner should wait till jobs get killed if the runner service is stoped (e.g machine shutdown/reboot).
• allows changing the working directory
• better module name

My proposal would be:

1. Replace the current module with this one but keeping all the currently supported options.
2. Add the .services attribute as proposed here to gitlab-runner
3. Allow .configFile to override the config. I think that would cover most usecases of the current module.
4. Deprecate .configOptions
5. Maybe rename packages to extraPackages as proposed here.

What do you think?

I think it would be better to have only one gitlab-runner module in nixpkgs.

[misuzu]

Replace the current module with this one but keeping all the currently supported options.

workDir option will not work correctly with DynamicUser. Should DynamicUser be dropped?

Allow .configFile to override the config. I think that would cover most usecases of the current module.

I'll look into this.

I think it would be better to have only one gitlab-runner module in nixpkgs.

The idea was to drop current one since it's broken anyway, but suggested changes sounds reasonable.

[flokli]

I'm also much in favour of improving/merging the existing module.

@misuzu workDir would still work in combination with DynamicUser=true, as long as it's inside /var/lib/gitlab-runner (which is the default).

I'd prefer to restrict the runner to this location over introducing all sort of system-wide requirements.
We could add an assertion asking the user to move things before switching if workDir doesn't currently point into a subdirectory of /var/lib/gitlab-runner.

[bachp]

As far as I remember workDir is only useful in combination with the shell runner, which I have no experience with. So from my point of view we can also drop the workDir option if nobody has a usecase for it.

[misuzu]

@misuzu workDir would still work in combination with DynamicUser=true, as long as it's inside /var/lib/gitlab-runner (which is the default).

We could add an assertion asking the user to move things before switching if workDir doesn't currently point into a subdirectory of /var/lib/gitlab-runner.

What's the difference if workDir in /var/lib/gitlab-runner or /var/lib/gitlab-runner/some/other/path? Don't really see the point of this option in this case.

As far as I remember workDir is only useful in combination with the shell runner, which I have no experience with. So from my point of view we can also drop the workDir option if nobody has a usecase for it.

I agree.

[flokli]

Yeah, okay, let's drop it, and in the mkRemovedOption module mention state (if any) would need to be migrated to /var/lib/gitlab-runner.

[bachp]

With my proposed change this modules works as a drop in replacement for the use cases I have tested.

All docker based runners tough.

[bachp]

Thanks a lot for the improvements to the module. All good from my side now!

FYI @max-wittig

[misuzu]

Added registrationFlags option for advanced use-cases.

[misuzu]

Added ability to change user for service using systemd.services.gitlab-runner.serviceConfig options. Configuration script will still be executed as root.
Tested changing User to root, unprivileged user and rolling back to DynamicUser, no manual chown's was needed.
Not sure if there should be dedicated options for this.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/call-for-testing-gitlab-runner-module/6765/1

[arianvp]

This looks great! what is left? I think this is ready to merge no?

[misuzu]

This looks great! what is left? I think this is ready to merge no?

I think it's feature-complete and people are saying it's working fine so yes, i think it is.
New options should be easy to add if necessary.

[flokli]

I'm still not a fan of running the configuration script as root, but that can be improved on in a later PR.

[flokli]

Thanks everyone!

[FRidh]

I reverted this when merging master into staging-next to avoid the merge conflict. Please rebase it unto staging-next and submit again.

[misuzu]

I reverted this when merging master into staging-next to avoid the merge conflict. Please rebase it unto staging-next and submit again.

Done! #86561