Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos/gitlab-runner: support multiple services #86561

Merged
merged 1 commit into from May 2, 2020

Conversation

@misuzu
Copy link
Contributor

misuzu commented May 2, 2020

Motivation for this change

This is a rebase of #84139

A more declarative module for gitlab-runner with support for multiple services.
Inspired by @arianvp's module.

Main changes:

  1. Multiple runner services can be registered.
  2. Automatic registration via registration token so same config can be used on many machines.
  3. Runner services is registered/unregistered only when needed (idempotent).
  4. Dead runner services are removed from config on start/reload.
  5. DynamicUser is used so no need for static user and group.
  6. Support for running nix in docker via host nix-daemon.

What is missing:

  1. Global configuration could only be changed by editing /var/lib/gitlab-runner/.gitlab-runner/config.toml. Fixed.

Please comment if you think that something else is missing or would be great to have.

Example config with four runner services:

{ config, pkgs, lib, ...}:
with lib;
{
  services.gitlab-runner = {
    enable = true;
    services = {
      # runner for building via nix in docker
      nix = {
        registrationConfigFile = pkgs.writeText "gitlab-runner-nix-registration" ''
          CI_SERVER_URL=https://gitlab.com/
          REGISTRATION_TOKEN=<token1>
        '';
        dockerImage = "alpine";
        dockerVolumes = [
          "/nix/store:/nix/store:ro"
          "/nix/var/nix/db:/nix/var/nix/db:ro"
          "/nix/var/nix/daemon-socket:/nix/var/nix/daemon-socket:ro"
        ];
        dockerDisableCache = true;
        preBuildScript = pkgs.writeScript "setup-container" ''
          mkdir -p -m 0755 /nix/var/log/nix/drvs
          mkdir -p -m 0755 /nix/var/nix/gcroots
          mkdir -p -m 0755 /nix/var/nix/profiles
          mkdir -p -m 0755 /nix/var/nix/temproots
          mkdir -p -m 0755 /nix/var/nix/userpool
          mkdir -p -m 1777 /nix/var/nix/gcroots/per-user
          mkdir -p -m 1777 /nix/var/nix/profiles/per-user
          mkdir -p -m 0755 /nix/var/nix/profiles/per-user/root
          mkdir -p -m 0700 "$HOME/.nix-defexpr"

          . ${pkgs.nix}/etc/profile.d/nix.sh

          ${pkgs.nix}/bin/nix-env -i ${concatStringsSep " " (with pkgs; [ nix cacert git openssh ])}

          ${pkgs.nix}/bin/nix-channel --add https://nixos.org/channels/nixpkgs-unstable
          ${pkgs.nix}/bin/nix-channel --update nixpkgs
        '';
        environmentVariables = {
          ENV = "/etc/profile";
          USER = "root";
          NIX_REMOTE = "daemon";
          PATH = "/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin:/bin:/sbin:/usr/bin:/usr/sbin";
          NIX_SSL_CERT_FILE = "/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt";
        };
        tagList = [ "nix" ];
      };
      # runner for building docker images
      docker-images = {
        registrationConfigFile = pkgs.writeText "gitlab-runner-docker-images-registration" ''
          CI_SERVER_URL=https://gitlab.com/
          REGISTRATION_TOKEN=<token2>
        '';
        dockerImage = "docker:stable";
        dockerVolumes = [
          "/var/run/docker.sock:/var/run/docker.sock"
        ];
        tagList = [ "docker-images" ];
      };
      # runner for executing stuff on host system
      # make sure to add required packages (including git!)
      # to `environment.systemPackages`
      shell = {
        registrationConfigFile = pkgs.writeText "gitlab-runner-shell-registration" ''
          CI_SERVER_URL=https://gitlab.com/
          REGISTRATION_TOKEN=<token3>
        '';
        executor = "shell";
        tagList = [ "shell" ];
      };
      # runner for everything else
      default = {
        registrationConfigFile = pkgs.writeText "gitlab-runner-default-registration" ''
          CI_SERVER_URL=https://gitlab.com/
          REGISTRATION_TOKEN=<token4>
        '';
        dockerImage = "debian:stable";
      };
    };
  };
  environment.systemPackages = with pkgs; [ git ];
}
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
@misuzu misuzu requested a review from flokli May 2, 2020
@misuzu misuzu mentioned this pull request May 2, 2020
3 of 10 tasks complete
@FRidh FRidh merged commit 0b0afca into NixOS:staging-next May 2, 2020
14 checks passed
14 checks passed
Evaluation Performance Report Evaluator Performance Report
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="8ccc3b2"; rev="8ccc3b2fd3fa1f208f0c9dcdf57d5c3b2720f354"; } ./pkgs/t
Details
grahamcofborg-eval-lib-tests nix-build --arg pkgs import ./. {} ./lib/tests/release.nix
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="8ccc3b2"; rev="8ccc3b2fd3fa1f208f0c9dcdf57d5c3b2720f354"; } ./nixos/
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="8ccc3b2"; rev="8ccc3b2fd3fa1f208f0c9dcdf57d5c3b2720f354"; } ./nixos/
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="8ccc3b2"; rev="8ccc3b2fd3fa1f208f0c9dcdf57d5c3b2720f354"; } ./nixos/
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="8ccc3b2"; rev="8ccc3b2fd3fa1f208f0c9dcdf57d5c3b2720f354"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="8ccc3b2"; rev="8ccc3b2fd3fa1f208f0c9dcdf57d5c3b2720f354"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="8ccc3b2"; rev="8ccc3b2fd3fa1f208f0c9dcdf57d5c3b2720f354"; } ./pkgs/t
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
@max-wittig
Copy link
Contributor

max-wittig commented May 5, 2020

@FRidh @misuzu Is there an option to define additional options? E.g. some things are missing:

  • S3 cache
  • Pull policy
  • Session server
  • Sentry DSN
@misuzu
Copy link
Contributor Author

misuzu commented May 5, 2020

@FRidh @misuzu Is there an option to define additional options? E.g. some things are missing:

  • S3 cache
  • Pull policy
  • Session server
  • Sentry DSN

Yes, there is registrationFlags.
Session server and Sentry DSN should be easy to add to module, in the meantime you can add them manually to /var/lib/gitlab-runner/.gitlab-runner/config.toml.

@max-wittig
Copy link
Contributor

max-wittig commented May 5, 2020

@misuzu Thanks for the fast response. I can't really add anything manually, as this is an automated setup. Will the remaining complex options like s3 cache and session server also be added soon?

@misuzu
Copy link
Contributor Author

misuzu commented May 5, 2020

@misuzu Thanks for the fast response. I can't really add anything manually, as this is an automated setup. Will the remaining complex options like s3 cache and session server also be added soon?

You can configure cache using registrationFlags:

   --cache-type value                                           Select caching method [$CACHE_TYPE]
   --cache-path value                                           Name of the path to prepend to the cache URL [$CACHE_PATH]
   --cache-shared                                               Enable cache sharing between runners. [$CACHE_SHARED]
   --cache-s3-server-address value                              A host:port to the used S3-compatible server [$CACHE_S3_SERVER_ADDRESS]
   --cache-s3-access-key value                                  S3 Access Key [$CACHE_S3_ACCESS_KEY]
   --cache-s3-secret-key value                                  S3 Secret Key [$CACHE_S3_SECRET_KEY]
   --cache-s3-bucket-name value                                 Name of the bucket where cache will be stored [$CACHE_S3_BUCKET_NAME]
   --cache-s3-bucket-location value                             Name of S3 region [$CACHE_S3_BUCKET_LOCATION]
   --cache-s3-insecure                                          Use insecure mode (without https) [$CACHE_S3_INSECURE]

I can try adding other global options.

@misuzu misuzu mentioned this pull request May 5, 2020
4 of 10 tasks complete
@misuzu
Copy link
Contributor Author

misuzu commented May 5, 2020

@max-wittig please take a look at #86946
See https://discourse.nixos.org/t/call-for-testing-gitlab-runner-module/6765 for how to quickly test it.

@max-wittig
Copy link
Contributor

max-wittig commented May 12, 2020

@misuzu I've tested it now with this MR, but I'm just getting the old gitlab-runner (12.4.1). Is there any way to specify the latest runner. I've seen the package option but I'm not sure how to bump something there. Should already be at pkgs.gitlab-runner

@misuzu
Copy link
Contributor Author

misuzu commented May 12, 2020

@misuzu I've tested it now with this MR, but I'm just getting the old gitlab-runner (12.4.1). Is there any way to specify the latest runner. I've seen the package option but I'm not sure how to bump something there. Should already be at pkgs.gitlab-runner

12.4.1 is in 19.09. You can upgrade to 20.03 or unstable for newer version.

@max-wittig
Copy link
Contributor

max-wittig commented May 12, 2020

@misuzu I just noticed that, but ran into another problem. Every 2nd deployment or so, the config.toml stays empty:
image

gitlab-runner is happy with an empty config, but I'm not 😄

@misuzu
Copy link
Contributor Author

misuzu commented May 12, 2020

the config.toml stays empty

Please share your config.

@max-wittig
Copy link
Contributor

max-wittig commented May 12, 2020

@misuzu Thanks for all the help!

services.gitlab-runner = {
    enable = true;
    checkInterval = 15;
    package = latestPkgs.gitlab-runner;
    #concurrent = ${vars.concurrent};
    services = {
      docker-images = {
        registrationFlags = [
          "--cache-type"
          "s3"
          "--cache-shared"
          "--cache-s3-server-address"
          "s3.amazonaws.com"
          "--cache-s3-access-key"
          "${vars.s3_access_key}"
          "--cache-s3-secret-key"
          "${vars.s3_secret_key}"
          "--cache-s3-bucket-name"
          "some-aws-bucket"
          "--cache-s3-bucket-location"
          "eu-central-1"
          "--docker-pull-policy"
          "always"
        ];
        registrationConfigFile = pkgs.writeText "gitlab-runner-nix-registration" ''
          CI_SERVER_URL=https://gitlab-url/
          REGISTRATION_TOKEN=${vars.token}
        '';
        #dockerAllowedImages = "vars.docker_whitelist}";
        dockerImage = "some-registry.com:some-image:stable";
        dockerVolumes = [
          "/mnt/projects/sstate-cache:/home/projects/sstate-cache:rw"
        ];
        executor = "docker";
        maximumTimeout = 86400;
        dockerDisableCache = true;
        environmentVariables = {
          CONCURRENCY_LEVEL = vars.concurrent;
        };
        #tagList = ["${vars.tags}"];
      };
    };
  };
@misuzu
Copy link
Contributor Author

misuzu commented May 12, 2020

@max-wittig this config is working fine on my server. Please check the logs: journalctl -u gitlab-runner.service.

@max-wittig
Copy link
Contributor

max-wittig commented May 12, 2020

No errors in the journal. Looks good, but doesn't work.

image

@misuzu
Copy link
Contributor Author

misuzu commented May 12, 2020

No errors in the journal. Looks good, but doesn't work.

Please share a full log: start journalctl -f -u gitlab-runner.service and in different console run systemctl restart gitlab-runner.service

@max-wittig
Copy link
Contributor

max-wittig commented May 12, 2020

Doesn't change anything, as it's the registration part that writes this empty config file that doesn't work, not the runner start part.

@misuzu
Copy link
Contributor Author

misuzu commented May 12, 2020

Doesn't change anything, as it's the registration part that writes this empty config file that doesn't work, not the runner start part.

That's exactly what i want to see, there is no logs from gitlab-runner-configure in your screenshot.

@max-wittig
Copy link
Contributor

max-wittig commented May 12, 2020

Ah I guess I had some registration parameter wrong. Can we implement some sort of error for this? Like fail the nixos switch if this happens?

@misuzu
Copy link
Contributor Author

misuzu commented May 12, 2020

Ah I guess I had some registration parameter wrong. Can we implement some sort of error for this? Like fail the nixos switch if this happens?

Maybe. I have to understand what happened to do that.

@max-wittig
Copy link
Contributor

max-wittig commented May 12, 2020

I think the registration failed. I had this message: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2862

@max-wittig
Copy link
Contributor

max-wittig commented May 12, 2020

@misuzu Thanks again for the great work you did here and sorry for my stupid questions 😉

@max-wittig
Copy link
Contributor

max-wittig commented May 12, 2020

@misuzu The registration error from the runner actually looks like this:
image

@misuzu
Copy link
Contributor Author

misuzu commented May 12, 2020

@misuzu Thanks again for the great work you did here and sorry for my stupid questions

Don't worry. Thanks for testing!

@misuzu
Copy link
Contributor Author

misuzu commented May 12, 2020

@misuzu The registration error from the runner actually looks like this:
image

I have updated pull request, now this should fail on activation.

@max-wittig
Copy link
Contributor

max-wittig commented May 12, 2020

Ah nice! Thanks. One more thing. The name of the runner cannot be overriden currently, right? Could that be configurable, too? 😄

@misuzu
Copy link
Contributor Author

misuzu commented May 12, 2020

Ah nice! Thanks. One more thing. The name of the runner cannot be overriden currently, right? Could that be configurable, too?

You can use something else instead of docker-images for prefix but suffix is needed to check if runner must be re-registered when configuration is changed.

@max-wittig
Copy link
Contributor

max-wittig commented May 12, 2020

Ah I see. Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.