end-to-end test example

[lelgenio]

One thing I've found to be pretty useful is running E2E tests as a nix derivation. I'm thinking of cleaning up and contributing this as an example, but requesting some comments before.

Maybe this sort of thing is best left to a separate project?

{
  description = "Example E2E testing";
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/release-23.05";

    crane = {
      url = "github:ipetkov/crane";
      inputs = {
        nixpkgs.follows = "nixpkgs";
        rust-overlay.follows = "rust-overlay";
        flake-utils.follows = "flake-utils";
      };
    };

    rust-overlay = {
      url = "github:oxalica/rust-overlay";
      inputs = {
        nixpkgs.follows = "nixpkgs";
        flake-utils.follows = "flake-utils";
      };
    };

    flake-utils.url = "github:numtide/flake-utils";
  };
  outputs = { self, nixpkgs, crane, rust-overlay, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
        };
        inherit (pkgs) lib;
        craneLib = crane.lib.${system};
        src = craneLib.cleanCargoSource (craneLib.path ./.);

        buildInputs = with pkgs; [ pkg-config openssl ];

        commonArgs = {
          inherit src buildInputs;
          pname = "example-e2e";
          version = "0.1";
          doCheck = false;
        };

        cargoArtifacts = craneLib.buildDepsOnly commonArgs;

        server = craneLib.buildPackage (commonArgs // {
          inherit cargoArtifacts;
          cargoExtraArgs = "--package=server";
        });
        e2eTester = craneLib.buildPackage (commonArgs // {
          inherit cargoArtifacts;
          cargoExtraArgs = "--package=e2e_tests";
        });

        runE2ETests = pkgs.runCommand "e2e-tests" { } ''
          HOME="$(mktemp -d)"

          wait-online() {
            timeout 5s \
              ${pkgs.retry}/bin/retry --until=success --delay "1,1,1,1,1" -- \
                ${pkgs.curl}/bin/curl -s "$@"
          }

          ${pkgs.postgresql}/bin/initdb postgres-data
          ${pkgs.postgresql}/bin/pg_ctl --pgdata=postgres-data --options "-c unix_socket_directories=$PWD" start
          export DATABASE_URL="postgres:///postgres?host=$PWD"

          ${server}/bin/server &
          wait-online --fail localhost:8000/ping

          ${pkgs.geckodriver}/bin/geckodriver --binary ${pkgs.firefox}/bin/firefox &
          wait-online localhost:4444

          export SSL_CERT_FILE="${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt"
          ${e2eTester}/bin/e2e_tests

          touch $out
        '';
      in
      {
        checks = {
          inherit server e2eTester;
          # ...
        } // (lib.optionalAttrs pkgs.stdenv.isLinux {
          inherit runE2ETests;
        });
      });
}

[dpc]

While this is fine, in my opinion, it's much more practical to have tests work on ./target/${CARGO_PROFILE}/ files.

In a Rust project, even with crane you want to avoid going through Nix whenever possible during normal development.

A developer should only need to run a nix develop shell which gives them all the dependencies and from there not worry about Nix at all. cargo build, cargo test, etc. should work. If there are any integration/e2e tests scripts they should also work in the shell. I tend to write them as ./script/some-tests.sh + Justfile to easily list&start them.

Obviously in your CI you also want to run them, bu then instead of building individual packages, just build a whole workspace in one derivation, and use it in test mkCargoDerivation with cargoArtifacts importing already fully built ./target, like here: https://github.com/fedimint/fedimint/blob/75f0b8f7021a9879201a98691ec6650a562c64ea/nix/craneBuild.nix#L154

IMO, it's a vital piece of advice, and if you don't follow it your developers are going to hate you, waiting for tens of minutes "for Nix to build, just to run the test", dreaming about "something faster" like in this story: https://mmapped.blog/posts/17-scaling-rust-builds-with-bazel.html

[ipetkov]

@lelgenio I'd be happy to include a potential example which shows off how to apply such a technique to those who really do want or need to run such tests as a derivation (or via hydra)! So long as the example is clear/self contained and doesn't get too complex to follow, but we can always iterate

Also worth noting that you can also set up nixos tests which basically spin up different VMs (e.g. you don't need to run the database in the same script, it could be a separate machine. not sure how reusing that machine with different tests works though since I haven't used it myself)!

[dpc]

BTW. (sorry for a bit tangential topics, but I think it's powerful and not well know). In our project we run all e2e tests (which includes various combinations of > 10 instances of various daemons, etc.) in parallel , each in a different Linux network namespace (so they don't collide w.r.t port numbers etc.).

No container rebuilding, no VMs, no waste. I don't know of any MacOS corresponding way (it's kind of an OS for media consumption, not real dev work... 😆), but MacOS developers can still run each script individually, and running everything in parallel is mostly useful for CI anyway.

[ipetkov]

Closed with #368 🎉