How do I build a custom bitcoin core branch?

[Sjors]

I'd like to be able to deploy non-release branches.

Something like services.bitcoind.gitURL, services.bitcoind.gitBranch and (for safety) services.bitcoind.commit or a sha256 hash of the source, would be handy.

[erikarvstedt]

It's not guaranteed that a bitcoind release derivation always works with a different source tree, so I don't think it makes sense to add a dedicated option for this.

But Nix is flexible enough support this use case without much boilerplate:

services.bitcoind.package = config.nix-bitcoin.pkgs.bitcoind.overrideAttrs (old: {
  version = "26-dev"; # Add version to the derivation name (optional)
  src = builtins.fetchTarball {
    url = "https://github.com/bitcoin/bitcoin/archive/<REV>.tar.gz";
    sha256 = "";
  };
});

Because bitcoind 26 can't be built with the bitcoind 25 derivation included in nix-bitcoin, we have to pull in the bitcoind 26 drv from a recent nixpkgs-unstable release:

services.bitcoind.package = let
  # nixpkgs-unstable as of 2023-12-27
  # This is needed because nix-bitcoin's nixpkgs-unstable only includes bitcoind 25 which
  # is incompatible with bitcoind 26.
  pkgsUnstable = import (builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/5f64a12a728902226210bf01d25ec6cbb9d9265b.tar.gz";
    sha256 = "0lq73nhcg11485ppazbnpz767qjifbydgqxn5xhj3xxlbfml39ba";
  }) {};
in pkgsUnstable.bitcoind.overrideAttrs (old: {
  version = "26-dev"; # Add version to the derivation name (optional)
  src = builtins.fetchTarball {
    url = "https://github.com/bitcoin/bitcoin/archive/4b1196a9855dcd188a24f393aa2fa21e2d61f061.tar.gz";
    # If you change the above `url`, add an empty hash ("") to auto-fetch the hash during evaluation
    sha256 = "18n5apgz4083r34hjwikgq8arycp8ysg1k2y7693b3b9b7qy8md4";
  };
});

[erikarvstedt]

It's also possible to use a local build (outside of the nix store) for the service.
Let me know if you need help with this.

[Sjors]

@erikarvstedt thanks! I'll give that a try.

[Sjors]

It seems to be building... Is there a way to pass -j$(nproc) to make? And to override the default configure settings, e.g. to add --disable-wallet --disable-tests --disable-fuzz-binary --without-upnp --disable-external-signer? (it's nice that it automatically runs the tests though)

(update: looks like it already uses -j$(nproc))

[Sjors]

Mmm, it gets pretty far but then fails:

make[1]: Entering directory '/build/source'
make[2]: Entering directory '/build/source'
make[2]: Nothing to be done for 'install-exec-am'.
 /nix/store/zx8aqgdy735qzk81glfwil6mbi6ddqb1-coreutils-9.4/bin/mkdir -p '/nix/store/q5cc48kxikdnw257sv1nsgjh91g6rawm-bitcoind-Master-branch-+-some-PRs/lib/pkgconfig'
 /nix/store/zx8aqgdy735qzk81glfwil6mbi6ddqb1-coreutils-9.4/bin/install -c -m 644 libbitcoinconsensus.pc '/nix/store/q5cc48kxikdnw257sv1nsgjh91g6rawm-bitcoind-Master-branch-+-some-PRs/lib/pkgconfig'
make[2]: Leaving directory '/build/source'
make[1]: Leaving directory '/build/source'
install: cannot stat 'contrib/completions/bash/bitcoin-cli.bash-completion': No such file or directory
/nix/store/d4jf1cbbk494zwgbqz31pxgigpsbh6w2-stdenv-linux/setup: line 131: pop_var_context: head of shell_variables not a function context
error: builder for '/nix/store/kq19nxfmmlh77c5fdm1rc4102n243a1s-bitcoind-Master-branch-+-some-PRs.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/vgrj6x5flk0mv54w5nqk8f293rmlxmy9-system-path.drv' failed to build
error: 1 dependencies of derivation '/nix/store/biqr6zxwaw2mi1zq0h7npzkzxkk2lv03-unit-bitcoind.service.drv' failed to build
error: 1 dependencies of derivation '/nix/store/gydpa569yihjqjh42l1q91p5v16m22gp-nixos-system-nixos-23.11.2130.d65bceaee0fb.drv' failed to build

My branch starts from the same 4b1196a9855dcd188a24f393aa2fa21e2d61f061 master commit as yours and doesn't touch bitcoin-cli or bash completion related stuff.

[Sjors]

I'm also trying to add ccache to speed up future builds, but with the instructions here ./configure is not picking it up. I'm assuming that for the package name I need to use "bitcoind"?

Here's the code that looks for it: https://github.com/bitcoin/bitcoin/blob/4b1196a9855dcd188a24f393aa2fa21e2d61f061/configure.ac#L1687-L1705

And above that there's a AC_PATH_PROG([CCACHE], [ccache])

[erikarvstedt]

Ah, sorry, nixpkgs-unstable is currently a few days behind master, so it doesn't yet include bitcoin 26.0.
Here's the fixed config (tested):

services.bitcoind.package = config.nix-bitcoin.pkgs.bitcoind.overrideAttrs (old: {
  version = "26-dev";
  src = builtins.fetchTarball {
    url = "https://github.com/bitcoin/bitcoin/archive/4b1196a9855dcd188a24f393aa2fa21e2d61f061.tar.gz";
    sha256 = "18n5apgz4083r34hjwikgq8arycp8ysg1k2y7693b3b9b7qy8md4";
  };
  postInstall = ''
    installShellCompletion --bash contrib/completions/bash/bitcoin-cli.bash
    installShellCompletion --bash contrib/completions/bash/bitcoind.bash
    installShellCompletion --bash contrib/completions/bash/bitcoin-tx.bash
  '';
});

CCache

Here's how to use ccache for building a nix-bitcoin system:

1. Add this to the config of the NixOS system that you're using to build the nix-bitcoin system:

   programs.ccache.enable = true;
   nix.settings.extra-sandbox-paths = [ config.programs.ccache.cacheDir ];

   If you're not using a NixOS system for building, see Non-NixOS in https://nixos.wiki/wiki/CCache.

2. Add this to your nix-bitcoin system config:

   programs.ccache.packageNames = [ "bitcoind" ];
   
   # Same as above
   services.bitcoind.package = config.nix-bitcoin.pkgs.bitcoind.overrideAttrs (old: {
     version = "26-dev";
     src = builtins.fetchTarball {
       url = "https://github.com/bitcoin/bitcoin/archive/4b1196a9855dcd188a24f393aa2fa21e2d61f061.tar.gz";
       sha256 = "18n5apgz4083r34hjwikgq8arycp8ysg1k2y7693b3b9b7qy8md4";
     };
     postInstall = ''
       installShellCompletion --bash contrib/completions/bash/bitcoin-cli.bash
       installShellCompletion --bash contrib/completions/bash/bitcoind.bash
       installShellCompletion --bash contrib/completions/bash/bitcoin-tx.bash
     '';
   });

   The above assumes you're using Flakes with useVersionLockedPkgs = false (the default).
   If this is not the case, use this config instead:

   nixpkgs.overlays = [
     (self: super: {
       ccacheWrapper = super.ccacheWrapper.override {
         extraConfig = ''
           export CCACHE_COMPRESS=1
           export CCACHE_DIR="${config.programs.ccache.cacheDir}"
           export CCACHE_UMASK=007
           if [ ! -d "$CCACHE_DIR" ]; then
             echo "====="
             echo "Directory '$CCACHE_DIR' does not exist"
             echo "Please create it with:"
             echo "  sudo mkdir -m0770 '$CCACHE_DIR'"
             echo "  sudo chown root:nixbld '$CCACHE_DIR'"
             echo "====="
             exit 1
           fi
           if [ ! -w "$CCACHE_DIR" ]; then
             echo "====="
             echo "Directory '$CCACHE_DIR' is not accessible for user $(whoami)"
             echo "Please verify its access permissions"
             echo "====="
             exit 1
           fi
         '';
       };
     })
   ];
   services.bitcoind.package = (config.nix-bitcoin.pkgs.bitcoind.override {
     stdenv = pkgs.ccacheStdenv;
   }).overrideAttrs (old: {
     version = "26-dev";
     src = builtins.fetchTarball {
       url = "https://github.com/bitcoin/bitcoin/archive/4b1196a9855dcd188a24f393aa2fa21e2d61f061.tar.gz";
       sha256 = "18n5apgz4083r34hjwikgq8arycp8ysg1k2y7693b3b9b7qy8md4";
     };
     postInstall = ''
       installShellCompletion --bash contrib/completions/bash/bitcoin-cli.bash
       installShellCompletion --bash contrib/completions/bash/bitcoind.bash
       installShellCompletion --bash contrib/completions/bash/bitcoin-tx.bash
     '';
   });