bee: make it able to start n instances

[attila-lendvai]

** ABANDONED see comment **

Motivation for this change

due to architectural reasons the way to scale up the storage that you provide in the Swarm is to run multiple Bee nodes (as opposed to increasing the storage capacity of a single Bee instance).

Status

it was ready for merging at one point, but i'm abandoning it. it's been months since i haven't played with this.

i have stopped working on it mostly because it was a struggle to work in the NixOS service system that was designed for much simpler use-cases than what's needed for bee.

i have pushed a WIP commit into this PR for prosperity. it contains useful, but untested/unbaked changes that i have made while i tried to introduce a new abstraction called swarm (bee nodes always join a swarm; e.g. testnet, mainnet, or a company-wide private network, etc).

my current pursuit is to add the same to Guix.

Notes

• i'm doing some magic in the nix files, reaching across them into each other, that may or may not be best practice. e.g. config.services.bee.settings.clef-signer-enable in bee-clef.nix, to default the instanceCount to that of the bee config.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/can-nix-module-support-multiple-instance/6119/10

[attila-lendvai]

FTR, i've converted this back into a draft because meanwhile a new bee release became imminent, and i'll also address this by using umask to avoid leaking secrets: #121293

[SuperSandro2000]

@aanderse can you take a look at the service?

[SuperSandro2000]

Suggested change

	{ lib, stdenv, buildGoModule, fetchFromGitHub, libobjc ? null, IOKit ? null }:
	{ lib, stdenv, buildGoModule, fetchFromGitHub, libobjc, IOKit }:

If this throws an error it is packaged incorrectly and does not work on Darwin. Did you inherit the packages correct?

[attila-lendvai]

@SuperSandro2000

i don't know what "inherit the packages correctly" means. this is what i use locally to invoke/test the build (this works):

nix-build --pure -E "with import <nixpkgs> {}; callPackage /[...]nixpkgs/pkgs/applications/networking/bee/bee.nix {}"

but when i use with nixos-rebuild, it doesn't work. this is my config (it's imported into configuration.nix):

{ config, pkgs, ... }:

let
  baseconfig = { allowUnfree = true; };
  masterBasePath = "/home/alendvai/workspace/nixos/nixpkgs";
  beeFromCheckout = true;
  unstableTarball = if beeFromCheckout then null else fetchTarball https://github.com/NixOS/nixpkgs/archive/nixos-unstable.tar.gz;
  latest =
    if beeFromCheckout then
      import masterBasePath { config = baseconfig; }
    else
      import unstableTarball { config = baseconfig; };
  beePortBase = 4500;
in
{
  imports =
    if beeFromCheckout then
      [
        "${masterBasePath}/nixos/modules/services/networking/bee/bee.nix"
        "${masterBasePath}/nixos/modules/services/networking/bee/bee-clef.nix"
      ]
    else
      [
        (unstableTarball + "/nixos/modules/services/networking/bee.nix")
        (unstableTarball + "/nixos/modules/services/networking/bee-clef.nix")
      ];

  # disable the ones coming from stable
  disabledModules = [
    "services/networking/bee.nix"
    "services/networking/bee-clef.nix"
  ];

  nixpkgs.overlays = [
    (self: super:
      if beeFromCheckout then {
        go-ethereum       = (latest.callPackage "${masterBasePath}/pkgs/applications/blockchains/go-ethereum.nix" { });
        bee               = (latest.callPackage "${masterBasePath}/pkgs/applications/networking/bee/bee.nix" { });
        bee-unstable      = (latest.callPackage "${masterBasePath}/pkgs/applications/networking/bee/bee.nix" { version = "unstable"; });
        bee-clef          = (latest.callPackage "${masterBasePath}/pkgs/applications/networking/bee/bee-clef.nix" { });
        bee-clef-unstable = (latest.callPackage "${masterBasePath}/pkgs/applications/networking/bee/bee-clef.nix" {  version = "unstable"; });
      }
      else {
        go-ethereum       = latest.go-ethereum;
        bee               = latest.bee;
        bee-unstable      = latest.bee-unstable;
        bee-clef          = latest.bee-clef;
      }
    )
  ];

  services.bee = {
    instanceCount = 2;
    #package = pkgs.bee;
    #package = pkgs.bee-unstable;
    p2pPortBase = beePortBase;
    daemonNiceLevel = 19;
    daemonIOSchedulingClass = "idle";

    settings = {
      full-node = true;
      debug-api-enable = true;
      global-pinning-enable = true;
      db-open-files-limit = 8000;
      db-block-cache-capacity = 134217728;
      db-write-buffer-size = 134217728;
    };
  };

  services.bee-clef = {
    daemonNiceLevel = 19;
    daemonIOSchedulingClass = "idle";
  };

[...]

the error:

# nixos-rebuild switch --show-trace
building Nix...
building the system configuration...
error: while evaluating the attribute 'activationScript' of the derivation 'nixos-system-serlap-21.05.906.cedcf2565c6' at /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/pkgs/stdenv/generic/make-derivation.nix:201:11:
while evaluating the attribute 'system.activationScripts.script' at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/activation-script.nix:80:9:
while evaluating 'textClosureMap' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/strings-with-deps.nix:75:35, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/activation-script.nix:101:18:
while evaluating 'id' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/trivial.nix:14:5, called from undefined position:
while evaluating the attribute 'text' at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/activation-script.nix:9:5:
while evaluating the attribute 'text' at undefined position:
while evaluating 'g' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:298:19, called from undefined position:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:140:72, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:301:20:
while evaluating the attribute 'value' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:525:9:
while evaluating the option `system.activationScripts.etc.text':
while evaluating the attribute 'mergedValue' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:557:5:
while evaluating the attribute 'values' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:551:9:
while evaluating the attribute 'values' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:650:7:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:537:28, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:537:17:
while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/etc/etc.nix':
while evaluating 'dischargeProperties' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:609:25, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:538:137:
while evaluating the attribute 'value' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:392:44:
while evaluating the attribute 'sources' of the derivation 'etc' at /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/pkgs/stdenv/generic/make-derivation.nix:201:11:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/etc/etc.nix:20:20, called from undefined position:
while evaluating the attribute 'source' at undefined position:
while evaluating 'g' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:298:19, called from undefined position:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:140:72, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:301:20:
while evaluating the attribute 'value' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:525:9:
while evaluating the option `environment.etc.dbus-1.source':
while evaluating the attribute 'mergedValue' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:557:5:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:559:17, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:559:12:
while evaluating 'check' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/types.nix:349:15, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:559:22:
while evaluating the attribute 'serviceDirectories' of the derivation 'dbus-1' at /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/pkgs/stdenv/generic/make-derivation.nix:201:11:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/types.nix:358:14, called from undefined position:
while evaluating the attribute 'value' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:570:27:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:559:17, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:559:12:
while evaluating 'check' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/types.nix:349:15, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:559:22:
while evaluating the attribute 'passAsFile' of the derivation 'system-path' at /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/pkgs/stdenv/generic/make-derivation.nix:201:11:
while evaluating the attribute 'passAsFile' at /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/pkgs/build-support/buildenv/default.nix:77:5:
while evaluating the attribute 'buildCommand' of the derivation 'nixos-manpages' at /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/pkgs/stdenv/generic/make-derivation.nix:201:11:
while evaluating the attribute 'buildCommand' of the derivation 'manual-olinkdb' at /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/pkgs/stdenv/generic/make-derivation.nix:201:11:
while evaluating the attribute 'buildCommand' of the derivation 'nixos-manual-combined' at /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/pkgs/stdenv/generic/make-derivation.nix:201:11:
while evaluating the attribute 'buildCommand' of the derivation 'generated-docbook' at /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/pkgs/stdenv/generic/make-derivation.nix:201:11:
while evaluating the attribute 'buildCommand' of the derivation 'options-docbook.xml' at /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/pkgs/stdenv/generic/make-derivation.nix:201:11:
while evaluating 'substFunction' at /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/nixos/lib/make-options-doc/default.nix:28:19, called from /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/nixos/lib/make-options-doc/default.nix:37:54:
while evaluating the attribute 'default' at /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/lib/options.nix:187:44:
while evaluating 'scrubOptionValue' at /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/lib/options.nix:206:22, called from /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/lib/options.nix:187:54:
while evaluating 'isDerivation' at /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/lib/attrsets.nix:327:18, called from /nix/store/bzp8hl2skgaj8qir935vprny6p0xiq0k-nixos-21.05.906.cedcf2565c6/nixos/lib/options.nix:207:8:
while evaluating the attribute 'default' at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/services/blockchain/ethereum/geth.nix:110:9:
while evaluating the attribute 'go-ethereum.geth' at undefined position:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/misc/documentation.nix:32:18, called from undefined position:
while evaluating 'callPackageWith' at /home/alendvai/workspace/nixos/nixpkgs/lib/customisation.nix:117:35, called from /etc/nixos/config-serlap.nix:42:30:
while evaluating 'makeOverridable' at /home/alendvai/workspace/nixos/nixpkgs/lib/customisation.nix:67:24, called from /home/alendvai/workspace/nixos/nixpkgs/lib/customisation.nix:121:8:
anonymous function at /home/alendvai/workspace/nixos/nixpkgs/pkgs/applications/blockchains/go-ethereum.nix:1:1 called without required argument 'libobjc', at /home/alendvai/workspace/nixos/nixpkgs/lib/customisation.nix:69:16

[SuperSandro2000]

You want to do something like

nixpkgs/pkgs/development/compilers/llvm/12/default.nix

Lines 123 to 126 in 687c03a

	inherit llvm_meta;
	inherit (darwin) libobjc bootstrap_cmds;
	inherit (darwin.apple_sdk.libs) xpc;
	inherit (darwin.apple_sdk.frameworks) Foundation Carbon Cocoa;

in the callPackage of the package.

[attila-lendvai]

@SuperSandro2000 do i get it right: to use the git HEAD version of go-ethereum on my linux install, i need to mess with darwin stuff? that's rather unfortunate.

isn't there a way to get darwin completely out of the picture when we are not on darwin?

[aanderse]

@aanderse can you take a look at the service?

@SuperSandro2000 I have looked a few times and I keep thinking we need someone who understands the software being packaged to give a meaningful review. Unfortunately, I think the time required to build that understanding is a bit too high for me to commit to as a casual contributor/reviewer to nixpkgs.

A few questions/comments I think worth mentioning:

• With this PR the module becomes so complex it probably would earn the rank of most difficult module to understand in all of NixOS.
• I wonder if/how other distros are effectively packaging this software when they don't have the power of a scripting language to generate OS configuration.
• What role, if any, might systemd template units provide in simplifying this module?
• If the module needs to be this complex does this module belong actually belong in NUR instead of nixpkgs?

[attila-lendvai]

some background info:

Swarm is going to go live on june 21st (i.e. it will switch to the Ethereum mainnet for incentives). v1.0.0 will be released on that day, and v1.0.0-rc2 is already tagged in the bee repo.

re other distros:

the main complexity comes from supporting multiple instances, but that's also the main benefit of the nix package. none of the other distros support running multiple instances. users are on their own if they want to run multiple bee nodes on one machine, which is the recommended way to increase the storage provided for the Swarm. they usually solve this by using Docker and virtualization.

let me know if i can clarify or simplify anything!

[SuperSandro2000]

The problem from a reviewer perspective is that we can't easily overlook things and guide you through many of the common pitfalls while creating packages and services. Not sure how to continue. I guess we can merge it but it could still contain common mistakes and not fully work under all scenarios.

[attila-lendvai]

FTR, i ran out of steam in the struggle. then i ended up trying Guix and found myself implementing this functionality there. it's available in a channel now: https://github.com/attila-lendvai/guix-crypto

it's unlikely that i'll get back to this PR.

[SuperSandro2000]

Might as well close it then.