Beat singularity-tools up to shape

[SomeoneSerge]

Issue description

I intend to start using nixpkgs' singularity-tools for hpc applications.
What follows is a list of hindrances and minor annoyances that I've immediately encountered.
The list is mostly for myself: I'm opening the issue to make this visible and maybe motivate people to voice ideas and comments.
Cf. this read on singularity with Nix for more inspiration

• VM-free image builds: Beat singularity-tools up to shape #177908 (comment)

• Singularity needs patching to make images reproducible: singularity-tools.buildImage: non-deterministic #279250

  • mkfs generates random UUIDs
  • SIF metadata (information about the builder, timestamps)

• Give users control over contents, in particular allow to remove bash: currently including bash manually results in singularity-tools.buildImage throwing obscure errors

• Annoyance: we can compute diskSize from the built contents instead of choosing an arbitrary constant

• Hindrance: failing to pack any cuda-enabled dependencies. The error says: ... Cannot allocate memory. My /tmp is on disk, and I don't seem to be running out of RAM, so this message might be just another version of "not enough space left on (squashfs) device"

• Hindrance: buildImage interface doesn't expose

  • Environment variables (or any other metadata, like apphelp)
  • Services ("container instances")

• ...

• Get this merged: singularity: fix defaultPath and reflect upstream changes #158486

CC (possibly interested) @ShamrockLee @jbedo

[ShamrockLee]

I would probably have more spare time after June 21.

To speed up the merging of #158486, I would prefer limiting the scope of this PR to:

1. Support multiple sources and non-vendored source build while
2. Not to break the previous functionality of singularity, singularity.nix and singularity-tools.

Further improvements can be done in successive PRs.

[jbedo]

I agree with limiting the scope of the PR, I'll have time to help in a couple of weeks.

[ShamrockLee]

[ ] Annoyance: we can compute diskSize from the built contents instead of choosing an arbitrary constant

Is there a way to compute diskSize from contents at eval time with no IFD?

[ShamrockLee]

Regarding the singularity-tools, a significant problem is the closure size being doubled unnecessarily by mkLayer.

singularity-tools.mkLayer generates all a new derivation by copying all the files and directories of each package into "$out", and then we use writeReferencesToFile to get the list of derivations in the dependency tree of the generated layer package.

Why don't we get a list of references of all the packages directly?

Here's my implementation which merges the writeReferencesToFiles result of all the packages in the list while removing the duplication. There should be a better implementation than the O(n^2) duplication removal, but it's much faster than the O(n) content copying of mkLayer anyway.

{
  writeMultipleReferencesToFile = paths: runCommand "runtime-deps-multiple" {
    referencesFiles = map writeReferencesToFile paths;
  } ''
    touch "$out"
    declare -a paths=();
    for refFile in $referencesFiles; do
      while read path; do
        isPathIncluded=0
        for pathIncluded in "''${paths[@]}"; do
          if [[ "$path" == "$pathIncluded" ]]; then
            isPathIncluded=1
            break
          fi
        done
        if (( ! isPathIncluded )); then
          echo "$path" >> "$out"
          paths+=( "$path" )
        fi
      done < "$refFile"
    done
  '';
}

[SomeoneSerge]

Is there a way to compute diskSize from contents at eval time with no IFD?

I cannot say what's a good way to compute it, but the trivial baseline is a derivation that takes, in buildInputs, a buildEnv with to-be image's contents, and dus it. The output times some constant is an upperbound on the diskSize

EDIT: i.e. we wouldn't know diskSize at nix eval time, but we'd know it at build time, which appears to be sufficient

[ShamrockLee]

EDIT: i.e. we wouldn't know diskSize at nix eval time, but we'd know it at build time, which appears to be sufficient

Then we would no longer be able to use

vmTools.runInLinuxVM (
  runCommand {
    preVM = vmTools.createEmptyImage {
      size = diskSize;
      fullName = "${projectName}-run-disk";
    };
  } ''
    mkfs -t ext3 -b 4096 /dev/${vmTools.hd}
    mount "/dev/${vm.hd}" disk
  ''
)

[SomeoneSerge]

I see now. It appears that createEmptyImage never uses size at eval time, so we could rewrite it to relax the constraint

nixpkgs/pkgs/build-support/vm/default.nix

Line 280 in 4b31cc7

${qemu}/bin/qemu-img create -f qcow2 $diskImage "${toString size}M"

[ShamrockLee]

I see now. It appears that createEmptyImage never uses size at eval time, so we could rewrite it to relax the constraint

nixpkgs/pkgs/build-support/vm/default.nix

Line 280 in 4b31cc7

${qemu}/bin/qemu-img create -f qcow2 $diskImage "${toString size}M"

Great! dockerTools would also benefits from that.

[dmadisetti]

For consistency with dockerTools.buildImage it would also be nice to change contents -> copyToRoot.

@SomeoneSerge any other HPC pain points?

[ShamrockLee]

I don't mind changing the interface of singularity-tools. (That would be a breaking change.) Sorry for not noticing the change of dockerTool.buildImage.

There's another change lineing up that builds the image through a Singularity definition (Apptainer recipe) file to make the image more declarative and the build process explainable. It could be a drop-in replacement of the current Singularity-sandbox-based implementation.

I also went on and made a generator function that turns a settings-like Nix attrset into a definition string. The parser, which does the reverse) is still work in progress.