singularity-tools: refactor and add definition-based image buider #224636
Conversation
This builder takes a list `path` and reduces the `writeMultipleReferencesToFile` result of each element to a file. It would be helpful when building a container image from a list of packages.
|
Questions:
|
It's the contents of the
Not quite, it's executed inside the container via a chroot, but has the host store available. So it allows arbitrary setup of things outside of the nix store prior to instantiating the container specific store. This is analagous to dockerTools.
Not sure how much isolation affects the process of building an image, more of a runtime thing.
One thing to consider is that many places have namespaces disabled so lighter alternatives are not possible. The squashfs approach of @posch is great and wasn't possible when I wrote the initial implementation, but runAsRoot isn't possible with this. This probably doesn't matter for a majority of use cases. |
What if we have user namespaces? posch's script is great, but it's a bit hacky as the neither Apptainer nor SingularityCE have published the specification about the image format. (Hand-made sandbox build also has similar problem.) |
(I haven't processed all of the previous notifications, but) I just wanted to note that this isn't too often the case for HPC clusters (Aalto's "Triton" only offers setuid)... |
Split the build steps used by buildImage into * buildSandboxFromContents * buildImageFromSandbox * buildImageInLinuxVM
ShamrockLee
force-pushed
the
singularity-tools
branch
from
5a88f78
to
53a1065
Compare
|
Just noticed that the layout of an Apptainer/Singularity "sandbox" is very similar to the extracted content of a SIF image. If we could figure out a way to create a sandbox, maybe we could just pack it into a squashfs image. On the other hand, I'm facing a strange |
| "${coreutils}/bin/ln" -s "${coreutils}/bin/env" "''${SINGULARITY_ROOTFS}/usr/bin/env" | ||
| ''; | ||
| environment = { | ||
| PATH = "${lib.makeBinPath contents}:\${PATH:-}"; |
There was a problem hiding this comment.
I suppose this is the alternative to the previous approach of generating a FHS-like tree (/bin and such). We can address this in a separate PR, but I wanted to explore (ab)using ext3 and populate both /nix/store and /bin using hardlinks. This way I could hide the container's /nix with --bind /tmp:/nix and still having working sh and nix (assuming I've built them statically linked)
There was a problem hiding this comment.
I suppose this is the alternative to the previous approach of generating a FHS-like tree (
/binand such).
Yes. The PATH-based approach seems cleaner for me, which is also how nix shell works. As both of them has its use cases (Nixy idiom vs. FHS PATH compatibility), we could allow users to decide which to use.
We can address this in a separate PR, but I wanted to explore (ab)using ext3 and populate both
/nix/storeand/binusing hardlinks. This way I could hide the container's/nixwith--bind /tmp:/nixand still having workingshandnix(assuming I've built them statically linked)
HPC might be based upon a distributed filesystems; great chance that they don't support hard links. Former versions of Singularity simply extracts the SquashFS image on the host. If the machine have some sort of TMPDIR where local filesystems like ext4 or tmpfs are mounted, you could have it extract there and (hopefully) enjoy the hard links. If you don't have access hard-link supported spaces, there might be problems extracting an image like that.
A good news is that both Apptainer and SingularityCE now tries to mount the image with the squashfuse command from the environment before falling back to extracting, and that SquashFS does support hard links on its own. I added squashfuse to the defaultPath and wrapper PATH of both apptainer and singularity along with the version bump #224683. Let's make it land on the stable release of Nixpkgs.
There was a problem hiding this comment.
great chance that they don't support hard links
I guess I don't really care if they're extracted as copies on the host (in fact, only one would be extracted, and the other would get hidden), as long as they are represented as hardlinks inside the image that we create.
I added squashfuse to the defaultPath and wrapper PATH of both apptainer and singularity along with the version bump
Awesome! I should have a look tomorrow!
Let's make it land on the stable release of Nixpkgs
Sounds great! But just in case it takes longer, we can and should backport later
Today I've tried to use "apptainer build" instead of "mksquashfs" to create the container from the sandbox-like tree. It worked with only a small change to apptainer (see apptainer/apptainer#1312 ). Maybe that's a little less hacky. |
Great! That would be more likely to be megred by the upstrem than mine (apptainer/apptainer#1284). |
|
During the work on #224828, I found that As most of the distributions (I have heard of) uses an interactive shell as its |
wegank
added
2.status: stale
stale
bot
removed
the
2.status: stale
Description of changes
Status: work in progress
The original
buildImageimplementation is commented out to make discussion easier.Also update trivial-builders.nix:
writeMultipleReferencesToFile: init (#178717)
writeScriptBin, writeShellScriptBin: add meta.mainProgram automatically
Things done
sandbox = trueset innix.conf? (See Nix manual)nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)