Implement more granular build process

[tazjin]

This implements the concept outlined in #50 and is a followup to #52.

Build logic has been moved from the monolithic Nix expression into a multi-staged process in which layer creation is handled by the server.

For now the primary advantages of this are subtle:

• after creating a layer grouping we can avoid duplicate layer builds by checking for previously built layers with the same contents
• less time is spent in Nix (which is the primary concurrency bottleneck for Nixery)
• the build process becomes easier to parallelise

In the future this might enable additional optimisations (such as additional caching steps, for example by requesting runtime closures of each package separately and applying caching at that level).

---

Note: This PR is currently WIP. There are many tasks left to do:

• Move group-layers into the server component
• Refactor build-image.nix to remove functionality that is moving into the server
• Deserialise the new build output correctly (including runtime graph)
• Add logic to check for missing layers (& define caching strategy)
• Add Nix derivation to build layers (build only!)
• Implement hashing & uploading of layers in one disk read
• Implement assembly of config layer
• Implement assembly of manifest
• Reimplement local manifest caching
• Load popularity data in server

[tazjin]

To answer my own question:

side note: is it possible to feed both hashes simultaneously from one disk read?

Yes!

package main

import (
	"crypto/md5"
	"crypto/sha256"
	"fmt"
	"io"
	"os"
)

func main() {
	f, err := os.Open(os.Args[1])
	if err != nil {
		fmt.Fprintf(os.Stderr, "failed to open file: %s\n", err)
		os.Exit(1)
	}

	md5hash := md5.New()
	sha256hash := sha256.New()
	multi := io.MultiWriter(md5hash, sha256hash)

	_, err = io.Copy(multi, f)
	if err != nil {
		fmt.Fprintf(os.Stderr, "failed to copy to multi hash: %s\n", err)
		os.Exit(1)
	}

	fmt.Printf("md5\t%x\n", md5hash.Sum([]byte{}))
	fmt.Printf("sha256\t%x\n", sha256hash.Sum([]byte{}))
}

Interestingly this ends up faster than a plain sha256sum-call on average, despite doing both hashes.

[tazjin]

Minor change of plans: I will keep the construction of layer tarballs in Nix, but invoked as a separate derivation. The tarballs will then be read, hashed & uploaded by Nixery in one disk read.

[tazjin]

This pull request is now missing two major things: Using the new caching strategy, and cleaning up builder.go.

[tazjin]

Almost there. Local manifest caching is currently gone, because the manifests are no longer written to local files and the local manifest cache used file paths.

[tazjin]

Alright, it's done.

The layer construction, as mentioned, is still in Nix (but in a separate invocation). A subsequent version could optimise this further to create the tarball in the server and avoid the last round of extra reads/writes on disk.