Investigate raw manpage deduplication

[stapelberg]

Manpages can be identical across different Debian versions. Our corpus of extracted manpages weighs about 2.5 GB. We should investigate whether it’d be worthwhile to de-duplicate content (by using hard links, most likely). If the savings are not substantial, it might be more trouble than it’s worth.

[stapelberg]

Using the following analysis program:

package main

import (
	"crypto/md5"
	"encoding/hex"
	"fmt"
	"io"
	"log"
	"os"
	"path/filepath"
)

func md5sum(fn string) ([]byte, error) {
	f, err := os.Open(fn)
	if err != nil {
		return nil, err
	}
	defer f.Close()

	h := md5.New()
	_, err = io.Copy(h, f)
	return h.Sum(nil), err
}

func main() {
	sums := make(map[string]int)
	var bytesSaved int64
	if err := filepath.Walk("/srv/man", func(path string, info os.FileInfo, err error) error {
		if info.IsDir() || info.Mode()&os.ModeSymlink != 0 {
			return nil
		}
		sumb, err := md5sum(path)
		if err != nil {
			return err
		}
		sum := hex.EncodeToString(sumb)
		num, ok := sums[sum]
		if !ok {
			sums[sum] = 0
			return nil
		}
		num++
		sums[sum] = num
		bytesSaved += info.Size()
		return nil
	}); err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%d bytes saved\n", bytesSaved)

}

…I get: 412926636 bytes saved.

So, 393 MB could be saved on disk if we implemented de-duplication. This is 4% of our current 8.3 GB footprint. Certainly a low-priority feature, if worthwhile at all.

[stapelberg]

With the additional experience of launching the service on the actual Debian hardware, I now know that we are not in a situation where we need to save disk and have extra CPU/RAM to spare in order to achieve that — it’s quite the opposite :).

Hence, let’s not do de-duplication.

[lahwaacz]

You could probably do much better with deduplicating the HTML, even if only in the same Debian version. And you don't need any expensive analysis to do that, see e.g. pwunconv.8.en.gz: it contains just .so jessie/passwd/pwconv.8.en.gz, so you can just create a symbolic link from pwunconv.8.en.html to pwconv.8.en.html on the filesystem level. That saves you storage as well as rendering time.