Allow to not have randomness in file_id

[bmwiedemann]

in order to make builds reproducible.
See https://reproducible-builds.org/ for why this is good.

Debian is approaching this with
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=835061
which fixes the rand seed, but I was afraid that this might produce
different files with identical file_id values.

There could be a third approach possible where all relevant inputs
are hashed into the file_id value, so that it differs when inputs change
but remains the same otherwise.

[bmwiedemann]

also note, that I only tested that it builds, not that it works.

[Dridi]

It's great to see the reproducible builds effort reaching out to us, thanks!

Summoning @ssm, your opinion?

[bmwiedemann]

Actually, projects could make their software more reproducible by just looking at results that I publish every few weeks - e.g. http://rb.zq1.de/compare.factory-20170208/varnish-compare.out but since I have all my tools setup and some experience in debugging reproducibility-issues, it is probably easier for me to do.

[bsdphk]

The randomness is a place-holder. What really should go there is the SHA256() over the .h files which constitute the VRT API.

Now that we hav $ABI VRT we need to decide exactly what those are (and clean them up)

[bmwiedemann]

So which .h files constitute that API? And I think this code was run multiple times - so how would one determine the right hash per call?

[Dridi]

@bmwiedemann, sorry for the lack of response lately but to answer your question this is still an ongoing effort. Somewhat related link, still an incomplete draft:

https://github.com/varnishcache/varnish-cache/wiki/VIP20%3A-Varnish-ABI-and-packaging

[bsdphk]

I guess at least I had misinterpreted this ticket: This is about the vmod's file_id, and the bugwash consensus was to SHA256 the vmods .vcc file, and leave it at that.

[bmwiedemann]

@bsdphk I tried your patch ontop of 6.0.0 and get an error during build:

/usr/bin/python3 ../../lib/libvcc/vmodtool.py  ../../lib/libvmod_debug/vmod.vcc
Traceback (most recent call last):
  File "../../lib/libvcc/vmodtool.py", line 1091, in <module>
    runmain(i_vcc, opts.rstdir, opts.output)
  File "../../lib/libvcc/vmodtool.py", line 1044, in runmain
    v.parse()
  File "../../lib/libvcc/vmodtool.py", line 856, in parse
    self.file_id = hashlib.sha256(a).hexdigest()
TypeError: Unicode-objects must be encoded before hashing
make[4]: *** [Makefile:736: vcc_if.c] Error 1

It probably needs to be

self.file_id = hashlib.sha256(a.encode('utf-8')).hexdigest()

For some reason both versions work in python2 (soon to be dropped anyway)

[bmwiedemann]

Made PR #2759 for the fixup. openSUSE's varnish package builds reproducibly with the patches. yay!