New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to not have randomness in file_id #2436

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
4 participants
@bmwiedemann
Contributor

bmwiedemann commented Sep 20, 2017

in order to make builds reproducible.
See https://reproducible-builds.org/ for why this is good.

Debian is approaching this with
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=835061
which fixes the rand seed, but I was afraid that this might produce
different files with identical file_id values.

There could be a third approach possible where all relevant inputs
are hashed into the file_id value, so that it differs when inputs change
but remains the same otherwise.

@bmwiedemann

This comment has been minimized.

Show comment
Hide comment
@bmwiedemann

bmwiedemann Sep 20, 2017

Contributor

also note, that I only tested that it builds, not that it works.

Contributor

bmwiedemann commented Sep 20, 2017

also note, that I only tested that it builds, not that it works.

@Dridi

This comment has been minimized.

Show comment
Hide comment
@Dridi

Dridi Sep 20, 2017

Member

It's great to see the reproducible builds effort reaching out to us, thanks!

Summoning @ssm, your opinion?

Member

Dridi commented Sep 20, 2017

It's great to see the reproducible builds effort reaching out to us, thanks!

Summoning @ssm, your opinion?

@bmwiedemann

This comment has been minimized.

Show comment
Hide comment
@bmwiedemann

bmwiedemann Sep 20, 2017

Contributor

Actually, projects could make their software more reproducible by just looking at results that I publish every few weeks - e.g. http://rb.zq1.de/compare.factory-20170208/varnish-compare.out but since I have all my tools setup and some experience in debugging reproducibility-issues, it is probably easier for me to do.

Contributor

bmwiedemann commented Sep 20, 2017

Actually, projects could make their software more reproducible by just looking at results that I publish every few weeks - e.g. http://rb.zq1.de/compare.factory-20170208/varnish-compare.out but since I have all my tools setup and some experience in debugging reproducibility-issues, it is probably easier for me to do.

@bsdphk

This comment has been minimized.

Show comment
Hide comment
@bsdphk

bsdphk Sep 22, 2017

Contributor

The randomness is a place-holder. What really should go there is the SHA256() over the .h files which constitute the VRT API.

Now that we hav $ABI VRT we need to decide exactly what those are (and clean them up)

Contributor

bsdphk commented Sep 22, 2017

The randomness is a place-holder. What really should go there is the SHA256() over the .h files which constitute the VRT API.

Now that we hav $ABI VRT we need to decide exactly what those are (and clean them up)

@bmwiedemann

This comment has been minimized.

Show comment
Hide comment
@bmwiedemann

bmwiedemann Sep 30, 2017

Contributor

So which .h files constitute that API? And I think this code was run multiple times - so how would one determine the right hash per call?

Contributor

bmwiedemann commented Sep 30, 2017

So which .h files constitute that API? And I think this code was run multiple times - so how would one determine the right hash per call?

@Dridi

This comment has been minimized.

Show comment
Hide comment
@Dridi

Dridi Oct 17, 2017

Member

@bmwiedemann, sorry for the lack of response lately but to answer your question this is still an ongoing effort. Somewhat related link, still an incomplete draft:

https://github.com/varnishcache/varnish-cache/wiki/VIP20%3A-Varnish-ABI-and-packaging

Member

Dridi commented Oct 17, 2017

@bmwiedemann, sorry for the lack of response lately but to answer your question this is still an ongoing effort. Somewhat related link, still an incomplete draft:

https://github.com/varnishcache/varnish-cache/wiki/VIP20%3A-Varnish-ABI-and-packaging

@bsdphk bsdphk added the a=bugwash label Jun 13, 2018

@bsdphk

This comment has been minimized.

Show comment
Hide comment
@bsdphk

bsdphk Jun 18, 2018

Contributor

I guess at least I had misinterpreted this ticket: This is about the vmod's file_id, and the bugwash consensus was to SHA256 the vmods .vcc file, and leave it at that.

Contributor

bsdphk commented Jun 18, 2018

I guess at least I had misinterpreted this ticket: This is about the vmod's file_id, and the bugwash consensus was to SHA256 the vmods .vcc file, and leave it at that.

@bsdphk bsdphk removed the a=bugwash label Jun 18, 2018

Allow to not have randomness in file_id
in order to make builds reproducible.
See https://reproducible-builds.org/ for why this is good.
@bmwiedemann

This comment has been minimized.

Show comment
Hide comment
@bmwiedemann

bmwiedemann Sep 3, 2018

Contributor

@bsdphk I tried your patch ontop of 6.0.0 and get an error during build:

/usr/bin/python3 ../../lib/libvcc/vmodtool.py  ../../lib/libvmod_debug/vmod.vcc
Traceback (most recent call last):
  File "../../lib/libvcc/vmodtool.py", line 1091, in <module>
    runmain(i_vcc, opts.rstdir, opts.output)
  File "../../lib/libvcc/vmodtool.py", line 1044, in runmain
    v.parse()
  File "../../lib/libvcc/vmodtool.py", line 856, in parse
    self.file_id = hashlib.sha256(a).hexdigest()
TypeError: Unicode-objects must be encoded before hashing
make[4]: *** [Makefile:736: vcc_if.c] Error 1

It probably needs to be

self.file_id = hashlib.sha256(a.encode('utf-8')).hexdigest()

For some reason both versions work in python2 (soon to be dropped anyway)

Contributor

bmwiedemann commented Sep 3, 2018

@bsdphk I tried your patch ontop of 6.0.0 and get an error during build:

/usr/bin/python3 ../../lib/libvcc/vmodtool.py  ../../lib/libvmod_debug/vmod.vcc
Traceback (most recent call last):
  File "../../lib/libvcc/vmodtool.py", line 1091, in <module>
    runmain(i_vcc, opts.rstdir, opts.output)
  File "../../lib/libvcc/vmodtool.py", line 1044, in runmain
    v.parse()
  File "../../lib/libvcc/vmodtool.py", line 856, in parse
    self.file_id = hashlib.sha256(a).hexdigest()
TypeError: Unicode-objects must be encoded before hashing
make[4]: *** [Makefile:736: vcc_if.c] Error 1

It probably needs to be

self.file_id = hashlib.sha256(a.encode('utf-8')).hexdigest()

For some reason both versions work in python2 (soon to be dropped anyway)

bmwiedemann added a commit to bmwiedemann/varnish-cache that referenced this pull request Sep 3, 2018

Fix vmodtool with python3
Without this patch, it did
TypeError: Unicode-objects must be encoded before hashing

Fixes: #2436
@bmwiedemann

This comment has been minimized.

Show comment
Hide comment
@bmwiedemann

bmwiedemann Sep 3, 2018

Contributor

Made PR #2759 for the fixup. openSUSE's varnish package builds reproducibly with the patches. yay!

Contributor

bmwiedemann commented Sep 3, 2018

Made PR #2759 for the fixup. openSUSE's varnish package builds reproducibly with the patches. yay!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment