howtodebug

This is a short guide on how to find, debug and fix reproducibility issues in openSUSE OBS package build results. Some of it will also apply to other subjects.

0. Fetch this git repo, follow the setup instructions in the README.md file.

1. find (detect) issues
1.1. run 'rbk'
1.2. for noarch packages run 'noarchcheck'
1.3. run 'nachbau' once the above is all good

all of them will store their outcome in dot files in the current dir for later use.
A result of '0' usually indicates 'success' or 'reproducible' in UNIX fashion.


2. debug (track down) issues
This section assumes you found a package that produced diffs in rbk

2.1. run autoclassify
this script will need a while and report a bitmask of what non-deterministic influences had to be disabled to produce reproducible results.
If it is "1 1 1 1 1 1 1 1 1 1 1 1 99" it remained unreproducible even with all tweaks applied.
Debugging the remaining diff can still be easier and once that is fixed, autoclassify can be rerun.
If there are multiple bits left, you can debug one of them at a time by running 'rbkt' with fewer bits set.
See rbkt source for the meaning of bits.

2.2. run autoprovenance $unreproduciblefile
this script will tell you what command produced the file along with a call trace. Requires 'stracebuild' to be run before once.

2.3 other ways to find provenance
2.3.1 review diffs in .rb.buildroot.diff - e.g. .c .h .a configure Makefile and track down their provenance
2.3.2 check build log (might need to use make V=1 VERB=1)
2.3.3 grep source for strings next to diff (e.g. grep "This file was autogenerated at")
2.3.4 grep source for known troublemakers (`date, $(date, %Y, year, strftime, ctime, asctime, stat (mtime,inode,dev) readdir, os.listdir, os.walk, glob.glob, find, random, fprofile-use)
2.3.5 use difflog to see if something interesting happens related to indeterministic files
2.3.6 review relevant files (configure{,.ac}, Makefile{,.in,.am}, CMakeLists.txt, setup.py)
2.3.7 manually add strace -f


3. fix issues
Once you know exactly which code line introduced non-determinism, fixing it can be pretty straightforward.
https://salsa.debian.org/reproducible-builds/reproducible-website/tree/master/_blog/posts numbers 100+ has 2 years worth of links to a large collection of Bernhard's patches along with short summaries like "date" or "readdir" or "ASLR" that map to one of the entries in
https://github.com/bmwiedemann/theunreproduciblepackage

The workflow for fixing issues usually involves:
 source $path_to_reproducibleopensuse/helperfuncs
 qs # shorthand for quilt setup + cd + apply old patches
 quilt new reproducible.patch
 quilt edit $file
 quilt diff # optional
 quilt refresh
 cd ..
 /usr/lib/build/spec_add_patch reproducible.patch
 rbk # retest

For patches that can not be upstreamed:
 bcocd
 cp -a ../../*.{patch,spec} .
 osc vc -m "Add reproducible.patch to make package build reproducible"
 osc addremove
 osc status # optional
 osc checkin --noservice
 osc submitrequest

3.1 drop unreproducible elements
  sometimes this is the simplest to get upstream because it can reduce the overall complexity of the build system

3.2 normalize unreproducible elements
  sort lists (that usually varied from filesystem readdir order or ASLR)
  replace build date with fixed date (e.g. SOURCE_DATE_EPOCH or ChangeLog file mtime)

3.3 fix bugs
  sometimes unreproducible output is the result of actual bugs and then actual upstream bugfixes rather than downstream distribution workarounds are recommended.
  https://github.com/bmwiedemann/theunreproduciblepackage/tree/master/race has several examples
  https://github.com/projectM-visualizer/projectm/pull/176

4. Do upstream patches, if possible
  sometimes non-determinism will be introduced by .spec files or downstream-only patches - then a osc submitrequest is enough.

4.1 locate the upstream git repo, bug tracker or website
4.1.1 grep -e ^Url -e ^Source *.spec
4.1.2 google $package git OR contribute OR bugtracker

4.2 check if the project had activity in the last 12 months
  if not, it may have moved or been orphaned. You can still submit a downstream patch.

4.3 make the patch
git clone $URL
patch -p1 < $path_to/reproducible.patch # or edit $file
git checkout -b $topic
# possibly check other contribution guidelines about "Signed-off-By" lines or where/how to submit fixes
git commit -a
# use good commit messages, that can include phrases like
 Do $SOMEFIX to make builds reproducible.
 See https://reproducible-builds.org/ for why this is good.
 Without this patch, build results varied in `$file` with this diff:
 ```diff
 $DIFF_EXAMPLE
 ```

4.4 submit the patch
4.4.1 For projects hosted on github this can use the `hub` package:
 hub fork
 git push $yourfork
 hub pull-request

4.4.2 For projects doing reviews on gerrit, use 'git-review' package
 git review -s
 git commit --amend # to add a ChangeID
 git review

4.4.3 For projects using phabricator use the 'arcanist' package
 arc diff # FIXME this is incomplete

4.4.4 For projects using mercurial, use hg or https://github.com/felipec/git-remote-hg

4.4.5 For projects using mailing lists for patches:
  if only members can post, subscribe to ML
  git send-email --to "$ML_ADDR" HEAD^
  check ML archive 10-30 minutes later
  unsubscribe from ML once patch discussion is over

4.4.6 If projects do not specify how to contribute, but you have a git repo, try
  git log $changedfile
  git send-email --to "$mainauthor" HEAD^

4.5 Tell the world about your fix
 using https://salsa.debian.org/reproducible-builds/reproducible-website monthly draft blog post entries to link to your fix and become famous - one patch at a time.

4.6 Track fixes for yourself
Often it takes maintainers some days or weeks to review and merge your fix, but sometimes fixes fall through cracks and then it helps to try and remind them.
Also when doing your next fix, you can re-use patches and submit message from previous submissions.

4.6.1 re-check after a day if there is work left for you to do like signing a CLA, adding ChangeLog/NEWS entries, addressing outstanding concerns (either with comments or code changes)
4.6.2 after a month, ping in your submission, especially if git history shows other patches going in
Can include polite helpful phrases like:
 Is there anything I can do to help get this simple patch merged?
4.6.3 try alternative routes of communication (not all at the same time): IRC, email, ...
4.6.4 after 6 months, if project appears orphaned:
4.6.4.1 look for active forks in github -> Insights -> Network and ask there
4.6.4.2 propose the package for inclusion in https://github.com/distropatches/
4.6.4.3 adopt it: fork it and collect patches from others and contact previous authors and downstreams
4.6.4.4 submit the patch downstream in a distribution