reproducible builds issues with noarch packages

[bmwiedemann]

Issue Description

OBS produces and uses different versions of (unreproducible) noarch packages, which makes verifying OBS builds hard.
See https://reproducible-builds.org/ for why this matters.

Symptom

comparing local builds of SUSE:SLE-15-SP1:GA/skelcd-ha with official ones gave an unexpected diff

/usr/lib/skelcd/CD1/docu/RELEASE-NOTES.en.pdf differs at offset '3555' (PDF document, version 1.4)
@@ -1,6 +1,6 @@
 00000dc0  20 20 20 20 20 20 20 20  20 20 3c 64 63 3a 64 61  |          <dc:da|
 00000dd0  74 65 3e 32 30 31 38 2d  30 37 2d 32 37 54 30 34  |te>2018-07-27T04|
-00000de0  3a 35 37 3a 31 38 5a 3c  2f 64 63 3a 64 61 74 65  |:57:18Z</dc:date|
+00000de0  3a 35 36 3a 31 36 5a 3c  2f 64 63 3a 64 61 74 65  |:56:16Z</dc:date|
 00000df0  3e 0a 20 20 20 20 20 20  20 20 3c 2f 72 64 66 3a  |>.        </rdf:|

neither the OBS build nor the local build happened in July and doing two local builds did not have this diff.

Analysis

The pdf is copied from the release-notes-ha noarch package

for arch in i586 x86_64 s390x ppc64le aarch64 ; do
  mkdir -p $arch
  ( cd $arch &&
    osc -A https://api.suse.de getbinaries SUSE:SLE-15:Update/release-notes-ha.8105 standard $arch
  )
done
md5sum */binaries/*rpm

shows that different versions of the noarch release-notes-ha exist and comparing with the published version, shows that the ppc64le version was published.

unrpm release-notes-ha-15.0.20180717-4.3.6.noarch.rpm
strings usr/share/doc/release-notes/SUSE_Linux_Enterprise_High_Availability_Extension_15/*pdf|grep date

shows the embedded date. The ppc64le noarch version is used for the local build.

Expected Result

OBS should ensure that only one version of a noarch package is published and used for builds later.
I heard, Fedora requires noarch packages to build reproducibly (so pkg-diff would return 0 when comparing x86 and ppc noarch builds).
The second best approach would be to discard the other builds of a noarch package and/or replace them with the canonical/published version.

How to Reproduce

1. iosc co SUSE:SLE-15-SP1:GA/skelcd-ha && cd $_
2. osc build --keep-pkg=binaries.nachbau
3. osc getbinaries standard x86_64
4. /usr/lib/build/pkg-diff.sh -a binaries{,.nachbau}/skelcd-ha-2018.08.14-*.x86_64.rpm

[Conan-Kudo]

I heard, Fedora requires noarch packages to build reproducibly (so pkg-diff would return 0 when comparing x86 and ppc noarch builds).

This is true. Fedora's Koji compares noarch package builds across arches and if they're built differently, the whole build is failed.

This is actually very difficult to do in OBS, since each architecture is an independent build job, whereas Koji ties the build job to a central task that takes the input source (git, svn, or srpm) as the key. This difference in behavior is critical to understanding why Koji can do this while OBS currently cannot.

[bmwiedemann]

Dirk Mueller pointed me to the possibility of an ExportFilter that takes a regexp of packages to copy from another build.
One downside would be that downstream build jobs for one architecture would need to wait on that build job to finish for another architecture.

[adrianschroeter]

On Montag, 10. September 2018, 09:34:16 CEST wrote Bernhard M. Wiedemann:

Dirk Mueller pointed me to the possibility of an [ExportFilter](https://en.opensuse.org/openSUSE:Build_Service_prjconf#ExportFilter) that takes a regexp of packages to copy from another build. One downside would be that downstream build jobs for one architecture would need to wait on that build job to finish for another architecture.

this would not happen and you could also create new cycles by using this mechanic. Cycles which can not be calculated before, so you won't see them, just notice endless builds.

…

-- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany

[mlschroe]

So we should aim to get reproducible noach packages. But that's not something where OBS itself is involved. Closing.