New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reproducible builds issues with noarch packages #5784

Open
bmwiedemann opened this Issue Sep 5, 2018 · 3 comments

Comments

Projects
None yet
3 participants
@bmwiedemann
Member

bmwiedemann commented Sep 5, 2018

Issue Description

OBS produces and uses different versions of (unreproducible) noarch packages, which makes verifying OBS builds hard.
See https://reproducible-builds.org/ for why this matters.

Symptom

comparing local builds of SUSE:SLE-15-SP1:GA/skelcd-ha with official ones gave an unexpected diff

/usr/lib/skelcd/CD1/docu/RELEASE-NOTES.en.pdf differs at offset '3555' (PDF document, version 1.4)
@@ -1,6 +1,6 @@
 00000dc0  20 20 20 20 20 20 20 20  20 20 3c 64 63 3a 64 61  |          <dc:da|
 00000dd0  74 65 3e 32 30 31 38 2d  30 37 2d 32 37 54 30 34  |te>2018-07-27T04|
-00000de0  3a 35 37 3a 31 38 5a 3c  2f 64 63 3a 64 61 74 65  |:57:18Z</dc:date|
+00000de0  3a 35 36 3a 31 36 5a 3c  2f 64 63 3a 64 61 74 65  |:56:16Z</dc:date|
 00000df0  3e 0a 20 20 20 20 20 20  20 20 3c 2f 72 64 66 3a  |>.        </rdf:|

neither the OBS build nor the local build happened in July and doing two local builds did not have this diff.

Analysis

The pdf is copied from the release-notes-ha noarch package

for arch in i586 x86_64 s390x ppc64le aarch64 ; do
  mkdir -p $arch
  ( cd $arch &&
    osc -A https://api.suse.de getbinaries SUSE:SLE-15:Update/release-notes-ha.8105 standard $arch
  )
done
md5sum */binaries/*rpm

shows that different versions of the noarch release-notes-ha exist and comparing with the published version, shows that the ppc64le version was published.

unrpm release-notes-ha-15.0.20180717-4.3.6.noarch.rpm
strings usr/share/doc/release-notes/SUSE_Linux_Enterprise_High_Availability_Extension_15/*pdf|grep date

shows the embedded date. The ppc64le noarch version is used for the local build.

Expected Result

OBS should ensure that only one version of a noarch package is published and used for builds later.
I heard, Fedora requires noarch packages to build reproducibly (so pkg-diff would return 0 when comparing x86 and ppc noarch builds).
The second best approach would be to discard the other builds of a noarch package and/or replace them with the canonical/published version.

How to Reproduce

  1. iosc co SUSE:SLE-15-SP1:GA/skelcd-ha && cd $_
  2. osc build --keep-pkg=binaries.nachbau
  3. osc getbinaries standard x86_64
  4. /usr/lib/build/pkg-diff.sh -a binaries{,.nachbau}/skelcd-ha-2018.08.14-*.x86_64.rpm
@Conan-Kudo

This comment has been minimized.

Show comment
Hide comment
@Conan-Kudo

Conan-Kudo Sep 10, 2018

Member

I heard, Fedora requires noarch packages to build reproducibly (so pkg-diff would return 0 when comparing x86 and ppc noarch builds).

This is true. Fedora's Koji compares noarch package builds across arches and if they're built differently, the whole build is failed.

This is actually very difficult to do in OBS, since each architecture is an independent build job, whereas Koji ties the build job to a central task that takes the input source (git, svn, or srpm) as the key. This difference in behavior is critical to understanding why Koji can do this while OBS currently cannot.

Member

Conan-Kudo commented Sep 10, 2018

I heard, Fedora requires noarch packages to build reproducibly (so pkg-diff would return 0 when comparing x86 and ppc noarch builds).

This is true. Fedora's Koji compares noarch package builds across arches and if they're built differently, the whole build is failed.

This is actually very difficult to do in OBS, since each architecture is an independent build job, whereas Koji ties the build job to a central task that takes the input source (git, svn, or srpm) as the key. This difference in behavior is critical to understanding why Koji can do this while OBS currently cannot.

@bmwiedemann

This comment has been minimized.

Show comment
Hide comment
@bmwiedemann

bmwiedemann Sep 10, 2018

Member

Dirk Mueller pointed me to the possibility of an ExportFilter that takes a regexp of packages to copy from another build.
One downside would be that downstream build jobs for one architecture would need to wait on that build job to finish for another architecture.

Member

bmwiedemann commented Sep 10, 2018

Dirk Mueller pointed me to the possibility of an ExportFilter that takes a regexp of packages to copy from another build.
One downside would be that downstream build jobs for one architecture would need to wait on that build job to finish for another architecture.

@adrianschroeter

This comment has been minimized.

Show comment
Hide comment
@adrianschroeter

adrianschroeter Sep 10, 2018

Member
Member

adrianschroeter commented Sep 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment