build proper src.rpm packages

[marmarek]

Currently building src.rpm packages is broken. Mostly because we're building directly from git checkout, without proper source tarball. But src.rpm packages are needed for #816 .
The work include:

• generate proper (deterministic) source tarball - git archive can be useful here
• create static spec file - currently it uses content of version file for getting package version - similar problem is already solved in Debian packages

Optional:

• generate proper changelog - similar code is already in Debian packages; additionally PLD Linux have a script exactly for that

[marmarek]

Work in progress version here: https://github.com/marmarek/qubes-builder-fedora/tree/reprobuild + https://github.com/marmarek/qubes-builder/tree/reprobuild

This require source component to be modified:

1. Rename .spec to .spec.in, use @VERSION@ instead of cat version hack.
2. Make sure that symlink hack in %prep is either under %if 0%{?qubes_builder}, or removed.
3. Same for %define _builddir %(pwd).
4. Add Source0: %{name}-%{version}.tar.gz
5. If component use BACKEND_VMM variable, set it in spec with @BACKEND_VMM@, for example export BACKEND_VMM=@BACKEND_VMM@.
6. Set RPM_USE_MOCKBUILD = 1 in Makefile.builder of that component.

Example modified component: https://github.com/marmarek/qubes-linux-utils/tree/srcrpm

Missing parts:

• replace git archive with tar and move it to builder-fedora plugin; to have more explicit format and archive layout, and also to take into account local changes (devel builds) - see reprobuild branch in builder-debian
• fill changelog in spec (see issue description)
• handle the case when source tarball is already provided (vmm-xen, linux-kernel, installer-qubes-os)
• handle multi-package components (installer-qubes-os) - related to replacing git archive
• fix building "new" type packages using "old" code - for fast devel builds (avoid installing build deps every time)
• verify if "old" code really build all "old" components - for example if all Qubes 3.2 can be built
• verify if INCREMENT_DEVEL_VERSIONS=1 still works
• verify if sign and update-repo/check-repo targets still works

cc @fepitre

[marmarek]

Work in progress version here: https://github.com/marmarek/qubes-builder-fedora/tree/reprobuild + https://github.com/marmarek/qubes-builder/tree/reprobuild

This require source component to be modified:

1. Rename .spec to .spec.in, use @VERSION@ instead of cat version hack.
2. Make sure that symlink hack in %prep is either under %if 0%{?qubes_builder}, or removed.
3. Same for %define _builddir %(pwd).
4. Add Source0: %{name}-%{version}.tar.gz
5. If component use BACKEND_VMM variable, set it in spec with @BACKEND_VMM@, for example export BACKEND_VMM=@BACKEND_VMM@.
6. Set RPM_USE_MOCKBUILD = 1 in Makefile.builder of that component.

Example modified component: https://github.com/marmarek/qubes-linux-utils/tree/srcrpm

Missing parts:

• replace git archive with tar and move it to builder-fedora plugin; to have more explicit format and archive layout, and also to take into account local changes (devel builds) - see reprobuild branch in builder-debian
• fill changelog in spec (see issue description)
• handle the case when source tarball is already provided (vmm-xen, linux-kernel, installer-qubes-os)
• handle multi-package components (installer-qubes-os) - related to replacing git archive
• fix building "new" type packages using "old" code - for fast devel builds (avoid installing build deps every time)
• verify if "old" code really build all "old" components - for example if all Qubes 3.2 can be built
• verify if INCREMENT_DEVEL_VERSIONS=1 still works
• verify if sign and update-repo/check-repo targets still works

cc @fepitre

[fepitre]

Great job! So the starting point here is to handle the missing parts and doing the "repack" of all the components?

[fepitre]

Great job! So the starting point here is to handle the missing parts and doing the "repack" of all the components?

[marmarek]

So the starting point here is to handle the missing parts and doing the "repack" of all the components?

Yes. While working on some of them it may turn out some better options for Makefile.builder in each component will suit there. Some alternative approach is to specify "builder compatibility version", instead of requested feature. This might ease handling similar changes in the future.

[marmarek]

So the starting point here is to handle the missing parts and doing the "repack" of all the components?

Yes. While working on some of them it may turn out some better options for Makefile.builder in each component will suit there. Some alternative approach is to specify "builder compatibility version", instead of requested feature. This might ease handling similar changes in the future.

[fepitre]

I'm looking at Mock possibilities and I'm testing the --buildsrpm using the spec file. Preliminaries tests of packaging work great but I encounter a problem with folders of patches. Is there a way to include easily folders of patches (e.g. patches.misc) with all the patches inside included? It seems that we have to list explicitly all the patches (like it is done in series.conf) and or Fedora upstream xen.spec. It that case it would rethink the way of applying patches?

[fepitre]

I'm looking at Mock possibilities and I'm testing the --buildsrpm using the spec file. Preliminaries tests of packaging work great but I encounter a problem with folders of patches. Is there a way to include easily folders of patches (e.g. patches.misc) with all the patches inside included? It seems that we have to list explicitly all the patches (like it is done in series.conf) and or Fedora upstream xen.spec. It that case it would rethink the way of applying patches?

[marmarek]

Is there a way to include easily folders of patches (e.g. patches.misc) with all the patches inside included?

No, listing a folder of patches is a bug in rpm packaging. It require explicit patch list. It works for us because of _builddir hack. Part of fixing components is about listing patches in spec file... See libvirt.spec for example how to not list them twice (the second one in %prep). I was thinking about generating that list (add @PATCHES@ or such), but finally abandoned that idea, because we don't have that much components with many patches.

[marmarek]

Is there a way to include easily folders of patches (e.g. patches.misc) with all the patches inside included?

No, listing a folder of patches is a bug in rpm packaging. It require explicit patch list. It works for us because of _builddir hack. Part of fixing components is about listing patches in spec file... See libvirt.spec for example how to not list them twice (the second one in %prep). I was thinking about generating that list (add @PATCHES@ or such), but finally abandoned that idea, because we don't have that much components with many patches.

[marmarek]

@fepitre vmm-xen build fails with RPM_USE_MOCKBUILD=1. Because git submodules (Source3[3-6] in spec) are not included in any tarball. I think each of them should have own tarball (created with create-archive in source-copy-in or such). Probably with some artificial version number, for example hash of last commit (or shortened hash), to avoid having same named tarball with different content.

stubdom-dhcp isn't a submodule, but there are only 3 files - IMO listing them directly should be ok.

[marmarek]

@fepitre vmm-xen build fails with RPM_USE_MOCKBUILD=1. Because git submodules (Source3[3-6] in spec) are not included in any tarball. I think each of them should have own tarball (created with create-archive in source-copy-in or such). Probably with some artificial version number, for example hash of last commit (or shortened hash), to avoid having same named tarball with different content.

stubdom-dhcp isn't a submodule, but there are only 3 files - IMO listing them directly should be ok.

[fepitre]

Oh yes. I will solve this problem as top priority.

[fepitre]

Oh yes. I will solve this problem as top priority.