Debian: update to dh compat 12, fix more serious packaging errors, correct copyright syntax #53546

mcv21 · 2023-09-20T12:40:35Z

This PR incorporates my two previous PRs (#53342 and #53397) working on improving the Debian packaging; it specifies a minimum g++ version (rather than hard-coding exactly 11), brings the debhelper compatibility up to level 12 (the most recent supported by the oldest Ubuntu LTS 20.04) making consequent updates to how systemd and sysvinit scripts are managed by dh. It also fixes some serious packaging errors; lintian highlighted a large number, and I have tried to just fix the ones that seem particularly critical for a backport to Reef:

Make sure built packages actually have a copyright file
Make sure packages with python scripts in declare a suitable python dependency

I think this should be backported for Reef (and it builds suitable packages for Reef in a clean bullseye chroot)

Fixes: https://tracker.ceph.com/issues/61845

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

tchaikov

thank you for your fixes and improvements. left a couple comments.

debian/control

debian/ceph-mon.postinst

debian/rules

debian/control

mcv21 · 2023-09-21T09:54:26Z

thank you for your fixes and improvements. left a couple comments.

Thanks for the useful review! I think I've made all the changes you suggested, and have split this PR into a number of smaller and self-contained commits.

debian/control

Rely on the packaging system to provide a suitable g++ of version 11 or greater, and removing the corresponding hard-coding from debian/rules, since cmake will then find a suitable version. This seems better than trying to hard-code a particular version in debian/rules, and Debian package building tools like e.g. sbuild will then do the right thing. This enables Reef (v18.2.0) to build on Debian bookworm in a clean chroot. Fixes: https://tracker.ceph.com/issues/61845 Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

These were previously missing. The requirement for interpreters is in Debian policy section 10.4: https://www.debian.org/doc/debian-policy/ch-files.html#s-scripts Debian's packaging already adds the #! to these two postinsts. In practice, a text executible without a #! line will likely be executed by the calling shell, so a lot of the time we'd get away with it unless the administrator is using an incompatible shell like tcsh. This behaviour of shells is documented in POSIX section 1(e)(i)(b) here: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01 Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

Unless there's a version requirement (which there isn't here), packages should not declare a Build-Depends: or Depends: relationship on essential packages. Policy link: https://www.debian.org/doc/debian-policy/ch-binary.html#dependencies Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

Update the header paragraph to link to the canonical URL for the format, and point to dev@ceph.io as the Contact. Also add License: stanzas to reflect the licences in use (and refer to fuller versions in /usr/share/common-licenses/ as appropriate). This means that packages containing this copyright file are better in compliance with the licences concerned. Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

Bring the dh compat level to 12, the most recent supported by the oldest supported Ubuntu LTS release, 20.04. This necessitates changes to how initscripts & systemd packaging are done. Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

This means that debian/control matches changelog entries, and that the Maintainer address is up to date. Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

debian/ceph-base.docs only referred to a README that doesn't exist, so remove it. Because dpkg-source doesn't reflect deletions from debian/ cf the orig.tar.gz, also remove the file in dh_auto_clean. Then do away with the removal of the empty override of dh_installdocs; the main benefit of which here is that debian/copyright gets installed in all of the built packages, which otherwise lack a copyright file. Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

cephadm is a compressed zipapp, and dh3_python3 doesn't understand this sort of binary file, so fails to produce the required python3 dependency. So specify this explicitly in debian/control Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

Installation of init scripts properly belongs with dh_installinit, so move the installation there. That means we no longer need the override of dh_auto_build, which simplifies the rules file. Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

In the cases of ceph-base, ceph-common, and ceph-fuse, this picks up that these packages contain python scripts and adds a necessary python3 dependency. In the case of ceph-volume it additionally parses the requirements.txt file. Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

tchaikov · 2023-09-22T16:37:21Z

i also pushed this change to ceph-ci, see https://shaman.ceph.com/builds/ceph/kefu-pr-53546/. so that it can be compile-tested on the ubuntu builders. and i'd like to check the generated debian/control file manually. once the build is green, and the generated debian/control file looks good. i will merge it.

tchaikov

lgtm

tchaikov · 2023-09-23T00:52:43Z

debian/rules

+	dh_python3 -p ceph-base --shebang=/usr/bin/python3
+	dh_python3 -p ceph-common --shebang=/usr/bin/python3
+	dh_python3 -p ceph-fuse --shebang=/usr/bin/python3
+	dh_python3 -p ceph-volume --shebang=/usr/bin/python3


for the posterity, despite that ceph-volume comes with a requirements_dev.txt. it is not installed or participates in the packaging. the generated DEBIAN/control of ceph-volume looks like

Package: ceph-volume Source: ceph Version: 18.0.0-6344-g4fd6c24f-1focal Architecture: all Maintainer: Ceph Maintainers <ceph-maintainers@ceph.io> Installed-Size: 724 Depends: ceph-osd (= 18.0.0-6344-g4fd6c24f-1focal), cryptsetup-bin, e2fsprogs, lvm2, parted, xfsprogs, python3:any Section: python Priority: optional Homepage: http://ceph.com/ Description: tool to facilidate OSD deployment Ceph is a massively scalable, open-source, distributed storage system that runs on commodity hardware and delivers object, block and file system storage. . This package contains a tool to deploy OSD with different devices like lvm or physical disks, and trying to follow a predictable, and robust way of preparing, activating, and starting the deployed OSD.

after auditing other packages' DEBIAN/control files, they all now have an additional python3:any dependency.

tchaikov · 2023-09-23T01:00:00Z

jenkins test api

mcv21 · 2023-09-25T09:04:51Z

@tchaikov shall I now open a PR to apply these changes to reef?

tchaikov · 2023-09-25T13:43:31Z

@mcv21 it was my fault marking https://tracker.ceph.com/issues/61845 resolved. should have marked it "Pending Backport". fixed it just now.

@tchaikov shall I now open a PR to apply these changes to reef?

thank you in advance! that'd be great. please take a look at https://github.com/ceph/ceph/blob/main/SubmittingPatches-backports.rst , if you haven't done so.

mcv21 · 2023-09-25T15:04:39Z

@tchaikov shall I now open a PR to apply these changes to reef?

thank you in advance! that's be great. please take a look at https://github.com/ceph/ceph/blob/main/SubmittingPatches-backports.rst , if you haven't done so.

I've done so (and thus opened #53654).

github-actions bot added the build/ops label Sep 20, 2023

tchaikov reviewed Sep 20, 2023

View reviewed changes

debian/control Show resolved Hide resolved

debian/ceph-mon.postinst Show resolved Hide resolved

debian/rules Outdated Show resolved Hide resolved

debian/control Outdated Show resolved Hide resolved

mcv21 force-pushed the deb_package_fixes branch from 11fc1c2 to fdeb77c Compare September 21, 2023 09:49

tchaikov reviewed Sep 22, 2023

View reviewed changes

debian/control Show resolved Hide resolved

mcv21 added 11 commits September 22, 2023 17:31

debian: add missing item separators in debian/control

7ec0f69

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

debian: correct maintainer address

74188c2

This means that debian/control matches changelog entries, and that the Maintainer address is up to date. Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

mcv21 force-pushed the deb_package_fixes branch from fdeb77c to e4393e5 Compare September 22, 2023 16:32

tchaikov self-assigned this Sep 22, 2023

tchaikov approved these changes Sep 22, 2023

View reviewed changes

tchaikov reviewed Sep 23, 2023

View reviewed changes

tchaikov merged commit cbb06d6 into ceph:main Sep 23, 2023
10 of 11 checks passed

This was referenced Sep 23, 2023

debian: unpin gcc version #52658

Closed

Debian: fix packaging errors #53397

Closed

debian: Build-Depend on g++ 11 or greater #53342

Closed

mcv21 mentioned this pull request Sep 25, 2023

reef: Debian: update to dh compat 12, fix more serious packaging errors, correct copyright syntax #53654

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Debian: update to dh compat 12, fix more serious packaging errors, correct copyright syntax #53546

Debian: update to dh compat 12, fix more serious packaging errors, correct copyright syntax #53546

mcv21 commented Sep 20, 2023 •

edited

tchaikov left a comment

mcv21 commented Sep 21, 2023

tchaikov commented Sep 22, 2023 •

edited

tchaikov left a comment

tchaikov Sep 23, 2023

tchaikov Sep 23, 2023

tchaikov commented Sep 23, 2023

mcv21 commented Sep 25, 2023

tchaikov commented Sep 25, 2023 •

edited

mcv21 commented Sep 25, 2023

Debian: update to dh compat 12, fix more serious packaging errors, correct copyright syntax #53546

Debian: update to dh compat 12, fix more serious packaging errors, correct copyright syntax #53546

Conversation

mcv21 commented Sep 20, 2023 • edited

Checklist

tchaikov left a comment

Choose a reason for hiding this comment

mcv21 commented Sep 21, 2023

tchaikov commented Sep 22, 2023 • edited

tchaikov left a comment

Choose a reason for hiding this comment

tchaikov Sep 23, 2023

Choose a reason for hiding this comment

tchaikov Sep 23, 2023

Choose a reason for hiding this comment

tchaikov commented Sep 23, 2023

mcv21 commented Sep 25, 2023

tchaikov commented Sep 25, 2023 • edited

mcv21 commented Sep 25, 2023

mcv21 commented Sep 20, 2023 •

edited

tchaikov commented Sep 22, 2023 •

edited

tchaikov commented Sep 25, 2023 •

edited