Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
534 lines (415 sloc) 19.7 KB
---
title: "Pragmatic Debian packaging (2019)"
description: |
Creating Debian packages using official tools could be as easy
as using fpm by following some pragmatic guidelines.
uuid: bf0bccf6-fac3-41fa-9f8d-b860dc8079d7
attachments:
"https://github.com/vincentbernat/pragmatic-debian-packages": "Packages examples"
tags:
- debian
---
!!! "Notice" This guide is an updated version of a [previous
edition]([[en/blog/2016-pragmatic-debian-packaging.html]]). If you
need to target distributions older than Debian Stretch and Ubuntu
Bionic, please have a look at the older version instead.
While the creation of Debian packages is [abundantly][newmaint]
[documented][newmaint], most [tutorials][tutorial] are targeted to
packages implementing the [Debian policy][policy]. Moreover, Debian
packaging has **a reputation of being unnecessarily
difficult**[^difficulty] and many people prefer to use less
constrained tools[^fpm] like [fpm][] or [CheckInstall][].
[^difficulty]: People may remember the time before *debhelper* 7.0.50
(circa 2009) where `debian/rules` was a daunting
beast. However, nowaday, the boilerplate is quite
reduced.
[^fpm]: The complexity is not the only reason. These alternative tools
enable the creation of RPM packages, something that Debian
tools don't.
However, building Debian packages with the official tools can become
straightforward if you bend some rules:
1. **No source package will be generated**. Packages will be built
directly from a checkout of a VCS repository.
2. **Additional dependencies can be downloaded during
build**. Packaging individually each dependency is a painstaking
work, notably when you have to deal with some fast-paced
ecosystems like Java, Javascript and Go.
3. The produced packages may **bundle dependencies**. This is likely
to raise some concerns about security and long-term maintenance,
but this is a common trade-off in many ecosystems, notably Java,
Javascript and Go.
[TOC]
# Pragmatic packages 101
In the Debian archive, you have two kinds of packages: the **source
packages** and the **binary packages**. Each binary package is built
from a source package. You need a name for each package.
As stated in the introduction, we won't generate a source package but
we will work with its unpacked form which is any source tree
containing a `debian/` directory. In our examples, we will start with
a source tree containing only a `debian/` directory but you are free
to include this `debian/` directory into an existing project.
As an example, we will package [memcached][], a distributed memory
cache. There are four files to create:
- `debian/compat`,
- `debian/changelog`,
- `debian/control`, and
- `debian/rules`.
The first one is easy. Put `11` in it:[^compat]
::sh
echo 11 > debian/compat
[^compat]: This compatibility level is available from Debian 9 and
Ubuntu Bionic. Therefore, this covers modern distributions.
The second one has the following content:
::text
memcached (0.0-0) UNRELEASED; urgency=medium
* Fake entry
-- Happy Packager <happy@example.com> Tue, 19 Apr 2016 22:27:05 +0200
The only important information is the name of the source package,
`memcached`, on the first line. Everything else can be left as is as
it won't influence the generated binary packages.
## The control file
`debian/control` describes the metadata of both the source package and
the generated binary packages. We have to write a block for each of
them.
::control
Source: memcached
Maintainer: Vincent Bernat <bernat@debian.org>
Package: memcached
Architecture: any
Description: high-performance memory object caching system
The source package is called `memcached`. We have to use the same name
as in `debian/changelog`.
We generate only one binary package: `memcached`. In the remaining of
the example, when you see `memcached`, this is the name of a binary
package. The `Architecture` field should be set to either `any` or
`all`. Use `all` exclusively if the package contains only
arch-independent files. In doubt, just stick to `any`.
The `Description` field contains a short description of the binary
package.
## The build recipe
The last mandatory file is `debian/rules`. It's the **recipe** of the
package. We need to retrieve *memcached*, build it and install its
file tree in `debian/memcached/`. It looks like this:
::make
#!/usr/bin/make -f
DISTRIBUTION = $(shell sed -n "s/^VERSION_CODENAME=//p" /etc/os-release)
VERSION = 1.5.16
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)
%:
dh $@
override_dh_auto_clean:
override_dh_auto_test:
override_dh_auto_build:
override_dh_auto_install:
wget -N --progress=dot:mega $(URL)
tar --strip-components=1 -xf $(TARBALL)
./configure --prefix=/usr
make
make install DESTDIR=debian/memcached
override_dh_gencontrol:
dh_gencontrol -- -v$(PACKAGEVERSION)
The empty targets `override_dh_auto_clean`, `override_dh_auto_test`
and `override_dh_auto_build` keep *debhelper* from being too
smart. The `override_dh_gencontrol` target sets the package
version[^version] without updating `debian/changelog`. If you ignore
the slight boilerplate, the recipe is quite similar to what you would
have done with `fpm`:
::sh
DISTRIBUTION=$(sed -n "s/^VERSION_CODENAME=//p" /etc/os-release)
VERSION=1.5.16
PACKAGEVERSION=${VERSION}-0~${DISTRIBUTION}0
TARBALL=memcached-${VERSION}.tar.gz
URL=http://www.memcached.org/files/${TARBALL}
wget -N --progress=dot:mega ${URL}
tar --strip-components=1 -xf ${TARBALL}
./configure --prefix=/usr
make
make install DESTDIR=/tmp/installdir
# Build the final package
fpm -s dir -t deb \
-n memcached \
-v ${PACKAGEVERSION} \
-C /tmp/installdir \
--description "high-performance memory object caching system"
[^version]: There are many ways to version a package. Again, if you
want to be pragmatic, the proposed solution should be good
enough for Ubuntu. On Debian, it doesn't cover upgrade
from one distribution version to another, but we assume
that nowadays, systems get reinstalled instead of being
upgraded.
You can review the whole package tree on [GitHub][memcached-basic-gh]
and build it with the `dpkg-buildpackage -us -uc -b` command or with
`GIT_PBUILDER_OPTIONS=--use-network=yes DIST=bionic git-pbuilder -us
-uc -b` if you already setup a tool like `pbuilder`.
# Pragmatic packages 102
At this point, we can iterate and add several improvements to our
*memcached* package. **None of them are mandatory** but they are
usually worth the additional effort.
## Build dependencies
Our initial build recipe only works when several packages are
installed, like `wget` and `libevent-dev`. They are not present on all
Debian systems. You can easily express that you need them by adding a
`Build-Depends` section for the source package in `debian/control`:
::control
Source: memcached
Build-Depends: debhelper (>= 11),
wget, ca-certificates,
libevent-dev
Always specify the `debhelper (>= 11)` dependency as we heavily rely on
it. We don't require `make` or a C compiler because it is assumed that
the `build-essential` meta-package is installed and it pulls
them. `dpkg-buildpackage` will complain if the dependencies are not
met. If you want to install these packages from your CI system, you
can use the following command:[^devscripts]
::sh
mk-build-deps \
-t 'apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -qqy' \
-i -r debian/control
[^devscripts]: You also need to install `devscripts` and `equivs` package.
You may also want to investigate [pbuilder][], [sbuild][] or
[whalebuilder][], three tools to build Debian packages in a clean
isolated environment.
## Runtime dependencies
If the resulting package is installed on a freshly installed machine,
it won't work because it will be missing *libevent*, a required
library for *memcached*. You can express the dependencies needed by
each binary package by adding a `Depends` field. Moreover, for dynamic
libraries, you can automatically get the right dependencies by using
some substitution variables:
::control
Package: memcached
Depends: ${misc:Depends}, ${shlibs:Depends}
The resulting package will contain the following information:
::console
$ dpkg -I ../memcached_1.5.16-0\~buster0_amd64.deb | grep Depends
Depends: libc6 (>= 2.17), libevent-2.1-6 (>= 2.1.8-stable)
## Integration with the init system
Most packaged daemons come with some integration with the init system.
This integration ensures the daemon will be started on boot and
restarted on upgrade. While Debian still supports several init sytems,
in my opinion, the most pragmatic one is **systemd**.
The content of a *systemd* unit should go in
`debian/memcached.service`. For example:
::ini
[Unit]
Description=memcached daemon
After=network.target
[Service]
Type=forking
Environment=PORT=11211
Environment=MAXCONN=1024
Environment=CACHESIZE=64
Environment=OPTIONS=
ExecStart=/usr/bin/memcached -d -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS
Restart=on-failure
User=_memcached
DynamicUser=yes
[Install]
WantedBy=multi-user.target
The `Type` directive is quite important. We used `forking` as
*memcached* is started with the `-d` flag and will fork when it is
ready to accept requests. If you use a non-forking daemon, you can
either use `notify` for a daemon with some support for *systemd* or
`simple` otherwise.
A user can customize one of the `Environment` directive by using
`systemctl edit memcached.service` to create an override which will be
installed in `/etc/systemd/system/memcached.service.d/override.conf`.
`systemctl cat memcached.service` can be used to check the final
definition of the unit.
The `DynamicUser` directive is also quite interesting. *systemd* will
automatically create a `_memcached` user[^naming] and will run the daemon as
this user. This frees us from managing a system user ourselves!
[^naming]: The Debian Policy doesn't provide any hint for the naming
convention of these system users. A common usage is to prefix the
daemon name with an underscore (like `_memcached`). Another common
usage is to use `Debian-` as a prefix. The main drawback of the
latter solution is that the name is likely to be replaced by the
UID in `ps` and `top` because of its length.
If no `User` directive is present, *systemd* will use the name
of the service as the name of the user. If you agree with this
choice, you can omit this directive.
You can review the whole package tree on
[GitHub][memcached-intermediate-gh] and build it with
the `dpkg-buildpackage -us -uc -b` command.
# Pragmatic packages 103
It is possible to leverage *debhelper* to **reduce the recipe size**
and to make it **more declarative**. This section is quite optional
and it requires understanding a bit more how a Debian package is
built. Feel free to skip it.
## The big picture
There are **four steps** to build a regular Debian package:
1. `debian/rules clean` should **clean** the source tree to make it
pristine.
2. `debian/rules build` should trigger the **build**. For an
autoconf-based software, like *memcached*, this step should
execute something like `./configure && make`.
3. `debian/rules install` should **install** the file tree of each
binary package. For an autoconf-based software, this step should
execute `make install DESTDIR=debian/memcached`.
4. `debian/rules binary` will **pack** the different file trees into
binary packages.
You don't directly write each of these targets. Instead, you let `dh`,
a component of *debhelper*, do most of the work. The following
`debian/rules` file should do almost everything correctly with many
source packages:
::make
#!/usr/bin/make -f
%:
dh $@
For each of the four targets described above, you can run `dh`
with `--no-act` to see what it would do. For example:
::console
$ dh build --no-act
dh_testdir
dh_update_autotools_config
dh_auto_configure
dh_auto_build
dh_auto_test
Each of these helpers has a manual page. Helpers starting with
`dh_auto_` are a bit "magic." For example, `dh_auto_configure` will
try to automatically configure a package prior to building: it will
detect the build system and invoke `./configure`, `cmake` or
`Makefile.PL`.
If one of the helpers do not do the "right" thing, you can replace it
by using an override target:
::make
override_dh_auto_configure:
./configure --with-some-grog
These helpers are also configurable, so you can just alter a bit their
behaviour by invoking them with additional options:
::make
override_dh_auto_configure:
dh_auto_configure -- --with-some-grog
This way, `./configure` will be called with your custom flag but also
with a lot of default flags like `--prefix=/usr` for better
integration. A manual page is available for each tool.
In the initial *memcached* example, we overrode all these "magic"
targets. `dh_auto_clean`, `dh_auto_configure` and `dh_auto_build` are
converted to no-ops to avoid any unexpected
behaviour. `dh_auto_install` is hijacked to do all the build
process. Additionally, we modified the behavior of the `dh_gencontrol`
helper by forcing the version number instead of using the one from
`debian/changelog`.
## Automatic builds
As *memcached* is an autoconf-enabled package, **`dh` knows how
to build it**: `./configure && make && make install`. Therefore, we
can let it handle most of the work with this `debian/rules` file:
::make
#!/usr/bin/make -f
DISTRIBUTION = $(shell sed -n "s/^VERSION_CODENAME=//p" /etc/os-release)
VERSION = 1.5.16
PACKAGEVERSION = $(VERSION)-0~$(DISTRIBUTION)0
TARBALL = memcached-$(VERSION).tar.gz
URL = http://www.memcached.org/files/$(TARBALL)
%:
dh $@ --with systemd
override_dh_update_autotools_config:
wget -N --progress=dot:mega $(URL)
tar --strip-components=1 -xf $(TARBALL)
override_dh_auto_test:
# Don't run the whitespace test
rm t/whitespace.t
dh_auto_test
override_dh_gencontrol:
dh_gencontrol -- -v$(PACKAGEVERSION)
The `dh_update_autotools_config` target is hijacked to download and
setup the source tree. We don't override the `dh_auto_configure` step,
so `dh` will execute the `./configure` script with the appropriate
options. We don't override the `dh_auto_build` step either: `dh` will
execute `make`. `dh_auto_test` is invoked after the build and it will
run the *memcached* test suite. We need to override it because one of
the test is complaining about odd whitespaces in the `debian/`
directory. We suppress this rogue test and let `dh_auto_test` executes
the test suite. `dh_auto_install` is not overriden either, so `dh`
will execute some variant of `make install`.
To get a better sense of the difference, here is a diff:
::diff
--- memcached-intermediate/debian/rules 2019-05-31 07:52:40.908868035 +0200
+++ memcached/debian/rules 2019-05-31 07:28:17.404380064 +0200
@@ -9,15 +9,14 @@
%:
dh $@
-override_dh_auto_clean:
-override_dh_auto_test:
-override_dh_auto_build:
-override_dh_auto_install:
+override_dh_update_autotools_config:
wget -N --progress=dot:mega $(URL)
tar --strip-components=1 -xf $(TARBALL)
- ./configure --prefix=/usr
- make
- make install DESTDIR=debian/memcached
+
+override_dh_auto_test:
+ # Don't run the whitespace test
+ rm t/whitespace.t
+ dh_auto_test
override_dh_gencontrol:
dh_gencontrol -- -v$(PACKAGEVERSION)
It is up to you to decide if `dh` can do some work for you, but
you could try to start from a minimal `debian/rules` and only override
some targets.
## Install additional files
While `make install` installed the essential files for *memcached*,
you may want to **put additional files** in the binary package. You
could use `cp` in your build recipe, but you can also declare them:
- files listed in `debian/memcached.docs` will be copied to
`/usr/share/doc/memcached` by `dh_installdocs`,
- files listed in `debian/memcached.examples` will be copied to
`/usr/share/doc/memcached/examples` by `dh_installexamples`,
- files listed in `debian/memcached.manpages` will be copied to the
appropriate subdirectory of `/usr/share/man` by `dh_installman`,
Here is an example using wildcards for `debian/memcached.docs`:
::text
doc/*.txt
If you need to copy some files to an arbitrary location, you can list
them along with their destination directories in
`debian/memcached.install` and `dh_install` will take care of the
copy. Here is an example:
::text
scripts/memcached-tool usr/bin
Using these files make the build process **more declarative**. It is a
matter of taste and you are free to use `cp` in `debian/rules`
instead. You can review the whole package tree on
[GitHub][memcached-gh].
# Other examples
The [GitHub repository][GitHub] contains some additional
examples. They all follow the same scheme:
- `dh_update_autotools_config` is hijacked to download and setup the source tree,
- `dh_gencontrol` is modified to use a computed version.
Notably, you'll find daemons in [Java][], [Go][], [Python][] and
[Node.js][]. The goal of these examples is to demonstrate that using
Debian tools to build Debian packages can be straightforward. Hope
this helps.
!!! "Update (2019.05)" This guide has been updated to use
`override_dh_update_autotools_config` to fetch source files instead of
`override_dh_auto_clean` as it interacts better with most build tools,
like `pbuilder` and `sbuild`.
*[VCS]: Version Control System
*[CI]: Continuous integration
[policy]: https://www.debian.org/doc/debian-policy/ "The Debian Policy Manual"
[newmaint]: https://www.debian.org/doc/manuals/maint-guide/ "New Maintainer's Guide"
[devref]: https://www.debian.org/doc/manuals/developers-reference/ "Developer's Reference"
[tutorial]: https://www.debian.org/doc/manuals/packaging-tutorial/packaging-tutorial "Lucas Nussbaum's packaging tutorial"
[Debian archive]: https://www.debian.org/distrib/packages "Debian archive"
[fpm]: https://github.com/jordansissel/fpm "Effing package management"
[CheckInstall]: http://checkinstall.izto.org/ "CheckInstall"
[GitHub]: https://github.com/vincentbernat/pragmatic-debian-packages "GitHub repository full of pragmatic Debian packages"
[memcached]: https://memcached.org/ "memcached - a distributed memory object caching system"
[memcached-basic-gh]: https://github.com/vincentbernat/pragmatic-debian-packages/tree/master/memcached-basic/debian
[memcached-intermediate-gh]: https://github.com/vincentbernat/pragmatic-debian-packages/tree/master/memcached-intermediate/debian
[memcached-gh]: https://github.com/vincentbernat/pragmatic-debian-packages/tree/master/memcached/debian
[pbuilder]: https://archive.today/d0i0A "pbuilder user's manual"
[sbuild]: https://wiki.debian.org/sbuild "sbuild page on Debian Wiki"
[whalebuilder]: https://www.uhoreg.ca/programming/debian/whalebuilder "Debian package builder using Docker"
[expect]: http://upstart.ubuntu.com/cookbook/#expect "Upstart cookbook: expect stanza"
[Java]: https://github.com/vincentbernat/pragmatic-debian-packages/tree/master/riemann/debian
[Go]: https://github.com/vincentbernat/pragmatic-debian-packages/tree/master/etcd/debian
[Python]: https://github.com/vincentbernat/pragmatic-debian-packages/tree/master/puppetboard/debian
[Node.js]: https://github.com/vincentbernat/pragmatic-debian-packages/tree/master/kiwiirc/debian
{# Local Variables: #}
{# mode: markdown #}
{# indent-tabs-mode: nil #}
{# End: #}
You can’t perform that action at this time.