Add build script to create .deb packages for Debian/Ubuntu style systems using FPM, changes to setup.py to facilitate this #73

gregcorbett · 2018-07-11T09:43:35Z

Resolves #60

This PR:

Adds a build script to build tagged versions of SSM. As with the RPM build script, testing on a non-tagged version requires some manual fudging (see manual testing below)
Modifies the setup.py script to facilitate generating packages via FPM by:
- Installing README.md under the directory /usr/share/doc/apel-ssm (rather than in a file)
- Preventing the install of the test package of SSM (i.e. the unittests)
- No longer installing system specific init files and moving apel directory creation into main()
Updates install documentation

Manual Testing

Running this stripped down version of the build script, that works within a checked out copy of the code base i.e. it doesn't go any fetch a version from GitHub.

apt-get install lintian
apt-get install ruby ruby-dev rubygems build-essential
gem install --no-ri --no-rdoc fpm -v 1.9.3

TAG=2.X.X-1

# Where to install the python lib files
PYTHON_INSTALL_LIB=/usr/lib/python2.7/dist-packages

# Split the tag into version and package number
# so they can be passed to fpm separately.
# This will work with RCs of the form X.Y.Z-0.A.rcA
VERSION=`echo "$TAG" | cut -d - -f 1`
ITERATION=`echo "$TAG" | cut -d - -f 2`

fpm -s python -t deb \
-n apel-ssm \
-v $VERSION \
--iteration $ITERATION \
-m "Apel Administrators <apel-admins@stfc.ac.uk>" \
--description "Secure Stomp Messenger (SSM)." \
--no-auto-depends \
--depends python2.7 \
--depends python-pip \
--depends python-ldap \
--depends libssl-dev \
--depends libsasl2-dev \
--deb-changelog CHANGELOG \
--python-install-bin /usr/bin \
--python-install-lib $PYTHON_INSTALL_LIB \
--exclude *.pyc \
setup.py

fpm -s pleaserun -t deb \
-n apel-ssm-service \
-v $VERSION \
--iteration $ITERATION \
-m "Apel Administrators <apel-admins@stfc.ac.uk>" \
--description "Secure Stomp Messenger (SSM) unit files." \
--no-auto-depends \
--depends apel-ssm \
/usr/bin/ssmreceive

lintian apel-ssm_${TAG}_all.deb

tofu-rocketry · 2018-08-13T12:46:54Z

README.md

+* `apt-get -f install`
+
+Install any missing python requirements that don't have system packages.
+* `pip install "stomp.py>=3.1.1" python-daemon python-ldap dirq`


None of these have Ubuntu packages?

stomp.py, no - there is a confusingly named python-stompy but its a complete different library
dirq - seemingly not

there does look like there are suitable packages for python-ldap and python-daemon, so I'll use them.

Mostly I just wanted to understand the differences. Can discuss later.

Ah I remember now, the python python-ldap requires the system python-ldap headers in order to install. That's why it's listed here and in the build script as a dependency.

And python-daemon is an optional requirement, so should stay out of the package dependencies as per #70.

I'll reword this line to be clearer

Infact, system python-ldap seems to be sufficient!

tofu-rocketry · 2018-08-13T12:47:58Z

README.md

+## Installing the DEB
+
+### Installation
+Install APEL SSM.


This and the following "Install..." lines should prob end in a colon as the code bit is a follow on.

tofu-rocketry · 2018-08-13T12:50:09Z

scripts/ssm-build-deb.sh

@@ -0,0 +1,84 @@
+#!/bin/bash
+
+# Execute the following as root to install lintian and fpm


Colon at end.

tofu-rocketry · 2018-08-13T12:50:33Z

scripts/ssm-build-deb.sh

+# apt-get install ruby ruby-dev rubygems build-essential
+# gem install --no-ri --no-rdoc fpm
+
+# Then run this file, altering the version.


Do you mean the version in TAG?

comment clarified

tofu-rocketry · 2018-08-13T12:51:21Z

scripts/ssm-build-deb.sh

+# This file will create two versions of the deb file:
+# - apel-ssm_<version>_all.deb contains all the files necessary to run a
+#   the SSM as a sender.
+# - apel-ssm-service_<version>_amd64.deb will install service files


Why is this one AMD 64 specific?

Because this is the more system specific / service files. The exact value should probably be a configurable option.

Oh, you mean just because this is what we use?

although it looks like we can package it as all, the deb equivalent of noarch

Ah, ok, that would seem to be more consistent with the RPMs.

tofu-rocketry · 2018-08-13T12:53:52Z

scripts/ssm-build-deb.sh

+
+# Split the tag into version and package number
+# so they can be passed to fpm separately.
+# This will work with RCs of the form X.Y.Z-0.A.rcA


And any other package that uses our version scheme? e.g 1.2.3-1.1.alpha1

it should do, I think it will split on the -. If it does I'll clarify the comment.

tofu-rocketry · 2018-08-13T12:55:06Z

scripts/ssm-build-deb.sh

+-v $VERSION \
+--iteration $ITERATION \
+-m "Apel Administrators <apel-admins@stfc.ac.uk>" \
+--description "Secure Stomp Messenger (SSM) unit files." \


Unit files?

the system files that let you do service apel-ssm start

I thought there weren't any for Ubuntu.

No there are, they just differ from the RedHat(6) style file the RPM would install

Renamed "unit files" to "Service Daemon files"

tofu-rocketry · 2018-08-13T12:56:02Z

setup.py

@@ -28,15 +30,6 @@ def main():
        copyfile('scripts/apel-ssm.logrotate', 'conf/apel-ssm')
        copyfile('README.md', 'apel-ssm')

-        if not path.exists('/var/log/apel'):


So this has been replaced by the bit below in data_files? Is that the better way to do it?

I believe so, it lets setuptools worry about the directory creation rather than doing it 'manually'

tofu-rocketry · 2018-08-13T12:56:35Z

setup.py

          scripts=['bin/ssmreceive', 'bin/ssmsend'],
          data_files=[(conf_dir, conf_files),
                      ('/etc/logrotate.d', ['conf/apel-ssm']),
-                      ('/etc/init.d', ['bin/apel-ssm']),
-                      ('/usr/share/doc', ['apel-ssm'])],
+                      ('/usr/share/doc/apel-ssm', ['apel-ssm']),


Where'd the docs go before this change?

it went in a file under /usr/share/doc rather than a file under /usr/share/doc/apel-ssm.

Ok, that sounds better now.

tofu-rocketry · 2018-08-13T12:59:08Z

scripts/ssm-build-deb.sh

+-n apel-ssm \
+-v $VERSION \
+--iteration $ITERATION \
+-m "Apel Administrators <apel-admins@stfc.ac.uk>" \


There's a lot of config here that seems like it would be better handled with a config file or as constants earlier in the script. Email is repeated.

Can the depends be given as a list?

It probably would be better as a config file, or at the very least reducing the duplication of the email.

I don't think so, the documentation around depends says A dependency. This flag can be specified multiple times.

Hmm, is there an option to read a list of dependencies or anything like that?

I don't think so. Ideally, dependencies would be read from the setup.py script but as some of the python packages don't have a system packages I didn't use that feature.

How is best to do this as a config file? Sourcing a separate file that list the variables? Like this:

TAG=... SOURCE_DIR=... BUILD_DIR=... # etc etc

?

As there's no built in way, I'd leave it as is.

tofu-rocketry · 2018-08-13T13:00:09Z

The Codacy suggestion could also be fixed: https://app.codacy.com/app/apel/ssm/pullRequest?prid=1888023

tofu-rocketry · 2018-08-16T10:29:58Z

scripts/ssm-build-deb.sh

@@ -61,21 +62,21 @@ fpm -s python -t deb \
 --python-install-bin /usr/bin \
 --python-install-lib $PYTHON_INSTALL_LIB \
 --exclude *.pyc \
+--package $BUILD_DIR \


This is a lot neater. 👍

tofu-rocketry · 2018-08-16T10:40:59Z

Think that's everything covered. Shall we rebase?

- ubuntu (and RedHat) seem to prefer this to file directly under /usr/share/doc/apel-ssm - and fpm errors because it seems to assume apel-ssm is a directory (see jordansissel/fpm#586)

- it is very RedHat/Centos/SL specific and whereas the setup.py script is meant to be a more general way of installing ssm - it also seems to cause problems with fpm and the packages generated by pleaserun as the bad init script conflicts with the unit files - add a note saying this script does not install init scripts

- so packages created by fpm will also create them

- so fpm generated package is appropriately named

- as it mangle them and those it doesn't mangle aren't packages under ubunutu so need to be installed by pip

- as pip is needed to install the python requirements and the system python-ldap package is needed for the python python-ldap to install

- to prevent a nasty surprise on systems where python 3 is the default python

- in its previous location, it seemed to have no effect. - it has been moved to before any "depends" options

- following review

- rather than moving it

- as suggested by Codacy

- as it is not limited to amd64 style Ubuntu systems - `all` is the deb equivalent of `noarch`

gregcorbett assigned tofu-rocketry Jul 11, 2018

gregcorbett requested a review from tofu-rocketry July 11, 2018 09:43

tofu-rocketry added this to the 2.3.0 milestone Jul 13, 2018

tofu-rocketry reviewed Aug 13, 2018

View reviewed changes

gregcorbett force-pushed the deb-fpm branch from ed6021f to 8726c0d Compare August 15, 2018 10:15

tofu-rocketry reviewed Aug 16, 2018

View reviewed changes

gregcorbett added 22 commits August 16, 2018 10:51

Install README.md under /usr/share/doc/apel-ssm

6dd75a2

- ubuntu (and RedHat) seem to prefer this to file directly under /usr/share/doc/apel-ssm - and fpm errors because it seems to assume apel-ssm is a directory (see jordansissel/fpm#586)

Prevent install of test as a python package

2ab8c24

Move directory making into the setup call

5a8125b

- so packages created by fpm will also create them

Add first attempt at debian building with fpm

ff165a1

Rename package to match rpm

d562ec2

Handle version and package number separately

2f051ab

- so fpm generated package is appropriately named

Add some extra cleanup

000bf2a

Add python as a dependency for the deb package

63f876a

Make the python lib directory configurable

d97bfbf

Prevent fpm parsing the setup.py requirements

02ff4d4

- as it mangle them and those it doesn't mangle aren't packages under ubunutu so need to be installed by pip

Exclude pyc files from built package

88fc35c

Make fpm generate unit files for Ubuntu in a .deb

52b8053

Add a note explaining how to install FPM

7612215

Move build script to scripts/ and add an extension

cbc69f7

Add pip and python-ldap as debian dependancies

a75e402

- as pip is needed to install the python requirements and the system python-ldap package is needed for the python python-ldap to install

Remove erroneous line escape

79c7341

Explain difference between the two debs produced

ecdebe5

Specify python version SSM depends on

cdb0aaf

- to prevent a nasty surprise on systems where python 3 is the default python

Add documentation around installing from the .deb

8e57f60

Move no-auto-depends option to after description

7b487de

- in its previous location, it seemed to have no effect. - it has been moved to before any "depends" options

Add remaining system requirements for python ldap

957725a

gregcorbett added 6 commits August 16, 2018 10:51

Comment and description clarifications

b40d763

- following review

Build package directly in $BUILD_DIR

46f1602

- rather than moving it

Use $(..) instead of legacy ..

f2fc605

- as suggested by Codacy

Make service deb architecture independant

90f675c

- as it is not limited to amd64 style Ubuntu systems - `all` is the deb equivalent of `noarch`

Clarify documentation on installing pip packages

3affd18

Update TAG to the current one

5448891

gregcorbett force-pushed the deb-fpm branch from f41af8e to 5448891 Compare August 16, 2018 10:54

tofu-rocketry approved these changes Aug 16, 2018

View reviewed changes

tofu-rocketry merged commit 6aee1f9 into apel:dev Aug 16, 2018

gregcorbett deleted the deb-fpm branch August 16, 2018 11:51

		@@ -0,0 +1,84 @@
		#!/bin/bash

		# Execute the following as root to install lintian and fpm

Add build script to create .deb packages for Debian/Ubuntu style systems using FPM, changes to setup.py to facilitate this #73

Add build script to create .deb packages for Debian/Ubuntu style systems using FPM, changes to setup.py to facilitate this #73

Conversation

gregcorbett commented Jul 11, 2018

Manual Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gregcorbett Aug 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tofu-rocketry Aug 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tofu-rocketry commented Aug 13, 2018

Choose a reason for hiding this comment

tofu-rocketry commented Aug 16, 2018

gregcorbett Aug 13, 2018 •

edited

Loading

tofu-rocketry Aug 13, 2018 •

edited

Loading