Support for Launchpad's PPA's

[brainstorm]

Hello Jordan,

I think it would be extremely useful if fpm generates a minimal yet correct .changes file for .deb packages. That way one could just run:

dput ppa:personalppa/repository package.changes

And a FPM-packaged package is automatically exposed to the debian world by just doing:

add-apt-repository ppa:personalppa/repository && apt-get update && apt-get install package

I have been trying with "dpkg-genchanges", but it requires correct "control" files, changelogs, etc... the typical hassle of a traditional (manual) package building. Is there already a way to have this process automated within FPM ?

Thanks for your tool(s) !

[jordansissel]

+1, this feature sounds awesome.

I'll see what I can do. In the meantime if anyone has specific implementation ideas or patches I'm happy to entertain those :)

[brainstorm]

Cool ! I have been further playing with "apt-get source tofrodos" and then dpkg-buildpackage to generate the .changes file.

Here's a sample .changes file we can use as a template for FPM.

Format: 1.8
Date: Thu, 21 Jan 2010 21:25:26 +0100
Source: tofrodos
Binary: tofrodos
Architecture: source amd64
Version: 1.7.8.debian.1-2
Distribution: unstable
Urgency: low
Maintainer: Alexander Reichle-Schmehl 
Changed-By: Alexander Reichle-Schmehl 
Description: 
 tofrodos   - Converts DOS <-> Unix text files, alias tofromdos
Changes: 
 tofrodos (1.7.8.debian.1-2) unstable; urgency=low
 .
   * Change maintainer name
   * Drop dos2unix and unix2dos symlinks, to allow introduction of original
     dos2unix package. (remove dh_link call and add NEWS about that)
   * Add README.source
   * Bump standards version to 3.8.3 (no further changes needed)
Checksums-Sha1: 
 20296fc485696be36e3471f6fa960698184ccd53 990 tofrodos_1.7.8.debian.1-2.dsc
 617eabee26c2c0531c0ed362afe81a322eab7d48 5919 tofrodos_1.7.8.debian.1-2.diff.gz
 b210ce619db57c9831b50f47b5bdf5ecb1aacf2e 20874 tofrodos_1.7.8.debian.1-2_amd64.deb
Checksums-Sha256: 
 89c09839cb84e340181762c50bfa77a266282146ce405bb6cb8d20714229d47a 990 tofrodos_1.7.8.debian.1-2.dsc
 c3eac567b6473d60f884f961d5db50bcf9fb57736164e183b1f01bc62786daad 5919 tofrodos_1.7.8.debian.1-2.diff.gz
 5c39de04230c6b05bb6c7d58fa026cc122ea12e1a66ca0bb48c1ef8e80c206bd 20874 tofrodos_1.7.8.debian.1-2_amd64.deb
Files: 
 64f74dac9de415f486342757464049ae 990 utils optional tofrodos_1.7.8.debian.1-2.dsc
 828cf376024253b30a63dbbc59094fd2 5919 utils optional tofrodos_1.7.8.debian.1-2.diff.gz
 938c5b941d8de0e552501cc63226beb9 20874 utils optional tofrodos_1.7.8.debian.1-2_amd64.deb

All of it has to be signed by the author's GPG key by using "gpg --clearsign template.changes", as far as I've seen. Fetching the info from the existing md5sums and control files (generated by FPM) should be sufficient to construct such a file and pass the different dput checks.

Cheers !
Roman

[brainstorm]

Apparently launchpad does not accept .deb files, so putting together a correct .changes file listing the .deb does not upload the file to the PPA :-(

$ tail -40 snpeff_2.0.5d.changes
Files:
 0908fb2b61df7fb48c5d8c4f02ce17d2 3951712 utils optional snpeff_2.0.5d_amd64.deb
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
(...)

$ dput ppa:brainstorm/galaxy snpeff_2.0.5d.changes
Uploading to ppa (via ftp to ppa.launchpad.net):
Successfully uploaded packages.

https://help.launchpad.net/Packaging/PPA/Uploading

(...) Note that Launchpad builds the packages onsite, and does not accept deb files.

[jordansissel]

yeah I think you have to upload a dsc file? I haven't looked much into it myself because building debian packages the 'debian way' requires so much silly ceremony.

[dougburks]

+1 This feature would be of tremendous value to us in the Security Onion distro (currently based on Xubuntu 10.04). We've been using fpm pretty heavily for the past year and uploading the debs to Sourceforge (faking a "repo" using a custom bash script). Now that Ubuntu 12.04 is out, we're going to be rebuilding the entire distro using a Launchpad PPA and if we could continue using fpm this would be a HUGE time savings for us. Please let me know if there is anything I can do to help! Thanks!

[jordansissel]

If someone can distill the PPA usage instructions into something simple (perhaps a small list of tasks) that could help. Everyt ime I look at the PPA and debian packaging docs, I get tremendously bored (they're long and obfuscated) and go on to doing something else ;)

[dougburks]

I definitely feel your pain :) I've been trying to wade into the ocean of PPA docs and it has been arduous.

These are the best documents I've found so far (still overly long, but I was able to follow the instructions and get a working deb):
http://developer.ubuntu.com/packaging/html/getting-set-up.html
http://developer.ubuntu.com/packaging/html/packaging-new-software.html

[dougburks]

Here are some notes from Peter Manev of the Suricata project:

1.
apt-get install wget dput build-essential autoconf automake autotools-dev dh-make debhelper devscripts fakeroot xutils lintian pbuilder
2. Create a Launchpad ID/account
3. Sign the ubuntu code of conduct
4.Upload your public PGP key - used to sign the pkgs and cross check with your upload.
5. Upload your ssh key (public) - used to upload/verify that you are the one uploading
6. create a PPA and name it (in our case "suricata-stable")
!!! NOTE: once you delete a PPA repo  - you CAN NOT create/recreate another one with the same name !!! so get use to renaming rather than deleting a name.
7. run the script locally on your PC as it is - it will create a source pkg  - have a look :) .(dpkg-buildpackage -S is important as i went through sleepless nights to battle things untill I found out that dpkg-buildpackage -S is what is needed not dpkg-buildpackage -F (full, creates binaries/source by default) ) - this is important when you upload, as multiple uploads WILL fail that way
8. in the script, at the bottom is explained how to upload it to PPA. - I personally used a simple ftp, you can try sftp if you would like.
9. after it is successfully uploaded it will build the pkg for the specified distribution (look in the script, " suricata (1.2.1-4ubuntu4) maverick; urgency=low' changelog") , you will receive an e-mail.
10. after that PPA will auto build the pkg for 32/64 bit Maverick (in our case).
11. after the build is done - you can copy (make sure when you copy, you copy just the binaries!!!) the binaries to other distros (your choice of distros) - use "copy package", from the "View package details" menu.
12. after you copy them - it will automatically build the pkgs - probably 30min to 1hr wait for that, there might be a queue on the particular server.

[dougburks]

As a followup to my previous comment, here is Peter's script for uploading Suricata to Launchpad:

#!/bin/bash
# make sure we have the following pkgs installed
# apt-get install wget dput build-essential autoconf automake
autotools-dev dh-make debhelper devscripts fakeroot xutils lintian
pbuilder

# below we check if we have all the pkgs needed:

# #get current rev number
# rev_num="$(git log |head -1 |awk '{print $2}' | cut -c1-7)"
# # one dir up, rename the directory to be debpkg suitable and make a
zip file out of it
# cd .. && mv oisf/ suricata-rev-$rev_num
# tar -czf suricata-rev-$rev_num.tar.gz suricata-rev-$rev_num/

if   ` wget http://www.openinfosecfoundation.org/download/suricata-1.2.1.tar.gz`
 ; then
       echo "Downloaded Suricata."

else
       echo "Could not download! Aborting. Check your connection and try again." 1>&2
       exit 1
fi

 tar -zxf suricata-1.2.1.tar.gz
 cd suricata-1.2.1

echo "" | dh_make -s -c gpl2 -n -e first.last@someemail.com -f
../suricata-1.2.1.tar.gz
#the echo above is in order to simulate pressed key "Enter", that
dh_make is waiting for ..so that it can continue

#get all the packages we need to add to the control file
dpkg-depcheck -d ./configure >> pkgstoadd.txt
to_add="$(sed -n '/Packages needed:/,$p' pkgstoadd.txt)"
rm pkgstoadd.txt

#prints all the packages that are needed, without "Pckages needed:"
and make the pkgs comma separated
pkgs_to_add="$(echo $to_add |awk '{$1=$2=""; print $0}' |awk
'{for(i=1;i<=NF;i++){$i=$i","}  }1')"
addme="libpcre3-dbg, libpcre3-dev, libpcap-dev, libnet1-dev,
libyaml-0-2, libyaml-dev, zlib1g, zlib1g-dev, libcap-ng-dev,
libcap-ng0, libhtp1, libhtp-dev "
pkgs_to_add_more=$pkgs_to_add$addme

cd debian

#add specific ./configure options before the package is build
#ex ./condifure --enable-sexy
# this is how it is done - period . !!
echo "override_dh_auto_configure:" >> rules
echo "  dh_auto_configure -- --enable-non-bundled-htp
--disable-gccmarch-native" >> rules
#http://www.debian.org/doc/manuals/maint-guide/dreq.en.html#rules

#add the proper distro name in changelog - Ubuntu PPA MUST have that
change !!!!!
sed -i '/suricata (1.2.1) unstable/c suricata (1.2.1-4ubuntu4)
maverick; urgency=low' changelog
#notice "4ubuntu4", this number should be incremented so that we can upload
#multiple pkg updates of ver 1.2.1 for example
#add the pkgs to line #5 in the control file
sed -i "/Build-Depends:/s/$/, $pkgs_to_add_more/" control

#changing the repo source to "devel" for ubuntu, must be "devel" for
PPA, does not accept anything else
sed -i '/Section: unknown/c Section: devel' control

#add proper Mantainer of the pkg so that it could be signed with the
pgp key!! You need UID - run " gpg --list-secret-keys " to get it
sed -i '/Maintainer:/c Maintainer: First Last <first.last@someemail.com>' control

# add website address - replace "Homepage:" with "Homepage:
http://www.openinfosecfoundation.org/"
sed -i '/Homepage:/c Homepage: http://www.openinfosecfoundation.org/' control

#the same for "Description:"
sed -i '/Description:/c Description: Suricata open source multi-thread
IDS/IPS.' control


#delete all lines after the Description, without the description itself
sed -i -n -e '1,/Description: Suricata open source multi-thread
IDS\/IPS/p' control

#append extra describtion
echo " The Suricata Engine is an Open Source Next Generation Intrusion
Detection and Prevention Engine.
 This engine is not intended to just replace or emulate the existing
tools in the industry, but will bring
 new ideas and technologies to the field. OISF is part of and funded
by the Department of
 Homeland Security's Directorate for Science and Technology HOST
program (Homeland Open Security
 Technology), by the the Navy's Space and Naval Warfare Systems
Command (SPAWAR), as well as through the
 very generous support of the members of the OISF Consortium. More
information about the Consortium is
 available, as well as a list of our current Consortium Members. The
Suricata Engine and the HTP Library
 are available to use under the GPLv2." >> control

cd ..
#git log |head -60 >> debian/changelog
#there is something that it does not like the way we edit the chanlog,
but it works non the less, we can just remove that line

# XXXXXXXX - is your key id - this is how you force signing
"-kXXXXXXXX", no space !!
# -S option build source only - it does not build the ****.deb pkg -
which we do not need for Ubuntu PPA
dpkg-buildpackage -S -rfakeroot -kXXXXXXXX

#uploads the file directly...
#cd ..
#dput suricata-ppa:oisf/suricata-ids-ips
suricata_1.2.1-1ubuntu14_source.changes

# I M P O R T A N T ! ! !
#
# MAKE YOUR dput.cf (located in /etc/dput.cf) look like this
#find [ppa] section and make it like that

# [suricata-ppa]
# fqdn                  = ppa.launchpad.net
# method                        = ftp
# incoming              = ~oisf/suricata-stable/ubuntu
# login                 = anonymous
# allow_unsigned_uploads        = 0

#also -
#http://www.debian.org/doc/debian-policy/ch-archive.html#s-subsections
#to check out specifically for debian "section" values in the "control" file
#http://www.debian.org/doc/manuals/maint-guide/dreq.en.html

[nlaurance]

+1. Would be a very nice feature

[rawberg]

+1. For now I'm going to just host a repo off of S3 so I can keep using FPM but this would be really nice to have!

[jordansissel]

I'm actually hoping for the success of another project - https://github.com/dnbert/prm - which aims to provide sane package repository management for rpm/deb/solaris.

[jordansissel]

I'm actually hoping for the success of another project - https://github.com/dnbert/prm - which aims to provide sane package repository management for rpm/deb/solaris.

[spaceapedev]

+1 fpm to ppa would be a massive win!

[joshskinner-wf]

+1 this feature due to the fact i'm using mini-dinstall to run my repo. would love to see if prm plays out, but for some reason prm fails to run for me. i'm a ruby noob tho

[dsturnbull]

prm worked excellently for this problem.

[thedrow]

+1!

[jordansissel]

I'll try answer the current state of things, first, and then answer what we can do about it.

For starters, PPA is a build system. It takes debian source packages (source code + instructions on how to build it using debian tooling) and produces binary packages for multiple platforms. This is distinctly different from what fpm does, which is to take a ready-to-package product and packages it.

PPA, in a way, is more similar to what fpm-cookery does, with the exception that PPA provides hosting for the package repositories as well.

I believe you folks are really wanting is simpler package repository maintenance and hosting. PPA is not simpler, in my opinion, but my opinions aside PPA still is technically limited to deb package formats. The dnbert/prm project is not limited to deb package formats and helps you host your own repos in S3 or over http.

---

We could hack it such that fpm could produce artifacts acceptable for upload into PPA, but because fpm starts from a ready-to-package origin, the artifacts produced by PPA would be limited to one architecture. This may be acceptable for architecture-independent code like ruby/python/java/perl projects. HOWEVER...

If you were to try publishing a single ruby app packaged by fpm, it would have dependencies that were probably not available anywhere else, so you would be inclined to package those with fpm and push them into your PPA which might get large or hilarious, depending on your perspective!

Launchpad PPA also threatens to delete any and all packages if they don't like what you're doing, and skipping Debian's packaging practices and ceremonies seems exactly like something someone with a vested interest in PPA would deem delete-worthy, but maybe I'm just paranoid ;)

I'm certain there are easier ways to provide package hosting and repo maintenance, and as said, prm gets you pretty close, and is open source.

[thedrow]

Nevertheless, PPAs makes it easy for the end user to install, manage and maintain dependencies when using Debian. It might be harder to maintain for the developers but it is much easier to use for the end-users which is what's important for most developers.
There should be a tool that allows you to one-click deploy to a PPA without actually bothering to learn how it's implemented.

[jordansissel]

WHen you say PPA, I really hear "apt repository" because PPA isn't the thing you say you are caring about, you want an easy way to install and maintain packages as a Debian user.

It isn't much work to use reprepro or apt-ftparchive to provide your own repository. Further, tools like prm and freight make this process even easier.

If you could clarify your requirements, I could more adequately respond to them. "Must support PPA" is a very specific requirement when I'm certain you're asking for easier generalized repository management.

[thedrow]

No, I say PPA because I want it to be on Launchpad because:

1. It's free and maintaining a repository on S3 isn't.
2. Every Debian user knows Launchpad.
3. Google knows Launchpad well which makes my packages easily findable.
4. It's easier to use using apt-add-repository.

[jordansissel]

It is worth approaching the launchpad folks separately and asking them to accept pre-built debs. If you are interested, please do this.

[thedrow]

Won't that take longer?

[jordansissel]

It isn't a simple thing to make fpm emit something consumable by launchpad for technical reasons already presented in this issue. I welcome any hacks or prototypes toward this effort, though

I will work on a hack for his eventually, time and energy permitting.

[CarlFK]

I have wallowed around trying to maintain my own PPA, so I appreciate the effort required.
given it doesn't look like the local devs are too interested in wallowing around (and I don't blame them, see above) I think it is best to stop bugging them and put that effort into finding someone proficient in packaging who will at a minimum get a single build up, and then submit the results here.

[klen]

+1 to the issue.

[brodock]

+1 thanks for all the documentation

[jordansissel]

There are two fantastic alternatives to Launchpad PPA now:

• bintray.com
• packagecloud.io

Both provide easy hosting for packages that work great with things produced with fpm. Muuuch simpler than PPA, imo, and more friendly to your target audience.

[indykish]

I was planning to setup my own repo using http://www.aptly.info/ behind nginx to upload debs but @jordansissel your info helped me to move to bintray.com. It simple and easy. packagecloud.io has a limit.

[burke]

In case it helps anyone that stumbles upon this issue in the future, I'm using this script as a simple-ish way to generate a .changes file:

#!/bin/bash
set -euo pipefail

distro=$1;shift
deb=$1;shift
test "${deb}"
test "${distro}"

dir=$(mktemp -d)
trap 'rm -rf "${dir}"' EXIT

cp "${deb}" "${dir}"
cd "${dir}"

ar x *.deb
tar xf control.tar.gz
tar xf data.tar.gz
cat usr/share/doc/*/changelog.Debian.gz | zcat > changelog

pkgname=$(grep -oP "(?<=Package: )(.*)" control)
echo "Source: ${pkgname}" >> control

dpkg-genchanges -lchangelog -ccontrol -u. -f<(echo *.deb admin extra) -b -q "-DDistribution=${distro}" "-DBinary=${pkgname}" 2>/dev/null

Call like fpm2changes trusty *.deb.

[jas99]

+1