Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Add Arch Linux packaging support #231

Closed
gtaylor opened this Issue May 30, 2012 · 39 comments

Comments

Projects
None yet
8 participants

gtaylor commented May 30, 2012

I'd settle for just the PKGBUILD file generation, but generation of the package archives would be trivial to add if we could generate PKGBUILD files.

Owner

jordansissel commented May 30, 2012

PKGBUILD files do what fpm does, so it seems odd to try and build those with fpm. fpm could be used to build the resulting package archives, though.

On the flipside, fpm-cookery could be used to generate PKGBUILD files since fpm-cookery is a general way to specify build/compile/download/etc steps

gtaylor commented May 30, 2012

Ah, I didn't realize that fpm-cookery was a thing. So what you're saying is I probably need to open an issue with fpm-cookery to support output, then get more specific here and ask for the actual package building (*.xz file) support, using the PKGBUILD support from fpk-cookery?

Owner

jordansissel commented May 30, 2012

well, fpm-cookery takes care of the 'download, build, install' steps, and it uses fpm as a final step to turn the result into a package. Likewise, PKGBUILD takes care of the 'download, build, install' steps, so it seems reasonable to have fpm-cookery output a PKGBUILD instead of actually executing those steps.

Would be cool, anyway :)

Contributor

hatt commented Aug 6, 2013

From duplicate #514, Pacman allows a pre/post upgrade script, which is different to install/remove, or calling both of them. Is it worth having a separate flag specific to Arch for that, or adding it to the pre/post hooks?

The biggest obstacle I can see is that OSX doesn't have GNU's du. gdu is an option (like gnutar I guess gets used?) but it doesn't read raw content sizes on disk properly. From what I can tell this is a constraint of HFS+, but I'm not sure why ext{2..4}, reiserFS, and at least XFS all report the same size for du -sk --apparent-size while HFS+ doesn't. I wonder if it's the way sparse files are handled? Pacman will reject packages where the size metadata doesn't match, so I'm open to any suggestions on how to handle this.

Owner

jordansissel commented Aug 6, 2013

the deb support in fpm already computes size: https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/deb.rb#L517-L522

We can probably reuse this?

Contributor

hatt commented Aug 6, 2013

I'm really confused by it now. I tried getting that size and comparing it to a known Arch package but three different size checks return three different values. From what I can tell, the deb way of getting it doesn't use 1024 byte sized blocks, so it's quite far off. Doing it raw with gdu seems to be off by exactly 1024 bytes in this test which is interesting, but this doesn't appear consistent across packages. If we ignore OSX for now, it seems to be ok in my testing on Ubuntu, but CentOS doesn't like it. Not sure what's up but I'm gonna put some time into Vagrant and filesystem experiments this week and dig deeper. It looks like makepkg tries to be FS agnostic by taking the apparent size across 1024 byte blocks and multiplying it by 1024, but as HFS+ doesn't support sparse files this seems arbitrary at best in OSX. Not sure what's up in CentOS yet.

hatt@momiji:~/namcap$ grep size .PKGINFO | awk '{print $3}'
447488
hatt@momiji:~/namcap$ ruby test.rb
439674
hatt@momiji:~/namcap$ gdu -sk --apparent-size usr/ | awk '{print $1*1024}'
448512
Contributor

djhaskin987 commented Dec 13, 2013

+1 for this ticket, it would be totally useful to make arch packages at work. Using docker, we can create containers for services, and Arch would be perfect OS for a container, since it's minimalistic and rolling-release.

Collaborator

r4um commented Feb 27, 2014

@hatt just doing a File.stat summing up the sizes and rounding to nearest multiple of 1024 should be enough.
The size metadata for archlinux package is just a hint doesn't influence any package behavior
(install/upgrade etc.).

Contributor

hatt commented Mar 7, 2014

Oh really? It didn't even occur to me to check if Arch actually cared that the metadata was valid or not. Now that I think about it, only the checksum should matter, as long as Pacman isn't overly strict. I can build packages now, will add a PR over the weekend.

As for pre/post scripts, do we want to handle the Arch-specific case of install/upgrade/remove?

Collaborator

r4um commented Mar 7, 2014

Oh really? It didn't even occur to me to check if Arch actually cared that the metadata was valid or not. Now that I think about it, only the checksum should matter, as long as Pacman isn't overly strict. I can build packages now, will add a PR over the weekend.

I only checked the significance of size metadata (in pacman source), majority of rest of them are important.

As for pre/post scripts, do we want to handle the Arch-specific case of install/upgrade/remove?

Would be nice to have :).

Contributor

hatt commented Mar 7, 2014

I'd just be worried about confusion for users. RPM/Deb just use the remove and then install scripts on an upgrade, only Arch has the special case of upgrade scripts. Would be strange to prefix it though I think.

Owner

jordansissel commented Mar 7, 2014

Well, both rpm and deb invoke the install script in a special way for
upgrades, similar for remove scripts. I forget the details though since I
never use this feature.

It's possible we can abstract this a bit to support arch, rpm, and deb
sanely

-Jordan

On Thursday, March 6, 2014, Matt Sharpe notifications@github.com wrote:

I'd just be worried about confusion for users. RPM/Deb just use the remove
and then install scripts on an upgrade, only Arch has the special case of
upgrade scripts. Would be strange to prefix it though I think.

Reply to this email directly or view it on GitHubhttps://github.com/jordansissel/fpm/issues/231#issuecomment-36969469
.

Contributor

djhaskin987 commented Mar 9, 2014

Although arch has install-upgrade-remove explicitly, actually RPM does, too. On upgrade, RPM passes the post-install script a "2" argument, meaning that two installations are on the same machine at once (the upgrade case). Based on the argument, RPM scripts can tell if it's an install case, an upgrade case, or a removal case. I can't speak to .deb packages, but maybe including install-upgrade-remove might be applicable to more than just Arch. It actually sounds quite useful, instead of having to muck about with "If upgrade, do this, else if install, do this" in my RPM post scripts.

Owner

jordansissel commented Mar 10, 2014

@djhaskin987 indeed! Thanks for providing the details on rpm that I had forgotten.

This describes how deb does things: http://www.debian.org/doc/debian-policy/ch-maintainerscripts.html - Deb has many ways it invokes the scripts, many of which I think aren't strictly useful for most humans.

It's likely we can abstract the common things (install, upgrade, remove) and make fpm create the appropriate madness in the underlying package to reflect our request. +1 on doing this

We should file a separate ticket for the install/upgrade/remove discussion though, for simplicity!

Owner

jordansissel commented Mar 10, 2014

For future install/upgrade/remove script actions, let's discuss them here: #661

Contributor

djhaskin987 commented Aug 27, 2014

Has there been any movement on this issue? It would truly make FPM better :)

Contributor

djhaskin987 commented Nov 27, 2014

K, I'm gonna get on this

Owner

jordansissel commented Nov 27, 2014

@djhaskin987 wooo <3

Contributor

hatt commented Nov 28, 2014

Awesome!

Contributor

djhaskin987 commented May 3, 2015

Just a status update. all my work for this ticket is on the branch feature/arch-support on my forked version of this repository. The work is nearly complete; I just need to implement an arch specific option, add some more unit tests, fix the ones I've added, and test my work generally. Almost there!

Contributor

djhaskin987 commented May 7, 2015

@hatt @jordansissel and anyone else who wants to help, I could use an extra pair of eyes.
My feature branch is so very near complete. I just hit this one last snag:

I wrote a test to make sure that permissions are properly preserved from the staging path's files:

Failures:

  1) FPM::Package::Pacman#output file permissions should preserve file permissions for /usr
     Failure/Error: dir)).mode & 07777 } == perm
     Insist::Failure:
       Expected 493, but got 509
     # ./spec/fpm/package/pacman_spec.rb:164:in `block (5 levels) in <top (required)>'

  2) FPM::Package::Pacman#output file permissions should preserve file permissions for /usr/bin
     Failure/Error: dir)).mode & 07777 } == perm
     Insist::Failure:
       Expected 493, but got 509
     # ./spec/fpm/package/pacman_spec.rb:164:in `block (5 levels) in <top (required)>'

  3) FPM::Package::Pacman#output file permissions should preserve file permissions for /usr/lib
     Failure/Error: dir)).mode & 07777 } == perm
     Insist::Failure:
       Expected 493, but got 509
     # ./spec/fpm/package/pacman_spec.rb:164:in `block (5 levels) in <top (required)>'

  4) FPM::Package::Pacman#output file permissions should preserve file permissions for /usr/share
     Failure/Error: dir)).mode & 07777 } == perm
     Insist::Failure:
       Expected 493, but got 509
     # ./spec/fpm/package/pacman_spec.rb:164:in `block (5 levels) in <top (required)>'

  5) FPM::Package::Pacman#output file permissions should preserve file permissions for /usr/share/doc
     Failure/Error: dir)).mode & 07777 } == perm
     Insist::Failure:
       Expected 493, but got 509
     # ./spec/fpm/package/pacman_spec.rb:164:in `block (5 levels) in <top (required)>'

  6) FPM::Package::Pacman#output file permissions should preserve file permissions for /etc
     Failure/Error: dir)).mode & 07777 } == perm
     Insist::Failure:
       Expected 493, but got 509
     # ./spec/fpm/package/pacman_spec.rb:164:in `block (5 levels) in <top (required)>'

  7) FPM::Package::Pacman#output file permissions should preserve file permissions for /var
     Failure/Error: dir)).mode & 07777 } == perm
     Insist::Failure:
       Expected 493, but got 509
     # ./spec/fpm/package/pacman_spec.rb:164:in `block (5 levels) in <top (required)>'

  8) FPM::Package::Pacman#output file permissions should preserve file permissions for /var/lib
     Failure/Error: dir)).mode & 07777 } == perm
     Insist::Failure:
       Expected 493, but got 509
     # ./spec/fpm/package/pacman_spec.rb:164:in `block (5 levels) in <top (required)>'

  9) FPM::Package::Pacman#output file permissions should preserve file permissions for /var/lib/foo
     Failure/Error: dir)).mode & 07777 } == perm
     Insist::Failure:
       Expected 493, but got 509
     # ./spec/fpm/package/pacman_spec.rb:164:in `block (5 levels) in <top (required)>'

Failure to properly preserve permissions makes arch linux sad:

resolving dependencies...
looking for inter-conflicts...

Packages (1): fu-:1.2.3-1

Total Installed Size: 0.06 MiB
Net Upgrade Size: 0.04 MiB

:: Proceed with installation? [Y/n] y
(1/1) checking keys in keyring                                            [#########################################] 100%
(1/1) checking package integrity                                          [#########################################] 100%
(1/1) loading package files                                               [#########################################] 100%
(1/1) checking for file conflicts                                         [#########################################] 100%
(1/1) checking available disk space                                       [#########################################] 100%
(1/1) downgrading fu                                                      [#########################################] 100%
warning: directory permissions differ on /etc/
filesystem: 755  package: 775
warning: directory permissions differ on /etc/profile.d/
filesystem: 755  package: 775
warning: directory permissions differ on /usr/
filesystem: 755  package: 775
warning: directory permissions differ on /usr/share/
filesystem: 755  package: 775
warning: directory permissions differ on /usr/share/doc/
filesystem: 755  package: 775
warning: directory permissions differ on /usr/lib/
filesystem: 755  package: 775
warning: directory permissions differ on /usr/bin/
filesystem: 755  package: 775
warning: directory permissions differ on /var/
filesystem: 755  package: 775
warning: directory permissions differ on /var/lib/
filesystem: 755  package: 775

Steps to reproduce:

  1. Make a tarball with correct permissions of files: usr/ => 755, usr/bin => 755, usr/share/ => 755, etc.
  2. Use the tarball to make a pacman package: fpm -s tar -t pacman -n fu -v 1.2.3 fu.tar.gz
  3. Your resulting pacman package was written out with the "group write" permission enabled where it otherwise was not enabled in the tarball.
  4. ARGH!

The functionality (and bug) is here: https://github.com/djhaskin987/fpm/tree/feature/arch-support . help? I don't know what to do.

Owner

jordansissel commented May 7, 2015

493 is 0755, 509 is 0775. This is usually due to your environment's umask, but I don't have data to support that hypothesis yet.

Contributor

djhaskin987 commented May 9, 2015

Fixed the problem. I needed to copy the metadata as well as the files from the staging_path to the build path. It's all ready to go! It both passes rspec and works with manual installation.

Contributor

hatt commented May 14, 2015

Good to hear, I was playing with your brand on my side and couldn't replicate the issue 😞

Nice work @djhaskin987! I got held up on the mtree stuff and couldn't find proof anywhere that the size was effectively ignored, then life took over.

Contributor

djhaskin987 commented May 14, 2015

Thanks for looking at that for me, I'm glad I'm not alone on this.

flaccid commented Sep 12, 2015

Thanks for the hard work here, I'm really looking forward to this!

Contributor

djhaskin987 commented Sep 17, 2015

This is, in my opinion, ready to go now. I just added compression option support.

So what's missing here? Anything?

Contributor

djhaskin987 commented Oct 6, 2015

@jordansissel is there anything else you would like me to add 🍰 ? Do you have any concerns 😟 👷 ?

Owner

jordansissel commented Oct 7, 2015

I love 🍰

Will review!

Contributor

djhaskin987 commented Oct 7, 2015

Thanks! I saw your review stuff, I'll take a look at it tonight :)

Daniel Jay Haskin
http://djhaskin987.github.io/

On Wed, Oct 7, 2015 at 9:11 AM, Jordan Sissel notifications@github.com
wrote:

I love [image: 🍰]

Will review!


Reply to this email directly or view it on GitHub
#231 (comment).

Is Arch Linux supposed to be supported as a target? I'm getting this error with fpm 1.4.0:

$ fpm -s gem -t pacman blah
Invalid output package (-t flag) type "pacman". Expected one of: cpan, deb, dir, empty, gem, npm, osxpkg, p5p, pear, pkgin, puppet, python, rpm, sh, solaris, tar, virtualenv, zip {:level=>:warn}
Contributor

djhaskin987 commented Mar 19, 2016

It is supported on the development branch but it hasn't been released yet.

On March 19, 2016 3:29:50 PM MDT, "Cédric Félizard" notifications@github.com wrote:

Is Arch Linux supposed to be supported as a target? I'm getting this
error with fpm 1.4.0:

$ fpm -s gem -t pacman blah
Invalid output package (-t flag) type "pacman". Expected one of: cpan,
deb, dir, empty, gem, npm, osxpkg, p5p, pear, pkgin, puppet, python,
rpm, sh, solaris, tar, virtualenv, zip {:level=>:warn}

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#231 (comment)

Sent from my Android device with K-9 Mail. Please excuse my brevity.

Contributor

hatt commented Mar 20, 2016

Maybe we need a tag on PR or issues for awaiting release, since it seems like quite a few at the moment.

I agree with @hatt 👍

Owner

jordansissel commented Mar 21, 2016

Good idea! Normally, there wouldn't be such delays between releases, but
master has several failing tests which, as a personal decision, have me
waiting until that is resolved before releasing.

❤️

On Monday, March 21, 2016, Anachron notifications@github.com wrote:

I agree with @hatt https://github.com/hatt [image: 👍]


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#231 (comment)

Thanks for the clarification guys, I can confirm -t pacman works when building the Gem from the master branch.

(Agreed you shouldn't make a release with failing tests ;))

Contributor

hatt commented Mar 22, 2016

I had a quick look into the failing tests, it seems to be because the tar command is hardcoded. The following diff makes tests pass. What worries me is that some stuff relies specifically on gnutar while others on bsdtar. On OSX, only gnutar works for the packaging, but only bsdtar works for generating the mtree. If we're going to force bsdtar, then maybe we should make it get used for everything? Diff incoming:

diff --git a/lib/fpm/package/pacman.rb b/lib/fpm/package/pacman.rb
index eebc9eb..40b715c 100644
--- a/lib/fpm/package/pacman.rb
+++ b/lib/fpm/package/pacman.rb
@@ -96,7 +96,7 @@ class FPM::Package::Pacman < FPM::Package
   def input(pacman_pkg_path)
     control = {}
     # Unpack the control tarball
-    safesystem("tar", "-C", staging_path, "-xf", pacman_pkg_path)
+    safesystem(tar_cmd, "-C", staging_path, "-xf", pacman_pkg_path)
     pkginfo = staging_path(".PKGINFO")
     mtree = staging_path(".MTREE")
     install = staging_path(".INSTALL")
@@ -264,7 +264,7 @@ class FPM::Package::Pacman < FPM::Package

     File.expand_path(output_path).tap do |path|
       ::Dir.chdir(build_path) do
-        safesystem(*(["tar",
+        safesystem(*([tar_cmd,
                       compression_option,
                       "-cf",
                       path] + data_tar_flags + \

The current flags used for gnutar to build the archive don't all work in bsdtar, but mainly it's just the ownership flags. Looks like gnutar uses --owner and --group, while bsdtar uses --uname/--uid and --gname/--gid, but only in the FreeBSD versions. OSX has no way of setting the group or owner other than via format attributes, so not with a flag. For every other package format, we still prioritise gnutar based on the tar_cmd utility function, but maybe here we should stick to just one. The above fix of using gnutar for the package creation and extraction and bsdtar for the mtree stuff works, it just feels ugly.

It would maybe be good to have a set of "depends on" binaries for each archive type, but I don't like the idea of non-ruby executors since the whole point is to remove platform dependency. Libarchive seems to handle all the gnutar stuff since the flag differences become irrelevant but the ruby bindings appear unmaintained for the past few years. What do you guys think?

Owner

jordansissel commented Nov 28, 2016

fpm 1.5.0 shipped with pacman support via #916.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment