New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add lzma support to dm.pl and use it on macOS/Linux #211

Closed
kirb opened this Issue Nov 12, 2016 · 27 comments

Comments

Projects
None yet
@kirb
Collaborator

kirb commented Nov 12, 2016

dpkg 1.18.11 finally pulled the plug on our saving grace; using -Zbzip2 or -Zlzma is now an error.

The most straightforward way to “fix” this would be to just replace it with dm.pl and add lzma support to it.

(Telesphoreo’s dpkg intentionally does not support xz due to various flaws with xz and LZMA2.)

Error output

dpkg-deb: error: obsolete compression type 'lzma'; use xz instead

Type dpkg-deb --help for help about manipulating *.deb files;
Type dpkg --help for help about installing and deinstalling packages.
make: *** [internal-package] Error 2

Workaround

For the moment, you’ll need to just use 1.18.10.

For Homebrew, you can downgrade like so:

$ brew remove dpkg  # remove latest dpkg
$ brew install --force-bottle https://raw.githubusercontent.com/Homebrew/homebrew-core/7a4dabfc1a2acd9f01a1670fde4f0094c4fb6ffa/Formula/dpkg.rb  # install dpkg as a bottle from the old commit
$ brew pin dpkg  # block homebrew from updating dpkg till you `brew unpin dpkg`

(Updated 01/03/17 using @Fr0stDev’s workaround because Debian delete old source releases. 1.18.10 is from July 2016.)

Fortunately, seems Debian/Ubuntu are safe for the moment: https://launchpad.net/dpkg/+packages. But I would expect them to be updating sometime soon.


Partly dupes #197.

@DHowett

This comment has been minimized.

Show comment
Hide comment
@DHowett

DHowett Nov 12, 2016

Collaborator

To facilitate this, I've transferred DHowett/dm.pl to theos/dm.pl.

Collaborator

DHowett commented Nov 12, 2016

To facilitate this, I've transferred DHowett/dm.pl to theos/dm.pl.

@kirb

This comment has been minimized.

Show comment
Hide comment
@kirb

kirb Nov 13, 2016

Collaborator

Cheers 👍

Collaborator

kirb commented Nov 13, 2016

Cheers 👍

@PrayagBhakar

This comment has been minimized.

Show comment
Hide comment
@PrayagBhakar

PrayagBhakar Nov 13, 2016

Following these instructions let me to an error where there was an undefined variable "@@HOMEBREW_CELLAR@@". Here is a solution and it worked for me. Just leaving this here in case anyone else faces the same problem.

PrayagBhakar commented Nov 13, 2016

Following these instructions let me to an error where there was an undefined variable "@@HOMEBREW_CELLAR@@". Here is a solution and it worked for me. Just leaving this here in case anyone else faces the same problem.

@kirb

This comment has been minimized.

Show comment
Hide comment
@kirb

kirb Nov 13, 2016

Collaborator

Whoops. Updated to include a step to replace @@HOMEBREW_CELLAR@@ with the cellar path.

Collaborator

kirb commented Nov 13, 2016

Whoops. Updated to include a step to replace @@HOMEBREW_CELLAR@@ with the cellar path.

@PrayagBhakar

This comment has been minimized.

Show comment
Hide comment
@PrayagBhakar

PrayagBhakar Nov 13, 2016

Just thought this would also be necessary in the process. To stop home-brew from updating dpkg just run:

$ brew pin dpkg

PrayagBhakar commented Nov 13, 2016

Just thought this would also be necessary in the process. To stop home-brew from updating dpkg just run:

$ brew pin dpkg
@kirb

This comment has been minimized.

Show comment
Hide comment
@kirb

kirb Nov 13, 2016

Collaborator

Good call, added that too.

Collaborator

kirb commented Nov 13, 2016

Good call, added that too.

@benjibobs

This comment has been minimized.

Show comment
Hide comment
@benjibobs

benjibobs Nov 16, 2016

The command for i in $(brew --cellar)/dpkg/1.18.10/bin/*; do sed -i s/'@@HOMEBREW_CELLAR@@'/$(brew --cellar)/g "$i"; done gives the following error:
sed: 1: "/usr/local/Cellar/dpkg/ ...": extra characters at the end of l command on macOS Sierra

benjibobs commented Nov 16, 2016

The command for i in $(brew --cellar)/dpkg/1.18.10/bin/*; do sed -i s/'@@HOMEBREW_CELLAR@@'/$(brew --cellar)/g "$i"; done gives the following error:
sed: 1: "/usr/local/Cellar/dpkg/ ...": extra characters at the end of l command on macOS Sierra

@uroboro

This comment has been minimized.

Show comment
Hide comment
@uroboro

uroboro Nov 16, 2016

Collaborator

Install sed from homebrew, Apple's Developer Tools package contains really old versions of such tools.

Collaborator

uroboro commented Nov 16, 2016

Install sed from homebrew, Apple's Developer Tools package contains really old versions of such tools.

@benjibobs

This comment has been minimized.

Show comment
Hide comment
@benjibobs

benjibobs Nov 16, 2016

Thank you 👍

If it matters, I got it working with this command using the stock sed:
for i in $(brew --cellar)/dpkg/1.18.10/bin/*; do sed -i '' s~'@@HOMEBREW_CELLAR@@'~$(brew --cellar)~g "$i"; done

benjibobs commented Nov 16, 2016

Thank you 👍

If it matters, I got it working with this command using the stock sed:
for i in $(brew --cellar)/dpkg/1.18.10/bin/*; do sed -i '' s~'@@HOMEBREW_CELLAR@@'~$(brew --cellar)~g "$i"; done

@infuse-location

This comment has been minimized.

Show comment
Hide comment
@infuse-location

infuse-location Dec 2, 2016

Hi kirb,

Thank you very much helping with this issue - I've followed the steps mentioned in Workaround and encountered the following error:

homebrew-core op$ git checkout 7a4dabf
error: pathspec '7a4dabf' did not match any file(s) known to git.

I also tried replacing dm.pl from the link and adding lzma support - but still get the "use xz instead" error:
(change in line 148)

sub compressed_fd {
	my $sref = shift;
	return IO::Compress::Gzip->new($sref, -Level => 9) if $::compression eq "gzip";
	return IO::Compress::Bzip2->new($sref) if $::compression eq "bzip2";
	return "|lzma -9c" if $::compression eq "lzma";
	open my $fh, ">", $sref;
	return $fh;
}

sub compressed_filename {
	my $fn = shift;
	my $suffix = "";
	$suffix = ".gz" if $::compression eq "gzip";
	$suffix = ".bz2" if $::compression eq "bzip2";
	$suffix = ".lzma" if $::compression eq "lzma";
	return $fn.$suffix;
}

Edit:
I've managed to switch to gzip (not tested yet - packaged but not yet installed on device)
In theos_path/makefiles/package/deb.mk

#_THEOS_PLATFORM_DPKG_DEB_COMPRESSION ?= lzma
_THEOS_PLATFORM_DPKG_DEB_COMPRESSION ?= gzip

Edit:
Packaging with gzip works without any issues

infuse-location commented Dec 2, 2016

Hi kirb,

Thank you very much helping with this issue - I've followed the steps mentioned in Workaround and encountered the following error:

homebrew-core op$ git checkout 7a4dabf
error: pathspec '7a4dabf' did not match any file(s) known to git.

I also tried replacing dm.pl from the link and adding lzma support - but still get the "use xz instead" error:
(change in line 148)

sub compressed_fd {
	my $sref = shift;
	return IO::Compress::Gzip->new($sref, -Level => 9) if $::compression eq "gzip";
	return IO::Compress::Bzip2->new($sref) if $::compression eq "bzip2";
	return "|lzma -9c" if $::compression eq "lzma";
	open my $fh, ">", $sref;
	return $fh;
}

sub compressed_filename {
	my $fn = shift;
	my $suffix = "";
	$suffix = ".gz" if $::compression eq "gzip";
	$suffix = ".bz2" if $::compression eq "bzip2";
	$suffix = ".lzma" if $::compression eq "lzma";
	return $fn.$suffix;
}

Edit:
I've managed to switch to gzip (not tested yet - packaged but not yet installed on device)
In theos_path/makefiles/package/deb.mk

#_THEOS_PLATFORM_DPKG_DEB_COMPRESSION ?= lzma
_THEOS_PLATFORM_DPKG_DEB_COMPRESSION ?= gzip

Edit:
Packaging with gzip works without any issues

@PeterPan507

This comment has been minimized.

Show comment
Hide comment
@PeterPan507

PeterPan507 Dec 7, 2016

When I checkout old commit and begin install.
Homebrew will execute 'Updating Homebrew...'
So I always install the newest version.

brew info:

Homebrew 1.1.2
Homebrew/homebrew-core (git revision e34b; last commit 2016-12-07)

PeterPan507 commented Dec 7, 2016

When I checkout old commit and begin install.
Homebrew will execute 'Updating Homebrew...'
So I always install the newest version.

brew info:

Homebrew 1.1.2
Homebrew/homebrew-core (git revision e34b; last commit 2016-12-07)
@kirb

This comment has been minimized.

Show comment
Hide comment
@kirb

kirb Dec 7, 2016

Collaborator

Aah I missed the disable updates var. Fixed now. Seems like something Homebrew should detect though (that you manually checked out a commit).

Collaborator

kirb commented Dec 7, 2016

Aah I missed the disable updates var. Fixed now. Seems like something Homebrew should detect though (that you manually checked out a commit).

@andrewwiik

This comment has been minimized.

Show comment
Hide comment
@andrewwiik

andrewwiik Dec 26, 2016

Contributor

also might wanna add "git fetch --unshallow" as some people may not have the dpkg commit referenced

Contributor

andrewwiik commented Dec 26, 2016

also might wanna add "git fetch --unshallow" as some people may not have the dpkg commit referenced

@guillermo-moran

This comment has been minimized.

Show comment
Hide comment
@guillermo-moran

guillermo-moran Dec 27, 2016

The following works as well, in case you're unable to reference an older commit:

brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/9e6d049c0d22824a4eedddf6f9d471b5ca2c5283/Formula/dpkg.rb

brew pin dpkg

This will install dpkg 1.18.9. It seems the formula for 1.18.10 no longer works.

guillermo-moran commented Dec 27, 2016

The following works as well, in case you're unable to reference an older commit:

brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/9e6d049c0d22824a4eedddf6f9d471b5ca2c5283/Formula/dpkg.rb

brew pin dpkg

This will install dpkg 1.18.9. It seems the formula for 1.18.10 no longer works.

@andrewwiik

This comment has been minimized.

Show comment
Hide comment
@andrewwiik

andrewwiik Dec 27, 2016

Contributor
Contributor

andrewwiik commented Dec 27, 2016

@abasefive

This comment has been minimized.

Show comment
Hide comment
@abasefive

abasefive Jan 12, 2017

@Fr0stDev thank you,but there is warning:
**```
dpkg-deb: warning: deprecated compression type 'lzma'; use xz instead
dpkg-deb: warning: ignoring 1 warning about the control file(s)

it doesn't matter?

abasefive commented Jan 12, 2017

@Fr0stDev thank you,but there is warning:
**```
dpkg-deb: warning: deprecated compression type 'lzma'; use xz instead
dpkg-deb: warning: ignoring 1 warning about the control file(s)

it doesn't matter?
@mstg

This comment has been minimized.

Show comment
Hide comment
@mstg

mstg Jan 12, 2017

Contributor

@abasefive Yes that is fine.

Contributor

mstg commented Jan 12, 2017

@abasefive Yes that is fine.

@supermamon

This comment has been minimized.

Show comment
Hide comment
@supermamon

supermamon Feb 8, 2017

Contributor

This allowed me to pull v1.18.10 from the 7a4dabf commit.

$ brew remove dpkg  
$ HOMEBREW_NO_AUTO_UPDATE=1 brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/7a4dabfc1a2acd9f01a1670fde4f0094c4fb6ffa/Formula/dpkg.rb
$ brew pin dpkg  
Contributor

supermamon commented Feb 8, 2017

This allowed me to pull v1.18.10 from the 7a4dabf commit.

$ brew remove dpkg  
$ HOMEBREW_NO_AUTO_UPDATE=1 brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/7a4dabfc1a2acd9f01a1670fde4f0094c4fb6ffa/Formula/dpkg.rb
$ brew pin dpkg  
@zhahao

This comment has been minimized.

Show comment
Hide comment
@zhahao

zhahao Feb 24, 2017

An error occurred,

$ git checkout 7a4dabf
error: pathspec '7a4dabf' did not match any file(s) known to git.

zhahao commented Feb 24, 2017

An error occurred,

$ git checkout 7a4dabf
error: pathspec '7a4dabf' did not match any file(s) known to git.
@KTSByJack

This comment has been minimized.

Show comment
Hide comment
@KTSByJack

KTSByJack Mar 22, 2017

I already downgrade dpkg,but I still can not make install success, do you guys have trouble about this

KTSByJack commented Mar 22, 2017

I already downgrade dpkg,but I still can not make install success, do you guys have trouble about this

uroboro added a commit to uroboro/Continuous-Integration-Setup that referenced this issue May 27, 2017

@uroboro

This comment has been minimized.

Show comment
Hide comment
@uroboro

uroboro Jun 7, 2017

Collaborator

Rather than making a new subprocess in #211 (comment) we can add proper support with Lzma.pm.

Collaborator

uroboro commented Jun 7, 2017

Rather than making a new subprocess in #211 (comment) we can add proper support with Lzma.pm.

@kirb

This comment has been minimized.

Show comment
Hide comment
@kirb

kirb Jun 8, 2017

Collaborator

@uroboro, agreed, though a little tricky because macOS doesn't include IO::Compress::Lzma (but does have most other modules of IO::Compress). IO::Compress also seems to be particular about the versions of each script being the same, which might force us to ship the entire module rather than just Lzma or the parts we use, because perl might import older or newer versions if they already exist on the machine. I'd be willing to accept piping to lzma for the moment just to get a workable solution out, though.

Collaborator

kirb commented Jun 8, 2017

@uroboro, agreed, though a little tricky because macOS doesn't include IO::Compress::Lzma (but does have most other modules of IO::Compress). IO::Compress also seems to be particular about the versions of each script being the same, which might force us to ship the entire module rather than just Lzma or the parts we use, because perl might import older or newer versions if they already exist on the machine. I'd be willing to accept piping to lzma for the moment just to get a workable solution out, though.

@DuIslingr

This comment has been minimized.

Show comment
Hide comment
@DuIslingr

DuIslingr Jul 1, 2017

Contributor

Why not just use gzip?

Contributor

DuIslingr commented Jul 1, 2017

Why not just use gzip?

@sterling0x1

This comment has been minimized.

Show comment
Hide comment
@sterling0x1

sterling0x1 Jul 2, 2017

Just install older dpkg and live with the warning. Does the job just fine.

sterling0x1 commented Jul 2, 2017

Just install older dpkg and live with the warning. Does the job just fine.

@kirb

This comment has been minimized.

Show comment
Hide comment
@kirb

kirb Jul 3, 2017

Collaborator

@DuIslingr, that's short sighted. If all compression formats except the oldest (gzip) and newest (xz) have been removed, them there's obviously a specific reason gzip is hanging around, and it might not be forever. As far as I'm aware, it's only still available because for some reason dpkg itself is packaged with gzip (on Debian, although I think it's the same in Telesphoreo). Seems strange if its dependencies are using xz, but maybe there's some compatibility detail I don't know. If Debian realises there's no need to do this any more, gzip will be gone, and we loop back to this same situation.

Also, lzma provides much better compression ratios than gzip/bzip2. You should be conscious of how much data you're asking your users to download. I know we're not packaging 250 MB Facebook here, but you have a duty for your users to save them some download usage by just having your tools set up correctly.

Collaborator

kirb commented Jul 3, 2017

@DuIslingr, that's short sighted. If all compression formats except the oldest (gzip) and newest (xz) have been removed, them there's obviously a specific reason gzip is hanging around, and it might not be forever. As far as I'm aware, it's only still available because for some reason dpkg itself is packaged with gzip (on Debian, although I think it's the same in Telesphoreo). Seems strange if its dependencies are using xz, but maybe there's some compatibility detail I don't know. If Debian realises there's no need to do this any more, gzip will be gone, and we loop back to this same situation.

Also, lzma provides much better compression ratios than gzip/bzip2. You should be conscious of how much data you're asking your users to download. I know we're not packaging 250 MB Facebook here, but you have a duty for your users to save them some download usage by just having your tools set up correctly.

@michyprima

This comment has been minimized.

Show comment
Hide comment
@michyprima

michyprima Oct 3, 2017

If anyone needs a working link to dpkg 1.18.10 replace url with this in dpkg.rb

michyprima commented Oct 3, 2017

If anyone needs a working link to dpkg 1.18.10 replace url with this in dpkg.rb

@kirb

This comment has been minimized.

Show comment
Hide comment
@kirb

kirb Dec 6, 2017

Collaborator

As of bad135c (release 2.1.0), the deb builder has changed from dpkg-deb to our built-in dm.pl. It has flags compatible with dpkg-deb, so you can also use it directly if needed. Worth noting its default compression format is gzip (dpkg-deb’s has been xz for a long time; previously was gzip). Theos will still manually specify -Zlzma, so packages built via Theos will compress with lzma.

There is (unfortunately) a binary dependency which provides bindings for liblzma to Perl. If that’s gibberish to you, don’t worry, it’ll be installed when you run make update-theos. The instructions to install it manually while installing Theos from scratch has been added to the Install page. It’s the best compromise I could come up with and I don’t want any pre-built binaries in Theos. If someone has a smarter solution than mine I’d love to discuss it.

Collaborator

kirb commented Dec 6, 2017

As of bad135c (release 2.1.0), the deb builder has changed from dpkg-deb to our built-in dm.pl. It has flags compatible with dpkg-deb, so you can also use it directly if needed. Worth noting its default compression format is gzip (dpkg-deb’s has been xz for a long time; previously was gzip). Theos will still manually specify -Zlzma, so packages built via Theos will compress with lzma.

There is (unfortunately) a binary dependency which provides bindings for liblzma to Perl. If that’s gibberish to you, don’t worry, it’ll be installed when you run make update-theos. The instructions to install it manually while installing Theos from scratch has been added to the Install page. It’s the best compromise I could come up with and I don’t want any pre-built binaries in Theos. If someone has a smarter solution than mine I’d love to discuss it.

@kirb kirb closed this Dec 6, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment