Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some binaries not replaced by 'pkg install -f' #1394

Closed
infracaninophile opened this issue Feb 16, 2016 · 15 comments
Closed

Some binaries not replaced by 'pkg install -f' #1394

infracaninophile opened this issue Feb 16, 2016 · 15 comments

Comments

@infracaninophile
Copy link
Member

So, I'm upgrading a machine from FreeBSD-9.3 to 10.2, and consequently reinstalling all packages.
However, it seems some binaries are not replaced even by pkg install -f:

ftp-2:...etc/pkg/repos:# pkg which /usr/local/bin/perl5 
/usr/local/bin/perl5 was installed by package perl5-5.22.1_4
ftp-2:...etc/pkg/repos:# pkg upgrade -f perl5-5.22.1_4 
Updating server repository catalogue...
Fetching meta.txz: 100%    264 B   0.3kB/s    00:01    
Fetching packagesite.txz: 100%  364 KiB 372.7kB/s    00:01    
Processing entries: 100%
server repository update completed. 1590 packages processed.
Updating apache24 repository catalogue...
apache24 repository is up-to-date.
Updating desktop repository catalogue...
desktop repository is up-to-date.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be REINSTALLED:
        perl5-5.22.1_4 [desktop]

13 MiB to be downloaded.

Proceed with this action? [y/N]: y
Fetching perl5-5.22.1_4.txz: 100%   13 MiB  13.8MB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Reinstalling perl5-5.22.1_4...
[1/1] Extracting perl5-5.22.1_4: 100%
Message from perl5-5.22.1_4:
The /usr/bin/perl symlink has been removed starting with Perl 5.20.
For shebangs, you should either use:

#!/usr/local/bin/perl

or

#!/usr/bin/env perl

The first one will only work if you have a /usr/local/bin/perl,
the second will work as long as perl is in PATH.
ftp-2:...etc/pkg/repos:# pkg which /usr/local/bin/perl5
/usr/local/bin/perl5 was installed by package perl5-5.22.1_4
ftp-2:...etc/pkg/repos:# file /usr/local/bin/perl5
/usr/local/bin/perl5: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD *9.2*, stripped

To fix this I had to:

ftp-2:...etc/pkg/repos:# rm /usr/local/bin/perl5 
ftp-2:...etc/pkg/repos:# pkg upgrade -f perl5-5.22.1_4
Updating server repository catalogue...
server repository is up-to-date.
Updating apache24 repository catalogue...
apache24 repository is up-to-date.
Updating desktop repository catalogue...
desktop repository is up-to-date.
All repositories are up-to-date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be REINSTALLED:
        perl5-5.22.1_4 [desktop]

Proceed with this action? [y/N]: y
[1/1] Reinstalling perl5-5.22.1_4...
[1/1] Extracting perl5-5.22.1_4: 100%
Message from perl5-5.22.1_4:
The /usr/bin/perl symlink has been removed starting with Perl 5.20.
For shebangs, you should either use:

#!/usr/local/bin/perl

or

#!/usr/bin/env perl

The first one will only work if you have a /usr/local/bin/perl,
the second will work as long as perl is in PATH.
ftp-2:...etc/pkg/repos:# file /usr/local/bin/perl5
/usr/local/bin/perl5: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 10.1, stripped

perl5 isn't the only pkg to suffer from this. Looks like git and binutils also do. (And, yes, these are with freshly installed 10.x versions of the pkgs.)

ftp-2:...etc/pkg/repos:# find /usr/local/bin -type f -print0 | xargs -0 file | grep 'for FreeBSD 9'
/usr/local/bin/git-upload-archive:                                ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 9.2, stripped
/usr/local/bin/git-receive-pack:                                  ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 9.2, stripped
/usr/local/bin/ld.bfd:                                            ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 9.2, stripped
@ohauer
Copy link

ohauer commented Feb 22, 2016

I was running today into the same issue by upgrading postfix on some systems to a new current version.
pkg upgrade was running without issues or notice, but smtp was crashing.
pkg delete -x postfix was showing the following error messages.

# pkg delete postfix-current
Proceed with deinstalling packages? [y/N]: y
[1/1] Deinstalling postfix-current-3.1.20160214,4...
You may need to manually remove /usr/local/etc/postfix/main.cf if it is no longer needed.
==> You should manually remove the "postfix" user.
[1/1] Deleting files for postfix-current-3.1.20160214,4:  15%
pkg: /usr/local/libexec/postfix/qmgr different from original checksum, not removing
[1/1] Deleting files for postfix-current-3.1.20160214,4:  16%
pkg: /usr/local/libexec/postfix/smtp different from original checksum, not removing
[1/1] Deleting files for postfix-current-3.1.20160214,4: 100%
pkg: /usr/local/libexec/postfix/smtp different from original checksum, not removing
[1/1] Deleting files for postfix-current-3.1.20160214,4:  79%
pkg: /usr/local/share/doc/postfix/defer.8.html different from original checksum, not removing
[1/1] Deleting files for postfix-current-3.1.20160214,4:  84%
pkg: /usr/local/share/doc/postfix/newaliases.1.html different from original checksum, not removing

And a bunch of files where not replaced during the upgrade

$ ls /usr/local/share/doc/postfix/                                                                                                                      
defer.8.html                    newaliases.1.html               sendmail.1.html                 smtp.8.html                     trace.8.html             
defer.8.html.HbCnJH90cCQM       newaliases.1.html.5gzZvAvUa0Pd  sendmail.1.html.BngMx61JoQPT    smtp.8.html.JwjSnlQwmgty        trace.8.html.eIGSTB6Av9IE
defer.8.html.fSoji81GGgKA       newaliases.1.html.ZyuIe805kapN  sendmail.1.html.I3GEDeBsRgRq    smtp.8.html.ad53jBYZdw5X        trace.8.html.igSKoUZE7Ny3

$ ll /usr/local/libexec/postfix/
total 564
-rwxr-xr-x  1 root  wheel   67800 Jan  4 12:54 qmgr
-rwxr-xr-x  1 root  wheel   67800 Feb 15 09:52 qmgr.5WmM6HPsrfcJ
-rwxr-xr-x  1 root  wheel   71568 Feb 22 09:14 qmgr.kuMCGEAoImB2
-rwxr-xr-x  1 root  wheel  115736 Jan  4 12:54 smtp
-rwxr-xr-x  1 root  wheel  124056 Feb 15 09:52 smtp.s5XXAy60oFWF
-rwxr-xr-x  1 root  wheel  124168 Feb 22 09:14 smtp.sslx3Q9AVpCC

During the second try postfix was not running, but old files where also not replaced.
Wiping the leftovers before installing again solved the issue and postfix was no longer crashing

@brd
Copy link
Member

brd commented Feb 22, 2016

@infracaninophile
Copy link
Member Author

This is a longstanding feature in the code -- pkg(8) will not touch a file where the checksum doesn't match what it has in its database. The idea being that the admin had overwritten some package files deliberately and pkg(8) didn't want to wipe out those changes.

You can see the logic there, but this doesn't make sense in all circumstances:

  • It's a major version upgrade and all the packages should be upgraded whatever
  • The mis-matched checksums have occurred by accident -- I believe it is possible for this to occur if pkg(8) failed mid-install of a package. It will cancel and rollback any updates to the pkgdb but it can leave files overwritten from the new package, hence invalidating the checksum records.
  • Even if the mis-match was deliberate the admin may choose to overwrite with a clean package.

At the moment, the only way to clear mismatched checksums is to remember to run 'pkg check -r ...' before trying to upgrade. There's no early warning that you're going to run into this problem, and once you see the error occurring there's no way to force pkg(8) to ignore the mismatched checksums and install the files anyway. Or to clean up the extra copies of file it leaves laying about.

Ideally the solution would be to have a full rollback capability (to avoid the accidental breakage), and change the default to always do a clean install of a package. I think any admin deliberately overwriting package files is sailing close to the anyhow (and maybe should investigate pkg-lock(8)). Plus maybe overload 'pkg install -f' or 'pkg upgrade -f' to force overwriting files irrespective of their current checksum.

@vstakhov
Copy link
Member

On conflicts resolution step (or just after), we could check files checksums as well. The only question is the cost of such an operation. For large files, we could have not a single checksum but a chain of checksums to detect issues early. We could also check mtime and compare it with ctime as well as perform sizes comparision. Transactional upgrade is far harder than my approach (but obviously better) - do we have sufficient manpower for this?

@ohauer
Copy link

ohauer commented Feb 22, 2016

Hm, don't know what works best, to prevent such issues. First I thought it was a issue because postfix was running during the update but second run didn't solve the issue.
Also I've tried at the moment a 'pkg upgrade -f' and fond also not all files where updated.

# pkg upgrade -f postfix-current
ls -l /usr/local/libexec/postfix/smtp*
-rwxr-xr-x  1 root  wheel  124168 Feb 22 09:14 /usr/local/libexec/postfix/smtp
-rwxr-xr-x  2 root  wheel  124168 Feb 22 20:10 /usr/local/libexec/postfix/smtp.xwd1aOSwnHTT
-rwxr-xr-x  1 root  wheel  237416 Feb 22 20:10 /usr/local/libexec/postfix/smtpd

# pkg delete postfix-current
... no issues reported, at last the checksum from the old smtp binary should be different, because the package was build again some min. before.

$ ls -l /usr/local/libexec/postfix/*
-rwxr-xr-x  1 root  wheel   71568 Feb 22 20:10 /usr/local/libexec/postfix/qmgr.KCNwUPIPsGHG
-rwxr-xr-x  1 root  wheel  124168 Feb 22 20:10 /usr/local/libexec/postfix/smtp.xwd1aOSwnHTT

At the moment I'm ranting if a pre-upgrade and post-upgrade script (stopping postfix and restarting postfix) could be helpful, since debugging issues on the user site seems harder.
But I don't think this will solve the issue after checking the postfix docs dir.

$ ls -l /usr/local/share/doc/postfix/ | wc -l
46
$ ls -l /usr/local/share/doc/postfix/ | grep 'Feb 22 20'
-rw-r--r--  2 root  wheel   9741 Feb 22 20:09 defer.8.html.Hh7NOBKrm0Z3
-rw-r--r--  2 root  wheel  24084 Feb 22 20:09 newaliases.1.html.STl9RSMj8ZKe
-rw-r--r--  2 root  wheel  24084 Feb 22 20:09 sendmail.1.html.DHUEA4cvSZMC
-rw-r--r--  1 root  wheel  51312 Feb 22 20:09 smtp.8.html.HyuWzLPHeP00
-rw-r--r--  2 root  wheel   9741 Feb 22 20:09 trace.8.html.9DYewYakmBKe

@ohauer
Copy link

ohauer commented Feb 22, 2016

PS: How can I format text the same as Matthew?
I found no hint on the Markdown pages ...

@brd
Copy link
Member

brd commented Feb 22, 2016

@ohauer use backticks

@ohauer
Copy link

ohauer commented Feb 22, 2016

@brd Thanks!
I always forget the backticks have be be on a line on it's one (I don't use github markdown regularly ...)

@infracaninophile
Copy link
Member Author

On 22/02/2016 20:08, ohauer wrote:

Hm, don't know what works best, to prevent such issues. First I thought it was a issue because postfix was running during the update but second run didn't solve the issue.

postfix being run at the time won't make any difference to the install.
You can trash the original file without affecting the running copy in
RAM. The way postfix works, it will tend to spawn new smtpd or similar
processes using the updated binaries very quickly, but that shouldn't
cause any real problems. You should 'postfix reload' fairly promptly
after the upgrade though, as some of the postfix processes live for
quite some time unless deliberately restarted.

Also I've tried at the moment a 'pkg upgrade -f' and fond also not all files where updated.

pkg upgrade -f postfix-current
ls -l /usr/local/libexec/postfix/smtp*
-rwxr-xr-x 1 root wheel 124168 Feb 22 09:14 /usr/local/libexec/postfix/smtp
-rwxr-xr-x 2 root wheel 124168 Feb 22 20:10 /usr/local/libexec/postfix/smtp.xwd1aOSwnHTT
-rwxr-xr-x 1 root wheel 237416 Feb 22 20:10 /usr/local/libexec/postfix/smtpd

I've used:

find -E /usr/local -regex '.*.[a-zA-Z0-9]{12}$'

to identify pkg(8)'s leavings. Beware of just deleting such files
blindly though -- there are quite a lot of legitimate files that end
with a 12-character word separated by a dot.

Cheers,

Matthew

vstakhov added a commit that referenced this issue Feb 23, 2016
In case of hardlinks, `pathname` var was incorrectly restored from `rpath`
which is intended to be a temporary name. Hence, renaming was meaningless
afterwards.

Issue: #1394
Reported by: many
@ohauer
Copy link

ohauer commented Feb 23, 2016

Maybe an interesting finding.
I had some spare min. and tested with an older pkg-1.6.1 version, and it seems the issue is not present.
Will do some more testing with pkg-1.6.1..1.6.3 this evening

# pkg info pkg
pkg-1.6.1
Name           : pkg
Version        : 1.6.1
Installed on   : Tue Feb 23 14:40:15 CET 2016
Origin         : ports-mgmt/pkg
Architecture   : freebsd:10:x86:64

@ohauer
Copy link

ohauer commented Feb 23, 2016

I've done some tests with pkg-static-1.6.1 .. 1.6.4 and suspect the issue is only in 1.6.4 present.

For the test I've build two postfix versions one was always the installed starting point and the other was the upgrade target.

pkg-static.1.6.(1-3) went without issues, all files where upgraded and no leftovers where found, 1.6.4 failed and some binaries and documents where not upgraded but left over, but checksums where updated in the database.

@vstakhov
Copy link
Member

I've added patch to the master that fixes this issue.

@ohauer
Copy link

ohauer commented Feb 23, 2016

@vstakhov seems commit 34f3802 solves the issue, will roll out your patch tomorrow on some dedicated systems

@ohauer
Copy link

ohauer commented Feb 24, 2016

@vstakhov
I've applied your patch today on 40+ systems (on 11 I found leftovers for manual cleanup)
Until now I haven't seen failed upgrades again (tested with downgrade/uprade/reinstall...)
@infracaninophile if the patch is also working for you the case can be closed.
I've uploaded the patch as drop in to pkg/files here: http://people.freebsd.org/~ohauer/diffs/patch-libpkg_pkg__add.c
It would be nice if someone from the pkg team / portmgr commits this patch to the tree (and bump PORTREVISION)
Thanks!

bapt pushed a commit that referenced this issue Feb 25, 2016
In case of hardlinks, `pathname` var was incorrectly restored from `rpath`
which is intended to be a temporary name. Hence, renaming was meaningless
afterwards.

Issue: #1394
Reported by: many
@bapt
Copy link
Member

bapt commented May 6, 2016

fixed

@bapt bapt closed this as completed May 6, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants