pkg update fail to unpack downloaded archive.

[AjaxOdessa]

Since about a week or two, cannot update packages — pkg update fails on unpacking downloaded file. There is plenty of disk space (over 50Gb) and permissions are correct everywhere.

Here is and machine info

root@server:~ # cat /usr/local/etc/pkg/repos/FreeBSD.conf
# $FreeBSD: release/10.0.0/etc/pkg/FreeBSD.conf 258710 2013-11-28 14:24:26Z gjb $
FreeBSD: {
    url: "pkg+http://pkg0.bme.FreeBSD.org/${ABI}/latest",
    mirror_type: "srv",
    signature_type: "fingerprints",
    fingerprints: "/usr/share/keys/pkg",
    enabled: yes
}

uname -a
FreeBSD server 10.0-RELEASE-p5 FreeBSD 10.0-RELEASE-p5 #2: Fri Oct 24 15:26:27 EEST 2014     root@server:/usr/obj/usr/src/sys/server  amd64

and pkg update output

root@server:~ #pkg update
Updating FreeBSD repository catalogue...
pkg: repository meta /var/db/pkg/FreeBSD.meta has wrong version or wrong format
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
pkg: No signature found
repository FreeBSD has no meta file, using default settings
Fetching packagesite.txz: 100%    5 MiB   1.4MB/s    00:04    
pkg: archive_read_extract(extract error): No error: 0
pkg: No signature found
Unable to update repository FreeBSD
root@server:~ #

[andrejzverev]

Which pkg version are you using? (pkg -vv output will be enough)
As I can see something is blocking exchange between pkg and remote site. Are you using proxy server?

[AjaxOdessa]

No proxy or any blocking software used. On the same server I can "lynx to" http://pkg0.bme.FreeBSD.org/FreeBSD:10:amd64/latest without any problems, download any archive and all downloaded archives are correct.

Running pkg --debug update -f also doesn't reveal much info, the same error about extract error and signature.

…
DBG(1)[63403]> Read status: 5409316 over 5409316
Fetching packagesite.txz: 100%    5 MiB 901.6kB/s    00:06    
DBG(1)[63403]> PkgRepo: extracting packagesite.yaml of repo FreeBSD
DBG(1)[64632]> PkgRepo: extracting signature of repo in a sandbox
pkg: archive_read_extract(extract error): No error: 0
pkg: No signature found
Unable to update repository FreeBSD

And here is output you asked for.

root@server:~ # pkg -vv
Version                 : 1.5.5
PKG_DBDIR = "/var/db/pkg";
PKG_CACHEDIR = "/var/cache/pkg";
PORTSDIR = "/usr/ports";
INDEXDIR = "";
INDEXFILE = "INDEX-10";
HANDLE_RC_SCRIPTS = false;
DEFAULT_ALWAYS_YES = false;
ASSUME_ALWAYS_YES = false;
REPOS_DIR [
    "/etc/pkg/",
    "/usr/local/etc/pkg/repos/",
]
PLIST_KEYWORDS_DIR = "";
SYSLOG = true;
ABI = "FreeBSD:10:amd64";
ALTABI = "freebsd:10:x86:64";
DEVELOPER_MODE = false;
VULNXML_SITE = "http://vuxml.freebsd.org/freebsd/vuln.xml.bz2";
FETCH_RETRY = 3;
PKG_PLUGINS_DIR = "/usr/local/lib/pkg/";
PKG_ENABLE_PLUGINS = true;
PLUGINS [
]
DEBUG_SCRIPTS = false;
PLUGINS_CONF_DIR = "/usr/local/etc/pkg/";
PERMISSIVE = false;
REPO_AUTOUPDATE = true;
NAMESERVER = "";
HTTP_USER_AGENT = "pkg/1.5.5";
EVENT_PIPE = "";
FETCH_TIMEOUT = 30;
UNSET_TIMESTAMP = false;
SSH_RESTRICT_DIR = "";
PKG_ENV {
}
PKG_SSH_ARGS = "";
DEBUG_LEVEL = 0;
ALIAS {
}
CUDF_SOLVER = "";
SAT_SOLVER = "";
RUN_SCRIPTS = true;
CASE_SENSITIVE_MATCH = false;
LOCK_WAIT = 1;
LOCK_RETRIES = 5;
SQLITE_PROFILE = false;
WORKERS_COUNT = 0;
READ_LOCK = false;
PLIST_ACCEPT_DIRECTORIES = false;
IP_VERSION = 0;
AUTOMERGE = true;
VERSION_SOURCE = "";
CONSERVATIVE_UPGRADE = true;
PKG_CREATE_VERBOSE = false;
AUTOCLEAN = false;


Repositories:
  FreeBSD: { 
    url             : "pkg+http://pkg0.bme.FreeBSD.org/FreeBSD:10:amd64/latest",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/share/keys/pkg"
  }

[andrejzverev]

That's strange. Can you please for comment one line in your /usr/local/etc/pkg/repos/FreeBSD.conf to make sure mirror is OK for sure (like this):

FreeBSD: {
#    url: "pkg+http://pkg0.bme.FreeBSD.org/${ABI}/latest",
     url: "pkg+http://pkg0.ydx.freebsd.org/${ABI}/latest",
    mirror_type: "srv",
    signature_type: "fingerprints",
    fingerprints: "/usr/share/keys/pkg",
    enabled: yes
}

At is possible:

1. fetch http://pkg0.bme.FreeBSD.org/FreeBSD:10:amd64/latest/packagesite.txz
2. fetch http://pkg0.bme.FreeBSD.org/FreeBSD:10:amd64/latest/meta.txz
3. md5 packagesite.txz
4. md5 meta.txz

[AjaxOdessa]

MD5 (packagesite.txz) = cae27b2be033ed4ef95a9321eae983b0
MD5 (meta.txz) = f06576a4e1314ad60ea207807fca0387

Yandex mirror gives the same. Upgrade to 10.1 didn't help. I'm not familiar with all procedures pkg does upon updating packages but it looks like a broken pkg update mechanism.

[AjaxOdessa]

BTW if there is a way to put reliably known correct FreeBSD.meta instead of zero-length one and try to run package update again? But where to get one?

[bapt]

hum can you show please show me uname -a and file /bin/sh
can you also
Why are you enforcing a mirror?
As that box have been upgraded from 9 to 10?
if yes can you remove your /usr/local/etc/pkg/repo/FreeBSD.conf and let the one already available in base to the job?

[AjaxOdessa]

Yes, it used to be 9.x and then 10.0 when it came to production and now 10.1

I guess mirror enforcing left since 9.x as there were some issued but that didn't appear to be a problem until the last week or so. Putting back default (at any location either /etc or /usr/local…) doesn't really help — produces the same error.

/bin/sh: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 10.1, stripped

Kernel is custom built one but options hasn't changed since 10.0 when pkg used to work

FreeBSD server 10.1-RELEASE-p14 FreeBSD 10.1-RELEASE-p14 #3: Wed Jul 22 00:21:40 EEST 2015     root@server:/usr/obj/usr/src/sys/SERVER  amd64

[bapt]

I suspect something went wrong during you upgrade, can you try first pkg bootstrap -f
then pkg update?

[AjaxOdessa]

Bootstrapping didn't help, unfortunately. As regard to upgrade, whole thing used to work and time to time I do run pkg upgrade to get recent packages. Just about the last week that didn't work and started to give me this error.

The message that disturbs me come from pkg/libpkg/pkg_repo.c# lines 141-144. To me it looks like a broken signature/hash checking routines on the particular installation but why that could have happened — pure mystery 😔

[andrejzverev]

i did try to repeat this case, but no luck.
Can you trace run pkg under ktrace and provide dump via some URL?

[AjaxOdessa]

You can grab dump from http://lasman.info/pkg-update.ktrace.txz

I'm not a guru of reading dumps but this part, which is just after receiving chunks of data from remote host looks disturbing

68276 pkg      RET   read 2888/0xb48
68276 pkg      CALL  stat(0x7fffffffcfb0,0x7fffffffcf38)
68276 pkg      RET   stat -1 errno 94 Not permitted in capability mode
68276 pkg      CALL  stat(0x7fffffffcfb0,0x7fffffffcf38)
68276 pkg      RET   stat -1 errno 94 Not permitted in capability mode
68276 pkg      CALL  stat(0x7fffffffcfb0,0x7fffffffcf38)
68276 pkg      RET   stat -1 errno 94 Not permitted in capability mode
68276 pkg      CALL  stat(0x7fffffffcfb0,0x7fffffffcf38)
68276 pkg      RET   stat -1 errno 94 Not permitted in capability mode
68276 pkg      CALL  write(0x2,0x7fffffffce20,0x5)
68276 pkg      GIO   fd 2 wrote 5 bytes
       "pkg: "
68276 pkg      RET   write 5
68276 pkg      CALL  write(0x2,0x7fffffffcf10,0x30)
68276 pkg      GIO   fd 2 wrote 48 bytes
       "archive_read_extract(extract error): No error: 0"

What's that capability mode is?

[bapt]

I really do not understand what tries to run stat and is not allowed to...

[bapt]

Sorry I misread the kdump, this failure you are showing are not related to your issue (even if they should be fixed) the issue happens before, All the "No signature found" beside I can see the signature being read in the sandbox, but somehow it is not passed to the application...

[bapt]

is it possible that you do grab the pkg distfiles, run ./configure and send me the config.log and pkg_config.h that has been created?

[andrejzverev]

Also can you show ldd /usr/local/sbin/pkg and try pkg-static update -f command, just for check. Will result will be same or not.

[AjaxOdessa]

So, ldd shows no problem

ldd /usr/local/sbin/pkg
/usr/local/sbin/pkg:
    libpkg.so.3 => /usr/local/lib/libpkg.so.3 (0x800845000)
    libutil.so.9 => /lib/libutil.so.9 (0x800c2a000)
    libssl.so.7 => /usr/lib/libssl.so.7 (0x800e3c000)
    libcrypto.so.7 => /lib/libcrypto.so.7 (0x8010a7000)
    libm.so.5 => /lib/libm.so.5 (0x80149b000)
    libelf.so.1 => /usr/lib/libelf.so.1 (0x8016c3000)
    libjail.so.1 => /lib/libjail.so.1 (0x8018d8000)
    libarchive.so.6 => /usr/lib/libarchive.so.6 (0x801add000)
    libz.so.6 => /lib/libz.so.6 (0x801d7f000)
    libbz2.so.4 => /usr/lib/libbz2.so.4 (0x801f95000)
    liblzma.so.5 => /usr/local/lib/liblzma.so.5 (0x8021a7000)
    libc.so.7 => /lib/libc.so.7 (0x8023ca000)
    libbsdxml.so.4 => /lib/libbsdxml.so.4 (0x802774000)
    libmd.so.6 => /lib/libmd.so.6 (0x80299a000)
    libthr.so.3 => /lib/libthr.so.3 (0x802baa000)

but… pkg-static worked! and updated FreeBSD.meta. Now when I ran pkg upgrade I got next "feedback":

 pkg upgrade
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
New version of pkg detected; it needs to be installed first.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be REINSTALLED:
    pkg-1.5.5 (needed shared library changed)

The operation will free 36 KiB.

Proceed with this action? [y/N]: y
[1/1] Reinstalling pkg-1.5.5...
[1/1] Extracting pkg-1.5.5:  96%
pkg: archive_read_next_header(): Lzma library error: Corrupted input data
[1/1] Extracting pkg-1.5.5: 100%
[1/1] Deleting files for pkg-1.5.5: 100%

Does it mean Lzma is broken?

[bapt]

Right so that confirms something is wrong with your upgrade! the only difference between pkg-static and pkg is that pkg-static the fact that pkg-static does not use local libraries.

How to you perform your upgrade?

[bapt]

Btw the is strange:

Installed packages to be REINSTALLED:
pkg-1.5.5 (needed shared library changed)

and is not supposed to happen given pkg "need libraries"

[AjaxOdessa]

Well, as usual: freebsd-update -r 10.1 upgrade.

BTW, @bapt do you still need ./configure output?

[bapt]

That will help yes

[AjaxOdessa]

@bapt available at http://lasman.info/pkg.config.tgz

P.S. If there are PMs at GitHub?

[bapt]

PMs?

hum I see nothing wrong in you configure...

[AjaxOdessa]

I meant Private Messages :)

[bapt]

nope

[andrejzverev]

Hm, same ldd from my box (10.1-RELEASE-p15)

ldd /usr/local/sbin/pkg
/usr/local/sbin/pkg:
    libpkg.so.3 => /usr/local/lib/libpkg.so.3 (0x800845000)
    libutil.so.9 => /lib/libutil.so.9 (0x800c27000)
    libssl.so.7 => /usr/lib/libssl.so.7 (0x800e39000)
    libcrypto.so.7 => /lib/libcrypto.so.7 (0x8010a4000)
    libm.so.5 => /lib/libm.so.5 (0x801498000)
    libelf.so.1 => /usr/lib/libelf.so.1 (0x8016c0000)
    libjail.so.1 => /lib/libjail.so.1 (0x8018d5000)
    libarchive.so.6 => /usr/lib/libarchive.so.6 (0x801ada000)
    libz.so.6 => /lib/libz.so.6 (0x801d7c000)
    libbz2.so.4 => /usr/lib/libbz2.so.4 (0x801f92000)
    liblzma.so.5 => /usr/lib/liblzma.so.5 (0x8021a4000)
    libc.so.7 => /lib/libc.so.7 (0x8023c9000)
    libbsdxml.so.4 => /lib/libbsdxml.so.4 (0x802773000)

You have some libthr...
Can you show cat /etc/libmap.conf and ls -la /usr/local/etc/libmap.d (if it exist)

[AjaxOdessa]

First file include second folder and both are empty.

[andrejzverev]

well, bad news here is that your system damaged..
freebsd-update as i can see don't work with custom kernels, so i don't know how exactly update was done or maybe it was aborted in the middle. But libthr sense is something from past..

I can only suggest you to try fetch 10.1-RELEASE sources from SVN and rebuild system(kernel/userbase) once more time. It strange what you don't having issues with other applications on this server.

[andrejzverev]

at i think you can try use pig-static till that moment as workaround. If it will work for you.

[infracaninophile]

freebsd-update will work with custom kernels. You should edit /etc/freebsd-update.conf and change:

Components src world kernel

to

Components src world

(or drop src as well if you're checking the system sources out of SVN.) Of course, given this you will need to build and install a new kernel when a patch level update affects its sources.

OP could also try freebsd-update's IDS mode to see if you can spot any libraries that don't match the known checksums from the freebsd-update distribution.

[andrejzverev]

as we found issue not related to pkg.

[chrisstone]

Apologies for digging up an old thread. I just ran into the same issue.

Figured out I had built a newer version of lzma, which placed a symlink in /usr/local/lib for liblzma.so.5 to liblzma.so.5.2. Running ldd showed pkg loading /usr/lib/liblzma.so.5, but this is not the case. Removing the errant symlink solved the issue.

[code2828]

I just ran into the same issue. I solved it by manually download the pkg binaries from pkg.freesd.org

[bapt]

which version of pkg are you using?

[code2828]

@bapt 1.13.2 I am using 13.0-CURRENT.