local repo path bug after multiple "solver iterations"

[courtland]

I ran into a strange issue that I think is worth reporting. Using FreeBSD-10.1-RC3 and pkg 1.3.8.

I am building a custom pkg using "pkg create".

I have two repos setup. One remote and one local.

cat /usr/local/etc/pkg/repos/*.conf
FreeBSD: { enabled: yes }
myco-FreeBSD: {
   url: "pkg+http://packages.myco.com:8080/packages/101-RC3amd64-default/"
   mirror_type: "srv",
   signature_type: "none",
   fingerprints: "/usr/share/keys/pkg",
   enabled: yes
}
myco-mypkg: {
    url             : "file:///tmp/upgrade_mypkg_repo",
    enabled         : yes,
    mirror_type     : NONE
}

The error happened when doing a pkg install against "mypkg" (the custom pkg), which changed the freeradius dependency. I had freeradius-3 installed, and "mypkg" required freeradius-2. pkg correctly detected this problem, which caused a second "solver iteration". At the end of the second solver iteration, the path to my custom "mypkg" pkg in repo "myco-mypkg" was wrong. See the log below...

bsd101-vm# pkg install mypkg
Updating myco-FreeBSD repository catalogue...
Fetching meta.txz: 100%   264 B   0.3k/s    00:01    
Fetching digests.txz: 100%   25 KB  25.4k/s    00:01    
Fetching packagesite.txz: 100%   83 KB  85.1k/s    00:01    
Processing new repository entries: 100%
myco-FreeBSD repository update completed. 282 packages processed:
  0 updated, 0 removed and 282 added.
Updating myco-mypkg repository catalogue...
Fetching meta.txz: 100%   264 B   0.3k/s    00:01    
Fetching digests.txz: 100%   268 B   0.3k/s    00:01    
Fetching packagesite.txz: 100%    2 KB   1.7k/s    00:01    
Processing new repository entries: 100%
myco-mypkg repository update completed. 1 packages processed:
  0 updated, 0 removed and 1 added.
Updating database digests format: 100%
The following 6 packages will be affected (of 0 checked):

New packages to be INSTALLED:
    mypkg: 7572 [myco-mypkg]
    subversion: 1.8.10_3 [myco-FreeBSD]
    serf: 1.3.8 [myco-FreeBSD]
    apr: 1.5.1.1.5.4 [myco-FreeBSD]
    db5: 5.3.28_2 [myco-FreeBSD]
    freeradius: 2.2.5_3 [myco-FreeBSD]

The process will require 123 MB more space.
31 MB to be downloaded.

Proceed with this action? [y/N]: y
Checking integrity... done (1 conflicting)
Checking integrity... done (0 conflicting)
Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.
The following 7 packages will be affected (of 0 checked):

Installed packages to be REMOVED:
    freeradius3-3.0.4

New packages to be INSTALLED:
    db5: 5.3.28_2 [myco-FreeBSD]
    apr: 1.5.1.1.5.4 [myco-FreeBSD]
    serf: 1.3.8 [myco-FreeBSD]
    subversion: 1.8.10_3 [myco-FreeBSD]
    freeradius: 2.2.5_3 [myco-FreeBSD]
    mypkg: 7572 [myco-mypkg]

The process will require 114 MB more space.
31 MB to be downloaded.

Proceed with this action? [y/N]: y
You should remove /usr/local/etc/raddb if you don't need it any more.
==> You should manually remove the "freeradius" user. 
[1/7] Deleting freeradius3-3.0.4: 100%
[2/7] Installing db5-5.3.28_2: 100%
[3/7] Installing apr-1.5.1.1.5.4: 100%
[4/7] Installing serf-1.3.8: 100%
[5/7] Installing subversion-1.8.10_3: 100%
===> Creating users and/or groups.
Using existing group 'freeradius'.
Using existing user 'freeradius'.
===> Setting user and group in radiusd.conf
[6/7] Installing freeradius-2.2.5_3: 100%
===> Adjusting ownership of directory /usr/local/etc/raddb
===> Adjusting ownership of directory /var/log/radacct
===> Adjusting ownership of directory /var/run/radiusd
===> Adjusting ownership of /var/log/radius.log
===> Adjusting ownership of /var/log/radutmp
===> Adjusting ownership of /var/log/radwtmp
===> Updating libdir in /usr/local/etc/raddb/radiusd.conf
pkg: archive_read_open_filename(/tmp/upgrade_mypkg_repo//tmp/upgrade_mypkg_repo/mypkg-7572.txz): Failed to open '/tmp/upgrade_mypkg_repo//tmp/upgrade_mypkg_repo/mypkg-7572.txz'
fbsd101-vm# 

Notice at the very end: "Failed to open '/tmp/upgrade_mypkg_repo//tmp/upgrade_mypkg_repo/mypkg-7572.txz'". The path should have been simply "/tmp/upgrade_mypkg_repo/mypkg-7572.txz". Somewhere the full path to the package file is not getting generated correctly.

If I fix the dependency problem manually by

pkg delete -x freeradius

Then...

pkg install mypkg

mypkg is correctly installed from the local repo. I believe this was because a second "solver iteration" was not required.

Here is the log after manually correcting the freeradius dependency and installing mypkg from the local repo.

fbsd101-vm# pkg install mypkg
Updating myco-FreeBSD repository catalogue...
myco-FreeBSD repository is up-to-date.
Updating myco-mypkg repository catalogue...
myco-mypkg repository is up-to-date.
All repositories are up-to-date.
The following 6 packages will be affected (of 0 checked):

New packages to be INSTALLED:
    mypkg: 7572 [myco-mypkg]
    subversion: 1.8.10_3 [myco-FreeBSD]
    serf: 1.3.8 [myco-FreeBSD]
    apr: 1.5.1.1.5.4 [myco-FreeBSD]
    db5: 5.3.28_2 [myco-FreeBSD]
    freeradius: 2.2.5_3 [myco-FreeBSD]

The process will require 123 MB more space.
31 MB to be downloaded.

Proceed with this action? [y/N]: y
Checking integrity... done (0 conflicting)
[1/6] Installing db5-5.3.28_2: 100%
[2/6] Installing apr-1.5.1.1.5.4: 100%
[3/6] Installing serf-1.3.8: 100%
[4/6] Installing subversion-1.8.10_3: 100%
===> Creating users and/or groups.
Using existing group 'freeradius'.
Using existing user 'freeradius'.
===> Setting user and group in radiusd.conf
[5/6] Installing freeradius-2.2.5_3: 100%
===> Adjusting ownership of directory /usr/local/etc/raddb
===> Adjusting ownership of directory /var/log/radacct
===> Adjusting ownership of directory /var/run/radiusd
===> Adjusting ownership of /var/log/radius.log
===> Adjusting ownership of /var/log/radutmp
===> Adjusting ownership of /var/log/radwtmp
===> Updating libdir in /usr/local/etc/raddb/radiusd.conf
Performing pre-install
[6/6] Installing mypkg-7572: 100%

[avg-I]

I can confirm this bug in similar circumstances with pkg 1.3.8_3 :

pkg: archive_read_open_filename(/usr/local/poudriere/data/packages/basejail-default//usr/local/poudriere/data/packages/basejail-default/All/libxml2-2.9.2_2.txz): Failed to open '/usr/local/poudriere/data/packages/basejail-default//usr/local/poudriere/data/packages/basejail-default/All/libxml2-2.9.2_2.txz

The actual repo URL is file:///usr/local/poudriere/data/packages/basejail-default
As the original reporter says the bug is only triggered when

Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.

The problem seems to come from pkg_repo_binary_try_fetch() function which has the following code:

    if (packagesite[strlen(packagesite) - 1] == '/')
        pkg_snprintf(url, sizeof(url), "%S%R", packagesite, pkg);
    else
        pkg_snprintf(url, sizeof(url), "%S/%R", packagesite, pkg);

    if (!mirror && strncasecmp(packagesite, "file://", 7) == 0) {
        pkg_set(pkg, PKG_REPOPATH, url + 7);
        return (EPKG_OK);
    }

The problem is that when another iteration is done that function is called again on the same pkg object. So, %R would expand to PKG_REPOPATH set in the previous iteration. Thus packagesite can be prepended multiple times when its URL scheme is file://.

I think that @vstakhov might know how to fix this bug.

[avg-I]

This is still a problem AFAICS.

[avg-I]

It seems that the problem is fixed in 1.4.0 RC.