.real_... directory in repo packagesite paths

[courtland]

I have a pretty standard poudriere repository setup. Everything has been working fine until yesterday, when I used the bulk command to update a single port. I was running poudriere 3.1.1, from the official FreeBSD packages repo. The new port built correctly and the resulting repo seemed fine. I later realized that the "repopath" and "path" keys in the resulting packagesite.txz were now prepended with ".real_1428515828/".

For example:

{"name":"wget","origin":"ftp/wget","version":"1.16","comment":"Retrieve files from the Net via HTTP(S) and FTP","maintainer":"vd@FreeBSD.org","www":"http://www.gnu.org/s/wget/","abi":"FreeBSD:10:amd64","arch":"freebsd:10:x86:64","prefix":"/usr/local","sum":"9931790f27a4ea32acdfe6c7a0d5dd6b61d66bc3065f16789e238fd8e19e2576","flatsize":1975752,"path":".real_1428515828/All/wget-1.16.txz","repopath":".real_1428515828/All/wget-1.16.txz"
...

This caused pkg upgrades on my FreeBSD machines to fail, with errors such as:

pkg: archive_read_open_filename(/space/dist/packages/101-RELEASE-amd64-official/.real_1428515828/All/xproto-7.0.26.txz): Failed to open '/space/dist/packages/101-RELEASE-amd64-official/.real_1428515828/All/xproto-7.0.26.txz'

I realize the ".real_1428515828" directory exists for atomic bulk build purposes, but the way I am mirroring the resulting repository drops directories beginning with dots.

In the past, the .real_ path has been absent from the packagesite.txz. This is the first time I've noticed this problem, and I've been running poudriere since before 3.0.

After upgrading to poudriere 3.1.3 and re-running bulk a few times, the new packagesite.txz is seemingly correct, and does not contain any references to ".real_...". For example:

{"name":"wget","origin":"ftp/wget","version":"1.16","comment":"Retrieve files from the Net via HTTP(S) and FTP","maintainer":"vd@FreeBSD.org","www":"http://www.gnu.org/s/wget/","abi":"FreeBSD:10:amd64","arch":"freebsd:10:x86:64","prefix":"/usr/local","sum":"9931790f27a4ea32acdfe6c7a0d5dd6b61d66bc3065f16789e238fd8e19e2576","flatsize":1975752,"path":"All/wget-1.16.txz","repopath":"All/wget-1.16.txz"
...

Note the path now lacks the .real_... directory (path":"All/wget-1.16.txz").

So I guess I have a few questions:

Should the .real_... directory be mentioned in the packagesite.txz? Or is this a bug?

Is this a known bug that has been fixed between 3.1.1 and 3.1.3? What is strange is that I was using 3.1.1 for some time, and it wasn't until recently that it started to add the .real directory to the packagesite. All I did was rebuild a port, which I had done before running 3.1.1 without resulting in a broken packagesite.txz.

Is there something I can do to ensure the .real_... paths do not end up in the packagesite.txz? Or should I fix my replication mechanism to handle the new "dotfile" directories?

Thanks.

[bdrewery]

Did you manually run pkg repo at all? Running it from the top-level PACKAGES directory can cause this to happen.

For mirroring the repo you should mirror the PACKAGES/.latest/ directory rather than PACKAGES/ since it contains all of the .real stuff. Ignoring dotdirs would do the trick too as long as you are following symlinks.

This had been fixed in freebsd/pkg@97a4693 but perhaps it is broken now.

[bdrewery]

I'm wrong about my referenced commit but I'll look into it more.

[courtland]

I was running an older version of pkg on the system running poudriere, so perhaps that has something to do with it? I don't believe I ever ran "pkg repo" at all, especially not in the poudriere PACKAGES directory.

Should the .real_... directory ever get included in the packagesite YAML?

I have an nginx server hosting the repo. The conf is similar to the example provided with poudriere.

        location /pkgrepo {
            alias      /poudriere/data/packages;
            index  index.html index.htm;
            autoindex on;
        }

As far as I can tell everything is symlinked to .real_1428609296. My mirroring happens over HTTP.

Thanks.

[bdrewery]

Did you accidentally run an older poudriere on the same repo that 3.1 created? No the .real dirs should not be in the packagesite.yaml but there is nothing preventing it currently. I'm working on a patch for pkg to prevent it.

[courtland]

No I've been running 3.1.1 for quite some time. I was running poudriere 3.1.1 and pkg-1.4.6 when the problem occurred. I have since upgraded to poudriere-3.1.3 and pkg-1.4.12.

Thanks for clearing up the confusion with .real dirs.

[bdrewery]

I can't explain how you ended up with .real dirs in your packagesite.

[courtland]

Thanks for the quick fixes. I will work around the problem for now on my end until the new stable pkg 1.4 is released. At the moment the packagesite in my repository is correct.