Published packages disappear after about a month

[Zanop]

Problem: Published snapshots made from mirroring official ubuntu repos stop working after about a month. Clients start getting 404, packages are missing.
Aptly version: 0.9.6, 0.9.7

So I want to create snapshots from public repost to sort of freeze package versions for our organisation and control when things get updated. This way I can always install new machines from our repo and know all things will be the same.

Here is what I do:

aptly mirror create -architectures=i386,amd64 xenial-main http://nl.archive.ubuntu.com/ubuntu/ xenial main
aptly mirror create -architectures=i386,amd64 xenial-universe http://nl.archive.ubuntu.com/ubuntu/ xenial universe
aptly mirror -architectures="i386,amd64" create xenial-updates-main http://nl.archive.ubuntu.com/ubuntu/ xenial-updates main
aptly mirror -architectures="i386,amd64" create xenial-updates-universe http://nl.archive.ubuntu.com/ubuntu/ xenial-updates universe

aptly snapshot create xenial-main-20160811 from mirror xenial-main
aptly snapshot create xenial-universe-20160811 from mirror xenial-universe
aptly snapshot create xenial-updates-main-20160811 from mirror xenial-updates-main
aptly snapshot create xenial-updates-universe-20160811 from mirror xenial-updates-universe


aptly publish snapshot -component=main,universe -distribution=xenial xenial-main-20160811 xenial-universe-20160811 ubuntu
aptly publish snapshot -component=main,universe -distribution=xenial-updates xenial-updates-main-20160811 xenial-updates-universe-20160811 ubuntu

So, after that things work well, I'm installing machines using this repo and preseed but, after about a month clients start throwing error 404, packages are not found.
When I try installing new server its usually failing on adduser, which is probably the first package that's missing and needed buy the installer.

I started looking into the issue

I have these mirrors:

 * [xenial-main]: http://nl.archive.ubuntu.com/ubuntu/ xenial
 * [xenial-universe]: http://nl.archive.ubuntu.com/ubuntu/ xenial
 * [xenial-updates-main]: http://nl.archive.ubuntu.com/ubuntu/ xenial-updates
 * [xenial-updates-universe]: http://nl.archive.ubuntu.com/ubuntu/ xenial-updates

from which I made these snapshots:

* [xenial-main-20160811]: Snapshot from mirror [xenial-main]: http://nl.archive.ubuntu.com/ubuntu/ xenial
 * [xenial-universe-20160811]: Snapshot from mirror [xenial-universe]: http://nl.archive.ubuntu.com/ubuntu/ xenial
 * [xenial-updates-main-20160811]: Snapshot from mirror [xenial-updates-main]: http://nl.archive.ubuntu.com/ubuntu/ xenial-updates
 * [xenial-updates-universe-20160811]: Snapshot from mirror [xenial-updates-universe]: http://nl.archive.ubuntu.com/ubuntu/ xenial-updates

and the publishes:

  * ubuntu/xenial [amd64, i386] publishes {main: [xenial-main-20160811]: Snapshot from mirror [xenial-main]: http://nl.archive.ubuntu.com/ubuntu/ xenial}, {universe: [xenial-universe-20160811]: Snapshot from mirror [xenial-universe]: http://nl.archive.ubuntu.com/ubuntu/ xenial}
  * ubuntu/xenial-updates [amd64, i386] publishes {main: [xenial-updates-main-20160811]: Snapshot from mirror [xenial-updates-main]: http://nl.archive.ubuntu.com/ubuntu/ xenial-updates}, {universe: [xenial-updates-universe-20160811]: Snapshot from mirror [xenial-updates-universe]: http://nl.archive.ubuntu.com/ubuntu/ xenial-updates}

There are missing packages in the public/ dir where things are published, for example

aptly@aptlyhost:~/.aptly$ ls -la public/ubuntu/pool/main/a/adduser/
total 28
drwxr-xr-x   2 aptly aptly  4096 Sep 12 06:44 .
drwxr-xr-x 782 aptly aptly 24576 Jul  6 14:06 ..
aptly@aptlyhost:~/.aptly$

is empty

The official repo from wich I mirror show there should be packages there

http://nl.archive.ubuntu.com/ubuntu/pool/main/a/adduser/

If I open the Packages file for the distro though 'public/ubuntu/dists/xenial/main/binary-amd64/Packages' its listed there:

Package: adduser
Priority: required
Section: admin
Installed-Size: 648
Maintainer: Ubuntu Core Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian Adduser Developers <adduser-devel@lists.alioth.debian.org>
Architecture: all
Version: 3.113+nmu3ubuntu4
Replaces: manpages-it (<< 0.3.4-2), manpages-pl (<= 20051117-1)
Depends: perl-base (>= 5.6.0), passwd (>= 1:4.1.5.1-1.1ubuntu6), debconf | debconf-2.0
Suggests: liblocale-gettext-perl, perl-modules, ecryptfs-utils (>= 67-1)
Filename: pool/main/a/adduser/adduser_3.113+nmu3ubuntu4_all.deb
Size: 161698
MD5sum: 36f79d952ced9bde3359b63cf9cf44fb
SHA1: 6a5b8f58e33d5c9a25f79c6da80a64bf104e6268
SHA256: ca6c86cb229082cc22874ed320eac8d128cc91f086fe5687946e7d05758516a3
Description: add and remove users and groups
Description-Md5: 7965b5cd83972a254552a570bcd32c93
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Homepage: http://alioth.debian.org/projects/adduser/
Supported: 5y
Task: minimal
Origin: Ubuntu
Multi-Arch: foreign

The package also exists in the original pool dir

aptly@aptlyhost:~/.aptly$ find pool/  -name \*adduser\*
pool/98/c5/adduser_3.113+nmu3ubuntu3_all.deb
pool/36/f7/adduser_3.113+nmu3ubuntu4_all.deb
aptly@aptlyhost:~/.aptly$

running find pool/ -links 1 and -links 2 and comparing them shows that most files have only 1 link, meaning they are missing from publish. Some are there tho.

Seems like the hard links from pool to publish are missing.

running

aptly snapshot verify xenial-updates-main-20160811 xenial-updates-universe-20160811 xenial-updates-universe-20160811 xenial-universe-20160811

returns a huge list of missing dependencies - 2399

I've checked all crons on the server several times and can't find anything that could remove links/files. All crons are from standard ubuntu/trusty, nothing custom added.

The only things running on the server is aptly serve, nginx which only serves .aptly/public.

This thing happened several times and every time packages start missing after about a month. The way I fix it is by deleting all publishes/snapshots, updating the mirror and creating snapshots and publishes again.

[Zanop]

I think I found the problem which seems to be ubuntu cron job from package 'apt'.
So basically apt-config variables Dir::Cache and Dir::Cache::archives are empty but instead of ignoring them the script /etc/cron.daily/apt uses '/' instead, which uses find to delete *.deb everywhere on the system.

Will check deeper and will close this upon confirmation.

https://bugs.launchpad.net/ubuntu/+source/apt/+bug/937951

[Zanop]

So, I think the issue is solved byt the apt config, Im closing.