repo.saltstack.com should retain old deb packages

[notpeter]

Description of Issue/Question

Currently only the newest debs are available from repo.saltstack.com for a given release channel. Any previous versions are purged when newer packages are published. Old releases are entirely moved into an archive folder (e.g. ubuntu14/archive/2015.8.8.2 and thus no longer available for install via apt-get.

No matter which repo I'm following (e.g. ubuntu14/latest, ubuntu14/2015.8 or ubuntu14/2015.8.8) if I have an issue with a particular release there is no easy way to revert to the previous version which was available hours earlier from the same repo.

For example, say I was running 2015.8.8.1 when 2015.8.8.2 was published. I upgraded but ran into a bug. I should be able to just run apt-get install salt-minion=2015.8.8.1 to go back to the previous version. But because my repo no longer contains the older version in it's package list (or underlying directories) I cannot.

Expected behavior

With the 2015.8 repo configured running apt-cache policy salt-minion should return package versions every 2015.8 release. Basically http://repo.saltstack.com/apt/ubuntu/14.04/amd64/archive/2015.8./.deb

apt-get install salt-minion=2015.8.8+ds-2 salt-common=2015.8.8+ds-2

Should allow reverting to an arbitrary previously packaged version.

Discussion

Yes, the debs haven't been purged, they have just been moved, but reverting now requires me to reconfigure my apt sources.lists. Naturally today it was double pain for me because SaltStack 2015.8.10 broke pkgrepo states (see #33447) so I had to hard code cmd.run states to rollback a broken saltstack version.

I should be able to pin my salt packages to a known working release without having to point my sources.lists definitions at that particular folder under archive/version.

This is not how sane vendor repos operate. Apt fully supports multiple versions of a given package available from a repository and can easily determine what's newest. Purging old packages save a few pennies in storage is extremely short sighted. Additionally, tools such as aptly make repository management simple.

[cachedout]

Sounds like we caused you some pain on that one. My sincere apologies.

@dmurphy18 can you comment on the repo layout, please?

[dmurphy18]

@notpeter @cachedout All previous releases (since repo.saltstack.com was finally up and running properly) are available and can be installed from. They are not moved, as they are always in archive to begin with, but the symbolic links to the releases are moved as new releases of Salt are made available, for example 'latest' was moved from archive/2015.8.9 to archive/2015.8.10 when the later was released.

On the landing page for repo.saltstack.com, it explains the various ways that a user can pick which release of Salt best suits them, whether they want the 'latest release', the latest release of a particular major version of Salt, that is, 2015.5 or 2015.8, or a particular minor release of Salt, for example: 2015.5.8, 2015.8.7. Pinning to the Minor Release allows users to migrate to later releases as it suits their needs without being affected by later releases, and this would appear to be the most suitable for your use case. I am sorry that the latest release broke your installation, SaltStack strives for no breakages with major point release updates, as in 2015.8.8.2 to 2015.8.10. I see that the issue #33447 is being resolved.

Which method to choose, 'latest', major' or 'minor' is left to the user as to which suits their circumstances best. If the instructions on repo.saltstack.com are unclear and need improvement, SaltStack would welcome contributions on improving them.

As for supporting multiple versions of a given package in a repository, that is possible with aptly, but we only maintain a single latest version of a package in a repository (use reprepro) in accordance with Debian guidelines and as requested by the Debian maintainer for SaltStack, noting that Debian also has Salt available from their repositories.

Lastly, SaltStack is considering functionality to allow for roll-back in the future, but this is in the very early stages and no time frame or even if, for that functionality to be available.

Hopefully this answers your concerns.

[notpeter]

Good morning @dmurphy18,

First, I fully consider this an "Enhancement Request" issue as obviously changing your repository management tools and procedures won't happen overnight. Second, let me take a moment to thank SaltStack for taking ownership of packaging rather than relying on downstream maintainers. I do not miss waiting weeks after the official release announcement for pre-build binaries to be made available. But I wanted to give you a little nudge in a direction which makes SaltStack an easy package for SysAdmins to support.

As would be expected I've switched to fully pegged versions from the repos out of the "archive" folder in my sources.list. I'm fine with micro-managing versions at work, but I think a healthier option for most people is to use the always up-to-date repo url (so apt-get upgrade gets them upgrades like every other pacakge) with the ability to rollback if they run into trouble.

If the Debian maintainer (Joe Healy or whomever it is now) would like a single set of packages to follow that's totally reasonable, but it used to be common (official?) practice for mirror maintainers to lazily keep around the previous version of packages for multiple days in case someone hadn't run apt-get update immediately preceeding an install or needed to rollback to verify if an issue was in fact caused by a new package. I'm curious where in the Debian guidelines it suggests only the single latest package should be available from a repo:

A Packages index may contain multiple versions of one binary package, for the same architecture and/or multiple architectures. Source

Again, no need to reply, I just wanted to offer some feedback on the current repo management procedures and suggestion of other tools (aptly) which might make it easier to offer the same packages on multiple different release channels.

[dmurphy18]

@notpeter using reprepro which only allows one, came across 'only latest' again yesterday, have to dig through history, when searching for setting hash to sha256 for Ubuntu 16.04.

But you are right, you can have multiple packages so long as versions / architectures etc are different, just Debian and Ubuntu do not do so, and respecting request from Debian maintainer to keep that for our repo too.

The roll-back, thinking about leveraging btrfs, given 12.04 should be dead next year.

With that, can I close this issue or do you want to turn it into a feature request ?, allowing for roll-back

[notpeter]

@dmurphy18 I would highly recommend aptly over reprepro. We use it internally for packages we roll and it produces signed static content which it can push directly to s3 or elsewhere. I'll likely end up signing the official debs in our repo that and stop relying on repo.saltstack.com all together, but I'd much prefer rely on just apt pinning rater than private repo manipulation.

Again, I'll reiterate, with aptly you could easily publish & sign distinct repos for debian and ubuntu for each of:

• main (multiple versions of each deb)
• archive/major - 2015.8.*
• archive/minor - 2015.8.8
• downstream (only newest supported deb, for downstream use by ubuntu & debian)

I know multiple other folks (@puneetk being one) would benefit from a single mirrorable repo where new versions get added without replacing all previously available debs. Again, just trying to push you towards a repo structure which is both easy for you to maintain and useful for those running saltstack at scale.

Filesystem level rollback (btrfs or zfs) is entirely unrelated and not something that I have any interest in given that apt can handle this perfectly all by itself.

[zerthimon]

+1

[dmurphy18]

@notpeter @zerthimon For the foreseeable future, the plan is too use reprepro. However I shall keep your enhancement request to use aptly instead as a background research item. I need to look into the aspects of keeping older versions of packages available in a repository for a specific version, even if they are available from older paths. As you are probably aware, changing repository layouts is froth with migration issues.

[cachedout]

@dmurphy18 I am moving this to the backlog, since I can't see anything that this is blocking on.

[dmurphy18]

I'll make this icebox

[cmclaughlin]

This applies to yum repos too - it would be nice to be able to downgrade without rewriting yum sources. Can we please update this issue title to apply to both repo types?

[dmurphy18]

@notpeter Salt has a new feature request process, hence if this is still of interest to you, can you please close this issue an make a request here https://github.com/saltstack/salt-enhancement-proposals

[gtmanfred]

That is not what that process is for. That process is for making enhancements to salt, not just feature requests.

So an example of using that repository, is if you decided to implement that featured, you may propose a SEP, to discuss how it would be implemented. And to get feedback from people that would be affected by it.

[dmurphy18]

@gtmanfred Had a quick chat and SEP process is for more than just feature requests to Salt but also 'processes and procedures'. I shall get that added in to the SEP process sometime this week, since only just got the clarification. For example: another issue where I replied similarly for ARM64 packaging support, which would be a major effort.

And actual clarification on this particular issue, yes, maybe SEP is overkill for it.

@notpeter Marked this as 'Won't fix for now' , since researching delivery of packages as part of Flatpack or Snap as a future direction, given this would allow for non-conflicting package version installation, that is, Salt and its dependencies self-contained and thus not conflict with versions available from the underlying OS or user's environment. There is no timeline as to when this will occur, since still researching the issue, but probably for Python 3 support given Python 2 EOL at the end of the year.

Hence closing this issue, but please re-open if the explanation is insufficient and further discussion is required.