Allow to pin packages to specific archive

[d12frosted]

Somewhat fixes or moves towards fixing #6531, #2057 and #3542.

Also #4165 is somewhat related.

Basically thanks to @tonylotts I realised that pinning packages to stable MELPA can be easily automated.

This leverages described problem. But doesn't solve it fully.

Hot it works

This PR does following.

1. Adds MELPA stable to list of package archives. This has minor drawback - longer update. But this is very minor.
2. Exposes function configuration-layer/pin-package which can be used to pin specific package to one of available archives. One can use it manually, but there is a better choice.
3. Adds custom handling for packages with :archive archive-name. To be accurate - it just pins that package.

Usage

(defun dotspacemacs/layers ()
  "Configuration Layers declaration."
  (setq-default
   ;; ...
   dotspacemacs-additional-packages '(;; ...
                                      (ensime :archive melpa-stable))
   ;; ...
   )

This will force Spacemacs to download ensime from MELPA stable.

[bmag]

Using a location recipe that isn't really a recipe kinda violates POLA. It may be better to implement a new syntax:

(ensime :archive "melpa-stable")

Or maybe:

(ensime :pin "melpa-stable")

In this case, configuration-layer/pin-stable-package should be renamed and take an additional archive parameter. The function could issue a warning if archive isn't listed in package-archives, mostly to protect from typos in the package declaration.

[d12frosted]

Hey @bmag,

What you say makes sense. I will update my PR to reflect this. Thanks for the suggestion.

[d12frosted]

I've updated PR.

Instead of using strings - let's use symbols 😸 Checkout updated PR description.

[bmag]

Should use package-archives instead of hard-coded archive names:

;; if archive is a string
(assoc archive package-archives)
;; if archive is a symbol
(assoc (symbol-name archive) package-archives)

[d12frosted]

Good catch. Thanks!

[bmag]

I don't understand why you prefer symbols to strings here, but either way works 😄

[d12frosted]

@bmag thanks for all suggestions once again 😸 💃

I don't understand why you prefer symbols to strings here, but either way works 😄

Yeah, both of them work. My main point is that symbol here is like enum. Only several predefined values are acceptable. On the other hand, string has another meaning 😸

[diego-alvarez-hs]

ping, would be awesome to have this on the develop branch

[d12frosted]

Fixed conflicts.

[d12frosted]

Fixed conflicts 😸

[robsonpeixoto]

A noob question. Can it be used on packages.el http://spacemacs.org/doc/LAYERS.html#packagesel

[d12frosted]

Yes

[diego-alvarez-hs]

ship it! :D

[liujoey]

Ship it please! 😸

[wwentland]

This sounds as if it is not possible to get all packages from stable by default, but as if one would have to enumerate all of them. Would be great if you could allow users to define both the default repository with package-archive-priorities along with per-package overrides as exemplified in this PR.

Thanks!

[clhodapp]

@BABILEN That sounds really likely to break things, as it's likely that the code within some layer that most users have enabled on their systems would break if you changed the default repo to pull packages from. It seems a lot more manageable for the layer itself to specify which package version (from which repo) it's intended to work against.

[wwentland]

@clhodapp Well, sure. It's just that at the moment all packages (appear?) to be daily snapshots from melpa. I don't quite see how that is considered to be less likely to break things than pulling properly released versions by default. :)

[clhodapp]

@BABILEN The problem is gradual vs all-at-once. Sure, some layers break some times, but by moving between melpa and melpa stable, you are likely changing the version of a whole bunch of packages at the same time (and changing them to a version that is potentially a lot different from what most users are running to boot). It seems like a much safer option would be to have separate "stable" and "unstable" versions of layers (potentially both coexisting in the tree at the same time) and letting you pick which set you wanted to use (at the potential cost of losing out on some layers, especially at first). It could maybe work to be able to mix-and-match from the sets, but then you could run the risk pre-inits and post-inits being broken (unless the system is crazy-smart and knows to use the pre-inits and post-inits from stable when a package is coming from a stable layer)... anyway, sorry for the rant ;)

[d12frosted]

@BABILEN AFAIK, MELPA builds all packages continuously. Once build is finished, new one is started. If a package doesn't have any changes between builds, there is no new version. MELPA builds from specified branch (with master being the default). Stable MELPA build from specified branch (with master being the default), but based on tags, not on commits. As practice shows many developers forget to tag 'stable' commits. Some packages have very slow 'release' policy. So even if a package has stable version (which is not true for most packages on MELPA) it might be long away from latest commit, which might bring so many troubles to Spacemacs users. Also keep in mind that Spacemacs is not officially released and it's still in heavy development. But yeah, having support of MELPA stable is a good thing. Not sure when it will come though.

[wwentland]

@d12frosted That's exactly how it works. With support for package-archive-priorities in Emacs 25 the ecosystem will, hopefully, provide proper releases more frequently. I would, personally, much prefer to track properly tested and coherent releases than following the whims of HEAD.

package-archive-priorities also doesn't exclude installations from other repositories, but simply allows you to assign a higher priority to some repositories rather than relying solely on version number comparisons.

I understand that this requires a more rigorous development process than many package developers are willing to adopt, but exposing package-archive-priorities and allowing users to specify exceptions as proposed in this PR would be useful. That way you could treat packages that do not provide proper releases as exceptions rather than the norm. And if this causes too many problems users can still opt to not tinker with this setting.

[d12frosted]

@BABILEN Unfortunately Spacemacs is not really suited for 'stable' releases, so package-archive-priorities will not solve problems at the moment. Because Spacemacs codebase targets latest available versions. If that version has any backward-incompatible changes - using older version breaks your setup. So you have to use new version.

We might provide something like 'snapshots'. The idea is that specific Spacemacs version is known to work with specific set of packages. There is a mirror of popular package archives we maintain. Ideally I would like it to be used for these snapshots.

[wwentland]

@d12frosted But isn't that more a result of the fact that melpa stable is a comparably new thing, rather than a conscious and intentional decision?

In a way you probably want to develop spacemacs against a defined set of dependencies, rather than targeting HEAD. The way it is done right now would also mean that you cannot use the current spacemacs version in one, two or five years, whereas a development process that targets (and potentially pins) stable package releases wouldn't "break" over time.

[d12frosted]

@BABILEN

But isn't that more a result of the fact that melpa stable is a comparably new thing, rather than a conscious and intentional decision?

It's hard to answer this question, but supporting Stable MELPA is not high on the priority list. There are several possible reasons for this.

1. Spacemacs is still not near it's release state. It just allows us to defer this issue 😸
2. There are several ways to provide real stable Spacemacs releases (using Stable MELPA, pinning packages to specific version, using package snapshots à la stack lts packages).
3. And most importantly - no one is working on this. I am planning to start, but I am busy with other things, so it might take some time until I begin doing anything 😸

But keep in mind that I understand the need of such thing. It's just not critical for me 😸 I update not that often and I know how to revert / rollback updates (thanks to Spacemacs it's easy to do).

Current implementation of stable Spacemacs releases is not ideal as it targets specific versions of packages, but these versions are not written anywhere. Our solution to this issue - package fix releases. So in case you wish to use 'stable' release you have to use latest stable release with latest packages. If something breaks - new release will come. In case you are using old version of Spacemacs - you are on your own 😸

P. S. Just in case. Everything I say is just my opinion and not something that can be treated as official project position. Other contributors might disagree with me. 😸

[wwentland]

@d12frosted Thank you for elaborating. I totally understand the reason why this is tricky to implement at the moment.

I also believe that functionality will be used if it is easy to do so, so by making it easier to install, configure and track stable versions the community will, inadvertently, start using it. This could result in a more stable working platform that does not break over time. Rather than having to catch up with every whim (and bug) in HEAD, spacemacs development could focus on targeting certain major package releases.

Thanks for all the work on spacemacs and comments on this issue. Much appreciated!

[syl20bnr]

I was working on this in late december but the solution was not satisfying, this is a tricky topic. The idea was to have package distribution channels:

• frozen (taken from a snapshot)
• stable
• latest

This is great but switching between channels may break Spacemacs since its configuration is bound to specific versions of packages.
Having priorities in Emacs 25 helps but I don't think we can leverage them given the limitation above.

So I start to think that it reduces only to a matter of choosing what we support in Spacemacs: either last stable packages or latest packages but not a mix of them.

There is no good choice to this, since some people want stability and others want latest.

I prefer latest with a fast living ecosystem but there is a major downside to this: sometimes a latest version of a package breaks fresh installs of Spacemacs. Can we find some solutions to this without pinning packages ?

I'm thinking about it, a sort of error detection + fallback may be a solution but not sure how hard it would be to implement.

[what-the-functor]

Is there anything holding this back from being merged?

[achikin]

Any updates on this? I'm running into issues with the latest cider.

[d12frosted]

seems like it's not going to happen 😸 closing PR