Upcoming version problem with the Emacs release

[Malabarba]

Hi Nico, I just realised a problem that we're about to face. Once Emacs 25 is released,there'll be a version of seq.el (2.x) in the wild, with a fixed set of features.
Later on, if you add new features to the seq.el that's on Elpa (whose version is 1.y) these features will be impossible to use without issue.

For instance, let's say you add seq-map-indexed on version 1.15, and I want to use it on a package. For that, I would require seq.el version 1.15 on my package.

However, I have no guarantee that I'll get that. If the user is running Emacs 25.1, they'll have a seq.el installed with version 2.something (which satisfies the dependency) but does not contain map indexed.

The same will happen for all newly added features. And seriously reduces the usefulness of seq as dependency package.

I understand that the version number seq.el inside Emacs core is marked as 2.x because it has the new generic functionality that the Elpa version will never have. However, maybe this generic functionality could be made available as a separate package (it could still be part of Emacs core) and the two seq.el packages could be allowed to converge to the same version number (even if their implementation differs).

Cheers

[NicolasPetton]

Artur Malabarba notifications@github.com writes:

Hi Nico, I just realised a problem that we're about to face. Once
Emacs 25 is released,there'll be a version of seq.el (2.x) in the
wild, with a fixed set of features. Later on, if you add new features
to the seq.el that's on Elpa (whose version is 1.y) these features
will be impossible to use without issue.

Indeed, I just realized that as well.

I understand that the version number seq.el inside Emacs core is
marked as 2.x because it has the new generic functionality that the
Elpa version will never have. However, maybe this generic
functionality could be made available as a separate package

I don't understand how you would do that.

what about putting seq.el 2.x into ELPA as well?

Then Emacs 25.x users could still get the latest version of seq.el with
the latest features. Imagine the following:

• Emacs 25.1 has seq 2.1
• ELPA has seq 2.12 and seq 1.12, both with the same set of features (apart from
  the generic stuff, and implemented differently).

Nico

[Malabarba]

I don't understand how you would do that.

I'm not sure either. =/

what about putting seq.el 2.x into ELPA as well?

Then Emacs 25.x users could still get the latest version of seq.el with
the latest features. Imagine the following:

• Emacs 25.1 has seq 2.1
• ELPA has seq 2.12 and seq 1.12, both with the same set of features (apart from
  the generic stuff, and implemented differently).

Elpa doesn't know how to advertise multiple versions of the same package. Even if it did, I don't think it would work.

Say a package foo depends on seq 1.12 (because it wants some very recent feature and wants to support Emacs 24). Now, if a user from Emacs 25 intalls foo, then package.el is not going to install seq 2.12 (nor 1.12), because Emacs 25 already has seq 2.1, which satisfies the 1.12 dependency (even though it has fewer features).

At the heart of this issue is the fact that seq 2.x doesn't necessarily include all the features in seq 1.y, even though it's a higher version.

---

Here's another idea. What if we had a single seq package on Elpa (carrying a 2.x version number), that offered both implementations. It would have 3 files:

- seq.el
- seq-25.el
- seq-24.el

• The file seq.el would be the main package file, and have the following contents:

  ;;; seq.el --- Sequence manipulation functions  -*- lexical-binding: t -*-
  
  ;; Copyright (C) 2014-2016 Free Software Foundation, Inc.
  
  ;; Author: Nicolas Petton <nicolas@petton.fr>
  ;; Keywords: sequences
  ;; Version: 2.3
  ;; Package: seq
  
  ;; Maintainer: emacs-devel@gnu.org
  
  ;; This file is part of GNU Emacs.
  
  ;; GNU Emacs is free software: [... usual jargon here ...]
  
  ;;; Commentary:
  
  ;; Sequence-manipulation functions that complement basic functions
  ;; provided by subr.el.
  ;;
  ;; All functions are prefixed with "seq-".
  ;;
  ;; All provided functions work on lists, strings and vectors.
  ;;
  ;; Functions taking a predicate or iterating over a sequence using a
  ;; function as argument take the function as their first argument and
  ;; the sequence as their second argument.  All other functions take
  ;; the sequence as their first argument.
  
  ;;; Code:
  
  (if (version< emacs-version "25")
      (require 'seq-24)
    (require 'seq-25))
  
  (provide 'seq)
  

• The file seq-24.el would have the exact same contents as the current seq-1.x we have on Elpa except:

  • The file-header and the provide statement would say seq-24
  • It wouldn't have the package pseudo-headers (no version number, no keywords, etc).

• The file seq-25.el would have the exact same contents as the current seq-2.x we have in Emacs except:

  • The file-header and the provide statement would say seq-25
  • It wouldn't have the package pseudo-headers (no version number, no keywords, etc).

This way there is only one version of the seq package. Packages that want some recent functionality added in minor-version 12 can simply depend on 2.12, regardless of the Emacs version.
Also, packages that want the generics functionality can simply depend on (seq "2.12") and on (emacs "25").

The only issue I can think of is that when seq gets installed on Emacs 24, the file seq-25.el will be byte-compiled (even though it's never used) and there'll be a compilation warning about “functions not known to be defined”.

[NicolasPetton]

Artur Malabarba notifications@github.com writes:

Say a package foo depends on seq 1.12 (because it wants some very
recent feature and wants to support Emacs 24). Now, if a user from
Emacs 25 intalls foo, then package.el is not going to install seq
2.12 (nor 1.12), because Emacs 25 already has seq 2.1, which satisfies
the 1.12 dependency (even though it has fewer features).

Of course, you are right.

Here's another idea. What if we had a single seq package on Elpa (carrying a 2.x version number), that offered both implementations. It would have 3 files:

- seq.el
- seq-25.el
- seq-24.el

I think that would work.

The only issue I can think of is that when seq gets installed on
Emacs 24, the file seq-25.el will be byte-compiled (even though it's
never used) and there'll be a compilation warning about “functions not
known to be defined”.

Could we declare these functions, just to get rid of the compilation
warnings?

[Malabarba]

Could we declare these functions, just to get rid of the compilation warnings?

Yes, that should work.

[Malabarba]

Actually, there will probably be more problems involved with this byte-compilation. For instance, these two lines will throw errors:

(eval-when-compile (require 'cl-generic))
(require 'cl-extra) ;; for cl-subseq

So you would have to wrap them with something like this (I think):

(eval-when-compile
  (when (version<= "25" emacs-version)
    (require 'cl-generic)))
(when (version<= "25" emacs-version)
  (require 'cl-extra))

This will prevent any errors, but there'll still be a ton of byte-compiler warnings for various reasons.
For instance, pcase-defmacro won't be defined, so the compiler assumes it is a function, and so it will complain that seq is an unbound variable and &rest is not a function.

An alternative would be to prevent the compilation of this file (by adding ;; no-byte-compile: t to the local variables at the bottom).
Then we can add something like this to seq.el (before the two requires) to ensure it gets compiled:

(eval-when-compile
  (when (version<= "25" emacs-version)
    (byte-compile-file "seq-25.el")))

The problem is that this last part won't work because byte-compile-file respects no-byte-compile too. :(

[NicolasPetton]

that's problematic. I think the best thing to do would be to have seq.el require seq-25.el or seq-24.el (as separate ELPA packages) depending on the version of Emacs, but I don't think we can to that today.

[Malabarba]

Yeah. Wonder if there's another way to prevent byte-compilation only on some emacs versions.

[NicolasPetton]

Artur Malabarba notifications@github.com writes:

Yeah. Wonder if there's another way to prevent byte-compilation only
on some emacs versions.

I think I'll explain the issue on emacs-devel.

[NicolasPetton]

Yes, that seems like a good idea!

[Malabarba]

What idea? 😮

[NicolasPetton]

nothing, I answered the to wrong issue :)

[NicolasPetton]

Artur Malabarba notifications@github.com writes:

This will prevent any errors, but there'll still be a ton of byte-compiler warnings for various reasons.
For instance, pcase-defmacro won't be defined, so the compiler
assumes it is a function, and so it will complain that seq is an
unbound variable and &rest is not a function.

An ugly solution would be to define the macro, to avoid any
byte-compilation warning:

(eval-when-compile
  (unless (fboundp 'pcase-defmacro)
    (defmacro pcase-defmarco (&rest _))))

[Malabarba]

Hm, that might work. Instead of actually defining the macro, it might be more polite to just add it to the variable byte-compile-macro-environment. That way it should only take effect on the current file.

[NicolasPetton]

Hm, that might work. Instead of actually defining the macro, it might
be more polite to just add it to the variable
byte-compile-macro-environment. That way it should only take effect
on the current file.

I've never used this variable, can I just add the definition as a list
there?

[Malabarba]

Yes, I think something like this should do

(push `(pcase-defmacro . ,(lambda (&rest _) nil))
      byte-compile-macro-environment)

(And similarly for any other macros we need to define.)

[cpitclaudel]

Hey Artur and Nicolas,

I must be misunderstanding something, but that confusion may be shared with other users: what would be wrong with making sure that the current MELPA seq has the same feature as the one in Emacs right now (so that the 2.x branch isn't missing any feature), and then just bumping the version number of this library to, say, 2.5 as soon as new features are added? Then packages would require that instead of 1.x.

IOW, my proposal is to make 2.current equal to 1.current by updating the version bundled with Emacs, and then use a higher number for this library. Would that not work? IIUC no packages would be broken by the update to Emacs 25 (since Emacs 25's seq would have everything they depend on) and no packages would break when updating their dependencies, because all new releases of seq would have higher version numbers than the one bundled with Emacs.

Is there something wrong with this?

[johnmastro]

@cpitclaudel, IIUC the fundamental issue is that ELPA seq is updated more frequently than Emacs makes releases.

[Malabarba]

what would be wrong with making sure that the current MELPA seq has the same feature as the one in Emacs right now

The version of seq that's in Emacs right now uses some new functionality that's part of Emacs 25. If we updated the seq package on Melpa to be the same, then it would have to drop support for Emacs 24. Which is not viable.

[cpitclaudel]

@johnmastro I don't think that's incompatible with the solution I outlined.

@Malabarba I was suggesting updating the Emacs package to be the same as the one on Melpa, not the other way around :) My suggestion was to update the Emacs one to be interface-equivalent to the current Melpa one (so that nothing breaks when users update to Emacs 25), then bump the version of the Melpa one to 2.5 or something, so everyone picks it up when updating.

[Malabarba]

@Malabarba I was suggesting updating the Emacs package to be the same as the one on Melpa, not the other way around :)

Oops. My bad.

My suggestion was to update the Emacs one to be interface-equivalent to the current Melpa one (so that nothing breaks when users update to Emacs 25), then bump the version of the Melpa one to 2.5 or something, so everyone picks it up when updating.

The Emacs version is already interface-equivalent with the Melpa version. The problem is that the Emacs version offers additional functionality that the Melpa version doesn't offer. Do you see what I mean?

So the problems we have are:

1. If the Melpa version does not contain the defgeneric or the pcase functionality, then it cannot be tagged 2.x (even if it's otherwise equivalent).
2. Developing two major versions in parallel (1.x for the Melpa version and 2.x for the Emacs version) is bad, because it means any Melpa version I depend on (like 1.10) will get satisfied by any Emacs version the user might have (like 2.3).
3. The Melpa version can't drop support for Emacs 24 (which didn't have the defgeneric or the pcase functionality).

Given those three restrictions, the only solution I see is to make the Melpa package provide both versions (the Emacs 25 and the Emacs 24 versions) which then allows it to follow the same versioning scheme as the Emacs 25 version (2.x).

But it's possible I misunderstood your suggestions.

[cpitclaudel]

If the Melpa version does not contain the defgeneric or the pcase functionality, then it cannot be tagged 2.x (even if it's otherwise equivalent).

Hmm. Can we implement the defgeneric and pcase functionality in Emacs 24?

[NicolasPetton]

Clément Pit--Claudel notifications@github.com writes:

Hmm. Can we implement the defgeneric and pcase functionality in Emacs
24?

That's not really required. The only thing that we'll miss in the Emacs
24.x version of seq.el is the extensibility (eg defining new types of
seqs, such as stream.el).