Preparing to release 4.0.0

[tarsius]

/cc @Titan-C @contrapunctus @cireu @jethrokuan @r0man @md11235 @manateelazycat @hyatt

I plan to release EmacSQL 4.0.0 toward the end of the month.

The big new feature is the addition of two new SQLite backends. emacsql-sqlite-builtin uses the built-in support in upcoming Emacs 29, provided that wasn't disabled when configuring the build. As a fallback, the new emacsql-sqlite-module can be used when Emacs was built with support for modules. It uses the module provided by the sqlite3 package.

These backends should be preferred over the existing emacsql-sqlite backend, which uses a custom binary, which is slower and more prone to failure. But it is a good final fallback and I don't intend to remove it any time soon, though eventually I probably will.

Users should not have to worry about which backend they use, which makes it necessary that packages are modified lightly. In most cases this should be as simple as:

 (defun demo-db (file)
   (or demo-db
-      (setq demo-db
-           (make-instance 'emacsql-sqlite :file file))))
+      (setq demo-db (emacsql-sqlite-open file))))

If you already have added an option to allow users to pick a backend, you should remove that to ensure that the best backend is used. emacsql-sqlite-default-connection is used to determine that.

The other big change is that going forward all backends are distributed with the emacsql package itself. If we didn't do that, then all packages that want users to use the best available SQLite backend would have to depend on emacsql, emacsql-sqlite, emacsql-sqlite-module and emacsql-sqlite-builtin, which would be silly. (See melpa/melpa@4872ef0 for why we include all backends.)

The emacsql packages on Melpa and NonGNU-devel Elpa started to include all libraries a little while ago, but for the time being the backends continue to also be available as separate packages. That's not a good situation, but in this transitional phase it is necessary to prevent packages that depend on emacsql-sqlite from breaking. That being said, we should complete the transition as soon as possible.

So please update the dependencies, like so now:

-;; Package-Requires: ((emacsql "3.1.1") (emacsql-sqlite "3.1.1"))
+;; Package-Requires: ((emacsql "20230228"))

That is, drop the dependency on emacsql-sqlite and temporarily depend on the Melpa snapshot version of emacsql. Doing the latter ensures that, along with your package, users update to a version of emacsql that contains emacsql-sqlite and the new backend libraries. That won't cause the emacs-sqlite to be uninstalled and I will have to deal with that later, by displaying a warning, instructing users to do so manually.

After depending on the snapshot, you also won't be able to create a new release until I have released EmacSQL 4.0.0, i.e. later this month (and of course you could create a release just before starting to depend on the snapshot). When you create the first new release after EmacSQL 4.0.0 was released, you will have to make sure to change from the snapshot version to 4.0.0.

[akirak]

Thank you for maintaining MELPA, magit, and other things. I have understood your point described in this thread.

As a fallback, the new emacsql-sqlite-module can be used when Emacs was built with support for modules. It uses the module provided by the sqlite3 package.

Now that the snapshot version of emacsql package contains both emacsql-sqlite-builtin.el and emacsql-sqlite-module.el, which depend on sqlite.el and sqlite3.el respectively, how should an Emacs package manager handle those dependencies?

For example, Emacs 29 will choose emacsql-sqlite-builtin so it shouldn't require sqlite3 package, but is it possible for package.el to conditionally install the dependency? I don't know of any mechanism to allow specifying conditional dependencies in Emacs Lisp, but is it possible?

I am working on an alternative package manager for Emacs, so I would like to know how the package is supposed to be built and distributed to its users.

[tarsius]

For example, Emacs 29 will choose emacsql-sqlite-builtin so it shouldn't require sqlite3 package, but is it possible for package.el to conditionally install the dependency? I don't know of any mechanism to allow specifying conditional dependencies in Emacs Lisp, but is it possible?

Unfortunately there's no such mechanism. So sqlite3.el along with some other files will always be installed. This is Melpa's recipe:

(sqlite3 :fetcher github
         :repo "pekingduck/emacs-sqlite3-api"
         :files (:defaults
                 "Makefile"
                 "consts.c"
                 "emacs-module.h"
                 "sqlite3-api.c"))

But that is acceptable. package.el won't actually attempt to build the module until sqlite3 is required. The above files sit around doing nothing, but they don't take up much space.

I am working on an alternative package manager for Emacs, so I would like to know how the package is supposed to be built and distributed to its users.

A gnu/linux or bsd distribution should probably only include the emacs-sqlite-* suitable for the emacs version they currently provide. The other extreme are package managers like package.el and my borg, which are quite a bit dumber, and have to resort to installing all backends and their dependencies. (Actually borg is even dumber and doesn't handle dependencies at all, so it is up to the user to decide whether to install sqlite3 or not.)

I don't know where your package manager falls on that spectrum, but my guess is that it would be best (and certainly simplest) to just install sqlite3. If your package manager makes sure to compile all modules when a package that ships in is installed, then it might be worth considering dropping the dependency when using emacs 29. On the other hand, the user might go back to emacs 28 for some reason and then it would be nice if sqlite3 were already installed.

[akirak]

A gnu/linux or bsd distribution should probably only include the emacs-sqlite-* suitable for the emacs version they currently provide.

Is it fine to even eliminate emacsql-sqlite-module.el from the output for Emacs 29? Without the dynamic module for sqlite3.el, the library fails to byte-compile, so one solution is to ignore the file and remove it entirely from the output. If it is possible, it will save time on byte-compiling sources ahead of installation, which is done by some package managers (e.g. straight.el and mine).

My package manager can be as flexible as the user wants it to be, because it allows working around using Nix, which is a turing complete language.

[tarsius]

Is it fine to even eliminate emacsql-sqlite-module.el from the output for Emacs 29?

It is fine if you know that no Emacs 29 user will ever go back to 28. Does your package manager have a "I am switching Emacs version, recompile all installed packages using the one I am switching to" command, and users are aware that they must always run that when switching?

I (and every other author of an emacs package) sometimes have to switch to old Emacs releases because users insist on sticking to those and occasionally report issues that are not present when using the latest Emacs release.

Right now it would be no problem if emacsql-sqlite-module were missing while being forced to use emacs 28; EmacSQL would just fall through even further to use the old emacs-sqlite backend. But eventually I would like to remove that backend.

Without the dynamic module for sqlite3.el, the library fails to byte-compile, so one solution is to ignore the file and remove it entirely from the output.

What library? emacsql-sqlite-module.el should compile just fine. Maybe we should add no-byte-compile: t to sqlite3.el.

[akirak]

Does your package manager have a "I am switching Emacs version, recompile all installed packages using the one I am switching to" command, and users are aware that they must always run that when switching?

My package manager rebuilds all Emacs Lisp packages every time the user updates Emacs. The artifacts are stored in a centralized Nix store, so multiple versions can exist simultaneously. Byte/native-compiling can be turned off, if the user wants to.

What library?

I am talking about emacsql-sqlite-module.el. It is possible to extend my package manager to remove emacsql-sqlite-module.el for Emacs 29 and emacsql-sqlite-builtin.el for Emacs 28 or before.

emacsql-sqlite-module.el should compile just fine. Maybe we should add no-byte-compile: t to sqlite3.el

Yes, it's probably sqlite3.el that failed to byte-compile.

[tarsius]

Sounds like I might have to do another round of testing. I probably won't do that today.

It is possible to extend it to remove emacsql-sqlite-module.el for Emacs 29 and emacsql-sqlite-builtin.el for Emacs 28 or before.

Err, I have to ask again what "it" refers too. Sounds like you are asking whether emacsql-sqlite-module.el can be extended to remove itself?!

In any case, I don't think EmacSQL can remove parts of itself (how would that even work)--you would have to do that in the nix package build steps.

[akirak]

Err, I have to ask again what "it" refers too. Sounds like you are asking whether emacsql-sqlite-module.el can be extended to remove itself?!

Sorry for this. I meant extending my package manager.

[r0man]

@tarsius I just updated the paimon.el package to use the latest snapshot versions of closql and emacsql.

[tarsius]

Thanks!

[tarsius]

Note to self: don't forget to update the documentation before release.

• Packages won't be split up the same as they are now.