Preview: Emacs enhancements

[ttuegel]

This is a preview for the community of some changes to Emacs packaging I have been preparing. I have been using this for some time already and I hope there is still enough time to get it into NixOS 15.05. There are two large changes:

1. Packages are generated automatically by downloading metadata from ELPA, Marmalade, MELPA, and org-mode ELPA.
2. A wrapper is provided so that the expression emacsWithPackages [ pkgA pkgB ... ] provides an Emacs which knows about the specified packages.

I look forward to your feedback!

Package Generation

TL;DR: Automatically-updated expressions for 1822 Emacs packages.

The package metadata can be downloaded from any package.el-compatible server. An unholy mixture of Elisp and Haskell converts the S-expression downloaded from the server to JSON; along the way, all the packages are downloaded and hashed. This JSON is loaded into a Nix expression using builtins.fromJSON and a set of packages is born! Two package sets are provided, emacs-packages includes packages from ELPA, Marmalade, and stable MELPA, while emacs-packages-unstable adds packages from unstable MELPA and org-mode ELPA.

Features:

• The package updater is fast. Downloading and hashing all the sources listed above, 1822 packages total, takes <10 minutes on my 7-year-old laptop over Wi-Fi. (The updater is a separate package and will be available shortly.)
• Package metadata is updated incrementally, i.e. only new packages are downloaded and hashed. Incremental updates take 30-50 seconds.
• The package expressions are built by the lovely melpaBuild.

Emacs Wrapper

The Emacs wrapper defines environments for Emacs without having packages installed in the user or system profiles. File collisions between packages and Emacs is a major cause of breakage. (I'm looking at you, org-mode!) Now, I just have this in my environment.systemPackages:

    (emacsWithPackages (with emacs-packages-unstable; [
      auctex company company-ghc diminish
      evil evil-leader flycheck git-auto-commit-mode
      git-timemachine haskell-mode helm magit
      markdown-mode monokai-theme
      org-plus-contrib rainbow-delimiters
      undo-tree use-package
    ]))

and everything Just Works!

The wrapper works by creating a custom site-start.el file which adds the packages to load-path and wrapping Emacs so the custom start-up file is found first. Adding packages to the load path this way comes at the price of start-up time: my load times increased about 5% (0.15 seconds) due to this change. Of course, all my packages actually work now, so that's a big bonus. Of course, it is still possible to use an un-wrapped Emacs, which will read packages from the user and system profiles as before.

[peti]

Wow, this looks really nice!

[the-kenny]

I don't think we should keep any packages from Melpa. Melpa uses daily snapshots - we just can't keep up with that.

[the-kenny]

Oh wait - this PR doesn't include any predefined package sets? Sorry, misunderstood that.

We should definitely ship packages from Marmalade and Melpa-Stable though.

[jgeerds]

@ttuegel Great! 🍻

[the-kenny]

Another thing: Is emacsWithPackages mandatory? I patched our Emacs some time ago to automatically search each entry in NIX_PROFILES so we should be able to just install packages via nix-env or environment.systemPackages.

Package collisions aside - I think we should support both.

[the-kenny]

Oh, and: Great! Nice work!

[edolstra]

Cool stuff :-)

Nitpick: emacs-packages should be emacsPackages IMHO, since the convention is to use camelCase for non-package variable names (like pythonPackages, haskellPackages, ...).

[ttuegel]

Oh wait - this PR doesn't include any predefined package sets? Sorry, misunderstood that.

We should definitely ship packages from Marmalade and Melpa-Stable though.

It defines emacs-packages for the stable packages (ELPA, Marmalade, stable MELPA) and emacs-packages-unstable for the unstable. I can probably live with dropping unstable MELPA, though.

[ttuegel]

Another thing: Is emacsWithPackages mandatory? I patched our Emacs some time ago to automatically search each entry in NIX_PROFILES so we should be able to just install packages via nix-env or environment.systemPackages.

emacsWithPackages is orthogonal to the package updates. If you install packages in the normal way, they will continue to work. In fact, I have not made emacsWithPackages completely pure. Emacs will still get packages from the active profiles, but it will only search there after the explicit package list is exhausted.

[domenkozar]

Adding documentation tag - we should update nixos release notes and write documentation for Nixpkgs manual.

[jgeerds]

I can probably live with dropping unstable MELPA, though.

I find MELPA unstable very useful because a lot of emacs packages don't have a stable version. There are about 1400 packages without a declared version number, that's more than the half

[ttuegel]

I find MELPA unstable very useful because a lot of emacs packages don't have a stable version. There are about 1400 packages without a declared version number, that's more than the half

I think there's a way I can keep everyone happy here. I can give each repository a priority in the updater, such that packages from stable repositories are chosen over packages from unstable repositories regardless of their version. If a package is in both stable and unstable MELPA, we would take the stable version; otherwise, we would take the unstable version. I think the packages without stable versions are not updated very often, so there isn't too much version churn to keep up with. @geerds @the-kenny Would this satisfy both of you?

[jgeerds]

Would this satisfy both of you?

Yes, good idea!

[gridaphobe]

First of all, thanks @ttuegel for working on this! I was going to start on my own version when I get back from my honeymoon, but you beat me to it :)

A couple comments:

1. It would be nice if the PR also added https://github.com/ttuegel/elpa2nix as a derivation so us other emacs maintainers have slightly easier access :)
2. I'm a bit concerned about keeping the package list up-to-date. I know melpa-unstable only keeps the most recent version (as was discussed above), but what about melpa-stable and gnu elpa, etc? If the package archives don't keep older versions around we may find ourselves with a regular source of broken packages, which could be especially problematic for stable branches of nixpkgs. My originally idea (which admittedly would only work for melpa) was to join their archive.json and recipes.json files to get the package metadata as well as the source info, and then build the archive from source using melpaBuild. This would circumvent any stale archive issues.

[ttuegel]

It would be nice if the PR also added https://github.com/ttuegel/elpa2nix as a derivation so us other emacs maintainers have slightly easier access :)

When it was finished, it was going to go on Hackage, where it would be kept up-to-date automatically, but...

I'm a bit concerned about keeping the package list up-to-date. I know melpa-unstable only keeps the most recent version (as was discussed above), but what about melpa-stable and gnu elpa, etc? If the package archives don't keep older versions around we may find ourselves with a regular source of broken packages, which could be especially problematic for stable branches of nixpkgs. My originally idea (which admittedly would only work for melpa) was to join their archive.json and recipes.json files to get the package metadata as well as the source info, and then build the archive from source using melpaBuild. This would circumvent any stale archive issues.

I hadn't considered that. The archives do not keep old versions, so what I'm doing simply won't work. And, as you point out, only MELPA exposes the information we need to build packages from the original source repositories; by my count, that's about half of the stable packages. Unless we want to mirror all these archives somewhere, there's really no way for us to capture all, or even most, of the available Emacs packages.

[ttuegel]

Thinking about it more, I see two paths forward:

1. As @gridaphobe suggests, we use the MELPA recipes to build packages directly from the original source repositories.
2. We can mirror the packages. Actually ELPA is well-behaved and keeps old versions, but it's the only one. Because Github has HTTP access to repositories, we could keep the mirrored files in a Git repository. (The packages are distributed as single Elisp files and small tar archives thereof; they are not very large files.) It would be simple to modify elpa2nix to update that repository when it downloads new packages. Each package version already has a unique filename, so we could keep old versions accessible through Github HTTP indefinitely.

The disadvantage to the first approach is that we miss all the ELPA and Marmalade packages; although MELPA is the most comprehensive archive, I think we would still miss about 1000 packages. I was initially against mirroring the packages. Now I think that although mirroring is a little ugly, the coverage gain may be worth it. Mirroring also lets us support our stable releases better: TeXLive has the same problem, and as a result, using TeX on a release branch is a slightly miserable experience.

I would appreciate some feedback here. I have to admit a bias: the first option above requires throwing out most of my efforts on elpa2nix! Am I crazy to consider mirroring the package archives?

[gridaphobe]

I was similarly opposed in spirit to mirroring packages when I first started thinking about this, but as long as we're not breaking GitHub's ToS I think I can live with the kludginess 😄

[ttuegel]

I just reviewed the ToS, and nothing forbids it. The ToS remarks that you must be legally able to redistribute whatever content you put up. The packages are all open-source just to be on *ELPA, so that's not a problem. However, the ToS also warns that repositories may be suspended for bandwidth use exceeding the typical, so we may want to avoid that. That reduces our options to mirroring somewhere else, or using the MELPA recipes.

I think using the MELPA recipes is not so bad. There seems to be a lot of duplication between MELPA and Marmalade. (ELPA is small and seems to keep old versions around indefinitely, so I don't think we should worry about that.) The reason that stable MELPA is so small seems to be mainly that unstable MELPA has many different plugins from emacswiki that don't have proper, stable versions. Some of the Marmalade packages on MELPA don't seem to have properly tagged Git repositories; I'm not sure what to do about that. If users do want packages in Nixpkgs that aren't in MELPA, we can submit recipe PRs upstream. I don't have a problem being the Debian of Emacs packages! :)

[ttuegel]

To stem the tide of bug reports: yes, the packages generated this way become outdated too quickly. As explained above, I am abandoning elpa2nix. I think we should generate packages based on MELPA's recipes. I intend to work on that in the near future, but I am not working on it at this time. The Emacs wrapper is still useful, though, and I will push it to master when I figure out some byte-compiling issues.

[ttuegel]

OK, emacsWithPackages is in master.

[purcell]

MELPA maintainer here.

I think using the MELPA recipes is not so bad. There seems to be a lot of duplication between MELPA and Marmalade.

Yes, MELPA is generally a superset of Marmalade, except for a few exceptions. The exceptions are split between packages which we declined on quality grounds, and ones which have been manually uploaded from a non-version-controlled source or a make result, such that MELPA can't host/build them.

(ELPA is small and seems to keep old versions around indefinitely, so I don't think we should worry about that.)

In case it's an issue, I should mention that we don't keep old versions currently. We'd like to start keeping just a few old versions.

The reason that stable MELPA is so small seems to be mainly that unstable MELPA has many different plugins from emacswiki that don't have proper, stable versions. Some of the Marmalade packages on MELPA don't seem to have properly tagged Git repositories; I'm not sure what to do about that.

There's a MELPA issue somewhere for that: the right course of action is to chase up the upstream maintainers and get them to add tags.

There are also a few packages which are in SCM systems which MELPA Stable does not yet handle. For better or worse, we refuse to trust versions embedded in files, since they're simply unreliable. Emacswiki-hosted packages are unlikely to ever make it into MELPA Stable, for exactly that reason.

Overall, due to these issues, MELPA Stable is fairly hit-and-miss, since dependencies for a listed package can sometimes be completely missing. The MELPA maintainers don't use Stable ourselves, because MELPA itself is not at all unstable in practice, so Stable is there primarily by popular demand on an "as-is" basis. I think many MELPA Stable users also have MELPA enabled, so that they can explicitly install "stable" packages on a package-by-package basis, while still having all the dependencies available for installation.

If users do want packages in Nixpkgs that aren't in MELPA, we can submit recipe PRs upstream. I don't have a problem being the Debian of Emacs packages! :)

Yes, please pass recipes on to us.

Regarding this effort in general (which I applaud!), I should point out that sometimes packages get removed from MELPA or even renamed. Not sure what impact that is likely to have on you. (Renaming is tracked via :old-names in the corresponding recipes.)

[ttuegel]

In case it's an issue, I should mention that we don't keep old versions currently. We'd like to start keeping just a few old versions.

That won't be a problem for us. Originally, I was just reading packages/archive-contents for all of the package archives. Then it would have been a problem, but when I realized that only ELPA keeps old versions, I abandoned that plan. It's probably better that we get packages from the original source anyway--we have a 12 month support cycle, and I don't expect any server to keep old packages that long.

I think many MELPA Stable users also have MELPA enabled, so that they can explicitly install "stable" packages on a package-by-package basis, while still having all the dependencies available for installation.

Right, there was some user feedback about that when I planned to just track MELPA Stable. My plan now is to use all the recipes, keeping stable versions where there is one and falling back to unstable versions when there isn't. We will be keeping our own snapshots, so I don't think there's a problem tracking unstable versions, but it would have been difficult to track daily snapshots from MELPA.

Regarding this effort in general (which I applaud!), I should point out that sometimes packages get removed from MELPA or even renamed. Not sure what impact that is likely to have on you. (Renaming is tracked via :old-names in the corresponding recipes.)

I don't think this will impact us much. It's no problem to have many names refer to the same package (the current name and all of :old-names). If a package is removed from MELPA, we can always add it back to Nixpkgs manually if there is demand for it, but I imagine packages are removed for good reason. There are already several packages that we must maintain manually, anyway. (Usually because they must be kept in-sync with some dependency, like ghc-mod.)

[purcell]

Great, sounds like that's all in hand, then. Just ping me or file a MELPA issue if you need anything.