Overlays don't seem to always be applied in the right order

[codygman]

The use of the module system merge with imports in combination with raked overlays usually puts those imported overlays first (proof tbd!), probably because of some "recursion-first" and thereby makes the other, usually external, overlays unavailable to them.

Since overlays more often than not depend on external overlays, this is mostly not intended and results in problems.

Quick workaround: don't use raked overlays, but import them each manually (in the correct order).

edited: @blaggacao

original report

I'm a little low on time, so I'm just pasting what I had from the discussion.

I'm not really sure what's going on here since I didn't change this piece. I use emacs overlay like this:

final: prev: {
  emacsGcc =
    let
      package = prev.emacsGcc.override ({ withXwidgets = true; });
      emacsPackages = prev.emacsPackagesNgGen package;
      emacsWithPackages = emacsPackages.emacsWithPackages;
    in
    emacsWithPackages (epkgs: with epkgs; [
      use-package
      magit
    ]);
}

It is tracked by the way:

devos2  try-upgrading-devos 「↪ 𝚫 ✔ 📁 」  ⎔ 
✦ ❯ git ls-files overlays/emacsGcc.nix 
overlays/emacsGcc.nix

It's not taking effect any more and seems to not apply to any pkgs such as those in my users/profiles/emacs/default.nix:

{ pkgs, ... }: {
  programs.emacs = {
    enable = true;
    package = pkgs.emacsGcc;
    # extraPackages = epkgs: [ epkgs.vterm epkgs.f ];
    extraPackages = import ./emacs-packages.nix;
  };

}

I think a while ago when I worked with @nrdxp over discord we had a similar issue where the order of overlays mattered for the emacs overlay.

So based on that hunch I'm about to try this in flake.nix:

digga.lib.mkFlake {
      inherit self inputs;

      channelsConfig = { allowUnfree = true; };

      channels = {
        nixos = {
          imports = [ (digga.lib.importers.overlays ./overlays) ];
          overlays = [
            ./pkgs/default.nix
+           emacs-overlay.overlay
            pkgs.overlay # for `srcs`
            nur.overlay
            agenix.overlay
-           emacs-overlay.overlay
          ];
        };
        latest = { };
      };

I have no idea if this will work, but the rationale is that maybe the pkg overlays have to happen after the emacs overlay is loaded.

Also, here is how I'm testing:

✦ ❯ emacs --batch --eval "(require 'magit)"
Debugger entered--Lisp error: (file-missing "Cannot open load file" "No such file or directory" "magit")
  require(magit)
  command-line-1(("--eval" "(require 'magit)"))
  command-line()
  normal-top-level()

Edit: Here is some extra context that could help:

✦ ❯ rg -C5 emacs-overlay -t nix
flake.nix
23-
24-      pkgs.url = "path:./pkgs";
25-      pkgs.inputs.nixpkgs.follows = "nixos";
26-
27-
28:      emacs-overlay.url = "github:nix-community/emacs-overlay";
29-    };
30-
31-  outputs =
32-    { self
33-    , pkgs
--
36-    , ci-agent
37-    , home
38-    , nixos-hardware
39-    , nur
40-    , agenix
41:    , emacs-overlay
42-    , ...
43-    } @ inputs:
44-    digga.lib.mkFlake {
45-      inherit self inputs;
46-
--
49-      channels = {
50-        nixos = {
51-          imports = [ (digga.lib.importers.overlays ./overlays) ];
52-          overlays = [
53-            ./pkgs/default.nix
54:            emacs-overlay.overlay
55-            pkgs.overlay # for `srcs`
56-            nur.overlay
57-            agenix.overlay
58-          ];
59-        };

devos2  try-upgrading-devos 「↪ ✘ 𝚫 ✔ 📁 」  ⎔ 
✦ ❯ rg emacsGcc -C5 -t nix
overlays/emacsGcc.nix
1-final: prev: {
2-
3:  emacsGcc =
4-    let
5:      package = prev.emacsGcc.override ({ withXwidgets = true; });
6-      emacsPackages = prev.emacsPackagesNgGen package;
7-      emacsWithPackages = emacsPackages.emacsWithPackages;
8-      app-launcher = emacsPackages.trivialBuild {
9-        pname = "app-launcher";
10-        src = prev.fetchurl {

users/profiles/emacs/default.nix
1-{ pkgs, ... }: {
2-  programs.emacs = {
3-    enable = true;
4:    package = pkgs.emacsGcc;
5-    # extraPackages = epkgs: [ epkgs.vterm epkgs.f ];
6-    extraPackages = import ./emacs-packages.nix;
7-  };
8-
9-#   home.file = {

devos2  try-upgrading-devos 「↪ ✘ 𝚫 ✔ 📁 」  ⎔ 
✦ ❯ rg profiles/emacs -C2 -t nix
users/cody/default.nix
28-
29-   home-manager.users.cody = {suites, ... }: {
30:    # imports = [ ../profiles/git ../profiles/direnv ../profiles/mail ../profiles/gpg ../profiles/emacs ];
31:    imports = suites.base ++ [ ../profiles/git ../profiles/direnv ../profiles/mail ../profiles/gpg ../profiles/emacs ];
32-
33-    # shouldn't this clash with environment.sessionVariables in develop/default.nix?

devos2  try-upgrading-devos 「↪ ✘ 𝚫 ✔ 📁 」  ⎔ 

[codygman]

As another data point I tried both using plain emacs from nixpkgs and instead using packages but still experienced the issue. It looks like those gifts both use overlays too so it's probably affected too.

Staged changes (4)
new file   pkgs/applications/editors/emacs/elisp-packages/apheleia/default.nix
@@ -0,0 +1,21 @@
+{ pkgs, lib, fetchFromGitHub }:
+
+pkgs.emacsPackages.trivialBuild {
+    pname = "apheleia";
+    packageRequires = [  ];
+    src = fetchFromGitHub {
+          owner = "raxod502";
+          repo = "apheleia";
+          rev = "876078f473cabd14129eb78d610ef2cc46ead4f1";
+          sha256 = "actIWr8fXYfPWRdMTmQFvn5PiJVRFIec7cwObjsn5lY=";
+    };
+
+    # INCOMPLETE
+
+    meta = with lib; {
+      description = "to do";
+      license = licenses.gpl2Plus;
+      maintainers = with maintainers; [ codygman ];
+      platforms = platforms.all;
+    };
+}
modified   pkgs/default.nix
@@ -1,4 +1,5 @@
 final: prev: {
   northwind-sql = prev.callPackage ./development/misc/northwind-sql { };
   shortuuid = prev.callPackage ./development/misc/shortuuid { };
+  apheleia-mode = prev.callPackage ./applications/editors/emacs/elisp-packages/apheleia { };
 }
modified   profiles/develop/emacs/config/vanilla.org
@@ -2212,24 +2212,8 @@ Perhaps even turning on code block execution for sql queries in general or those
 
 *** TODO transparent formatting for everything
 
-
-#+begin_src nix :noweb-ref emacs-package-in-let :tangle no
-  apheleia = emacsPackages.trivialBuild {
-    pname = "apheleia";
-    packageRequires = [  ];
-    src = prev.fetchFromGitHub {
-          owner = "raxod502";
-          repo = "apheleia";
-          rev = "876078f473cabd14129eb78d610ef2cc46ead4f1";
-          sha256 = "actIWr8fXYfPWRdMTmQFvn5PiJVRFIec7cwObjsn5lY=";
-        };
-  };
-#+end_src
-
-I guess I could live without the other ones, but not having format on save like this would really be painful
-
-  #+begin_src nix :noweb-ref disabled-emacs-package :tangle no
-apheleia
+#+begin_src nix :noweb-ref emacs-package :tangle no
+apheleia-mode
   #+end_src
 
 #+begin_src emacs-lisp :noweb yes
modified   users/profiles/emacs/emacs-packages.nix
@@ -29,6 +29,7 @@ epkgs: with epkgs; [
   lsp-mode
   lsp-haskell
   lsp-ui
+  apheleia-mode
     restclient
   
   ob-restclient

[blaggacao]

Fixed in #55. Awaiting update, closing nevertheless.