nix-straight.el cannot read Doom package locks

[ckiee]

Doom has its own mechanism to lock packages to specific revisions. Currently, nix-straight.el only knows how to extract package names and not additional metadata for versions / tags / git revs as this would require writing multiple fetchers for all the *ELPAS and probably some Git special-case stuff too. Additionally, Emacs Lisp is terrifying and requires a tonne of patience.

This being left unimplemented so far has left Doom working just well enough that no one (update: a few people) have bothered since my (inexperienced, messy, researching) attempt a year or so ago, and I'm hoping this issue might incite some action:

• Here's the issue on the original vlaci repository where vlaci gives a few hints.
• Many many errors, this is like the leading cause of open issues¹... [Request] :recipe support #177 (package! nano-theme) not installing correctly #70 Package not available: idris2-mode #259 can't Install copilot.el using emacsPackagesOverlay #212 Fails to build due to 'emacs-ghub' melpa package #147 (maybe Navigation keys with modifiers working incorrectly. #352)
• PR documenting some of this (and current hacks) for end-users at Documentation overhaul #266
• If implemented with one derivation per package, perhaps improvements (on Nativecomp especially) in Any change requires a long rebuild of straight-emacs-env #324?
• Open: Should we go monorepo with nix-straight.el for easier integration? We're may need more hacks for things like Using plists #344. Can't find anyone else depending on it but not on nix-doom-emacs on cs.github.com.

This whole thing is like, big and difficult², but very worth it for the community if someone decides to do it.

cc @thiagokokada @YellowOnion @ribosomerocker @evertedsphere

Footnotes

1. …even if each user will only notice one or two things ↩

2. Doesn't split nicely into different work units except the fetchers even. ↩

[leifhelm]

Have I understood the problem correctly nix-straight.el uses straight-use-package to determine which packages are used, but there is no way to extract further information like the pin or recipe?

[ckiee]

@leifhelm: Yes, except Doom is the one calling straight-use-package and it does have the pin info. It's just a lot of unfun specialized Elisp hacking to figure out the details.

[YellowOnion]

I have no clue how to code elisp, So I can't really confirm if this will reduce work load, but why can't we just use doom itself?

there's a bunch of dooms functions for package management in the file lisp/doom-packages.el

And the function doom-packages-install in lisp/cli/packages.el#L273, calls a few of these against straight-use-package.

[ribosomerocker]

Generally, because we want to use nix for package management, which is why this project exists.

[ckiee]

Specifically what we want to do is keep our Doom build reproducible, so we can have some hashes -- possibly as many as yarn2nix keeps so all the builds of that Emacs will have the exact same package sources.

We basically have to use some of Doom's package manager, but we need someone with enough patience to setup an advice hook or something to be able to grab hashes and source URLs for all the packages so we can have Nix fetch and verify them for us instead of Doom's ~black box.

[leifhelm]

The doom-package-pinned-list contains all pinned packages after the initialization. This however does not include the hash of the commit. So a sort of commit to hash resolution is necessary to get the hash of the commit. Recipes can be obtained by hooking into straight-override-recipe.

Generated pinned packages of my doom config

{
  "link-hint": "676dac6621e321b33a8d396fa27dd0ea619d21e3",
  "drag-stuff": "6d06d846cd37c052d79acd0f372c13006aa7e7c8",
  "zig-mode": "aa20d630b8c413dab8d6bd120ec3ed5db5c9da70",
  "yaml-mode": "535273d5a1eb76999d20afbcf4d9f056d8ffd2da",
  "company-shell": "a77f4de75912aa87314cde92c603b831d5050246",
  "rustic": "6eec9713876d698510ee3715698fd42c1571e307",
  "py-isort": "e67306f459c47c53a65604e4eea88a3914596560",
  "pyimport": "a6f63cf7ed93f0c0f7c207e6595813966f8852b9",
  "python-pytest": "9bf8db38bf18feb0484931877210cecfaa96bfc6",
  "nose": "f8528297519eba911696c4e68fa88892de9a7b72",
  "pyvenv": "31ea715f2164dd611e7fc77b26390ef3ca93509b",
  "pipenv": "3af159749824c03f59176aff7f66ddd6a5785a10",
  "company-anaconda": "da1566db41a68809ef7f91ebf2de28118067c89b",
  "anaconda-mode": "160e4e7185881233d96da6722332bd6d038187a9",
  "lsp-python-ms": "f8e7c4bcaefbc3fd96e1ca53d17589be0403b828",
  "pip-requirements": "216cd1690f80cc965d4ae47b8753fc185f778ff6",
  "ob-async": "9aac486073f5c356ada20e716571be33a350a982",
  "orgit": "b33b916915db5f91d2c9da4cb1a2457ccbb09332",
  "org-pdftools": "967f48fb5038bba32915ee9da8dc4e8b10ba3376",
  "evil-org-mode": "a9706da260c45b98601bcd72b1d2c0a24a017700",
  "org-cliplink": "13e0940b65d22bec34e2de4bc8cba1412a7abfbc",
  "toc-org": "bf2e4b358efbd860ecafe6e74776de0885d9d100",
  "ox-clip": "ff117cf3c619eef12eccc0ccbfa3f11adb73ea68",
  "org-yt": "40cc1ac76d741055cbefa13860d9f070a7ade001",
  "htmlize": "dd27bc3f26efd728f2b1f01f9e4ac4f61f2ffbf9",
  "avy": "ba5f035be33693d1a136a5cbeedb24327f551a92",
  "org-contrib": "39e2abc5629c1be6186bb6489ec4f76524edf82a",
  "org-mode": "00adad9357b9123a7b11ecf12c148a5196debcb7",
  "company-nixos-options": "053a2d5110ce05b7f99bcc2ac4804b70cbe87916",
  "nix-update": "fc6c39c2da3fcfa62f4796816c084a6389c8b6e7",
  "nix-mode": "20ee8d88900b169831d6b0783bd82d2625e940c7",
  "evil-markdown": "8e6cc68af83914b2fa9fd3a3b8472573dbcef477",
  "edit-indirect": "f80f63822ffae78de38dbe72cacaeb1aaa96c732",
  "markdown-toc": "3d724e518a897343b5ede0b976d6fb46c46bcc01",
  "markdown-mode": "1f709778ac7990f4a07fdf11fe37bc6541810b29",
  "company-math": "45778f5731c97a21a83e3b965cbde42018709afd",
  "company-reftex": "42eb98c6504e65989635d95ab81b65b9d5798e76",
  "company-auctex": "9400a2ec7459dde8cbf1a5d50dfee4e300ed7e18",
  "evil-tex": "0fa85c3fc88d96621002b5a1b79efcc06776642f",
  "latex-preview-pane": "5297668a89996b50b2b62f99cba01cc544dbed2e",
  "adaptive-wrap": "0d5b4a07de76d87dd64333a566a8a0a845f2b9f0",
  "auctex": "830e40a0639aedc6c362a3a01e160aaa8246bb99",
  "lsp-java": "0968038b9aea52ef3bf499e597cf3800d913c059",
  "groovy-mode": "bf732d367b16887f81d404481c11ed1a58671d4e",
  "android-mode": "d5332e339a1f5e30559a53feffb8442ca79265d6",
  "counsel-jq": "8cadd2e96470402ede4881b4e955872976443689",
  "json-snatcher": "b28d1c0670636da6db508d03872d96ffddbc10f2",
  "json-mode": "eedb4560034f795a7950fa07016bd4347c368873",
  "lsp-haskell": "7cf64944ab3a25ea5d6f8d5e0cd33124182df991",
  "haskell-mode": "cb573c8db5b856eb37473009f2c62e0717a1cd02",
  "poly-R": "e4a39caaf48e1c2e5afab3865644267b10610537",
  "polymode": "2094c92403fe395dfb2b8b2521da1012a966e9ab",
  "ess-R-data-view": "d6e98d3ae1e2a2ea39a56eebcdb73e99d29562e9",
  "ess": "39eba283000a7b0220303d7c5a4f3ee05efc1e9c",
  "buttercup": "ceedad5efa797e860dbb356bc2c3028a4e0321ec",
  "flycheck-cask": "4b2ede6362ded4a45678dfbef1876faa42edbd58",
  "flycheck-package": "615c1ed8c6fb7c73abec6aaa73d3fef498d231bc",
  "elisp-demos": "01c301b516e9949d0239d20f6834afbc9acf0abb",
  "elisp-def": "dfca043ec0cbead67bd9c526cb009daf771d0fa2",
  "overseer": "02d49f582e80e36b4334c9187801c5ecfb027789",
  "macrostep": "424e3734a1ee526a1bd7b5c3cd1d3ef19d184267",
  "highlight-quoted": "24103478158cd19fbcfb4339a3f1fa1f054f1469",
  "flycheck-elm": "1b60050efd4729bfba548f3e5adbcb58436667cb",
  "elm-mode": "d4e434fa1857ba2f58d27c7520ebeac0515cd140",
  "company-coq": "a6e349e0131f676a885bd14c908fd26054b2df42",
  "proof-general": "e1e29acb04f0707fb2b323727d08f0fe85fc0c02",
  "ccls": "675a5704c14a27931e835a431beea3631d92e8e6",
  "company-glsl": "404cd0694ab34971f9c01eb22126cd2e7d3f9dc4",
  "glsl-mode": "9b2e5f28e489a1f73c4aed734105618ac0dc0c43",
  "opencl-mode": "15091eff92c33ee0d1ece40eb99299ef79fee92d",
  "modern-cpp-font-lock": "43c6b68ff58fccdf9deef11674a172e4eaa8455c",
  "disaster": "10a785facc60d89d78e0d5177985ab1af1741bb4",
  "demangle-mode": "04f545adab066708d6151f13da65aaf519f8ac4e",
  "cuda-mode": "7f593518fd135fc6af994024bcb47986dfa502d2",
  "cmake-mode": "d5d77de8c4c69e348b182eeb30222b2f1ba8db7b",
  "saveplace-pdf-view": "54ed966b842501c3c092dbf57b372e37b033c578",
  "pdf-tools": "bb0b71f5bafd81d0b5647c4ec48fafa0bb6f6c21",
  "code-review": "d38fbe59304ed31c759ce733cda16f69a8ef2d8c",
  "magit-todos": "67fd80c2f10aec4d5b2a24b5d3d53c08cc1f05dc",
  "magit-gitflow": "cc41b561ec6eea947fe9a176349fb4f771ed865b",
  "magit": "c1fb53d3de6390961ccd8dfb1cc135383508d0fc",
  "lsp-ivy": "3e87441a625d65ced5a208a0b0442d573596ffa3",
  "lsp-ui": "0a6368d38dc32e52abcbd52b63d1e557e42c66e6",
  "lsp-mode": "3fa645c0397b8f438f2db2dd288b899ba330d410",
  "request": "38ed1d2e64138eb16a9d8ed2987cff2e01b4a93b",
  "ivy-xref": "a82e8e117d2dd62c28b6a3e3d6e4cfb11c0bda38",
  "dumb-jump": "1dd583011f4025b1b8c75fd785691851b6c5dfa3",
  "eros": "dd8910279226259e100dab798b073a52f9b4233a",
  "quickrun": "314beae43cac2e4943e9ed4850e8e147bc3d2fac",
  "envrc": "7f36664fc6d97a7ca77c6c3e0c6577b72fa0b70d",
  "realgud": "3c88611c4ed59069093187c2a039b8d05cbe53e8",
  "flycheck-popup-tip": "ef86aad907f27ca076859d8d9416f4f7727619c6",
  "flycheck": "784f184cdd9f9cb4e3dbb997c09d93e954142842",
  "git-modes": "eca3bb42ea8abed9ef8549b2ac91bbea445c5bb5",
  "git-timemachine": "ca09684e94767cc0b2339b77b778b4de4f9d104f",
  "git-commit": "8a0cc83eff98489d3685b8585afdcebbb47c1393",
  "browse-at-remote": "cef26f2c063f2473af42d0e126c8613fe2f709e4",
  "undo-fu-session": "3e810c7c9ab75d2b6f92c7c876290abbc164e750",
  "undo-fu": "ab8bc10e424bccc847800c31ab41888db789d55d",
  "fd-dired": "458464771bb220b6eb87ccfd4c985c436e57dc7e",
  "dired-rsync": "7940d9154d0a908693999b0e1ea351a6d365c93d",
  "dired-git-info": "9461476a28a5fec0784260f6e318237c662c3430",
  "diredfl": "f9140b2c42151dca669003d685c9f079b2e3dc37",
  "snippets": "2d031f723164c6500ce68dcafe3dd30e6d3b23b1",
  "auto-yasnippet": "db9e0dd4335b2202cd5dac95bbbc87a1032d9bbe",
  "evil-vimish-fold": "b6e0e6b91b8cd047e80debef1a536d9d49eef31a",
  "vimish-fold": "a6501cbfe3db791f9ca17fd986c7202a87f3adb8",
  "yasnippet": "5cbdbf0d2015540c59ed8ee0fcf4788effdf75b6",
  "evil-collection": "6cc02b238cde28d850ac68ce429d24dd1e354ad4",
  "evil-quick-diff": "69c883720b30a892c63bc89f49d4f0e8b8028908",
  "exato": "aee7af7b7a0e7551478f453d1de7d5b9cb2e06c4",
  "evil-visualstar": "06c053d8f7381f91c53311b1234872ca96ced752",
  "evil-traces": "290b5323542c46af364ec485c8ec9000040acf90",
  "evil-textobj-anyblock": "29280cd71a05429364cdceef2ff595ae8afade4d",
  "evil-surround": "c9e1449bf3f740b5e9b99e7820df4eca7fc7cf02",
  "evil-snipe": "c07788c35cf8cd8e652a494322fdc0643e30a89f",
  "evil-numbers": "08f0c1ee93b8a563770eaefaf21ab9087fca7bdb",
  "evil-nerd-commenter": "386cd758a477d1b1ba742ef698ecc19916b43fbe",
  "evil-lion": "4da660e124731ed65e7aaa6c067c30e876619429",
  "evil-indent-plus": "b4dacbfdb57f474f798bfbf5026d434d549eb65c",
  "evil-exchange": "5f0a2d41434c17c6fb02e4f744043775de1c63a2",
  "evil-escape": "819f1ee1cf3f69a1ae920e6004f2c0baeebbe077",
  "evil-embrace": "7b5a539cfe7db238d860122c793a0cb2d329cc6e",
  "evil-easymotion": "f96c2ed38ddc07908db7c3c11bcd6285a3e8c2e9",
  "evil-args": "2671071a4a57eaee7cc8c27b9e4b6fc60fd2ccd3",
  "evil": "157af04d2cf466e301e82b0e667c5e7203fd96a2",
  "persp-mode": "7a594a3d8f1c4ba9234dcd831a589e87f3f4ae86",
  "vi-tilde-fringe": "f1597a8d54535bb1d84b442577b2024e6f910308",
  "git-gutter-fringe": "648cb5b57faec55711803cdc9434e55a733c3eba",
  "evil-goggles": "8f20a16e74016f37ad76dc4f2230d9b00c6df3c2",
  "neotree": "98fe21334affaffe2334bf7c987edaf1980d2d0b",
  "evil-anzu": "d3f6ed4773b48767bd5f4708c7f083336a8a8a86",
  "anzu": "5abb37455ea44fa401d5f4c1bdc58adb2448db67",
  "compat": "cc1924fd8b3f9b75b26bf93f084ea938c06f9615",
  "doom-modeline": "ce9899f00af40edb78f58b9af5c3685d67c8eed2",
  "hl-todo": "e52285965b5ee89c18080661d4f80270143ae8dc",
  "solaire-mode": "8af65fbdc50b25ed3214da949b8a484527c7cc14",
  "doom-themes": "83fd9545c2823b4b2610947802fa3a52995517b4",
  "wgrep": "f9687c28bbc2e84f87a479b6ce04407bb97cfb23",
  "ivy-rich": "600b8183ed0be8668dcc548cc2c8cb94b001363b",
  "counsel-projectile": "40d1e1d4bb70acb00fddd6f4df9778bf2c52734b",
  "amx": "37f9c7ae55eb0331b27200fb745206fc58ceffc0",
  "swiper": "764e0d35ba63adb893743f27a979144477d9bfb9",
  "company-dict": "cd7b8394f6014c57897f65d335d6b2bd65dab1f4",
  "company": "1005540b1cdf176cbcf893b2fa83d2075cbbe3ca",
  "which-key": "1ab1d0cc88843c9a614ed3226c5a1070e32e4823",
  "general": "9651024e7f40a8ac5c3f31f8675d3ebe2b667344",
  "project": "2e8da1b811d5213bd143d9d036bf7ea14741a1ae",
  "projectile": "dc6e7ff658789e4c6cf7c32a4b4fdf22ef9bc5a3",
  "ws-butler": "572a10c11b6cb88293de48acbb59a059d36f9ba5",
  "smartparens": "8b6a3c3b31afd5b3f846e09859b5fc62eb06e7c1",
  "pcre2el": "0b5b2a2c173aab3fd14aac6cf5e90ad3bf58fa7d",
  "helpful": "94a07d49a80f66f8ebc54a49a4b4f6899a65fbe3",
  "dtrt-indent": "d4fd1b4977eb0d534844fddf01c3c51c70c57205",
  "better-jumper": "47622213783ece37d5337dc28d33b530540fc319",
  "restart-emacs": "1607da2bc657fe05ae01f7fdf26f716eafead02c",
  "rainbow-delimiters": "a32b39bdfe6c61c322c37226d66e1b6d4f107ed0",
  "highlight-numbers": "8b4744c7f46c72b1d3d599d4fb75ef8183dee307",
  "hide-mode-line": "bc5d293576c5e08c29e694078b96a5ed85631942",
  "all-the-icons": "b18db6be0a290e8f91fd8d8340b2b44c8b75049a",
  "use-package": "0ad5d9d5d8a61517a207ab04bf69e71c081149eb",
  "straight.el": "fed215348076ba9182f634e0770a175220474b5a",
  "explain-pause-mode": "2356c8c3639cbeeb9751744dbe737267849b4b51",
  "gcmh": "0089f9c3a6d4e9a310d0791cf6fa8f35642ecfd9",
  "auto-minor-mode": "17cfa1b54800fdef2975c0c0531dad34846a5065"
}

[ckiee]

Finding the repositories where these commits are stored is also a bit of an adventure I think. There's a few sources; IIRC the *ELPA's and some Git forges are the common ones.

[badly-drawn-wizards]

I want to pick this up considering I still use NDE and am a bit fed up, but don't want to be a Cookie Licker. I have made some attempts at understanding the doom-packages.el for this issue a while back, but got dragged into other things.

[cooperpellaton]

I have some free time and I'd love to help; please let me know what I can do!

[YellowOnion]

Maybe it's time to consider tossing out doom cli, all together and just directly parse doom's modules, or creating something completely seperate...the packages.el files have the pins if we need them.

We all know Nix, but probably want doom because we don't know elisp well, but doom still suffers from not having a sandbox and reproducable builds (trying to sync laptop and desktop was a PITA).

At this stage I'm probably going to move away from doom and try to config emacs with home-manager, so many things are currently broken in this repo, and doom itself doesn't seem to care about nix friendly tooling despite being inspired by it.

[marienz]

Wanting to learn Nix and not wanting to keep running doom sync, I ended up writing https://github.com/marienz/nix-doom-emacs-unstraightened, a reimplementation taking a different approach:

• Use doom's CLI to export dependencies / pins
• Customize emacs-overlay to apply Doom's pins and recipes, and build an emacsWithPackages containing them
• Hook into Doom's profile generation to use the "built-in" version for all packages we just built (while keeping Doom's startup optimizations)

The end result does not use straight.el (other than to write an empty build cache to keep some bits of Doom working). All Emacs package-building is handled by Nix.

Applying pins on top of emacs-overlay is a hack, and I had to specialcase a few packages, but it does seem to work so far (on my machine... feedback appreciated!), and I'm cautiously optimistic the entire contraption is maintainable. Or at least it should be once I write tests and set up CI.

[joelmccracken]

... I ended up writing https://github.com/marienz/nix-doom-emacs-unstraightened, a reimplementation taking a different approach:

* Use doom's CLI to export dependencies / pins

* Customize emacs-overlay to apply Doom's pins and recipes, and build an emacsWithPackages containing them

* Hook into Doom's profile generation to use the "built-in" version for all packages we just built (while keeping Doom's startup optimizations)

This is awesome. I started looking into doing this at some point but ran into the problem of the pins referring to repos/projects that are not obvious from the get-go, kept hacking a while, but ended up giving up as I was out of my depth. Looking forward to reading through your project, I really hope I can use it.

[marienz]

This is awesome.

Thanks! :)

I started looking into doing this at some point but ran into the problem of the pins referring to repos/projects that are not obvious from the get-go

That was by far the most bang-head-against-wall trial-and-error bit of development. Tracing through Doom startup to figure out my options for hooking into it also took some time.

Tricky bits include:

• Can't use Emacs packages from NixOS 23.11, at least one package Doom pins moved its upstream. nixpkgs-unstable might work but emacs-overlay seemed more reasonable (because if a repo moves and it takes Doom a few days to update you just pin emacs-overlay to before the move, no need to pin all of nixpkgs).
• Doom fetches ELPA packages from the https://github.com/emacs-straight mirror repositories. You have to do the same for the pins to work. And that means you can't just override src for those packages, because nixpkgs elpaBuild expects a tarball, not a directory (melpaBuild seems to work). There's enough of those you can't just rely on emacs-overlay + a manageable number of handwritten derivations, generating per-package derivations (for ELPA + a few packages not in emacs-overlay for which Doom provides a recipe) is more or less required.
• straight.el applies pins per repository, not package. I thought I could get away with ignoring that, but then almost immediately afterwards discovered ignoring it triggers the behavior described in Cleanup expansion of magit-insert-section magit/magit#5130 (because Doom pins magit but does not explicitly pin the magit-section package, and I assume but did not confirm building the magit package uses magit-section.el from that package's source, not the magit-section dependency). Fixed now but that's the one where straight.el being structured differently really hurt.

Looking forward to reading through your project

It's... not the most readable code. I blame simultaneously learning Nix and having to add an increasing number of hacks to make it work: it could do with a cleanup pass now that it does.

Happy to answer questions if anything's unclear (if you don't understand it now, probably I won't understand it a year from now when it inevitably breaks...)