Emacs can do this:
nix profile install nixpkgs#hello --profile emacs-profile
export PATH=$PATH:./emacs-profile/bin
which hello
# /home/user/.emacs.d/emacs-profile/bin/hello
hello
# Hello, world!
Binaries, reproducible, when you need them. No need for restarting Emacs or using Nix or Guix commands. Let Emacs control everything, as the world should be. Portable across Darwin, WSL2, Linux etc. Pure-build system agnostic so both Guix and Nix users can play together. Re-use work in CI & sanbox tests.
This package is very incomplete and this repo is being rushed out as a proof-of-concept.
Too much of the Nix & Guix work out there has adopted a workflow that forces users to restart or run Nix builds just to change Elisp dependencies. Read the Design Vision to understand what we should be doing and what this package will be. Read Current Status to figure out how to where to start working on it.
- Contents
- Introduction to pure profiles
- The Design Vision
- Install yakspkgs
- Current Status
- Hacking Guide
- Contributing
- Shout-outs
- Footnote on FSF and Emacs Core Licensing
Pure dependency management tools (Nix & Guix) have a concept of a profile, a way to present entire sets of programs and their dependencies an environment. This package gives Emacs control of its very own profile. Emacs with a profile can do all of its own 3rd party dependency management without affecting the rest of the user shell. Freedom. Independence. Reproducibility and portability. This use manner will revolutionize Emacs’s relationship with binary packages.
Emacs should be it’s own package manager for 3rd party packages, using Nix / Guix as backends. Emacs should be in the driver seat. No restarts necessary.
While nixpkgs and guixpkgs include a lot of packages, Emacs may need special builds or bleeding edge upstreams, and this repository can provide those packages. Yakspkgs is an especially good place for dynamically loaded modules that are otherwise a major portability headache.
- Infrastructure for using Nix & Guix profiles from within Emacs
- Packages tuned for Emacs whenever package upstreams need some help
- Live or fully declarative profile management
- Integration infrastructure for elisp to predictably consume 3rd party deps from profiles
- Home manager integration driven from within Emacs
- Lots of packaging examples for people to provide their binaries without requiring hosting
Note
The interactive profile can be edited by the user live using Emacs commands to experiment with different binary dependencies from yakspkgs
Record the current state of the interactive profile into a portable lock file, using elisp, consumed in Emacs or through interpretation in batch workflows.
Re-hydrating the lock file into a profile results in a reproducible profile. It only changes when the lock file changes.
After a frozen interactive profile is working well enough, a declarative profile can be created for human-maintained fully reproducible 3rd party dependency sets.
Either we say what elisp packages depend on or elisp packges tell us what they depend on. Both workflows will be needed during adoption.
Elisp packages like emacs-vterm can adopt a new header instructing yakspkgs about what 3rd party dependencies to install into the profile.
Not all elisp packages will declare their 3rd party dependencies overnight. It takes time to normalize this and make it as ubiquitous as declaring elisp dependencies. Package decoration allows elisp package managers to get 3rd party dependencies when yakspkgs moves faster than elisp packages.
;; straight-use-package-by-default t and you want a git version
(use-package yakspkgs
:straight
(yakspkgs :type git :host github :repo "positron-solutions/yakspkgs"))
;; if straight-use-package-by-default is nil
(straight-use-package '(yakspkgs :type git :host github
:repo "positron-solutions/yakspkgs"))
;; using elpaca (recommended to add a hash for reproducibility)
(elpaca-use-package
(example :host github
:repo "positron-solutions/yaskpkgs")
:demand t)
An initial package is being created to demonstrate all the necessary capabilities and figure out what the user interface should be.
There’s a lot to get on top of. To start, a profile is being created inside the user Emacs directory, possibly using no-littering conventions. The initialization will verify the sanity of the package manager (nix) and install a dummy package that will cause a manifest to be created so that the other commands will provide feedback.
Dynamic profile management is pretty easy. Any nix flake style package path is easy to install and they can be listed and uninstalled. This will complete the very basic POC.
After that, some more planning is needed. We can create package sets. We can generate Nix expressions from elisp. We can install them and continue supporting more of the Nix profile API.
The best user experience and utility for hard to do things is going to drive most decisions.
This is a short re-index of the information related to providing Emacs with an independent profile, similar to what home manager does. Links and examples work in Emacs, as literate org content.
- The regular envionment data for nix is carried into to the Emacs process (unless you do something to stop this of course)
- You can list the environment using the
list-environmentpackage. - Emacs does not by default use buffer local environments. The envrc package
by Steve Purcell is very useful for buffer-local style direnv integration.
This does mean that
process-variablewill change for buffers that have activated a direnv. The errata caused by functions using a temp buffer without propagating the environment are not that hard to clean up. - Inside of vterm, you can run
env. The environment inheritsprocess-variablebut the two are decoupled after this inheritance, so updating the Emacs global profile variables will not change the local variables. - Modifying the
PATHinprocess-variablehas the desired effect so thatshell-commandand other functions correctly discover binaries within the nix profile.
Profiles are versioned sets of packages. We want to give Emacs a profile. To do a batch style update to a profile, we can use symlinkJoin to create a package that points to other packages and install that into the profile.
Nix profiles are maintained using the nix binary. You can create a basic
profile manually and “activate” it by just appending the $PATH.
nix profile install nixpkgs#hello --profile emacs-profile
export PATH=$PATH:./emacs-profile/bin
which hello # /home/user/.emacs.d/emacs-profile/bin/hello
hello Hello, world!Note, for libraries, such as Emacs modules, you will need to state the path in a nix variable and export that dynamic path to a static path.
You can also link the dynamic path to a predictable location and use that location from within elisp. This is done now in Posimacs to expose the vterm.so to Emacs.
The file manifest.json records the provenance of the packages that are installed in this version of the profile.
So where is our new manifest?
cat emacs-profile/manifest.json | jq
# {
# "elements": [
# {
# "active": true,
# "attrPath": "legacyPackages.x86_64-linux.hello",
# "originalUrl": "flake:nixpkgs",
# "outputs": null,
# "priority": 5,
# "storePaths": [
# "/nix/store/g2m8kfw7kpgpph05v2fxcx4d5an09hl3-hello-2.12.1"
# ],
# "url": "github:NixOS/nixpkgs/a2d2f70b82ada0eadbcb1df2bca32d841a3c1bf1"
# }
# ],
# "version": 1
# }
Flake registries contain flakes mappings between a short name and a full path, which are used when we specify packages using shorthand paths such as nixpkgs#hello.
Registry support is kind of a replacement for the older nix channel configuration style. However, installing packages in thi way is still a bit unspecific. In order to rehydrate such a path, we need the expanded URL. Pinning a flake is an option.
Yakspkgs will need registry suport to make it easy to switch between different package sets for dynamic profile usage.
For the declarative style, we usually specify nixpkgs as an input and pin it using the flake lock. There is no question what version of nixpkgs is in use.
We can use a custom registry specific to Emacs with the --registry or
--option flag on most commands. This could be maintained in the
yakspkgs-profile-dir. It’s just a registry.json file.
# Specifying a registry to nix search
nix search --option flake-registry "file://$(pwd)/registry.json" nixpkgs hello
# Specifying a registry in nix add
nix registry add --registry ./custom-flake-registry.json nixpkgs github:nixos/nixpkgs
The registry is obviously convenient for times when the user approximately knows the package name and doesn’t really care which nixpkgs version the package is from, usually because the tool is very stable across versions.
We can offer similar shortcuts in Emacs, using completions across flake attributes from a frozen flake in the registry.
The profile manifest, stored in manifest.json, can also be used to uniquely describe package sources.
Versioned flakes and packages very nearly approximate the same information as a declaraed package set.
Registries are somewhat related to search in that they specify flakes from which we might install other packages. In the case of something like nixpkgs, the user probably wants to search without having to declare nixpkgs as a flake input only to re-export the packages for local completions. The search interface is good for this.
Searching the flakes in the registry is also valuable becuase flakes that are added to a regsitry are added essentially to provide convenient access.
We don’t want to use the old school (dirty) nix-env -i style stuff all the
time. Use flakes. Be pure.
We can use pkgs.symlinkJoin to bundle together a pile of packages into one
super package. Then, this super-package has all the versioned goodness of
the profile. Thus we can roll back to a previous working declaration if
needed.
An example is in the profile-contents directory.
Beautiful Nix Profile diagrams. It is a superposition of outputs basically. If we install this package into the profile, the profile will have a new version, one per each time we update and install our overall declaration.
Home manager also builds itself as a package. You can see in the activation script that it executes commands like below:
# The docs in the help are good
nix profile --help
nix profile install /nix/store/yz31iyfqdw4n20l6bpbbxx1y5hrxz4l7-home-manager-path
# You can see the derivation that outputs this path. Nix infers
# this fact from the /nix directory in the path.
nix show-derivation /nix/store/yz31iyfqdw4n20l6bpbbxx1y5hrxz4l7-home-manager-path
Look inside the profile-contents directory for a flake. This flake just provides a super-package including cowsay and hello. Let’s install this super-package into our Emacs profile:
nix profile install ./profile-contents#profile-contents --profile emacs-profile
# If you get a collision, you should remove the hello package or just delete the profile and start over
# nix profile remove /nix/store/g2m8kfw7kpgpph05v2fxcx4d5an09hl3-hello-2.12.1 --profile emacs-profile
# after these get installed, the manifest will be updated. If the profile is on
# the $PATH, you will be able to run cowsay and hello. No magic, but cool.
# Note, the structure includes our silly `emacs-god-profile' package name from the flake.nix
tree emacs-profile/bin -L 2
emacs-profile/bin
├── cowsay -> /nix/store/3d54xbrqj5zixa0cfnyki09jrffr0g3a-cowsay-3.04/bin/cowsay
├── cowthink -> cowsay
└── hello -> /nix/store/g2m8kfw7kpgpph05v2fxcx4d5an09hl3-hello-2.12.1/bin/hello
So there it is. You can build a set of packages. You can attach them to a profile. You can update them declaratively. You can make the available on the bin path. This is almost a POC for giving Emacs an independent profile already.
Let’s reverse-engineer home manager’s activation to get some ideas. If you
have home manager installed, after building a completed home manager
profile, just take a look at ~/.nixpkgs/config/result/activate. It
contains a lot of stuff that applies to starting systemd processes. The
path and environment stuff is what we’re interested in.
The “activation” script basically:
- Check for file symlink collisions on any files we plan to link into place
- Check that the profile we are upgrading to is not a downgrade
- Link stuff into place and make the profile active
The money hunk here updates the PATH, desktop environment, and systemd
state. It’s not very magical.
XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)} \
PATH=/nix/store/9fhmhbfkdcarrl1d75h1zbfsnbmwrw57-systemd-250.4/bin:$PATH \
bash /nix/store/lyvazadz3v9nck27nwcczqi4s9m402ix-systemd-activate.sh "${oldGenPath=}" "$newGenPath"Yes, updating the PATH is 99% of “activation.” For Emacs to obtain
binaries, it’s almost 100%. We only need to reproduce the file linking
support for linking libraries into place if elisp scripts expect them to be
in a known. (The slightly longer way is to include the generated paths into
the elisp files, basically using nix to finish the elisp files as
tempaltes.)
Note it is not recommended to use most elisp files as generated outputs of Nix because they are immutable. We want to be able to modify our elisp without running a bunch of (sometimes costly) nix builds.
We can reverse-engineer the symlink system and even perform “activation” using elisp instead of yucky bash or less common (to Emacs users) Nix lang.
We are gluing nix packages onto Emacs packages. Either the package asks for it or we provide the package when we see its dependent.
Emacs packages have package headers. They can contain 3rd party dependency declaration or we can provide a map. We can consume header information or we can decorate those packages with information they don’t know yet.
If you want Emacs to run when the DE starts, home manager has this integration ready to go with its Emacs module.
If we try to drive this stuff from within Emacs’ own profile, it may introduce some bootstrap issues with low value, so that’s not a goal right now.
There’s an analog for Darwin. (Sorry WSL2).
First decide if you want to work on this repository or fork it to something entirely different. Non-exhaustive list of changes that are very welcome:
- Guix package declarations
- Guix translation of elisp package headers
- Guix elisp package decoration
- Guix profile generation from lock file
- Guix commands to interact with Guix daemon
Changes will likely be rejected if it is aimed at:
- Batch style workflows that require the user to restart Emacs except for CI & sandbox cases, which must use a lock file
The CI files in the project are distributed with the MIT license. For elisp files, Nix expressions, and Guix expressios, only files with GPL3 headers will be accepted. DCO sign-off is mandatory.
A copy of the DCO is distributed with this project. Read its text to understand the significance of configuring for sign-off.
A sign-off means adding a “trailer” to your commit that looks like the following:
Signed-off-by: Random J Developer <random@developer.example.org>
A GPG signed commit shows that the owner of the private key submitted the changes. Wherever signatures are recorded in chains, they can demonstrate participation in changes elsewhere and awareness of what the submitter is participating in. The lack of such a proof elsewhere and the presence of a verifiable proof in this repo’s history prevent improper claims of originating source code or introducing relabelled source code.
Follow these instructions before you get ready to submit a pull-request.
Refer to the Github signing commits instructions to set up your git client to add GPG signatures. File issues if you run into Emacs-specific problems.
Because signing is intended to be a conscious process, please remember to read and understand the Developer Certificate of Origin before confinguring your client to automatically sign-off on commits.
In magit, set the -s switch. Use C-x C-s (transient-save) to
preserve this switch on future uses. (Note, this is not per-project).You
can also set the signature flag this way.
In order to specify which projects you intend to sign with which keys, you will want to configure your git client using path-specific configurations.
Configuing git for this can be done with the following directory structure:
/home/rjdeveloper/
├── .gitconfig
└── .gitconfig.d
├── sco-linux-projects.conf
├── other-projects.conf
└── gpg-signing-projects.conf
In your root config, .gitconfig, add an includeIf directive that will
load the configuration you use for projects you intend to GPG sign commits
for.
[includeIf "gitdir:/home/rjdeveloper/**/gpg-signing/**/.git"] path = "~/.gitconfig.d/gpg-signing-projects.conf"
In the gpg-signing-projects.conf add your GPG signing configuration from
earlier. sign adds the GPG signature automatically. File an issue if you
need help with multiple GPG homes or other configurations.
[user] name = "Random J Developer" email = "random@developer.example.org" signingkey = "5FF0EBDC623B3AD4" [commit] sign = true gpgSign = true
If you don’t like these configurations and want to individually indicate you have read and intend to apply the DCO to your changes, these commands are equivalent:
git commit -s -S --message "I don't like using .gitconfig"
# To clean up a commit
git commit --amend -s -S --no-message
# Combine with rebase to sign / sign-off multiple existing commits
git rebase -i- alphapapa for being super prolific at everything, including package writing, documentation, and activity on various social platforms
- adisbladis for the Nix overlay that makes the CI and local development so nice
- FSF for the Yak shaving club
- NobbZ for being all over the Nix & Emacs interwebs
Free Software Foundation currently requires copyright assignment on all code that goes into Emacs core. Many GNU projects have since switched to using a Developer Certificate of Origin. DCO sign-off is a practice accepted by git, GCC, and the Linux Kernel. Doing DCO sign-off is not the same as copyright assignment, and serves a slightlly different purpose. DCO is more defensive of any users while copyright assignment is offensive in the case of GPL non-compliance. In any case, with DCO sign-off, you can be assured that changes submitted to a code base you control are incontrovertibly covered by the license you chose. Using the DCO may make it easier for code in your project to be included in Emacs core later.