- Agreements and a repository for a shared GNU/Linux power-user environment.
- So that it’s easy to work on shared projects (e.g. the haskell
header can be
#!/opt/ceh/bin/runhaskell -i /opt/ceh/p/ourproject, instead of some complicated perl one-liner). - So that the packages are available with the same version on our
machines (e.g.
emacs,tmux,firefox, etc). - These agreements are intentionally based on simple unifications
instead of abstractions (e.g.
/opt/ceh/homehas to be a symlink for the user’s home directory, no need to$HOMEor tilde lookups anymore). - We use amd64 Nix for package management and if you can decide on the distribution that you use as a base system, we recommend amd64 Debian. But the point is that this should work on any modern amd64 GNU/Linux, we also tested Ubuntu trusty and precise.
- Modern GNU/Linux distro (ubuntu, debian, etc.).
- Intel architecture (amd64, no i386/i686 or arm).
- The machine only has one user who wants to use ceh and nix and she
will be the owner of the files in
/opt/cehand/nix. - If the machine is running NixOS already, then
/nixshould exist already of course and Ceh is able to handle that. - Root privileges for the installation. Actually only needed for the
creation of
/opt/cehand/nix, not needed for everyday usage or package installation. /etc/passwdcontains your user, so nix doesn’t have to use any fancy NSS modules.- The user uses bash or zsh. This is because he has to source
/opt/ceh/scripts/ceh-profile.shwhich supports these two shells. Patches are welcome for other shells.
sudo mkdir /opt/ceh /nix sudo chown $USER. /opt/ceh /nix chmod 0700 /opt/ceh /nix cd /opt/ceh git clone git://github.com/nilcons/ceh.git . ln -s $HOME home /opt/ceh/scripts/ceh-init.sh
sudo mkdir /opt/ceh /nix/var/nix/gcroots/auto/ceh sudo chown $USER. /opt/ceh /nix/var/nix/gcroots/auto/ceh chmod 0700 /opt/ceh /nix/var/nix/gcroots/auto/ceh cd /opt/ceh git clone git://github.com/nilcons/ceh.git . ln -s $HOME home /opt/ceh/scripts/ceh-init.sh
In the /opt/ceh/bin directory there are small wrapper scripts. Those
install the required pacakages for the wrapped binaries on first run.
/opt/ceh/bin is put into $PATH by /opt/ceh/scripts/ceh-profile.sh,
that you’re supposed to source from your ~/.bashrc.
If for any reason you prefer to ignore one of our provided binaries,
you can use the ceh_exclude script to create a symlink from
/opt/ceh/bin-user/ to the other version of the binary in your PATH.
If you don’t want the exclusion anymore, just delete the symlink.
As previously mentioned, the scripts in /opt/ceh/bin just install
the programs on demand as you run them. Therefore once ran, they take
up disk space on your machine. If you tried out something big and
would like to get rid of it, then go to /opt/ceh/installed and feel
free to delete any symlinks there. After deleting the symlinks, run
nix-garbage-collect to free up the disk space. If you decide to use
some deleted software again, it will be redownloaded.
If you delete from /opt/ceh/installed/essential, then you will have
to run ceh-init.sh, but Ceh will show a warning message on stderr
about that.
Sometimes there are changes that are not backward compatible with the
previous ceh-init.sh. In those cases we do our best to provide
warning messages on stderr from scripts, and those messages will ask
you to rerun ceh-init.sh. That operation is idempotent, feel free
to run it you think it can fix up something for you.
You can also read the “Upgrade notes” section to know about changes.
You may just want to see if it’s still possible to build a software from source, instead of using the nix binary cache. Or the nix binary cache is down and you still want to install packages instead of having to wait until it’s back.
In both cases, you can just set $CEH_NO_BIN_CACHE to 1 and then the
wrapper scripts will start building packages instead of using the
binary cache.
When you do this, you may also want to set $CEH_BUILD_MAX_JOBS to
the number of your processor cores in the system, because usually
building of tools (like compilers, linkers, make, etc.) can be
parallelized by nix.
This is the first Ceh release with Nix 2, if there are issues, please keep us posted!
Nix 2 removed the nix-mode.el emacs bindings, instead you have to use it from Melpa, as you do with all your other emacs modules.
We removed Firefox, because it was a constant maintenance burden, and the reasoning from many years ago (easy to use and install java, flash, googletalk plugin) is not strong enough anymore. The Firefox versions shipped by standard desktop distibutions are good enough.
Vanitygen is removed from nixpkgs, so we remove it from ceh. :(
For the shiny new Nix2 2 gigs of ram is not enough to download ghc-8.2.2 from the binary cache. Therefore we can’t use 32-bit nix anymore. So the base perl and nix package has been changed to 64-bit versions. The bit32 option has also been removed, ceh is now 64-bit only. Rest in peace ia32.
TWS is also removed, because Interactive Brokers started to release usable linux desktop installation packages, which includes the correct version of java and just works.
We think that for Haskell, stack is a clear winner in the ecosystem.
Therefore we may add stack support to ceh in the future, but we have removed our nix based haskell support.
As preparation for a baseline upgrade, we had to do a refactoring: in new nixpkgs versions, most of the derivations result in multiple output, not just one. Previously this happened only rarely, so we handled the special cases with the outFilter hack. This hack is gone, and instead we refactored the gclink symlinks a little bit to support multiple derivations.
For the end user, the change shouldn’t be notable and everything should continue working as expected.
NixOps was not working after the 32-bit -> 64-bit migration, I do not think this is anything to do with the architecture, something has changed in the git repo.
Anyways, we know of no users who uses Ceh and nixops together, so it was easier to remove it. Tell us if you need it!
I love i686, programs are so fluffy and cute. But unfortunately the world started to ignore that platform and stuff is supported only on amd64 nowadays. E.g. the haskell “community” ignores i686 altogether and unit tests routinely break. The nix binary cache is also only maintained for amd64 and if the glibc is missing for i686, nobody will fix it and you will have to build everything from source.
So we decided to move ceh to 64-bit, and live with the memory and disk usage of amd64.
This transition should be seamless from the user point of view, but if something goes bad, of course you can always just purge Ceh and reinstall it…
Also, if you want to install 32-bit stuff from some packages, you have to use the bit32 flag, instead of the old bit64 flag that we had for 64-bit. For Haskell, you can use the envvar CEH_GHC32 to get a 32-bit GHC and tools.
As a last resort, the base perl and nix package is still i686, so you
can run ceh-init.sh even on a i686 system and write your own
wrappers by using the bit32 flag for every package. But we are ready
to remove this convenience the first time it causes some problems.
Sorry guys, i686 is dead.
From now on, we don’t use profiles for Ceh’s internal purposes.
Instead we use the /opt/ceh/installed directory.
Profiles essentially give you two features. First, it’s a way to
track different states of what is installed and an easy way to roll
back and forward between those states. As we have git, we don’t need
this. The other feature is that profiles give you a huge symlink farm
that mimics a directory tree where only the currently installed
packages are seen. We don’t need this feature either, because we have
the wrappers in /opt/ceh/bin and we can just introduce a constant
symlink farm for the manpages in /opt/ceh/man.
This refactoring will also solve the constant issue around file collissions in profiles and package renames in upstream nixpkgs.
You can safely delete /nix/var/nix/profiles/ceh and run a
nix-garbage-collect to get rid of some old stuff that never would be
deleted otherwise.
The new place for the nixpkgs checkout of Ceh is /opt/ceh/nixpkgs,
you can run rm -rf /nix/var/ceh_nixpkgs to remove the old cruft.
Also, from now on it’s not necessary (or allowed) to specify
derivation hashes when calling ceh_nixpkgs_install_* functions.
Derivation hashes are always figured out automatically when on-demand
installations is needed and only the output hash is consulted for
binary consistency.
Another change is that we made ceh-init.sh idempotent and it should
fix up stuff from any state that is reasonable after a git pull or
after deletion of all the profiles in /nix/var/nix/profiles/ceh.
This means that ceh-upgrade.sh is no more and you should always use
ceh-init.sh if you think Ceh should be reinitialized.
On existing installations, run the following:
nix-channel --remove nixpkgs nix-channel --add http://nixos.org/channels/nixpkgs-unstable nix-channel --update
Unfortunately the nixpkgs guys changed the name of the xpra packages from python2.7-xpra to xpra. As described in the Troubleshooting section, this is causes a filename collision. This can be fixed:
/opt/ceh/bin/nix-env -p /nix/var/nix/profiles/ceh/bin -e python2.7-xpra /opt/ceh/bin/xpra
The /nix/var/nix/profiles/per-user/root/profile profile (which is
symlinked by default from $HOME/.nix-profile) is not used for
installations anymore. Earlier versions of ceh used to install
packages there, but now that profile is reserved for the user.
This is good, because if you need some package (e.g. vim) quickly
from the nixpkgs version that is current the ceh baseline, you can
just do:
ceh_nixpkgs_install vim ~/.nix-profile/bin/vim
Alternatively, you can use the most up-to-date nixpkgs:
nix-channel --add http://nixos.org/channels/nixpkgs-unstable nix-channel --update nix-env -i vim ~/.nix-profile/bin/vim
No need to create a wrapper in /opt/ceh just for a quick test. If
you want to get rid of your user profile, simply delete
/nix/var/nix/profiles/per-user/$USER/profile*. This is from now on
is totally separate from ceh’s own nix profiles. Those are located in
/nix/var/nix/profiles/ceh/*.
If you’re upgrading ceh from a version, where the default profile
was used for package installations, please delete
/nix/var/nix/profiles/per-user/root/profile*.
nix-init.sh -> ceh-init.sh nix-profile.sh -> ceh-profile.sh nix-purge.sh -> ceh-purge.sh nix-upgrade.sh -> ceh-upgrade.sh
And install-world.sh and predict-binary-cache.sh moved to scripts/maintainer.
- Have an LD_PRELOAD library that hides
/usr/lib/mozilla/pluginsfrom Firefox’s view.
/opt/ceh/bin/: wrapper scripts, they make sure that the wanted
package is already installed into /nix/store and symlinked from
/opt/ceh/installed. After this initialization they just exec
the binary from /nix/store,
/opt/ceh/bin-user/: gitignored directory for local exclusions, so the
end-user has the final say. The scripts/ceh-profile.sh script puts
this directory in front of the bin/ directory in PATH,
/opt/ceh/emacs.d/: contains nix-mode.el only, we used to
distribute emacs modules via Ceh, but it didn’t work too well, it’s
simpler to just use melpa,
/opt/ceh/home: gitignored symlink to the user’s home, this is
extremely useful, because if Ceh is installed, you can simply use
/opt/ceh/home as a path to point to the user’s home directory
without using the tilde expansion of the shell (that doesn’t work in
syscalls or on the shebang line),
/opt/ceh/lib/: the Ceh (mainly written in Perl) internals,
/opt/ceh/p/: gitignored symlinks to the user’s projects, so every project can
have an absolute symlink in /opt/ceh/p/whatever-project, like we
have /opt/ceh/home as an absolute path for the user’s home directory,
/opt/ceh/scripts/: Ceh end-user scripts (install, purge, profile, upgrade),
/opt/ceh/scripts/maintainer: Ceh maintainer scripts,
/opt/ceh/nixpkgs: contains checked out versions of the nixpkgs
package collection, used and maintained by lib/CehInstall.pm,
gitignored,
/nix: not in Git, but doesn’t store any important data. Deleting
all the content and reinstalling Ceh will result in some compilation
and downloading, but everything will be recreated eventually,
/nix/store: contains the installed package roots. Managed by nix
commands, like nix-store, nix-env and nix-instantiate,
/opt/ceh/installed/packages: gitignored directory storing the currently
installed packages on a machine. The state is stored as symlinks to
/nix/store and there are symlinks to here from
/nix/var/nix/gcroots/auto/ceh. So the purpose of this directory is
two-fold: first, it protects the packages that the user use from being
garbage collected. Secondly, we can determine with a simple stat
whether a package is already installed.
/opt/ceh/installed/essential: nix and perl that is used by Ceh,
/opt/ceh/installed/tools: the tools used by Ceh itself (e.g. which).
- Simply create a wrapper script in
/opt/ceh/bin, usetmuxas an example, - use the autoinit feature to get the initial hash values:
ceh_nixpkgs_install_bin('newpkg', AUTOINIT);After running this wrapper once, the AUTOINIT will be replaced with the current nixpkgs version and out hash value.
- add it to
scripts/maintainer/install-world.sh, - symlink manpages in
/opt/ceh/manif appropriate, - send a github pull request.
- Update CEH_BASELINE_NIXPKGS in
lib/CehBase.pmwith a new hash, that you should get from http://hydra.nixos.org/jobset/nixpkgs/trunk, by choosing the most recent build that has no pending (gray) packages anymore. Clicking inputs, you can get the git revision you need. - Update the used Nix and Perl to the new baseline:
CEH_AUTO_UPGRADE=1 \ ENSURE_BASE_PERL=/opt/ceh/installed/essential/perl/MAIN/bin/perl \ ENSURE_BASE_NIXPATH=/opt/ceh/installed/essential/nix/MAIN /opt/ceh/lib/ensure_base_installed.pl
- Update CEH_BASELINE_NIXPATH in
lib/CehBase.pm(peek from CehInstall.pl:ensure_base_installed). - Update CEH_BASELINE_PERL in
lib/CehBase.pm(peek from CehInstall.pl:ensure_base_installed). - Update all the wrappers to have the new baseline versions. For example to upgrade git:
export CEH_AUTO_UPGRADE=1 /opt/ceh/bin/gitk
This will update the hash in lib/Packages/Git.pm.
- To update all the packages, use scripts/maintainer/install-world.sh. You may also find scripts/maintainer/predict-binary-cache.sh useful in selecting and checking a baseline.
git grepthe old baseline hash to see that everything has been updated.- git commit, git push.
This should only be done if you have some strong reason for this. E.g. it’s no longer compatible with current glibc on standard linuxes or the communication protocol for nix-daemon changed and therefore Ceh is unusable on NixOS.
- Choose a new release from http://nixos.org/nix/download.html, read release notes, etc.
- Look into the install shell script, to figure out the current URL for i686-linux, e.g. for 1.11.2 it was `https://nixos.org/releases/nix/nix-1.11.2/nix-1.11.2-i686-linux.tar.bz2`
- tar xvfj nix-x.x.x-i686-linux.tar.bz2
- Update
CEH_NIX_DOWNLOAD,NIX_TARDIR_NAMEandCEH_NIXinscripts/ceh-init.sh. - Test if
scripts/ceh-init.shstill works on a clean Debian and on a clean NixOS. - git commit, git push.
Title: Nix & ceh: reproducible power-user environment Abstract: It's common for engineers to use multiple GNU/Linux based computers daily, however it's not easy to achieve the same working environment on each of them. Computers owned by the company may run the company's linux variant, while computers at home run some common distribution. Some of the running variants are already 64-bit, some are not. Some of them is 2 years old, some of them is fresh. On top of this mixture, it's hard to imagine to run exactly the same version of GNU Emacs or Mozilla Firefox (with all the hard to install plugins: flash, googletalk, java), without installing a common base operating system. This is the problem that I wanted a solution for, when I looked into the Nix package manager and the Nixpkgs package collection. After a month of exploration I now have a working setup (called ceh), that I run on all of my computers to synchronize (via Git) the choice of software between them. In the techtalk I will present the fundamentals of Nix and show ceh, my setup around Nix.