Skip to content

nilcons/ceh

Repository files navigation

https://travis-ci.org/nilcons/ceh.png

Power-user environment to share

  • Agreements and a repository for a shared GNU/Linux power-user environment.
  • So that it’s easy to work on shared projects (e.g. the haskell header can be #!/opt/ceh/bin/runhaskell -i /opt/ceh/p/ourproject, instead of some complicated perl one-liner).
  • So that the packages are available with the same version on our machines (e.g. emacs, tmux, firefox, etc).
  • These agreements are intentionally based on simple unifications instead of abstractions (e.g. /opt/ceh/home has to be a symlink for the user’s home directory, no need to $HOME or tilde lookups anymore).
  • We use amd64 Nix for package management and if you can decide on the distribution that you use as a base system, we recommend amd64 Debian. But the point is that this should work on any modern amd64 GNU/Linux, we also tested Ubuntu trusty and precise.

Requirements

  • Modern GNU/Linux distro (ubuntu, debian, etc.).
  • Intel architecture (amd64, no i386/i686 or arm).
  • The machine only has one user who wants to use ceh and nix and she will be the owner of the files in /opt/ceh and /nix.
  • If the machine is running NixOS already, then /nix should exist already of course and Ceh is able to handle that.
  • Root privileges for the installation. Actually only needed for the creation of /opt/ceh and /nix, not needed for everyday usage or package installation.
  • /etc/passwd contains your user, so nix doesn’t have to use any fancy NSS modules.
  • The user uses bash or zsh. This is because he has to source /opt/ceh/scripts/ceh-profile.sh which supports these two shells. Patches are welcome for other shells.

Installation instructions

sudo mkdir /opt/ceh /nix
sudo chown $USER. /opt/ceh /nix
chmod 0700 /opt/ceh /nix
cd /opt/ceh
git clone git://github.com/nilcons/ceh.git .
ln -s $HOME home
/opt/ceh/scripts/ceh-init.sh

Installation instructions for NixOS

sudo mkdir /opt/ceh /nix/var/nix/gcroots/auto/ceh
sudo chown $USER. /opt/ceh /nix/var/nix/gcroots/auto/ceh
chmod 0700 /opt/ceh /nix/var/nix/gcroots/auto/ceh
cd /opt/ceh
git clone git://github.com/nilcons/ceh.git .
ln -s $HOME home
/opt/ceh/scripts/ceh-init.sh

Everyday usage

In the /opt/ceh/bin directory there are small wrapper scripts. Those install the required pacakages for the wrapped binaries on first run.

/opt/ceh/bin is put into $PATH by /opt/ceh/scripts/ceh-profile.sh, that you’re supposed to source from your ~/.bashrc.

Exclusions: ignoring programs provided by ceh

If for any reason you prefer to ignore one of our provided binaries, you can use the ceh_exclude script to create a symlink from /opt/ceh/bin-user/ to the other version of the binary in your PATH.

If you don’t want the exclusion anymore, just delete the symlink.

Removing packages that got installed on-demand

As previously mentioned, the scripts in /opt/ceh/bin just install the programs on demand as you run them. Therefore once ran, they take up disk space on your machine. If you tried out something big and would like to get rid of it, then go to /opt/ceh/installed and feel free to delete any symlinks there. After deleting the symlinks, run nix-garbage-collect to free up the disk space. If you decide to use some deleted software again, it will be redownloaded.

If you delete from /opt/ceh/installed/essential, then you will have to run ceh-init.sh, but Ceh will show a warning message on stderr about that.

Troubleshooting if something is broken after git pull

Sometimes there are changes that are not backward compatible with the previous ceh-init.sh. In those cases we do our best to provide warning messages on stderr from scripts, and those messages will ask you to rerun ceh-init.sh. That operation is idempotent, feel free to run it you think it can fix up something for you.

You can also read the “Upgrade notes” section to know about changes.

Disable the binary cache, for fun and profit

You may just want to see if it’s still possible to build a software from source, instead of using the nix binary cache. Or the nix binary cache is down and you still want to install packages instead of having to wait until it’s back.

In both cases, you can just set $CEH_NO_BIN_CACHE to 1 and then the wrapper scripts will start building packages instead of using the binary cache.

When you do this, you may also want to set $CEH_BUILD_MAX_JOBS to the number of your processor cores in the system, because usually building of tools (like compilers, linkers, make, etc.) can be parallelized by nix.

Upgrade notes

2018-05-23: Nix 2, no nix-mode.el, no firefox, no vanitygen, no 32-bit, no TWS

This is the first Ceh release with Nix 2, if there are issues, please keep us posted!

Nix 2 removed the nix-mode.el emacs bindings, instead you have to use it from Melpa, as you do with all your other emacs modules.

We removed Firefox, because it was a constant maintenance burden, and the reasoning from many years ago (easy to use and install java, flash, googletalk plugin) is not strong enough anymore. The Firefox versions shipped by standard desktop distibutions are good enough.

Vanitygen is removed from nixpkgs, so we remove it from ceh. :(

For the shiny new Nix2 2 gigs of ram is not enough to download ghc-8.2.2 from the binary cache. Therefore we can’t use 32-bit nix anymore. So the base perl and nix package has been changed to 64-bit versions. The bit32 option has also been removed, ceh is now 64-bit only. Rest in peace ia32.

TWS is also removed, because Interactive Brokers started to release usable linux desktop installation packages, which includes the correct version of java and just works.

2017-07-14: no Haskell anymore

We think that for Haskell, stack is a clear winner in the ecosystem.

Therefore we may add stack support to ceh in the future, but we have removed our nix based haskell support.

2016-04-14: support for multiple outputs

As preparation for a baseline upgrade, we had to do a refactoring: in new nixpkgs versions, most of the derivations result in multiple output, not just one. Previously this happened only rarely, so we handled the special cases with the outFilter hack. This hack is gone, and instead we refactored the gclink symlinks a little bit to support multiple derivations.

For the end user, the change shouldn’t be notable and everything should continue working as expected.

2014-12-20: no more nixops

NixOps was not working after the 32-bit -> 64-bit migration, I do not think this is anything to do with the architecture, something has changed in the git repo.

Anyways, we know of no users who uses Ceh and nixops together, so it was easier to remove it. Tell us if you need it!

2014-12-20: 32-bit -> 64-bit migration

I love i686, programs are so fluffy and cute. But unfortunately the world started to ignore that platform and stuff is supported only on amd64 nowadays. E.g. the haskell “community” ignores i686 altogether and unit tests routinely break. The nix binary cache is also only maintained for amd64 and if the glibc is missing for i686, nobody will fix it and you will have to build everything from source.

So we decided to move ceh to 64-bit, and live with the memory and disk usage of amd64.

This transition should be seamless from the user point of view, but if something goes bad, of course you can always just purge Ceh and reinstall it…

Also, if you want to install 32-bit stuff from some packages, you have to use the bit32 flag, instead of the old bit64 flag that we had for 64-bit. For Haskell, you can use the envvar CEH_GHC32 to get a 32-bit GHC and tools.

As a last resort, the base perl and nix package is still i686, so you can run ceh-init.sh even on a i686 system and write your own wrappers by using the bit32 flag for every package. But we are ready to remove this convenience the first time it causes some problems.

Sorry guys, i686 is dead.

2014-05-08: refactoring

From now on, we don’t use profiles for Ceh’s internal purposes. Instead we use the /opt/ceh/installed directory.

Profiles essentially give you two features. First, it’s a way to track different states of what is installed and an easy way to roll back and forward between those states. As we have git, we don’t need this. The other feature is that profiles give you a huge symlink farm that mimics a directory tree where only the currently installed packages are seen. We don’t need this feature either, because we have the wrappers in /opt/ceh/bin and we can just introduce a constant symlink farm for the manpages in /opt/ceh/man.

This refactoring will also solve the constant issue around file collissions in profiles and package renames in upstream nixpkgs.

You can safely delete /nix/var/nix/profiles/ceh and run a nix-garbage-collect to get rid of some old stuff that never would be deleted otherwise.

2014-03-04: refactoring

The new place for the nixpkgs checkout of Ceh is /opt/ceh/nixpkgs, you can run rm -rf /nix/var/ceh_nixpkgs to remove the old cruft.

Also, from now on it’s not necessary (or allowed) to specify derivation hashes when calling ceh_nixpkgs_install_* functions. Derivation hashes are always figured out automatically when on-demand installations is needed and only the output hash is consulted for binary consistency.

Another change is that we made ceh-init.sh idempotent and it should fix up stuff from any state that is reasonable after a git pull or after deletion of all the profiles in /nix/var/nix/profiles/ceh. This means that ceh-upgrade.sh is no more and you should always use ceh-init.sh if you think Ceh should be reinitialized.

2014-01-07: nix channels moved

On existing installations, run the following:

nix-channel --remove nixpkgs
nix-channel --add http://nixos.org/channels/nixpkgs-unstable
nix-channel --update

2014-02-03: xpra upgrade broken

Unfortunately the nixpkgs guys changed the name of the xpra packages from python2.7-xpra to xpra. As described in the Troubleshooting section, this is causes a filename collision. This can be fixed:

/opt/ceh/bin/nix-env -p /nix/var/nix/profiles/ceh/bin -e python2.7-xpra
/opt/ceh/bin/xpra

2013-08-05: ~/.nix-profile reserved for the user

The /nix/var/nix/profiles/per-user/root/profile profile (which is symlinked by default from $HOME/.nix-profile) is not used for installations anymore. Earlier versions of ceh used to install packages there, but now that profile is reserved for the user.

This is good, because if you need some package (e.g. vim) quickly from the nixpkgs version that is current the ceh baseline, you can just do:

ceh_nixpkgs_install vim
~/.nix-profile/bin/vim

Alternatively, you can use the most up-to-date nixpkgs:

nix-channel --add http://nixos.org/channels/nixpkgs-unstable
nix-channel --update
nix-env -i vim
~/.nix-profile/bin/vim

No need to create a wrapper in /opt/ceh just for a quick test. If you want to get rid of your user profile, simply delete /nix/var/nix/profiles/per-user/$USER/profile*. This is from now on is totally separate from ceh’s own nix profiles. Those are located in /nix/var/nix/profiles/ceh/*.

If you’re upgrading ceh from a version, where the default profile was used for package installations, please delete /nix/var/nix/profiles/per-user/root/profile*.

2013-08-15: renames

nix-init.sh -> ceh-init.sh nix-profile.sh -> ceh-profile.sh nix-purge.sh -> ceh-purge.sh nix-upgrade.sh -> ceh-upgrade.sh

And install-world.sh and predict-binary-cache.sh moved to scripts/maintainer.

TODOs

  • Have an LD_PRELOAD library that hides /usr/lib/mozilla/plugins from Firefox’s view.

Design and other info for Ceh hackers, maintainers

Directory structure

/opt/ceh/bin/: wrapper scripts, they make sure that the wanted package is already installed into /nix/store and symlinked from /opt/ceh/installed. After this initialization they just exec the binary from /nix/store,

/opt/ceh/bin-user/: gitignored directory for local exclusions, so the end-user has the final say. The scripts/ceh-profile.sh script puts this directory in front of the bin/ directory in PATH,

/opt/ceh/emacs.d/: contains nix-mode.el only, we used to distribute emacs modules via Ceh, but it didn’t work too well, it’s simpler to just use melpa,

/opt/ceh/home: gitignored symlink to the user’s home, this is extremely useful, because if Ceh is installed, you can simply use /opt/ceh/home as a path to point to the user’s home directory without using the tilde expansion of the shell (that doesn’t work in syscalls or on the shebang line),

/opt/ceh/lib/: the Ceh (mainly written in Perl) internals,

/opt/ceh/p/: gitignored symlinks to the user’s projects, so every project can have an absolute symlink in /opt/ceh/p/whatever-project, like we have /opt/ceh/home as an absolute path for the user’s home directory,

/opt/ceh/scripts/: Ceh end-user scripts (install, purge, profile, upgrade),

/opt/ceh/scripts/maintainer: Ceh maintainer scripts,

/opt/ceh/nixpkgs: contains checked out versions of the nixpkgs package collection, used and maintained by lib/CehInstall.pm, gitignored,

/nix: not in Git, but doesn’t store any important data. Deleting all the content and reinstalling Ceh will result in some compilation and downloading, but everything will be recreated eventually,

/nix/store: contains the installed package roots. Managed by nix commands, like nix-store, nix-env and nix-instantiate,

/opt/ceh/installed/packages: gitignored directory storing the currently installed packages on a machine. The state is stored as symlinks to /nix/store and there are symlinks to here from /nix/var/nix/gcroots/auto/ceh. So the purpose of this directory is two-fold: first, it protects the packages that the user use from being garbage collected. Secondly, we can determine with a simple stat whether a package is already installed.

/opt/ceh/installed/essential: nix and perl that is used by Ceh,

/opt/ceh/installed/tools: the tools used by Ceh itself (e.g. which).

Adding a new package to ceh

  • Simply create a wrapper script in /opt/ceh/bin, use tmux as an example,
  • use the autoinit feature to get the initial hash values:
    ceh_nixpkgs_install_bin('newpkg', AUTOINIT);
        

    After running this wrapper once, the AUTOINIT will be replaced with the current nixpkgs version and out hash value.

  • add it to scripts/maintainer/install-world.sh,
  • symlink manpages in /opt/ceh/man if appropriate,
  • send a github pull request.

Upgrading ceh to a new baseline

  • Update CEH_BASELINE_NIXPKGS in lib/CehBase.pm with a new hash, that you should get from http://hydra.nixos.org/jobset/nixpkgs/trunk, by choosing the most recent build that has no pending (gray) packages anymore. Clicking inputs, you can get the git revision you need.
  • Update the used Nix and Perl to the new baseline:
CEH_AUTO_UPGRADE=1 \
ENSURE_BASE_PERL=/opt/ceh/installed/essential/perl/MAIN/bin/perl \
ENSURE_BASE_NIXPATH=/opt/ceh/installed/essential/nix/MAIN /opt/ceh/lib/ensure_base_installed.pl
  • Update CEH_BASELINE_NIXPATH in lib/CehBase.pm (peek from CehInstall.pl:ensure_base_installed).
  • Update CEH_BASELINE_PERL in lib/CehBase.pm (peek from CehInstall.pl:ensure_base_installed).
  • Update all the wrappers to have the new baseline versions. For example to upgrade git:
export CEH_AUTO_UPGRADE=1
/opt/ceh/bin/gitk

This will update the hash in lib/Packages/Git.pm.

  • To update all the packages, use scripts/maintainer/install-world.sh. You may also find scripts/maintainer/predict-binary-cache.sh useful in selecting and checking a baseline.
  • git grep the old baseline hash to see that everything has been updated.
  • git commit, git push.

Upgrading nix inside ceh-init.sh

This should only be done if you have some strong reason for this. E.g. it’s no longer compatible with current glibc on standard linuxes or the communication protocol for nix-daemon changed and therefore Ceh is unusable on NixOS.

Techtalk

Title: Nix & ceh: reproducible power-user environment

Abstract:

It's common for engineers to use multiple GNU/Linux based computers
daily, however it's not easy to achieve the same working environment
on each of them.

Computers owned by the company may run the company's linux variant,
while computers at home run some common distribution.  Some of the
running variants are already 64-bit, some are not.  Some of them is 2
years old, some of them is fresh.

On top of this mixture, it's hard to imagine to run exactly the same
version of GNU Emacs or Mozilla Firefox (with all the hard to install
plugins: flash, googletalk, java), without installing a common base
operating system.

This is the problem that I wanted a solution for, when I looked into
the Nix package manager and the Nixpkgs package collection.  After a
month of exploration I now have a working setup (called ceh), that I
run on all of my computers to synchronize (via Git) the choice of
software between them.

In the techtalk I will present the fundamentals of Nix and show ceh,
my setup around Nix.

About

A guild for people who like Nix

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •