BIMHAW: BashIng My Head Against a Wall

Cut the Gordian know of shell profiles on linux.

Whats the problem?

Despite it being the most used interface as a linux user, configuration of shells is unnecessarily obtuse and lacks many important features.

I mean more specifically, and less ranty?

Refactor shell startup scripts so that you can actually understand and edit without thinking about it too much.

Particularly the strange and subtle interactions between the system shell (called “sh” or “posix shell” throughout despite whatever true identity it has…).

Check out this diagram which shows the startup process (that is documented, implementations are known to significantly deviate from this…):

*Nix shell apologists please with a straight face tell me this is okay and you should think about this everytime you want to add a little something to your dotfiles?

Provide for different shell “profiles”

Having multiple profiles that are easily changed between is something that I found a great need for in a system. There are multiple use cases for this:

• Different profiles for different computers or environments.
• Always having a simple fall-back profile in case something breaks and you really shouldn’t be working on debugging your dotfiles.
• Experimenting with new shell configurations without committing to it.
• Running demos, tutorials, or presentations where you want as little as possible to be able to go wrong.
• Activating your rice at the cafe, but then returning to your uncool comfort setup in the comfort of your home.

Proper (re)factoring of shell configuration scripts

In addition to being very aesthetically messy and difficult to debug dumping all configuration into a single .bash_profile (or was it .bashrc?) doesn’t really work well for multiple profiles. Multiple profiles would require a lot of duplicated code/configuration since many of them will share the same features. So we provide a rough module system. Where modules are just scripts for different roles like environment, aliases, and autocompletion placed in certain directories. The modules for each profile are specified by name in configuration file.

Sounds complicated, whats the solution?

Getting Started

Installation

Install it from the git repo directly:

pip install git+https://github.com/salotz/bimhaw.git

Initialization

Once you have it installed you will need to create a configuration directory at $HOME/.bimhaw.

You can and should do this with the initialization command:

python -m bimhaw.init

This will fail if the directory already exists.

Now you should have a $HOME/.bimhaw directory with some stuff:

an empty profiles dir

this is where the generated profiles shell scripts (from the templates; see below) will be placed. These files are not meant to be permanant, editing should be for debugging purposes only.

• a dir shell_dotfiles with:

  profile

  where $HOME/.profile will be symlinked

  bash_profile

  ditto but for .bash_profile

  bashrc

  ditto

  bash_logout

  ditto

env.sh

a shell script that will be sourced for all bimhaw shells, which contains a crap-ton of variables that bimhaw uses for making writing module scripts easier, but some are also necessary. This is auto-generated, do not edit.

config.py

the configuration for your profiles. You can edit, but we suggest symlinking to another config file controlled under another personal config dir so that this directory can be regenerated whenever (for updates and screwups).

lib

a directory that has all the module scripts and other configuration files etc. Ditto for config.py editability, see below.

As mentioned in the notes above we highly suggest linking to config.py and lib directories of your own somewhere else. There are two main reasons for this:

1. It is likely that you will want to either update bimhaw in the future or have to remake the $HOME/.bimhaw directory (because you screwed up you ninny) in the future and you don’t want to accidentally want to overwrite your carefully crafted config files. Think of $HOME/.bimhaw as a build or cache directory. You don’t want to unnecessarily remake it but its not really a big deal if you do. And you definitely don’t want them in version control and writing the .gitignores for them would be unnecessary and annoying.
2. Dotfiles are not meant to be forked. While personally we try to make bimhaw fit all of our pragmatic needs in managing environments and profiles in a *nix environment not everyone else will share this with us. Invariably you will have your own configurations that don’t fit into bimhaw that you will want to have a place for. Also you might store secrets or something there.

Personally we have a separate repo at $HOME/.salotz.d with all this kind of stuff. This is actually where bimhaw was born as a prototype that I used and built over about a year. We highly recommend such a place and a private repo server to house it (and git-crypt for passwords etc.).

To make this work with bimhaw in the easiest possible way, just symlink config.py and lib. You can do this at the beginning with:

python -m bimhaw.init --config "$HOME/.myhandle.d/config" --lib "$HOME/.myhandle.d/lib"

Loading Configurations

Now that we have the configuration directory setup we want to set up a profile to use.

To create profiles we run the main command line application:

bimhaw profile.gen --name 'bimhaw'

If you run it like this you will create the default and only built-in profile that bimhaw has called ‘bimhaw’. If you want to create your own profiles you will need to edit the ~~/.bimhaw/config.py~ file which is discussed in Creating new profiles.

This command will read .bimhaw/config.py (yes it uses exec so make sure you only have trusted code which shouldn’t be a problem since it is just setting some strings and may change in the future to some non-problematic configuration system).

Now we need to activate this profile which is simply done by running:

bimhaw profile.load --name 'bimhaw'

This simply makes the symbolic link .bimhaw/active point to the specified profile directory.

And technically it will also rerun the profile.gen subcommand again, so in the future you only have to run the one command.

bimhaw profile --name bimhaw

To actually get this profile loaded upon startup of your shell we need to link the actual shell startup files to the ones where the shell programs expect them to be.

Please go and back up your current dotfiles, these dotfiles will need to be turned into symbolic links:

• $HOME/.profile
• $HOME/.bash_profile
• $HOME/.bash_logout
• $HOME/.bashrc

Once they are backed up you can run:

bimhaw link-shells

If you didn’t delete or move the listed startup files then either do so or run with the ‘force’ flag:

bimhaw link-shells --force

Creating new profiles

Lets say we want to create a common use profile.

We would edit the config.py file so that it looked something like this:

docs/example_config.py

Apologies for the verboseness, but I didn’t want to commit to any configuration system as of now.

Since there are no config files to differentiate the ‘common’ profile from the ‘bimhaw’ it will be the same. You can add new content for it by adding files to the folders in the lib directory (see below).

To create this profile run:

bimhaw profile.gen --name common

Writing modules

Your “modules” should be referenced by the $HOME/.bimhaw/lib target and must maintain a specific directory structure in order to work properly.

Some might see this as onerous as they must conform to a specific way of thinking. Try it out and see for yourself. In the process of creating these distinctions we learned a lot about the startup process actually works and maybe you will too. It is perfectly in your rights to remain in your basement and disparage everything that isn’t from 1989 (we promise to only nickname you troglodytes behind your back. Even the new mac OS is using ZShell…)

The most critical modules are directly related to the configuration of your *nix shells. bimhaw has some support for other features which will be talked about later (shell configuration is big enough of a topic.)

For shells there are a few different roles of modules:

envs

define environmental variables, call other executables, etc. most of the “code” part of configuration

funcs

functions in the bash or sh sense, whatever that means

aliases

things you create with the shell builtin alias

logouts

scripts to run when you logout

prompts

choice of a prompt

autocompletion

autocompletion scripts (bash only)

The modules for posix shell are in .bimhaw/lib/shell/sh and for bash they are in .bimhaw/lib/shell/bash.

It’s important to note that in bimhaw all modules in a profile that are loaded for ‘sh’ will also be loaded for ‘bash’. This is part of the bash startup sequence (in most cases; see diagram). The modules specified for bash will be bash only. This allows for “pure” and “portable” posix sh scripts (we all know you sh scripts are only of the highest quality and POSIX compliant…) and also configurations with bashisms.

To add or edit modules you probably will just want to go to $HOME/.bimhaw/lib/shell/sh, unless you know it is something particular to bash.

In here you will see the folders for the different module types outlined above. If you want to create a module for a development environment called, for example, dev; create the file sh/envs/dev.sh.

Then put whatever configuration you want in it and add the module to whatever profile needs it in $HOME/.bimhaw/config.py under the SH_LOGIN_ENVS or SH_NONLOGIN_ENVS groups.

Same goes for all the other kinds of modules.

How it works

To refactor the shell startup we provide a single set of replacement contentless startup files located at $HOME/.bimhaw/shell_dotfiles. These in turn source another set of startup scripts with clean sensible names that are expected to be at $HOME/.bimhaw/active. These clean sensible ones are provided as templates (using jinja2) so you can easily generate many of them. These scripts are also relatively contentless and will only contain listings of which modules they are to use (which come from the configuration file). This also allows for the templates to be updated (although there probably we won’t be too many of those) since you shouldn’t be storing any of your state there.

We’ve solved the terrible legacy cruft with a couple levels of indirection (i.e. symlinks) and so is more complicated (i.e. not KISS; some ugliness is necessary to get shit done I’m afraid). Here is the high level linking structure (roughly) to help you understand starting with the file targets listed above:

~.profile~ --> ~.bimhaw/shell_dotfiles/profile~ --> ~.bimhaw/active~ --> ~.bimhaw/profiles/profile/shells/sh/login.sh~
~.bash_profile~ --> ~.bimhaw/shell_dotfiles/profile~ --> ~.bimhaw/active~ --> ~.bimhaw/profiles/profile/shells/bash/login.sh~
~.bashrc~ --> ~.bimhaw/shell_dotfiles/profile~ --> ~.bimhaw/active~ --> ~.bimhaw/profiles/profile/shells/bash/interactive.sh~

Diatribe

Wherein I try to come to some sort of enlightenment of what a healthy relationship with *nix shells looks like.

Newbs read on. Hopefully, it will save you much suffering.

Shells

A system is interacted with by a shell and so the shells must be the first thing to be configured.

Unfortunately, on unix-like systems, such as linux, the configuration of shells is extremely and unnecessarily complex for historical reasons.

However, before I start denigrating shell languages too much, one should consider some unique challenges shells face in their design.

First, shell languages being at the heart of a distro are going to have more pressure to maintain backwards compatibility and work in consensus with the actual distro maintainers. Who – understandably – are more interested in maintaining rather than adding features. Thus, the datedness of the shell languages usually shows when compared with newer, saner languages like python (which interestingly was originally developed as a shell language for the Amoeba distributed operating system project) and tcl (which again, interestingly was developed originally as the shell scripting language for another distributed operating system of the same era, neither of which succeeded).

Second, the obvious and glaring warts on the languages like the control flow structures (such as if...fi) tend to obscure the fact that much of the language is dealing with things that don’t really have first-class treatments in higher level languages. Things like piping, redirection, and subshells. These are actually very difficult and subtle things to get right in any programming language or system. So take some time to understand them without getting too flustered about the more procedural programming elements being un-ergonomic.

Introduction to the complexity of unix shells

The first thing we must consider is the features implementated by the shell (or shells) that will be on the systems you will be configuring.

Furthermore, the capabilities, features, and syntax of shells is highly divergent. This would be okay if there were value-added shells that could be used on specific systems and setups, were there a standard shell that could be used across different systems.

This is why the POSIX shell standard was developed, which is only a specification and not an implementation.

This specification furthermore is dated and idiosyncratic, and no shell implements the specified features on a 1:1 basis.

The most used shells, bash and zsh, implement many more features than the POSIX shell and the use of scripts written with them will vary from system to system.

A shell script written adhering to the POSIX standards should run in all shells, but it requires that the author understand that all features of the shell they are using should not be used.

Of course this is not entirely true in practice and there are discrepencies between the shells.

So the best advice for writing shell scripts that are meant to be portable is to keep them very simple.

If you need more advanced control flow, consider using a shell or programming language (that is a more sane language than a bash-type language) which is easily available on all systems you wish to configure (such as python or xonsh).

To make matters worse (at least) bash has the ability to run in a separate mode that alters its behavior to adhere more to a pure POSIX shell.

This can happen if an explicit flag is raised during invocation, or if the /bin/sh file is a symlink to the bash binary (I know a rather strange and special case behavior). Details on the change in behavior can be found here: https://www.gnu.org/software/bash/manual/html_node/Bash-POSIX-Mode.html

The shell executable located at /bin/sh is for all POSIX systems assumed to be a POSIX compliant shell and should be declared as the runner for all scripts intended to be executed by a POSIX shell. That is scripts that are intended to be portable across POSIX operating systems.

Different distros use different shells for /bin/sh but the most popular seem to be dash (Debian Almquist Shell; in e.g. Ubuntu) and bash.

Bash is used because it is the most widely used shell, however, it will be run in compatibility mode as described above and will not be the shell you know, and perhaps love.

The dash shell is used because it is much faster than other shells and is an attempt to be as compliant as possible to the POSIX standard. I will assume for my purposes that dash is the reference implementation of the POSIX shell. So any script that is written should be tested against the dash shell before being considered portable.

Configuring shell startups

Configuring the runtime properties and environment of a shell is also a convoluted mess, in part due to a complex set of different modes that shells can be started in and a mistrust on the adherence to the behavior that is documented in the manual.

The documented behavior is shown in this diagram:

Basically a shell can be started with 3 options:

• login or nonlogin
• interactive or batch
• local or remote

The files that get read and executed as part of the configuration changes with which modes are activated.

Here is a table showing which configuration scripts are executed and in what order for different options. The labelled columns are the different states that trigger sourcing of files.

Files are sourced in alphabetic order. Numbers indicate choices for sourcing at a given stage, and the lowest number will be searched for first. The first one encountered will be sourced and the rest will be skipped.

	Interactive	Interactive	Batch	Batch	Remote
	login	non-login	login	non-login
/etc/profile	A			A
/etc/bash.bashrc		A
~/.bashrc		B			A
~/.bash_profile	B1			B1
~/.bash_login	B2			B2
~/.profile	B3			B3
BASH_ENV			A	C
~/.bash_logout	logout		logout

Here is the table for POSIX shells:

	Interactive	Interactive	Batch	Batch
	login	non-login	login	non-login
~/.profile	A		A
ENV	B	A

Because of the complexity of this process and the fact that we don’t want to duplicate the coding of environments which will be the same between shells, we code these common configurations in POSIX shell.

To map onto these logical categories of different shell stages we write shell scripts for each shell in the ‘shells’ directory.

These scripts include:

• interactive
• login_env
• nonlogin_env
• logout

Through clever sourcing of these dependencies between these files among different shells we can separate configurations common to all POSIX shells (in the sh dir) and for each specific shell, each having a dir with it’s namesake.

These initialization files are intended to encapsulate the logic necessary for this trick of dependencies, and most of the actual content of the configurations is found, logically, in with the rest of the configurations under the name of the shell, which may have configurations added as if it were any other program.

Alternatives

• https://drewdevault.com/2019/12/30/dotfiles.html
• Chezmoi