Spec for init scripts and sandbox updates

Overview

opam has a series of embedded shell scripts and shell commands which are used in three contexts as part of:

• opam init to set-up automatic switch detection based on CWD and command completion
• opam env to set-up the shell environment
• opam install to set-up the sandbox

Updating these requires updating every supported version of opam. The present mechanism is a little cumbersome (requiring the user to opam init --reinit) and can also result in unusable opam releases (e.g. from fish syntax removal).

This spec proposes moving the scripts and shell data to a separate repository which can be updated and released separately, allowing alterations to the sandbox and fixes to shell commands without having to re-release opam.

src/state/opamEnv.ml and src/client/opamInitDefaults.ml are where these scripts are presently used from.

Implementation

init and update

A new root branch scripts is added to ocaml/opam. Commits to this branch should be signed with the same GPG key as opam binaries. The branch should have linear history (i.e. no merge commits).

opam init gains a new --init-scripts <giturl>[#<branch>] parameter whose default value is https://github.com/ocaml/opam.git#scripts. If no <branch> is given then main is assumed (NOT the default branch of the repo). Two fields are added to .opam/config:

scripts-repository: "https://github.com/ocaml/opam.git"
scripts-branch: "scripts"

Two new options are added to both opam init and opam update: --fetch-scripts and --no-fetch-scripts (the default is --fetch-scripts) and additionally --update-scripts and --no-update-scripts to opam update (the default is --update-scripts). The opam binary at release time will contain an embedded git bundle of the scripts branch.

At opam init, as long as the --init-scripts parameter was either omitted or is (https://github.com/|git://github.com/|ssh://git@github.com/|git@github.com:)ocaml/opam[.git]#scripts, then the internal bundle is cloned to ./opam/opam-init/scripts. Otherwise, the supplied repository URL is cloned. Note that --no-fetch-scripts conflicts --init-scripts if the <giturl> is not the same as the git-bundle's. The cloned repository is referenced as upstream in the cloned repo (and will always point to <giturl> not the bundle) and the branch is called main. If the bundle is extracted, and --no-fetch-scripts was not specified, then upstream is pulled.

At opam update, unless --no-fetch-scripts is given, upstream is fetched. If the .opam/opam-init/scripts contains untracked or uncommitted changes or the branch main is not checked out then opam update emits a warning and does not merge or update the repo. After fetching, if there are new commits and --no-update-scripts was not given, opam update either:

• fast-forwards main to upstream/<branch>, if the HEAD of main is a commit on upstream/<branch> and informs the user that the scripts were updated.
• attempts an automatic rebase of main onto upstream/<branch>. If this succeeds, and custom commits remain, then the user is informed that the scripts were updated and custom patches have been retained. If this fails, then main is left unaltered and the user is warned that the upstream scripts have been updated but the patches made to them do not rebase cleanly.

Note that both processes may cause the variables and init scripts to be regenerated. opam will display the first line of each commit messages of the new commits which have been adopted.

Sandboxes

opamrc's init-scripts field contains the actual sandbox scripts. The default will be updated to:

init-scripts: [
  [
    "sandbox.sh"
    """\
#!/usr/bin/env bash

. "$(dirname "$0")/scripts/sandboxes-v1/bwrap.sh"
"""
  ] {os = "linux"}
  [
    "sandbox.sh"
    """\
#!/usr/bin/env bash

. "$(dirname "$0")/scripts/sandboxes-v1/sandbox-exec.sh"
"""
  ] {os = "macos"}
]

and the two scripts presently in src/state/shellscripts/ get moved to sandboxes-v1/ in the new scripts branch.

Note: versioning is achieved here by changing the name of sandbox scripts and updating the path in opamrc to point to it.

Shells

The shells are a little more involved. The existing env_hook and complete scripts get moved from src/state/shellscripts/ to env/ and complete/ in the new scripts branch.

The root of the scripts branch then contains a single config file:

opam-version: "2.2"
shells: [
  "bash" {opam-version >= "4.0"}
  "sh"
  "csh"
  "zsh"
  "fish" {os = "macos"}
]
shell "bash" {
  command: "bash"
  profile: [ ".bash_profile" ".bash_login" ".profile" ".bash_profile" ]
  profile-message: "Make sure that ~/%{profile}% is well sourced in your ~/.bashrc" {profile != ".bashrc}
  eval: "eval $(%{cmd}%)"
  comment: "# "
  export: "%{name}%=%{value}; export %{name}%;"
  source: "test -r %{name}% && . %{name}% > /dev/null 2> /dev/null || true"
  env-updates: [
    "%{single-quote-value}%" # =
    "%{single-quote-value}%:\"$%{name}%\"" # += / := / =+=
    "\"$%{name}%\":%{single-quote-value}%" # =: =+
  ]
}
shell "sh" {
  command: "sh"
  profile: ".profile"
  eval: "eval $(%{cmd}%)"
  init {
    complete: [
      "complete/2.0-complete.sh" {opam-version = "2.0"}
      "complete/2.1-complete.sh" {opam-version = "2.1"}
    ]
    env-hook: "env/hook.sh"
    tty: [ "if [ -t 0 ]; then" "else" "fi" ]
  }
  source: "test -r %{name}% && . %{name}% > /dev/null 2> /dev/null || true"
  comment: "# "
  export: "%{name}%=%{value}; export %{name}%;"
  env-updates: [
    "%{single-quote-value}%" # =
    "%{single-quote-value}%:\"$%{name}%\"" # += / := / =+=
    "\"$%{name}%\":%{single-quote-value}%" # =: =+
  ]
}
shell "csh" {
  command: "csh"
  aliases: [ "tcsh" "bsd-csh" ]
  profile: [ ".cshrc" ".tcshrc" ]
  eval: "eval `%{cmd}%`"
  init {
    env-hook: "env/hook.csh"
    tty: [ "if ( $?prompt ) then" "else" "endif" ]
  }
  comment: "# "
  export: """\
if ( ! ${?%{name}%} ) setenv %{name}% ""
setenv %{name}% %{value}%
"""
  source: "if ( -f %{name}% ) source %{name}% >& /dev/null"
  env-updates: [
    "%{single-quote-value}%" # =
    "%{single-quote-value}%:\"$%{name}%\"" # += / := / =+=
    "\"$%{name}%\":%{single-quote-value}%" # =: =+
  ]
}
shell "zsh" {
  command: "zsh"
  profile: ".zshrc"
  eval: "eval $(%{cmd}%)"
  init {
    variables: "variables.sh"
    complete: "complete/complete.zsh"
    env-hook: "env/hook.zsh"
    tty: [ "if [[ -o interactive ]] then" "else" "fi" ]
  }
  comment: "# "
  export: "%{name}%=%{value}; export %{name}%;"
  source: "[[ ! -r %{name}% ]] || source %{name}% > /dev/null 2> /dev/null"
  env-updates: [
    "%{single-quote-value}%" # =
    "%{single-quote-value}%:\"$%{name}%\"" # += / := / =+=
    "\"$%{name}%\":%{single-quote-value}%" # =: =+
  ]
}
shell "fish" {
  command: "fish"
  profile: ".config/fish/config.fish"
  eval: "eval (%{cmd}%)"
  init {
    env-hook: "env/hook.fish"
    tty: [ "if isatty" "else" "end" ]
  }
  comment: "# "
  export: [
    "set -gx %{name}% %{fish-array-value}%;" {name = "PATH"}
    "if [ (count $MANPATH) -gt 0 ]; set -gx MANPATH %{fish-array-value}%; end;" {name = "MANPATH"}
    "set -gx %{name}% %{value}%"
  ]
  source: "source %{name}% > /dev/null 2> /dev/null; or true"
  env-updates: [
    "%{fish-single-quote-value}%" # =
    "%{fish-single-quote-value}%:\"$%{name}%\"" # += / := / =+=
    "\"$%{name}%\":%{fish-single-quote-value}%" # =: =+
  ]
}

The fields in this file control the entire algorithm used in OpamEnv. The shells field is a filtered list of names of shell sections. The filtering allows for both platform-specific shells (e.g. for Windows) and also for different versions of opam. The format of this file will always be opam-version: "2.2" - any breaking future version will have to use a file other than config. Fields within a shell section which are not recognised should be silently ignored. The shell section defines the following fields:

• command: the command name of the shell
• aliases (optional): other possible names of the shell (cf. OpamStd.Env.shell_of_string)
• profile: list of paths relative to $HOME to search for a profile. First matching one is used or the last entry (cf. ".bash_profile" appearing twice in the bash list, therefore).
• profile-message (optional): filterable message to display (cf. .bashrc message in opam init)
• eval: the command to use to run the literal text of cmd variable
• init (optional). If given, init.%{command}% is generated subject to:
  • variables (optional): if given, this is the name of variables script to source in init. If not given, variables.%{command}% will be generated (and sourced).
  • complete (optional): relative path to a completion script for this shell. This will be symlinked in .opam/opam-init and sourced in init.%{command}%.
  • env-hook (optional): as for complete with the env-hook script.
  • tty: a three-member list giving the syntax for a conditional for whether the shell is interactive. The first element is the test, the second element is the line for "else" and the third element is the line for "endif".
• comment: the prefix for comment lines
• export: the syntax for exporting a variable. The variables name, value and fish-array-value are defined and this can be a filterable list where the first matching filter is used.
• source: the syntax for sourcing name without error
• env-updates: a three-element list of the syntax for the = operator, the +=/:=/=+= operators and the =:/=+ operators. value, single-quote-value (single-quoted to sh-conventions using ") and fish-single-quote-value (single-quoted to fish-conventions so with \ and ' backslash-quoted) are available along with name.