RFC: FPM packaging

docs/contributing/contrib.md

@@ -1,19 +1,36 @@
 ## PaSh on Docker: A Pocket Guide
 
-This is a pocket guide for running PaSh in a docker container.
+This is a pocket guide for running PaSh in a Docker container.
 
 ### Loading image
 
-The following will pull and launch the latest image (tag: `pash:latest`) using the name `pash-play`
+#### From Docker Hub
+
+The following will pull the latest Docker image (tag: `pash:latest`),
+and start a new container using the name `pash-play`.
 
 ```sh
 docker pull binpash/pash
 docker run --name pash-play -it pash
 ```
+
 This will use `pash-play` for the container (to be able to start/stop it later) and `-it` runs it interactively (subsuming the next command).
 
 To restart after you exit, run `docker start -i pash-play` (flag `-i` starts it interactively.)
 
+
+#### From source code
+
+If you want to use or develop PaSh for a specific commit, this will
+build the latest Docker image.
+
+```sh
+git clone git@github.com:binpash/pash.git
+cd pash/scripts
+docker build -t "pash:latest" .
+```
+
+
 ### Customizing image
 
 To be used for continuous integration and testing, this image has been configured to have _read-only_ access to the repo through a different user.

docs/tutorial/tutorial.md

@@ -49,92 +49,93 @@ The new parallel script has POSIX constructs added to explicitly guide paralleli
 
 ## Installation
 
-#### Natively on Linux
+Regardless of your target platform, PaSh requires the `PASH_TOP`
+environment variable to point to a PaSh installation on disk.  The
+installation method impacts the value of this variable.
 
-_Ubuntu, Fedora, Debian, Arch:_
-Use `up`:
 
-```sh
-curl up.binpa.sh | sh
-```
+### Unix-likes
 
-_Other distros_:
-Use the system's package manager to install the following packages (some of which might already be installed in your system):
+Download your preferred system package (all architectures).
 
-```
-git libtool m4 curl automake pkg-config libffi-dev python python3 python3-pip wamerican-insane bc bsdmainutils python3-testresources python3-setuptools locales locales-all wget netcat-openbsd
-```
+| Package                           | Install command               |
+|-----------------------------------|-------------------------------|
+| [Red Hat (`pash.rpm`)](#link-tbd) | `yum localinstall ./pash.rpm` |
+| [Debian (`pash.deb`)](#link-tbd)  | `apt install ./pash.deb`      |
 
-Then clone the PaSh repository and run `setup-pash.sh` as follows:
 
-```sh
-git clone git@github.com:binpash/pash.git
-./pash/scripts/setup-pash.sh
-```
+Run `pa.sh` to verify installation of the following paths.
 
-As noted at the end of `setup-pash.sh`, make sure you set `PASH_TOP` pointing to the absolute path of the directory `pa.sh` resides (you can optionally place that in your `PATH`).
+| Path             | Purpose                                 |
+|------------------|-----------------------------------------|
+| `/usr/bin/pa.sh` | symlink to `/opt/pash/pa.sh`            |
+| `/usr/lib/pash`  | source code (as seen in the repository) |
+| `/var/log/pash`  | package build logs                      |
 
-_Installation Flags_:
+If something goes wrong, please submit your build logs with a bug
+report.
 
-PaSh offers the following installation flags:
-* `-p/--prepare` that installs all the required dependencies if your system is supported
-* `-o/--opt-agg` that downloads g++-10 and builds an optimized version of some aggregators (not currently implemented in the main PaSh workflow)
 
-To use these flags, see the following installation procedure:
-```sh
+### Source
+[FPM]: https://fpm.readthedocs.io/en/latest/cli-reference.html#general-options
+
+Source deployments start with a clone.
+
+```
 git clone git@github.com:binpash/pash.git
-cd pash/ && bash scripts/install.sh -p --opti-agg
 ```
 
-This will install all the dependencies for PaSh on your machine (sudo is used) and the g++-10 toolchain, but also build the respective aggregators.
-
-#### Docker
+You can either use the source code to create a Docker image, or deploy
+PaSh on the current host. For Docker instructions, see the
+[pash-on-docker
+guide](../contributing/contrib.md#pash-on-docker-a-pocket-guide).
 
-PaSh on Docker is useful when native installation is not an option -- for example, to allow development on Windows or OS X.
-First [make sure you install Docker](https://docs.docker.com/get-docker/) on your machine.
-Note that, depending on the configuration, PaSh on Docker may or may not be able to exploit all available hardware resources.
-There are two main options for installing PaSh via Docker.
+For native deployment, install [FPM][] to prepare native packages. For
+a Debian-based distribution (as an example), you can deploy PaSh on
+the current host for some label `X`.
 
-_Major Releases:_
-The easiest is to `pull` the docker image [from Docker Hub](https://hub.docker.com/r/binpash/pash):
-```sh
-docker pull binpash/pash
 ```
-We refresh this image (as well as other images) on every major release.
+./pash/scripts/package/build-packages.sh X
+sudo apt install ./pash/scripts/package/output/pash-X.deb
+```
 
-[//]: # "TODO(@nvasilakis, @dkarnikis): Need to automate this per release."
+RHEL is no different.
 
-_Latest Commit:_
-To built the latest Docker container from scratch, run `docker build` in scripts:
-```sh
-git clone git@github.com:binpash/pash.git
-cd pash/scripts
-docker build -t "pash:latest" .
 ```
-This will build a fresh Docker image using the latest commit---recommended for development.
-
-In all the above cases, launching the container is done via:
-```sh
-docker run --name pash-play -it pash
+./pash/scripts/package/build-packages.sh X
+sudo yum localinstall ./pash/scripts/package/output/pash-X.rpm
 ```
-PaSh can be found in the container's `/opt/pash` directory, so run `cd pash; git pull` to fetch the latest updates.
-More information in the [pash-on-docker guide](../contributing/contrib.md#pash-on-docker-a-pocket-guide).
 
-#### Windows using WSL
+[`build-packages.sh`](../../scripts/package/build-packages.sh) defines
+supported systems and distribution-specific modifications.
+
+
+### Docker
+[Docker]: https://docs.docker.com/get-docker/
+
+In the context of a [Docker][] container, `PASH_TOP` is `/opt/pash`. The
+same directory holds is a copy of the Git repository. Use `git -C
+/opt/pash pull` to update the code in a container. PaSh may not be
+able to exploit all available hardware resources when used in a
+container.
+
+For instructions, see the [pash-on-docker
+guide](../contributing/contrib.md#pash-on-docker-a-pocket-guide).
+
+[//]: # "TODO(@nvasilakis, @dkarnikis): Need to automate this per release."
+
+
+### Windows using WSL
 
 To run PaSh on windows without Docker, install [WSL](https://docs.microsoft.com/en-us/windows/wsl/install-win10).
 A short tutorial is included in the [contributing](../contributing/contrib.md) guide.
 
-## Running Scripts
 
-All scripts in this guide assume that `$PASH_TOP` is set to the top directory of the PaSh codebase (e.g., `/opt/pash` on docker)
 
-**To run scripts in this section of the tutorial, make sure you are in the `intro` directory of the `evaluation`:**
-```sh
-cd $PASH_TOP/evaluation/intro
-```
+## Running Scripts
+
+To use this section, run `cd $PASH_TOP/evaluation/intro` first.
 
-> In the following examples, you can avoid including `$PASH_TOP` before `pa.sh` by adding `PASH_TOP` in your `PATH`, which amounts to adding an `export PATH=$PATH:$PASH_TOP` in your shell configuration file.
 
 #### Intro: Hello, Parallel World!
 

pa.sh

@@ -1,6 +1,32 @@
 #!/usr/bin/env bash
 
-export PASH_TOP=${PASH_TOP:-${BASH_SOURCE%/*}}
+# Help out with shell configuration
+if [ -z "$PASH_TOP" ]; then
+    me="$(readlink -f "${BASH_SOURCE[0]}")"
+    export PASH_TOP="$(dirname "$me")"
+    export PATH="$PASH_TOP:$PATH"
+    printf '# Add this to your shell configuration.\n\n'
+    printf "  export PASH_TOP='%s'\n" "$PASH_TOP"
+    printf '  export PATH="$PASH_TOP:$PATH"\n\n'
+    printf '# If your shell supports process substitution (bash, zsh, etc.),\n'
+    printf '# then run `. <(%s)` to use this config in your current shell.\n' "$0"
+    exit 0
+fi
+
+# Halt if script was sourced. The above is only a setup aid that
+# should not interfere with the script's normal operation.
+return 2>/dev/null
+
+
+rcfile=~/.pash_init
+if [ ! -f "$rcfile" ]; then
+     printf '# Include system-level rules\n' > "$rcfile"
+     printf ". '%s/scripts/pash-init.sh'\n" "$PASH_TOP" >> "$rcfile"
+     chmod +x "$rcfile"
+     printf 'Created %s\n' "$rcfile"
+fi
+
+
 export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib/"
 # point to the local downloaded folders
 export PYTHONPATH=${PASH_TOP}/python_pkgs/

requirements.txt

@@ -0,0 +1,4 @@
+jsonpickle
+matplotlib
+numpy
+pexpect

scripts/distro.sh

@@ -0,0 +1,13 @@
+#! /usr/bin/env bash
+# Infer Unix-like distribution
+#
+# TODO: This logic appears verbatim in a few places.
+# Deduplicate by having other scripts call this one.
+
+if type lsb_release >/dev/null 2>&1 ; then
+    distro=$(lsb_release -i -s)
+elif [ -e /etc/os-release ] ; then
+    distro=$(awk -F= '$1 == "ID" {print $2}' /etc/os-release)
+fi
+
+printf '%s\n' "$distro" | LC_ALL=C tr '[:upper:]' '[:lower:]'

scripts/package/.gitignore

@@ -0,0 +1,2 @@
+fpm
+output

scripts/package/after-install.sh

@@ -0,0 +1,15 @@
+#! /usr/bin/env bash
+# Called by system package manager to request on-site setup on host.
+
+set -euo pipefail
+
+pashd=/usr/lib/pash
+logd=/var/log/pash
+
+say_task() {
+    printf 'pash: %s (log: %s/%s)\n' "$1" "$logd" "$2"
+}
+
+mkdir -vp "$logd"
+cd "$pashd/compiler/parser"
+ln -fs "$pashd/pa.sh" /usr/bin/pa.sh

scripts/package/shell.sh

@@ -0,0 +1,33 @@
+#! /usr/bin/env bash
+# Launch REPL for building PaSh system packages.
+
+# We need FPM's source code to build a key Docker image.
+if [ ! -d fpm ]; then
+    git clone --depth 1 git@github.com:jordansissel/fpm.git
+fi
+
+# Make a Docker image that knows about building packages.
+if ! docker image inspect fpm 2>&1 >/dev/null; then
+    make -C fpm docker-release-everything
+fi
+
+here="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")"
+PASH_TOP=$(readlink -f "$here/../..")
+mkdir -p "$here/output"
+echo $here
+
+main() {
+    docker run \
+	   --entrypoint /bin/bash \
+	   --interactive \
+	   --tty \
+	   --rm \
+	   --user "$(id -u):$(id -g)" \
+	   --volume "$here/output:/out" \
+	   --volume "$here/tools:/tools" \
+	   --volume "$PASH_TOP:/src" \
+	   fpm --rcfile /tools/start "$@"
+}
+
+# Only enter if script was not sourced
+(return 2>/dev/null) || main "$@"

scripts/package/tools/build

@@ -0,0 +1,100 @@
+#! /usr/bin/env bash
+
+# Builds a specific package in the context of FPM's Docker container.
+# https://fpm.readthedocs.io/
+
+# Enforce preconditions
+set -ex
+version="$1"
+install_dir=/usr/lib/pash
+set +x -u; : "$version"
+shift # for remaining $@
+
+
+print_dotfpm() {
+    output_format="$1"
+    output_file="$2"
+
+    # I/O rules, with Docker container paths
+    flag --chdir /src
+    flag --input-type dir
+    flag --output-type "$output_format"
+    flag --package "$output_file"
+    flag --log info
+    flag --verbose
+    flag --force
+
+    # Discovery information
+    flag --name pash
+    flag --description "Parallelize POSIX shell scripts"
+    flag --license mit
+    flag --architecture all
+    flag --url 'https://github.com/binpash/pash'
+    flag --version "$version"
+
+
+    # System-level dependencies
+    flag --depends automake
+    flag --depends bash
+    flag --depends bc
+    flag --depends bsdmainutils
+    flag --depends curl
+    flag --depends git
+    flag --depends g++-10
+    flag --depends libffi-dev
+    flag --depends libtool
+    flag --depends locales
+    flag --depends locales-all
+    flag --depends m4
+    flag --depends netcat-openbsd
+    flag --depends pkg-config
+    flag --depends python # Redundant?
+    flag --depends python3
+    flag --depends python3-pip
+    flag --depends python3-setuptools
+    flag --depends python3-testresources
+    flag --depends wamerican-insane
+    flag --depends wget
+
+    # Handle requirements.txt with `python3 -m pip`
+    flag --python-bin /bin/python3
+    flag --python-obey-requirements-txt
+    flag --python-internal-pip
+
+    # Tell PaSh to set itself up on a target system.
+    flag --after-install "$PASH_TOP/scripts/package/after-install.sh"
+
+    # Map contents from here to target system.
+    flag --exclude '*.git*'
+    flag --exclude '*.pyc'
+    flag --exclude '__pycache__'
+    flag --exclude '*package/output*'
+    printf ".=%s\n" "$install_dir"
+}
+
+flag() {
+    if [ "$#" -gt 1 ]; then
+	printf "%s '%s'\n" "$1" "$2"
+    else
+	printf "%s\n" "$1"
+    fi	    
+}
+
+trap 'rm -f .fpm' EXIT
+
+for output_format in "$@"; do
+    output_file="/out/pash-${version}"
+
+    # Map formats to extensions, when they differ.
+    case "$output_format" in
+	*)
+	    output_file="${output_file}.${output_format}"
+	    ;;
+    esac
+
+    print_dotfpm "$output_format" "$output_file" > .fpm
+    set +e
+    /fpm/bin/fpm && chmod 440 "$output_file"
+    set -e
+    printf '\n'
+done