Metabash

Metabash (or M#! for short) is a GNU Guile module that provides domain specific language (DSL) for running distributed shell pipelines. “What are distributed shell pipelines?” you may ask. Well, the idea is that we can implement some sort of Unix pipeline that is spanned across multiple hosts over a network.

Using such pipeline you can run a command on one host and then pass the output to a command that is run on another.

Metabash is a research project of mine that was born as a Guile-SSH spinoff.

License

Metabash is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. Please see COPYING file for the terms of GNU General Public License.

The logo (doc/logo.svg and rasterised versions) is distributed under terms of Creative Commons Attribution-ShareAlike 4.0 International license.

Requirements

• GNU Guile 2.2+
• Guile-SSH 0.13.0+

Installation

$ autoreconf -vif
$ ./configure
$ make install

Main concepts

Process

Running local or remote process. Each process runs a specified command and has an input and an output port.

A process is described by <process> class from the (metabash core process) module.

Pipe

Pipe connects two running processes by means of connecting output port (stdout) of the first process with the input port (stdin) of the second.

For the clarity sake we will assume that the process that produces the data is always placed on the “left” side of the pipe and all the processes that read the data is always placed on the “right” side of the pipe. With that assumption in mind we can talk about “left” and “right” processes in relation to a pipe.

A pipe is described by <pipe> class from the (metabash core plumbing pipe) module.

Plumber

Plumber builds a pipeline upon the processes and pipes that are described above. It implemented as plumb command and M#! macro.

When a plumber is called it spawns a process for each specified command and connects the spawned processes with pipes. The output of the plumber is a pipeline.

Pipeline

Pipeline describes a sequence of running processes connected by pipes. Each pipeline has an input port and output port.

A pipeline is described by <pipeline> class from the (metabash core pipeline) module.

Examples

(use-modules (ssh session)
             (ssh auth)
             (oop goops)
             (metabash core plumbing pipe)
             (metabash core pipeline)
             (metabash core process)
             (metabash plumber))

;; Create a Guile-SSH session:
(define session (make-session #:host "example.org"))

;; Connect to the remote host:
(connect! session)

;; Authenticate with the host using the SSH Agent:
(userauth-agent! session)

(let* ((pipeline (plumb
                   ;; This commands runs on a remote host
                   `(remote ,session "cat /etc/passwd")
                   ;; This commands runs on the local host
                   `(local "sort")
                   ;; This command runs on the local host as well,
                   ;; but this time it's a Guile process
                   `(guile #f
                            (begin
                              (use-modules (ice-9 rdelim))
                              (let loop ((line (read-line)))
                                (if (eof-object? line)
                                  (exit)
                                  (begin
                                    (write-line (string-join (string-split line #\:) ","))
                                    (loop (read-line)))))))))
       ;; Plumber will connect output of the 1st (remote) command
       ;; with the input of the 2nd (local) command with a pipe.
       ;; The list of commands passed to the plumber can be arbitrary
       ;; long.
       ;;
       ;; To read data from the pipeline, we connect the pipeline output
       ;; with the current output port:
       (pipe     (make <pipe> 
                       #:input-port (pipeline-output-port pipeline)
                       #:output-port (current-output-port))))
  (while #t
    (sleep 1)))

The plumbing can be done with M#! macro which adds some syntactic sugar upon the plumb command:

(use-modules (oop goops)
             (ssh session)
             (ssh auth)
             (metabash core plumbing pipe)
             (metabash core pipeline)
             (metabash plumber))

(define (main args)
  (define session (make-session #:host "example.org"))
  (connect! session)
  (userauth-agent! session)
  (let ((pipeline (M#! remote session "cat /etc/passwd"
                       => local "sort"
                       => local "grep avp")))

    (let ((pipe (make <pipe>
                  #:input-port    (pipeline-output-port pipeline)
                  #:output-port   (current-output-port)
                  #:on-disconnect (lambda (pipe)
                                    (pipe-close! pipe)
                                    (disconnect! session)))))
      (pipe-connect! pipe)
      (while (pipe-connected? pipe)
        (sleep 1)))))