TUTORIAL.org

Unix in Lisp shell tutorial

The missing semester

Shell Tools and Scripting

mcd () {
    mkdir -p "$1"
    cd "$1"
}

(defmacro mcd (dir)
  `(fg
     (mkdir -p ,dir)
     (cd ,dir)))

Data Wrangling

To get the data, in POSIX shell they write:

ssh myserver journalctl

In Unix in Lisp, we write something very similar:

(def journal (ssh myserver journalctl))

Note that we conveniently store the result into journal so we can use it later. In POSIX shell they may use a temporary file, and they may be relunctant to do so to avoid littering in the file system. In any case, if we also want to store it in a file, just

(defile journal.log (ssh myserver journalctl))

In POSIX shell, they do the following to narrow down the lines.

ssh myserver journalctl | grep sshd

In Unix in Lisp, we may do something similar

(pipe journal (grep sshd))

We may do the same thing in a way that looks like more “common” Lisp:

(filter (lambda (line) (ppcre:scan "sshd" line)) journal)

In POSIX shell, they do the following to narrow down more:

ssh myserver journalctl
 | grep sshd
 | grep "Disconnected from"

We can translate it “literally” into Unix in Lisp:

(pipe journal (grep sshd) (grep "Disconnected from"))

Or do it the more “common” way:

(filter
 (lambda (line)
   (and (ppcre:scan "sshd" line)
        (ppcre:scan "Disconnected from" line)))
 journal)

Now, POSIX shell people write their intermediate result into a temporary file:

$ ssh myserver 'journalctl | grep sshd | grep "Disconnected from"' > ssh.log
$ less ssh.log

We have our intermediate result ready for use in our REPL via *,**,*** variables. We can also store them into variables via def and alike. If we really want to store it persistently, just

(defile ssh.log ...)

POSIX shell people now use sed magic to pick out user names.

ssh myserver journalctl
 | grep sshd
 | grep "Disconnected from"
 | sed -E 's/.*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \[preauth\])?$/\2/'

We still use regex magic here (using cl-ppcre), but no need for sed line editing witchery. We collect the matches for the user field into a list:

(def users
  (collecting
   (do-each (line (ssh myserver journalctl))
     (ppcre:register-groups-bind (_ user)
         (".*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \\[preauth\\])?$"
          line)
       (when user (collect user))))))

Now, POSIX shell people use sort and uniq to count time of occurrences:

ssh myserver journalctl
 | grep sshd
 | grep "Disconnected from"
 | sed -E 's/.*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \[preauth\])?$/\2/'
 | sort | uniq -c

We may do something just as insane as the above:

(pipe users (sort) (uniq -c))

The output is a list of table rows, each of them is a string separated by white spaces. Rather than meddling with these unstructured text, I recommend using structured data consisting of real lists and numbers.

(def fail-users-alist (hash-table-alist (frequencies users)))

In POSIX shell, they use some more cryptic text-manipulating spells to sort the table and display the 10 users with most occurrences:

ssh myserver journalctl
 | grep sshd
 | grep "Disconnected from"
 | sed -E 's/.*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \[preauth\])?$/\2/'
 | sort | uniq -c
 | sort -nk1,1 | tail -n10

In Lisp, we do it the structured data way:

(def sus-users (subseq (sort fail-users-alist #'> :key #'cdr) 0 10))

In POSIX shell, awk and paste to construct a comma separated list:

ssh myserver journalctl
 | grep sshd
 | grep "Disconnected from"
 | sed -E 's/.*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \[preauth\])?$/\2/'
 | sort | uniq -c
 | sort -nk1,1 | tail -n10
 | awk '{print $2}' | paste -sd,

In Lisp, we:

(string-join (mapcar #'car sus-users) ",")

Analyzing data

Now the POSIX crowd spirals deeper into insanity, constructing math expressions using string concatenation and passing to bs:

... | paste -sd+ | bc -l

In Lisp, we simply:

(reduce #'+ (mapcar #'cdr fail-users-alist))

Data wrangling to make arguments

The POSIX shell way:

rustup toolchain list | grep nightly | grep -vE "nightly-x86" | sed 's/-x86.*//' | xargs rustup toolchain uninstall

The Lisp way:

(def unused-crabs
  (mapcan
   (lambda (l)
     (when (and (ppcre:scan "nightly" l)
                (not (ppcre:scan "nightly-x86" l)))
       (list (ppcre:regex-replace "-x86.*" l ""))))
   (rustup toolchain list)))
(rustup toolchain uninstall ,@unused-crabs)