Skip to content
Yet Another Portable Library for Process Handling / Subshell Invokation
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
compat
src
t
.dir-locals.el
.gitignore
.travis.yml
README.org
ccl-fd0-failure
eazy-process.asd
eazy-process.test.asd
install-lfp.sh
simple-build-test.sh
testscr.lisp

README.org

  • Simplified API/implementation, achieved by REMOVING some features in run-program
  • Declarative process handling
  • Subprocesses spawned by POSIX fork&exec, no impl-specific interface
  • Memory / CPU-time management of the child processes
  • [-] Compatibility layer to the existing State-of-the-Art libraries
    • [X] trivial-shell
    • [ ] inferior-shell
    • [ ] sb-ext:run-program

Usage

(defvar pipe (pipe))
(defvar in "t/test-input")
(defvar out "t/test-output")

(defvar p1 (shell '("cat") `(,in  ,pipe))) ; fd 0:pathname, 1:pipe
(defvar p2 (shell '("cat") `(,pipe ,out))) ; fd 0:pipe, 1:pathname

(wait p1)
(wait p2)

Processes are async by default. Even if you forget waiting the process, it is handled by trivial-garbage — when a process object (here, p1 and p2) is GC-ed, it is automatically finalized – killed and waited.

To ensure the process is finalized, with-process might help you. It ensures the subprocess is terminated and no zombie remains. However, it does not synchronize with the subprocess by itself — synchronization should be done manually.

(with-process (p1 '("sleep" "3"))
  (print :x))
;; p1 is killed right after the execution of (print :x),
;; which means that the subprocess does not actually run for 3 seconds.

(with-process (p1 '("sleep" "3"))
  (print :x)
  (wait p1)) ;; synchronization. Waits for 3 sec

Note that if you leave too many processes alive, the total number of file descriptors might hit the system-provided limit (around 1000 by default).

Implicit Piping

If you do not specify pipes nor pathnames, a new pipe is created for each fd, and the other end of the pipe is accessible with (fd process fdnum). When nil is specified, it means /dev/null.

(let* ((in "t/test-input")
       (p1 (shell '("cat") `(,in :output))) ; fd 0,1 where 1 is an implicit pipe
       (p2 (shell '("cat") `(,(fd p1 1) :out :o :input :in :i nil)))) ; fd 0-6
  (wait p1)
  (wait p2))

Reading the output

The output of the subprocess is NOT directly accessible with EAZY-PROCESS. Instead, open the pathname returned by fd-as-pathname, which is /dev/fd/[fd].

(test read
  (let ((p1 (shell `("hostname"))))
    (with-open-file (s (fd-as-pathname p1 1))
      (is (string= (machine-instance)
                   (read-line s))))))

This has greatly simplified the API of eazy-process.

Resource management

Macro with-rlimit dynamically binds the current rlimit resource limitation. As noted in *rlimit-resources* docstring, this does not affect the lisp process itself. — It only affect the new child processes spawned by shell.

Example below shows the usecase where *spendtime* contains a path to a simple C program that busy-waits for 10 seconds. The execution is terminated in 3 seconds. TERMSIG is set to 24 because the program is killed by SIGXCPU.

(with-rlimit ((+rlimit-cpu-time+ 3)) ; 3 sec
  (let ((p (shell `(,(namestring *spendtime*))))) 
    (multiple-value-bind
           (exited exitstatus ifsignalled termsig ...)
           (wait p)
      (is-false exited)
      (is-false exitstatus)
      (is-true ifsignalled)
      (is (= 24 termsig)))))

This macro can be nested, and the new subprocess reflects the inntermost limitation.

(with-rlimit ((+rlimit-cpu-time+ 3))
  (shell ...) ; 3 sec
  (with-rlimit ((+rlimit-cpu-time+ 5)
                (+rlimit-as+ 500000))
    (shell ...))) ; 5 sec, 500 MB

Somewhat Longer Description

In `run-program` interface in the popular implementations, piping between subprocesses are hard. It requires either reading the entire output stream and packing the contents as a new string-input-stream, or using some other implementation-specific functions. Also, compatibility libraries e.g. trivial-shell or inferior-shell, often depend on these functions, implying the same problem.

Iolib also has `run-program` that allows easy piping, but it is restricted to 3 fds: `input,output,error`.

Eazy-process provides a clean, declarative and thin layer for the processes. It depends on the concept of “everything is a file” and do not provide interfaces to streams.

Tested Impl

This library is at least tested on implementation listed below:

  • SBCL 1.2.1 on X86-64 Linux 3.13.0-39-generic (author’s environment)
  • SBCL 1.1.14 on X86 Linux 3.13.0-44-generic (author’s environment)
  • CCL 1.10 on linux currently has a problem reading /dev/fd/[fd], which is actually a symlink to /proc/[pid]/fd/[fd], and the test does not pass. Do not use (fd-as-pathname process fd) and use temorary files instead.
  • ECL opens /dev/fd/[fd] correctly, but it fails to load CFFI…
  • ABCL has more problems than CCL. It fails to open /proc/[pid]/fd/[fd] and also have problems with CFFI.

Test reports on other OS’es/impls are greatly appreciated. Run ./simple-build-test.sh, assuming it already loads quicklisp in your init files.

Dependencies

It depends on the latest libfixposix available at https://github.com/sionescu/libfixposix .

Also, it depends on the following libraries:

  • iterate by Jonathan Amsterdam : Jonathan Amsterdam’s iterator/gatherer/accumulator facility
  • Alexandria by ** : Alexandria is a collection of portable public domain utilities.
  • cffi by James Bielman <jamesjb@jamesjb.com> : The Common Foreign Function Interface
  • optima by Tomohiro Matsuyama : Optimized Pattern Matching Library
  • iolib
  • trivial-garbage
  • cl-rlimit

Syntax

(defun shell (argv &optional
               (fdspecs '(:input :output :output))
               (environments nil env-p)
               (search t))
    ...)

When search is nil, it disables the pathname resolving using PATH.

Fdspecs

fdspecs := {fdspec}*
fdspec  := output | input | fd | path-or-pipe | openspec
output  := :output | :out | :o
input   := :input | :in | :i
fd      := <fixnum>
openspec := (path-or-pipe &key direction if-exists if-does-not-exist)
path-or-pipe := <pipe object> | <pathname>
direction := :input | :output | :io | :probe
if-exists := :overwrite | :supersede | :append | :error
if-does-not-exist := :create | :error
  • output form and input form implicitly create a new pipe.
  • The fixnum fd should be a value of function (fd process fdnum).
  • Openfilespec is almost identical to the argument list of OPEN in ANSI spec, however :rename, :rename-and-delete, :new-version are not supported and signals an error.
  • Function pipe generates a new pipe object that can be used in an fdspec.
  • If a <pipe object> or a <pathname> are given without options, it uses a default direction, which is :input for fd 0 and :output for fd 1 and fd 2. For fd > 2, missing direction signals an error.
  • Be careful when you open a fifo, the process will be blocked.

Environments

environments := {environment}*
environment  := env-pair | env-string
env-pair     := (name . value)
env-string   := "name=value"
name, value -- string

If we omit the second argument environments, the subprocess inherits the environment of the parent lisp process. unset -ting the environment value is not available.

Compatibility Layers

There are compatibility layers for trivial-shell and inferior-shell. For documentation, see compat/README.org .

run-program compatibility

Abandoned , since the design of run-program is not thoroughly delibelated, and any form of its descendants is not acceptable. It is a mistake to combine processes with streams in a tightly coupled manner.

Author & Copyright

Copyright (c) 2014 Masataro Asai (guicho2.71828@gmail.com)

You can’t perform that action at this time.