find a way to make `psub --fifo` safe from deadlock

[geoff-nixon]

There are actually a couple of bugs here.

The easy one is here — use_fifo is missing a sigil and is therefore a string comparison, causing psub to always act as psub -f.

Unfortunately, it's not as simple as fixing that typo as doing so will cause a non-interruptable hang under certain circumstances. I believe it occurs when the pipe buffer is exceeded? But I'm not sure how to actually determine the pipe buffer in fish. Maybe forking another process is needed? Or... something.

Anyway, here's a (hopefully) cross-platform test case (about 1.5MiB and requires Java, uses openssl to decode base64). It's a standalone wrapper for rhino. There's a big hunk of base64 in the middle of it, but the script is just:
java -jar (echo 'BIGHUNKOFBASE64' | openssl base64 -d | psub)

[ridiculousfish]

Nice diagnosis and test case!

[zanchey]

I've pushed a fix to a topic branch (19217f3 on psub_fix), but I don't want to merge to master until the buffering issue is worked out.

[faho]

Any progress on this?

[geoff-nixon]

@faho Yes. And no. But, yes.

But before we get to the good stuff, lets review:

1. Disregard everything in the psub man page.
   Virtually unchanged from when it was written 10 years ago for fish 1.0.0, literally (and by literally, I mean literally) every sentence in man psub is wrong (except for the example). With that out of the way...
2. What is 'process substitution'?
   Process substitution (which itself is something of a misnomer) was a concept introduced in the Korn shell (ksh88) as a shorthand for the often messy process of using file descriptors and named pipes with a command or program when the program expects a file as an argument. In other words, it is a way to emulate including these programs in a UNIX pipeline, where one otherwise could not, either simply by design (man dd), or for some other clever reason (man tee). The syntax used by shells that support this is:

command <(process) ...

The standard output of process is fed into a file descriptor or named pipe, which is passed as an argument to command.

and

process >(command) ...

The standard input of command is read from whatever process is doing with that argument (ideally, producing output).

This syntax is by no means standard, and it certainly is not POSIX. It is supported by ksh88 and ksh93, but not pdksh or mksh; bash supports it, but not in sh mode (bash --posix). It is supported in zsh when not in an emulation mode that proscribes it; and zsh also has another syntax, =(process), which we'll get to in a bit.

The fish psub function:

command (process | psub)

(purportedly) capitalizes on the fact that no special syntax is needed to perform "process substitution"; the ordinary syntax of command substitution (command or $(command) in Bourne shells, (command) is fish) can be used to accomplish the same thing pipeline within the command substitution whose standard output culminates in a file descriptor or named pipe.

1. So, why has this bug stayed in place since it was introduced seven years ago (c6ebb23), two years since I opened this issue, followed by about a dozen others?
   This is where it gets tricky, and there are two factors at play.

   a. Many utilities which take files as arguments (i.e., for input or output), do so for a reason. A buffered UNIX pipeline often simply is not acceptable ... blah blah blah I've been up all night hacking on this so I'll finish my blustering treatise later. 💀

Basically, I propose we use a ramdisk. It combines to the "durability", ability to handle large files and non-streamable data of "file substitution" (?) (zsh =(command), what we're doing with psub at present) with the speed and ephemeral nature of using a fd or fifo.

@ridiculousfish @zanchey @anyone @anyone @bueller... Initial thoughts?

function psub --description "Process substitution, revisited."

    set -l filename
    set -l funcname

    set -l halfmem
    set -l sectors
    set -l ramdisk
    set -l mountpoint
    set -l psubdir

    set -l use_file 0

    while set -q argv[1]
        switch $argv[1]
            case -h --help
                __fish_print_help psub
                return 0

            case -f --file
                set use_file 1
        end
        set -e argv[1]
    end

    if not status --is-command-substitution
        echo psub: Not inside of command substitution >&2
        return 1
    end

    if test -z "$TMPDIR"
        set TMPDIR /tmp
    end

    # Implemention for other systems is left as an exercise for the reader.
    if test (uname) != Darwin
        set use_file 1                            # ... mount -t tmpfs ...
    end

    if test $use_file -eq 1
        while not set psubdir (mktemp -d $TMPDIR/.psub.XXXXXXXXXX); end
        chmod 0300 $psubdir

        while not set filename (mktemp $psubdir/temp.XXXXXXXXXX); end
        chmod 0100 $psubdir
        chmod 0200 $filename
    else
        # Basically: detect memory and use 1/2 of it (the default with tmpfs
        # on other platforms) as a ramdisk. The memory is allocated as needed.
        # `hdid -nomount ram://SECTORS` (of 512 bytes) on Darwin.

        set halfmem (math (sysctl -n hw.memsize) / 2)
        set sectors (math $halfmem / 512)
        set ramdisk (hdid -nomount ram://$sectors | tr -d [:space:])
        chmod 0600 $ramdisk

        # $ramdisk is now something like '/dev/disk2'; it would be nice if we
        # could just use the raw device file as $filename, but if we do that
        # there's no EOF. So we format, mount, and use a tempfile on our
        # ramdisk. UDF or is probably as good as anything. We probably just
        # don't want any filesystem that's journaled to reduce overhead.


        while not set mountpoint (mktemp -d /$TMPDIR/.psub.XXXXXXXXXX); end
        chmod 0300 $mountpoint
        newfs_udf $ramdisk >/dev/null 2>&1
        mount_udf -o nobrowse $ramdisk $mountpoint

        while not set psubdir (mktemp -d $mountpoint/.psub.XXXXXXXXXX); end
        chmod 0300 $psubdir        

        while not set filename (mktemp $psubdir/temp.XXXXXXXXXX); end
        chmod 0100 $mountpoint
        chmod 0100 $psubdir
        chmod 0200 $filename
    end

    # Write stdin to tempfile
    cat > $filename
    chmod 0400 $filename


    # Write filename to stdout
    echo $filename

    # Find unique function name
    while true
        set funcname __fish_psub_(random)
        if not functions $funcname >/dev/null 2>&1
            break
        end
    end

    # Make sure we unmount and detatch when caller exits.
    function $funcname --on-job-exit caller --inherit-variable filename --inherit-variable funcname --inherit-variable use_file --inherit-variable ramdisk --inherit-variable mountpoint --inherit-variable psubdir
        chmod 0700 $psubdir $filename
        if test $use_file -eq 0
            umount -f $mountpoint >/dev/null 2>&1
            which hdiutil >/dev/null 2>&1
            and hdiutil detach $ramdisk >/dev/null 2>&1
            chmod 0700 $mountpoint
        end
        command rm -rf $mountpoint $psubdir $filename
        functions -e $funcname
    end
end