>(Process substitution) b0rken when combined with redirection

[McDutchie]

Process substitution of the form >(list) is utterly broken:

echo ok > >(sed s/ok/good/)

Expected output: good (as on bash and zsh)

Actual output:

• either none, with a file with a bizarre binary file name created
  in the current working directory that contains ok plus a newline
  (which is what happens on Linux and Solaris);
• or (depending on the file system and/or OS and/or locale) an
  error message indicating the refusal to create that file:
  ksh: ?: cannot create [Illegal byte sequence]
  (which is what happens on my Mac).

Before I spend hours searching for the cause, does anyone have any idea where to begin debugging this?

[JohnoKing]

I did some quick tests and running that command will not produce a strange file in ksh93v- if run on Linux, although ksh93v- on FreeBSD still does cause an odd file to be created. Both ksh93u+ and ksh2020 will produce a file. All versions since ksh93u+ will not produce output on Linux or FreeBSD.

[JohnoKing]

Update: Apparently ksh93v- is still creating a file, but the file name was not showing up when I used ls or echo *. Using Dolphin (the KDE Plasma 5 file manager) shows a file without a name.

[McDutchie]

It only breaks when you combine it with an output redirection. When used without that, e.g. when passing the process substitution as a filename to tee, it works correctly:

$ ksh -c 'echo hi | tee >(cat)'
hi
hi

(output is as on bash and zsh)

[JohnoKing]

I found the reason why process substitution doesn't work with redirections. The code for handling process substitution with redirections is inside of an if statement that assumes IORAW is not set:

ksh/src/cmd/ksh93/sh/io.c

Line 1175 in de2b4a6

if(!(iof&IORAW))

ksh/src/cmd/ksh93/sh/io.c

Lines 1185 to 1196 in de2b4a6

	else if(iof&IOPROCSUB)
	{
	struct argnod *ap = (struct argnod*)stakalloc(ARGVAL+strlen(iop->ioname));
	memset(ap, 0, ARGVAL);
	if(iof&IOPUT)
	ap->argflag = ARG_RAW;
	else if(shp->subshell)
	sh_subtmpfile(shp);
	ap->argchn.ap = (struct argnod*)fname;
	ap = sh_argprocsub(shp,ap);
	fname = ap->argval;
	}

This is never run for redirections as IOPROCSUB usually causes IORAW to be set. This is because process_sub sets the bit for ARG_RAW when the given token is '>' (i.e. OPROCSYM):

ksh/src/cmd/ksh93/sh/parse.c

Line 1386 in de2b4a6

int mode = (tok==OPROCSYM);

ksh/src/cmd/ksh93/sh/parse.c

Line 1391 in de2b4a6

argp->argflag = (ARG_EXP|mode);

ksh/src/cmd/ksh93/sh/parse.c

Lines 1743 to 1747 in de2b4a6

	else if(((token==IPROCSYM && !(iof&IOPUT)) || (token==OPROCSYM && (iof&IOPUT))) && !(iof&(IOLSEEK|IOREWRITE|IOMOV|IODOC)))
	{
	lexp->arg = process_sub(lexp,token);
	iof |= IOPROCSUB;
	}

ksh/src/cmd/ksh93/sh/parse.c

Lines 1781 to 1782 in de2b4a6

	if(lexp->arg->argflag&ARG_RAW)
	iof |= IORAW;

The IOPROCSUB section in io.c can be moved out of the IORAW if statement as a workaround for this, although a background job will spawn ksh will print the process ID of the asynchronous process:

$ echo ok > >(sed s/ok/good/)
[1] 19032
$ good

[McDutchie]

That's some great sleuthing! Process substitution is actually documented to run the process asynchronously (meaning: as a background process), so that bit is fine – it's just that those processes should be exempt from job control, so the [1] 19032 should not have been printed.

[McDutchie]

After moving the IOPROCSUB section as you described above, it now appears to work correctly on non-interactive shells, e.g.:

$ ksh -c 'echo ok > >(sed s/ok/good/); echo $!; wait'
79626
good

The wait is for the parent shell to wait for the asynchronous process to be finished before exiting. $! is set to its PID, as expected.

Unfortunately, moving the block introduces a memory fault somewhere in the io.sh regression tests, which I can't localise right now as it's 3am here. Thanks and goodnight.

[JohnoKing]

Ksh prints the process ID of the asynchronous process when process substitution is used in general (in interactive shells), which suggests it is a separate bug:

$ echo ok >(/bin/true)
[1]	11717
ok /dev/fd/5

This doesn't happen in Bash or zsh.