runtime processing of stderr from a command

[tgianko]

Hi,

I need to run an external command and at the same time to process the output as it is generated.

In this setting, the external command is the producer (with async=True and capture_stderr=True), and my program is the consumer. The stderr should be consumed by a function in a thread that pulls from the shared resource until the command finishes (is_finished=True).

In my first attempt, I passed a StringIO to ExternalCommand.stderr_file. My idea was to keep everything in memory, without external files. This does not work as ExternalCommand tries to get the fileno from the file object. StringIO does not support that... so exception. I looked at the code of CachedStream and I couldn't see an obvious reason why to retrive the fileno and file name from the file object. Wouldn't be better to keep a local copy of the file object and write on it?

In my second attempt, I thought to use a FIFO file. However, here there is another shortcoming. The built-in function open returns such a file object only when the producer opens the same FIFO for writing operations. Until then, the open blocks the consumer. This prevents me to pass the file object to ExternalCommand.stderr_file. An alternative could be os.open that allows to open with the flag os.O_NONBLOCK, but this won't work as the returned descriptor is an int.

Do you have any idea on how to go around that? I would like to avoid to use regular files and actually I'd prefer to keep everything in memory. StringIO is actually my favorite one, but FIFO are also fine.

[xolox]

Hi Giancarlo and thanks for the feedback. I've just released executor 18.1 which adds the buffered property that can be used to disable the buffering of output to temporary files. The documentation contains an example that reads the output of an external command in realtime. I hope this helps!