mkbigfifo

mkbigfifo is a program that supplements mkfifo by allowing to create named pipes that have by default the maximum allowed buffer size (typically increasing the size from 64 KiB to 1 MiB).

Introduction

mkfifo creates named pipes on the operating system. These pipes default to the default pipe size. On recent versions of the linux kernel this is 64K. When large volumes of data are processed and piped into another program the pipe size can be limiting.

For example program_a my_big_file | program_b -o result_file. If program_b momentarily stops input processing while it writes a chunk to the result file this will block program_a at the moment the pipe buffer is full. By using a bigger pipe buffer these moments where the pipe buffer is full will occur less frequently.

Why not mkfifo?

Unfortunately mkfifo can NOT be used to set the size in advance. The size is determined by the kernel and can only set by a program that has opened the fifo.

Solutions have been presented on stackoverflow but these require the use of fcntl kernel calls. A working example in python can be found here. Mkbigfifo was made to provide an easy interface to these solutions.

Usage

Since pipe size can only be modified by running programs mkbigfifo must run until all the programs using the named pipes are finished. Mkbigfifo will exit gracefully and remove the pipes after it receives a SIGINT (ctrl-c) or SIGTERM signal. It can be used in scripts like this:

mkbigfifo fifo1 fifo2 &
FIFO_PID=$!
program1 -o fifo1 -i /some/input &
program2 -o fifo2 -i fifo1 &
program3 -i fifo2 -o /some/output
kill $FIFO_PID

Alternatively you can create the pipes in a separate terminal window: mkbigfifo mypipe. And then terminate with ctrl-c when done.

CLI

usage: mkbigfifo [-h] [-s SIZE] [-v] [-q] FIFO [FIFO ...]

positional arguments:
  FIFO                  One or multiple named pipes to create.

optional arguments:
  -h, --help            show this help message and exit
  -s SIZE, --size SIZE  Size in bytes for the fifo files. Default is the
                        maximum user-allowed pipe size on the system. On this
                        system that is 1048576 bytes.
  -v, --verbose         Increase verbosity. Multiple flags allowed.
  -q, --quiet           Decrease verbosity. Multiple flags allowed.

Acknowledgements

Thanks to Stéphane Chazelas for outlining how to increase the size of a named pipe.

Huge thanks to golinuxhub for creating the python example. This was the basis of this program.

Thanks to Mayank Jaiswal for providing an example how to process signals in python. This example was used to properly handle SIGINT and SIGTERM signals in this program.