Support implementing the jobserver using named pipes

Using anonymous pipes for jobserver support has some advantages:
for example there is nothing on disk that needs to be cleaned up.
However it has many obscure problems, related to the fact that in
order for it to work we need to ensure these resources are properly
passed through to child processes that want to use the jobserver.
At the same time we don't want to pass the pipe to process which
DON'T know about the jobserver.

Other processes can open file descriptors which we then think are
our jobserver, but aren't.  And, we open the pipe file descriptors
in blocking mode which doesn't work for all users.

See issues such as SV 57178, SV 57242, and SV 62397

To avoid these issues, use named pipes (on systems where they are
available) instead of anonoymous pipes.  This simplifies many things:
we never need to pass open file descriptors to our children; they
can open the jobserver named pipe.  We don't need to worry about
recursive vs. non-recursive children.  Users don't have to "pass
through" the resources if they are invoking sub-makes.  Each child
can open its own file descriptor and set blocking as needed.

The downside is the named pipe exists on disk and so must be cleaned
up when the "top-level" make instance exits.

In order to allow make to continue to be used in build systems where
older versions of GNU make, or other tools that want to use the
jobserver, but don't understand named pipes, introduce a new option
--jobserver-style that allows the user to choose anonymous pipes.

* NEWS: Announce the change and the --jobserver-style option.
* doc/make.1: Add --jobserver-style documentation.
* doc/make.texi (Special Variables): Add missing items to .FEATURES.
(Options Summary): Add --jobserver-style.
(POSIX Jobserver): Named pipes, changes to --jobserver-auth, and the
--jobserver-style option.
(Windows Jobserver): Document --jobserver-style for Windows.
* configure.ac: Check for mkfifo.
* src/config.h-vms.template: Undefined HAVE_MKFIFO.
* src/config.h.W32.template: Ditto.
* src/main.c: Add jobserver-style as a new command line option.
(main): Add jobserver-fifo to .FEATURES if supported.  Pass the style
option to jobserver_setup().
* src/os.h (jobserver_setup): Accept a style string option.
* src/posixos.c (enum js_type): Enumeration of the jobserver style.
(js_type): Which style we are currently using.
(fifo_name): The path to the named pipe (if in use).
(jobserver_setup): If no style is given, or "fifo" is given, set up a
named pipe: get a temporary file and use mkfifo() on it, then open it
for reading and writing.  If something fails fall back to anonymous
pipes.
(jobserver_parse_auth): Parse jobserver-auth to determine the style.
If we are using a named pipe, open it.  If we're using anonymous pipes
ensure they're valid as before.
(jobserver_get_invalid_auth): Don't invalidate the jobserver when
using named pipes.
(jobserver_clear): Clean up memory used for named pipes.
(jobserver_acquire_all): Unlink the named pipe when done.
* src/w32/w32os.c (jobserver_setup): Check the style argument.
* tests/scripts/features/jobserver: Use --jobserver-style to test
the anonymous pipe behavior, and also test named pipe/semaphore
behavior.  Check invalid jobserver-style options.
* tests/scripts/functions/shell: Use --jobserver-style to test the
anonymous pipe behavior, and also test named pipe/semaphore
behavior.

NEWS

@@ -74,6 +74,19 @@ https://sv.gnu.org/bugs/index.php?group=make&report_id=111&fix_release_id=109&se
   described in the makefile.
   Implementation provided by Sergei Trofimovich <siarheit@google.com>
 
+* New feature: The --jobserver-style command line option and named pipes
+  A new jobserver method is used on systems where mkfifo(3) is supported.
+  This solves a number of obscure issues related to using the jobserver
+  and recursive invocations of GNU make.  This change means that sub-makes
+  will connect to the jobserver even if they are not marked as recursive.
+  It also means that other tools that want to participate in the jobserver
+  will need to be enhanced as described in the GNU make manual.
+  You can force GNU make to use the simple pipe-based jobserver (perhaps if
+  you are integrating with other tools or older versions of GNU make) by
+  adding the '--jobserver-style=pipe' option to the command line of the
+  top-level invocation of GNU make, or via MAKEFLAGS or GNUMAKEFLAGS.
+  To detect this change search for 'jobserver-fifo' in the .FEATURES variable.
+
 * GNU make has sometimes chosen unexpected, and sub-optimal, chains of
   implicit rules due to the definition of "ought to exist" in the implicit
   rule search algorithm, which considered any prerequisite mentioned in the

configure.ac

@@ -133,7 +133,7 @@ AS_IF([test "$ac_cv_func_gettimeofday" = yes],
 AC_CHECK_FUNCS([strtoll strdup strndup stpcpy memrchr mempcpy umask mkstemp \
                 mktemp fdopen dup dup2 getcwd realpath sigsetmask sigaction \
                 getgroups seteuid setegid setlinebuf setreuid setregid \
-                getrlimit setrlimit setvbuf pipe strerror strsignal \
+                mkfifo getrlimit setrlimit setvbuf pipe strerror strsignal \
                 lstat readlink atexit isatty ttyname pselect posix_spawn \
                 posix_spawnattr_setsigmask])
 

doc/make.1

@@ -205,6 +205,16 @@ option is given without an argument,
 .BR make
 will not limit the number of jobs that can run simultaneously.
 .TP 0.5i
+\fB\--jobserver-style=\fR\fIstyle\fR
+The style of jobserver to use.  The
+.I style
+may be one of
+.BR fifo ,
+.BR pipe ,
+or
+.B sem
+(Windows only).
+.TP 0.5i
 \fB\-k\fR, \fB\-\-keep\-going\fR
 Continue as much as possible after an error.
 While the target that failed, and those that depend on it, cannot

doc/make.texi

@@ -6800,10 +6800,33 @@ Supports the @code{-L} (@code{--check-symlink-times}) flag.
 Supports ``else if'' non-nested conditionals.  @xref{Conditional
 Syntax, ,Syntax of Conditionals}.
 
+@item extra-prereqs
+Supports the @code{.EXTRA_PREREQS} special target.
+
+@item grouped-target
+Supports grouped target syntax for explicit rules.  @xref{Multiple Targets,
+,Multiple Targets in a Rule}.
+
+@item guile
+Has GNU Guile available as an embedded extension language.
+@xref{Guile Integration, ,GNU Guile Integration}.
+
 @item jobserver
 Supports ``job server'' enhanced parallel builds.  @xref{Parallel,
 ,Parallel Execution}.
 
+@item jobserver-fifo
+Supports ``job server'' enhanced parallel builds using named pipes.
+@xref{Integrating make, ,Integrating GNU @code{make}}.
+
+@item load
+Supports dynamically loadable objects for creating custom extensions.
+@xref{Loading Objects, ,Loading Dynamic Objects}.
+
+@item notintermediate
+Supports the @code{.NOTINTERMEDIATE} special target.
+@xref{Integrating make, ,Integrating GNU @code{make}}.
+
 @item oneshell
 Supports the @code{.ONESHELL} special target.  @xref{One Shell, ,Using
 One Shell}.
@@ -6812,9 +6835,16 @@ One Shell}.
 Supports order-only prerequisites.  @xref{Prerequisite Types, ,Types
 of Prerequisites}.
 
+@item output-sync
+Supports the @code{--output-sync} command line option.  @xref{Options Summary,
+,Summary of Options}.
+
 @item second-expansion
 Supports secondary expansion of prerequisite lists.
 
+@item shell-export
+Supports exporting @code{make} variables to @code{shell} functions.
+
 @item shortest-stem
 Uses the ``shortest stem'' method of choosing which pattern, of
 multiple applicable options, will be used.  @xref{Pattern Match, ,How
@@ -6826,14 +6856,6 @@ Supports target-specific and pattern-specific variable assignments.
 
 @item undefine
 Supports the @code{undefine} directive.  @xref{Undefine Directive}.
-
-@item guile
-Has GNU Guile available as an embedded extension language.
-@xref{Guile Integration, ,GNU Guile Integration}.
-
-@item load
-Supports dynamically loadable objects for creating custom extensions.
-@xref{Loading Objects, ,Loading Dynamic Objects}.
 @end table
 
 @vindex .INCLUDE_DIRS @r{(list of include directories)}
@@ -9286,6 +9308,15 @@ If there is more than one @samp{-j} option, the last one is effective.
 @xref{Parallel, ,Parallel Execution}, for more information on how
 recipes are run.  Note that this option is ignored on MS-DOS.
 
+@item --jobserver-style=[@var{style}]
+@cindex @code{--jobserver-style}
+Chooses the style of jobserver to use.  This option only has effect if
+parallel builds are enabled (@pxref{Parallel, ,Parallel Execution}).  On POSIX
+systems @var{style} can be one of @code{fifo} (the default) or @code{pipe}.
+On Windows the only acceptable @var{style} is @code{sem} (the default).  This
+option is useful if you need to use an older versions of GNU @code{make}, or a
+different tool that requires a specific jobserver style.
+
 @item -k
 @cindex @code{-k}
 @itemx --keep-going
@@ -12025,20 +12056,20 @@ number of active jobs across recursive invocations.  The actual
 implementation of the jobserver varies across different operating
 systems, but some fundamental aspects are always true.
 
-First, only command lines that @code{make} understands to be recursive
-invocations of @code{make} (@pxref{MAKE Variable, ,How the @code{MAKE}
-Variable Works}) will have access to the jobserver.  When writing
-makefiles you must be sure to mark the command as recursive (most
-commonly by prefixing the command line with the @code{+} indicator
-(@pxref{Recursion, ,Recursive Use of @code{make}}).
+@cindex @code{--jobserver-auth}
+First, @code{make} will provide information necessary for accessing the
+jobserver through the environment to its children, in the @code{MAKEFLAGS}
+environment variable.  Tools which want to participate in the jobserver
+protocol will need to parse this environment variable and find the word
+starting with @code{--jobserver-auth=}.  The value of this option will
+describe how to communicate with the jobserver.  The interpretation of this
+value is described in the sections below.
 
-Second, @code{make} will provide information necessary for accessing
-the jobserver through the environment to its children, in the
-@code{MAKEFLAGS} environment variable.  Tools which want to
-participate in the jobserver protocol will need to parse this
-environment variable, as described in subsequent sections.
+Be aware that the @code{MAKEFLAGS} variable may contain multiple instances of
+the @code{--jobserver-auth=} option.  Only the @emph{last} instance is
+relevant.
 
-Third, every command @code{make} starts has one implicit job slot
+Second, every command @code{make} starts has one implicit job slot
 reserved for it before it starts.  Any tool which wants to participate
 in the jobserver protocol should assume it can always run one job
 without having to contact the jobserver at all.
@@ -12075,54 +12106,75 @@ the jobserver.
 @subsection POSIX Jobserver Interaction
 @cindex jobserver on POSIX
 
-On POSIX systems the jobserver is implemented as a simple UNIX pipe.
-The pipe will be pre-loaded with one single-character token for each
-available job.  To obtain an extra slot you must read a single
-character from the jobserver pipe; to release a slot you must write a
-single character back into the jobserver pipe.  Note that the read
-side of the jobserver pipe is set to ``blocking'' mode.
-
-To access the pipe you must parse the @code{MAKEFLAGS} variable and
-look for the argument string @code{--jobserver-auth=R,W} where
-@samp{R} and @samp{W} are non-negative integers representing file
-descriptors: @samp{R} is the read file descriptor and @samp{W} is the
-write file descriptor.
-
-It's important that when you release the job slot, you write back the
-same character you read from the pipe for that slot.  Don't assume
-that all tokens are the same character; different characters may have
-different meanings to GNU @code{make}.  The order is not important,
-since @code{make} has no idea in what order jobs will complete anyway.
+On POSIX systems the jobserver is implemented in one of two ways: on systems
+that support it, GNU @code{make} will create a named pipe and use that for the
+jobserver.  In this case the auth option will have the form
+@code{--jobserver-auth=fifo:PATH} where @samp{PATH} is the pathname of the
+named pipe.  To access the jobserver you should open the named pipe path and
+read/write to it as described below.
+
+@cindex @code{--jobserver-style}
+If the system doesn't support named pipes, or if the user provided the
+@code{--jobserver-style} option and specified @samp{pipe}, then the jobserver
+will be implemented as a simple UNIX pipe.  In this case the auth option will
+have the form @code{--jobserver-auth=R,W} where @samp{R} and @samp{W} are
+non-negative integers representing file descriptors: @samp{R} is the read file
+descriptor and @samp{W} is the write file descriptor.  If either or both of
+these file descriptors are negative, it means the jobserver is disabled for
+this process.
+
+When using a simple pipe, only command lines that @code{make} understands to
+be recursive invocations of @code{make} (@pxref{MAKE Variable, ,How the
+@code{MAKE} Variable Works}) will have access to the jobserver.  When writing
+makefiles you must be sure to mark the command as recursive (most commonly by
+prefixing the command line with the @code{+} indicator (@pxref{Recursion,
+,Recursive Use of @code{make}}).  Note that the read side of the jobserver
+pipe is set to ``blocking'' mode.  This should not be changed.
+
+In both implementations of the jobserver, the pipe will be pre-loaded with one
+single-character token for each available job.  To obtain an extra slot you
+must read a single character from the jobserver; to release a slot you must
+write a single character back into the jobserver.
+
+It's important that when you release the job slot, you write back the same
+character you read.  Don't assume that all tokens are the same character;
+different characters may have different meanings to GNU @code{make}.  The
+order is not important, since @code{make} has no idea in what order jobs will
+complete anyway.
 
 There are various error conditions you must consider to ensure your
 implementation is robust:
 
 @itemize @bullet
 @item
-Usually you will have a command-line argument controlling the parallel
-operation of your tool.  Consider whether your tool should detect
-situations where both the jobserver and the command-line argument are
-specified, and how it should react.
+If you have a command-line argument controlling the parallel operation of your
+tool, consider whether your tool should detect situations where both the
+jobserver and the command-line argument are specified, and how it should
+react.
 
 @item
-If your tool determines that the @code{--jobserver-auth} option is
-available in @code{MAKEFLAGS} but that the file descriptors specified
-are closed, this means that the calling @code{make} process did not
-think that your tool was a recursive @code{make} invocation (e.g., the
-command line was not prefixed with a @code{+} character).  You should
-notify your users of this situation.
+If your tool does not recognize the format of the @code{--jobserver-auth}
+string, it should assume the jobserver is using a different style and it
+cannot connect.
 
 @item
-Your tool should also examine the first word of the @code{MAKEFLAGS}
-variable and look for the character @code{n}.  If this character is
-present then @code{make} was invoked with the @samp{-n} option and
-your tool should stop without performing any operations.
+If your tool determines that the @code{--jobserver-auth} option references a
+simple pipe but that the file descriptors specified are closed, this means
+that the calling @code{make} process did not think that your tool was a
+recursive @code{make} invocation (e.g., the command line was not prefixed with
+a @code{+} character).  You should notify your users of this situation.
 
 @item
-Your tool should be sure to write back the tokens it read, even under
-error conditions.  This includes not only errors in your tool but also
-outside influences such as interrupts (@code{SIGINT}), etc.  You may
-want to install signal handlers to manage this write-back.
+Your tool should be sure to write back the tokens it read, even under error
+conditions.  This includes not only errors in your tool but also outside
+influences such as interrupts (@code{SIGINT}), etc.  You may want to install
+signal handlers to manage this write-back.
+
+@item
+Your tool may also examine the first word of the @code{MAKEFLAGS} variable and
+look for the character @code{n}.  If this character is present then
+@code{make} was invoked with the @samp{-n} option and your tool may want to
+stop without performing any operations.
 @end itemize
 
 @node Windows Jobserver,  , POSIX Jobserver, Job Slots
@@ -12139,6 +12191,9 @@ look for the argument string @code{--jobserver-auth=NAME} where
 @samp{NAME} is the name of the named semaphore.  Use this name with
 @code{OpenSemaphore} to create a handle to the semaphore.
 
+@cindex @code{--jobserver-style} for Windows
+The only valid style for @code{--jobserver-style} is @samp{sem}.
+
 There are various error conditions you must consider to ensure your
 implementation is robust:
 

src/config.h-vms.template

@@ -260,6 +260,9 @@ this program.  If not, see <http://www.gnu.org/licenses/>.  */
 /* Define to 1 if you have the memmove function.  */
 #define HAVE_MEMMOVE 1
 
+/* Define to 1 if you have the 'mkfifo' function.  */
+/* #undef HAVE_MKFIFO */
+
 /* Define to 1 if you have the mktemp function.  */
 #define HAVE_MKTEMP 1
 

src/config.h.W32.template

@@ -195,6 +195,9 @@ this program.  If not, see <http://www.gnu.org/licenses/>.  */
 /* Define to 1 if you have the <minix/config.h> header file. */
 /* #undef HAVE_MINIX_CONFIG_H */
 
+/* Define to 1 if you have the 'mkfifo' function. */
+/* #undef HAVE_MKFIFO */
+
 /* Define to 1 if you have the 'mkstemp' function. */
 /* #undef HAVE_MKSTEMP */
 

src/main.c

@@ -234,6 +234,9 @@ static const int inf_jobs = 0;
 
 char *jobserver_auth = NULL;
 
+/* Style for the jobserver.  */
+static char *jobserver_style = NULL;
+
 /* Shuffle mode for goals and prerequisites.  */
 
 static char *shuffle_mode = NULL;
@@ -342,6 +345,8 @@ static const char *const usage[] =
     N_("\
   -j [N], --jobs[=N]          Allow N jobs at once; infinite jobs with no arg.\n"),
     N_("\
+  --jobserver-style=STYLE     Select the style of jobserver to use.\n"),
+    N_("\
   -k, --keep-going            Keep going when some targets can't be made.\n"),
     N_("\
   -l [N], --load-average[=N], --max-load[=N]\n\
@@ -488,6 +493,7 @@ static const struct command_switch switches[] =
     /* There is special-case handling for this in decode_switches() as well.  */
     { TEMP_STDIN_OPT, filename, &makefiles, 0, 0, 0, 0, 0, "temp-stdin" },
     { CHAR_MAX+11, string, &shuffle_mode, 1, 1, 0, "random", 0, "shuffle" },
+    { CHAR_MAX+12, string, &jobserver_style, 1, 0, 0, 0, 0, "jobserver-style" },
     { 0, 0, 0, 0, 0, 0, 0, 0, 0 }
   };
 
@@ -1347,6 +1353,9 @@ main (int argc, char **argv, char **envp)
 #endif
 #ifdef MAKE_JOBSERVER
                            " jobserver"
+#ifdef HAVE_MKFIFO
+                           " jobserver-fifo"
+#endif
 #endif
 #ifndef NO_OUTPUT_SYNC
                            " output-sync"
@@ -2094,7 +2103,7 @@ main (int argc, char **argv, char **envp)
      submakes it's the token they were given by their parent.  For the top
      make, we just subtract one from the number the user wants.  */
 
-  if (job_slots > 1 && jobserver_setup (job_slots - 1))
+  if (job_slots > 1 && jobserver_setup (job_slots - 1, jobserver_style))
     {
       /* Fill in the jobserver_auth for our children.  */
       jobserver_auth = jobserver_get_auth ();