Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
A shell wrapper to re-serialize output from parallel software builds
C Perl
branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
.gitignore
Makefile
README
jmake
syncsh.c

README

Parallelism in software builds can be extremely valuable for
cutting build time.  Unfortunately, a side effect of parallelism
is that the textual build logs may become scrambled as results
are reported asynchronously.  This can make it hard - or even
impossible - to debug build errors.

The "syncsh" program is a tiny program which looks enough like
a shell that make programs such as GNU make can specify it as
their SHELL value, though in fact it just forks a "real" shell
such as /bin/sh to do the actual work. Its value-added, in return
for that extra process, is that it holds onto the output of the
real shell and coordinates with other running instances of itself
before printing those results to stdout. The result is that the
output of each make "recipe" is written atomically.

Synchronization is achieved by use of a semaphore.  It's important
to understand that syncsh does not hold the semaphore for the
entire time its job is running; that would force jobs to run
serially and make parallel builds a no-op. Instead, each job is
allowed to run freely and the semaphore is only acquired for as
long as required to write its output.

Thus syncsh will re-synchronize the output of asynchronous jobs.
Note that no guarantee is made that jobs will report in the "right"
order (since in parallel processing there is no right order), only
that the order will be sensible and that the results of different
tasks will not be intermingled. It would probably be possible to
enhance syncsh to guarantee results in the same order as a serial
build but that would require some help from, and thus changes to,
the make program.

Though GNU make is not an architectural requirement, syncsh was
written with the expectation of use with GNU make. Any make program
typically runs its commands by calling the shell like this

    /bin/sh -c "command"

Which is the only usage pattern syncsh currently understands. It
_should_ work with any parallel build program which runs its
commands as (SHELL FLAGS "COMMAND").

For example, you could enable syncsh by adding the following line
to a GNU makefile:

SHELL		:= /full/path/to/syncsh

Syncsh can also be used without modifying makefiles at all:

% gmake SHELL=/full/path/to/syncsh -j12 ...

There are a few other useful SYNCSH_* environment variables; see
the usage message or the source code. Run with no arguments for
usage.

This package includes a sample "jmake" script which shows how
this can be bound up in a wrapper.

In order for syncsh to work reliably we must ensure that each
recipe is passed to a single instance of $(SHELL). With GNU make
3.82 or above this can be done by setting .ONESHELL mode but
in any other situation it may be necessary to fix the makefile(s)
by adding shell metacharacters like ; or &&. E.g. this:

	rm -f $@
	cp $< $@

would result in two shell invocations (or none, if we hadn't
overridden SHELL) but

	rm -f $@ &&\
	cp $< $@

will guarantee a single shell and is better practice anyway. Note
that GNU make will normally avoid using an intermediate shell at
all when possible but this behavior is suppressed when SHELL is
assigned to. Thus by setting SHELL=<syncsh> we guarantee syncsh
will be used for every recipe.

As an advanced feature, you can force a certain set of recipes to
run serially by setting the environment variable SYNCSH_SERIALIZE
to a regular expression. Any recipe which matches the regexp will
cause syncsh to hold a special semaphore derived from the regexp
*for its entire duration*.  The net effect should be to guarantee
that only one recipe which matches the regexp is running at a
time.  Recipes which do not match use the default semaphore and
thus do not get stuck behind these "serialized" jobs.  For
example:

    export SYNCSH_SERIALIZE='[AC]'

will serialize all recipes containing the characters 'A' or 'C'. This
variable, of course, could also be exported directly from the makefile:

    export SYNCSH_SERIALIZE := [AC]
Something went wrong with that request. Please try again.