build.nw

% -*- mode: Noweb; noweb-code-mode: c-mode; -*-
% Build with noweb:
%  notangle -t8 build.nw > makefile
%  make
\documentclass[twoside,english]{article}
\usepackage[letterpaper,rmargin=1.5in,bmargin=1in]{geometry}
%%% latex preamble
% Begin-doc build-rcs
\RCS $Id$
\RCS $Revision$
\RCS $Date$
% End-doc build-rcs

\begin{document}

\title{Syntax Highlighting Modular Build System for noweb}
\author{Thomas J. Moore}
% Begin-doc build-ver
\date{Version 3.17.\RCSRevision\\\RCSDate}
% End-doc build-ver
\maketitle

\begin{abstract}

This document describes, implements, and is built using a modular ``literate
programming%
\footnote{\url{http://www.literateprogramming.com/articles.html}
is a good starting point.}%
'' build system with syntax highlighting, based on Norman Ramsey's noweb%
\footnote{\url{http://www.eecs.harvard.edu/~nr/noweb/}}%
, GNU%
\footnote{\url{http://www.gnu.org}}%
 make, and a number of other freely available support tools.

This document, its contents and all prior versions published by the
same author are granted to the Public Domain in 2021 by Thomas J.
Moore.

\end{abstract}

\tableofcontents

\section{Overview}

% Begin-doc literate-intro
Most of my old personal programming projects started out as a general
idea, along with numerous scattered pieces of paper with notes
detaling some particular aspects of the design.  Most of my
professional programming projects instead started with formal design
documents containing similar information.  Over time, the design
documents and notes were lost and/or forgotten, and/or they diverged
from the actual product.  In any case, the organization of any design
documentation rarely corresponds with the actual code, as efficient
code follows much different rules than efficient human communication.
Literate programming provides two main features that alleviate these
problems: the design document is kept with the source code, and the
code can be written in the order of the design document, to be
reordered later for more efficient compilation.

Literate programming is a method of programming devised and named by
Donald Knuth in 1983, along with the tools to help implement that
method (Web).  It mainly emphasizes writing a human-readable document
with embedded code rather than the other way around.  Over the years,
as others have adopted and adapted the style, the same term has been
used to mean slightly different things, and the tools have emphasized
different features.  To me, the important features are mixing of
documentation and code, and reordering of code chunks to match the
text, rather than the other way around.  Systems which do not display
the source code and documentation at the same time (e.g. Doxygen,
perl's pod) and/or do not provide code reordering (e.g. dtx, lgrind,
code2html) do not support literate programming.  Knuth would probably
be even pickier, and reject my use of large code chunks, short
variable names for temporary variables, long sequences of multiple
code chunks, and multiple source files and programs in a single
document, as well as my lack of a language-specific symbol cross
reference and index (in addition to the chunk index).

Knuth felt that programs should be pleasant to read, like documents
which happen to have executable code in them.  He believed that his
method would allow creation of programs which are by their nature
easier to maintain and less error-prone.  I disagree.  Whether or not
a document is pleasant to read is always very dependent on the writer
(and, to a lesser extent, the reader).  Easier maintenance and fewer
bugs are but a pipe dream: people will in fact be even more tempted to
ignore the source code in favor of the commentary, thus missing
obvious bugs. Code reordering, and especially small code chunks may
also make it difficult for some to follow the code to find out what
\emph{really} gets executed.  Instead of these lofty goals, my own
reasons for doing this are mainly to keep my scattered design notes in
one place (the source) and to keep the design document in sync with
the implementation (since they are the same thing).  

Creating programs as documents using literate programming principles
requires a system that supports easy-to-write documentation, and code
that is both structured like the documentation and actually part of
the documentation.  For simple programs, a set of tools which support
only a single source language and a single program file can be used.
As programs become more complicated, though, the number of source
files may increase, and when designing entire products, the number of
source languages may increase as well.  Current common public
documentation standards also require a system which produces at least
HTML and PDF output.  After long searching, I have not found any
systems which meet these requirements. Instead, the closest is the
noweb package%
\footnote{In particular, version 2.  Version 3 is available and
functional, but has been in ``alpha'' for many years, has a different
command syntax, and is missing a few important bits (like the style
files for weaving and a [[noroots]]-like command, although those can
just be obtained from a version 2 archive).  I'm also not particularly
fond of using a custom version of Lua (I converted it to use a
standard Lua 5.1 a while back, but lost all of that work somehow).
}.  It supports \LaTeX{} and HTML output, and an arbitrary number of
output files and source languages. However, it does little to make the
code look like part of the documentation.
% End-doc literate-intro
The code in this document serves to provide a build system that
includes syntax-highlighted code, at the very least.

While the build system could simply be a carefully crafted shell script, a
makefile is used instead.  This allows use of implicit rules and the
system-wide [[CC]] and [[CFLAGS]] defaults.  The default chunk is set to the
makefile, requiring no chunk name to extract.
% Begin-doc build-doc
To build, place the noweb source files into their own directory (optionally
extracting them from the document you are reading first), extract the
makefile, and make using GNU make:

% Uses \jobname instead of \emph{<this file>} so that cut & paste works mostly
\begin{quote}{\ttfamily
\begin{rawhtml}
<!-->
\end{rawhtml}
\ifpdf
\#{} if your PDF viewer supports attachments, save the attachment\\*
\#{} otherwise, use pdfdetach:\\*
pdfdetach -saveall \jobname.pdf\\*
\#{} or pdftk:\\*
pdftk \jobname.pdf unpack\_files output .\\*
\else
\begin{rawhtml}
<-->
\end{rawhtml}
uudecode \jobname.html\\*
\#{} if uudecode chokes, use this instead:\\*
\#{} tr '\textbar$\sim$m`' '<>- ' < \jobname.html | uudecode -o \jobname.tar.gz\\*
gzip -dc \jobname.tar.gz | tar xf -\\*
\fi
\#{} then extract the makefile and build\\*
notangle -t8 build.nw > makefile\\*
make install
}\end{quote}

Note that while some dependency-based package managers may pull in a
full \TeX{} distribution as a dependency for noweb, none of that is
actually needed when only compiling code (i.e., tangling).

% End-doc build-doc
As noted below, additional makefile components are created in separate files.
% Begin-doc build-doc
Additional build configuration can be done in \texttt{makefile.config}
before installing (in particular, the install locations).  This file can be
generated using either \texttt{make makefile.config} or \texttt{make -n}.
On the other hand, to avoid having to modify this file after cleaning,
\texttt{makefile.config.local} can be created for this purpose instead.
% End-doc build-doc

\subsection{Makefile}

The makefile is for building the support files, executable binaries, and
printable source code (HTML and PDF), maybe installing them, and cleaning up
afterwards. Configuration information, variable definitions, and
miscellaneous rules are extracted from not only this noweb file, but also
any others that use it. The makefile itself can be used to include that
information, but to simplify makefile extraction, all of that information is
kept in separate files.  Modules that wish to extend the makefile can do so
using [[<<makefile.config>>]] for user-adjustable variables,
[[<<makefile.vars>>]] for any other variable definitions, and
[[<<makefile.rules>>]] for new rules.  Some configuration variables may need
values before the configuration file is built, though, so defaults are
included in the makefile, to be overridden by the configuration file.  Note
that any variable which does not depend on unknowns (noweb chunks or
variables not yet defined) should be defined using GNU make's ``simply
expanded'' variable flavor (i.e., [[:=]] instead of plain [[=]]) for
improved performance.  This has a disadvantage in that overriding simply
defined variables will require overriding any of its dependents as well, but
the performance improvement is usually worth it.

I have made some adjustments to the ordering of this in order to
support Daniel Pfeiffer's Makepp%
\footnote{\url{http://makepp.sourceforge.net}}%
, a GNU make clone which adds MD5 checksums as a file change detection
mechanism.  This magically fixes the problem of rebuilding every time,
at the expense of dealing with Makepp-specific issues.  The first of
these is that include files cannot be automatically (re)built unless
they appear after the rules which create them.  This means that the
two variables which control their creation can no longer come from the
configuration file.  They can only be set using the command line.
Another, related issue is that even though Makepp solves the rebuild
problem for most files, it makes the problem worse for the makefile
and its include files.  These are unconditionally rebuilt every single
time makepp is invoked.  The use of Makepp is detected by checking for
the existence of a non-empty [[$(MAKEPP_VERSION)]].

\lstset{language=make}
<<*>>=
<<makefile>>
@

<<makefile>>=
# See makefile.config for variables to override
# Put local config in makefile.config or makefile.config.local
<<Common noweb Warning>>

# default rule
all: misc bin doc

ifneq ($(MAKEPP_VERSION),)
# not much point, since it doesn't filter out #line directives
# and therefore most changes to noweb source will result in file changes
signature c_compilation_md5
endif

<<Defaults for [[makefile.config]]>>

# This is the right place to include them, but makepp can't handle it
#-include makefile.config
-include makefile.config.local

<<Build variables for makefile includes>>
<<Build rules for makefile includes>>

-include makefile.config
# reinclude to ensure overrides
-include makefile.config.local

-include makefile.vars

# keep intermediate files
.SECONDARY:
# mark a few phonies - not really necessary
.PHONY: all misc bin doc install clean distclean

bin: $(LIB_FILES) $(EXEC_FILES)

doc: $(DOC_FILES)

misc: $(MISC_FILES)

-include makefile.rules
@

<<Common noweb Warning>>=
# GENERATED FILE:  DO NOT EDIT OR READ THIS FILE
# Instead, read or edit the noweb file(s) from which this was generated,
# listed below.  Copyright notice and license terms can be found there.
# $Id$
@

\lstset{language=txt}
<<Sources>>=
$Id$
@

\lstset{language=make}
<<makefile.rules>>=
<<Common noweb Warning>>
install: misc bin
	@for x in $(EXEC_FILES); do set -e; \
	  echo "install $(DESTDIR)$(BIN_DIR)/$$x"; \
	  mkdir -p $(DESTDIR)$(BIN_DIR); \
          rm -f $(DESTDIR)$(BIN_DIR)/$$x; \
          cp -p $$x $(DESTDIR)$(BIN_DIR); \
        done
	<<Install other files>>

clean:
	<<Clean temporary files>>
	rm -f $(filter-out makefile, $(MAKEFILES))

distclean: clean
	<<Clean built files>>
	rm -rf .makepp
	<<Remove makefile>>
@

<<makefile.config>>=
# Configuration parameters
# Change here only if:
#  - you do not use makepp (makepp will overwrite on every make)
#  - you do not modify the noweb sources (otherwise even GNU make overwrites)
# Better to copy to makefile.config.local and make changes there.
# You can, of course, also specify overrides on the command line

<<Installation Directory Configuration Options>>
@

<<Installation Directory Configuration Options>>=
# Installation prefix to apply to all install locations
# This is not applied to hard-coded paths within installed files
DESTDIR:=

# Installation prefix to apply to all install locations by default
# This is applied after DESTDIR.
prefix:=/usr/local

# Installation directory for binaries
BIN_DIR:=$(prefix)/bin

@

<<makefile.vars>>=
<<Common noweb Warning>>
MAKEFILES = makefile makefile.config makefile.vars makefile.rules
EXEC_FILES = <<Executables>>

LIB_FILES = <<Libraries>>

DOC_FILES = <<Source Code Documentation Files>>

MISC_FILES = <<Plain Files>>

@

<<Install other files>>=
@

<<Plain Files>>=
\
@

<<Clean built files>>=
rm -rf $(DOC_FILES)
@

The makefile can make itself, as well.  This is dependent only on its
source file; there is little point in making this dependent on the
included files, as they will be automatically rebuilt as needed,
anyway.  A quick check before writing out the file ensures that a
blank or otherwise seriously invalid makefile will never be created
due to errors in the source file.  However, [[-include]] statements
are filtered out, so that GNU make doesn't try to automatically build
the include files.  This was never a perfect check, and this filtering
makes it less perfect:  each file must be independently error-free, and
errors introduced from other files (such as variable definitions)
which may affect the file being tested will not be detected.

Note that Makepp has issues with recursive make invocation for the
verification of the makefile, so this step is skipped.  More recent
versions claim to have fixed this, but I have yet to test it.

<<makefile.config>>=
# The name of the file containing the makefile
# Note: due to Makepp restrictions, this can only be set on the command line
# or in makefile.config.local
#BUILD_NOWEB:=build.nw

@

<<Defaults for [[makefile.config]]>>=
BUILD_NOWEB=build.nw
@

<<Build rules for makefile includes>>=
makefile: $(BUILD_NOWEB)
	notangle -t8 -R$@ $(BUILD_NOWEB) 2>/dev/null | grep -v '^$$' >/dev/null
ifeq ($(MAKEPP_VERSION),)
	notangle -t8 -R$@ $(BUILD_NOWEB) 2>/dev/null | \
	    grep -v '^-include' | \
	      env -i PATH="$${PATH}" $(MAKE) -n -f- /dev/null >/dev/null
endif
	-notangle -t8 -R$@ $(BUILD_NOWEB) > $@
@

<<Remove makefile>>=
rm -f makefile
@@echo
@@echo Regenerate the makefile with:
@@echo notangle -t8 $(BUILD_NOWEB) \> makefile
@

Generating the other files requires the ability to correctly assemble them
from all the other noweb files.

<<makefile.config>>=
# The name of the source files
# Any time you change this, makefile.* should be removed and rebuilt
# Note: due to Makepp restrictions, this can only be set on the command line
# or in makefile.config.local
#NOWEB:=$(wildcard *.nw)

@

<<Defaults for [[makefile.config]]>>=
NOWEB:=$(wildcard *.nw)
@

\subsection{Merging Sources}

From these files, an order of building must be derived.  This is done using
a dependency tree created from special comments in the source files.  These
comments are at the beginning of a line, and are of the form
\texttt{\%\%\% requires \emph{X}}, where \emph{X} is either the name of a
noweb file or the name with the \texttt{.nw} removed.  An explicit
\verb|\input{|\emph{X}\verb|.nw| creates a dependency as well.
Top-level build files are those on which no others depend.  The tree
order consists of the top-level files, followed by their direct
dependencies, in the order found in those files, followed by those
files' direct dependencies, and so forth, with no file repeated.

Finding the dependency directives requires [[egrep]], and removing the
pattern requires [[sed]].  Due to the sloppiness of the patterns and other
parsing, no file names can contain colons or spaces.  This is all done using
GNU make's internal functions so that the results can be easily used in
other rules.

<<Build variables for makefile includes>>=
NOWEB_DEPS:=$(shell egrep '^%%% requires |^\\input{.*\.nw}$$' $(NOWEB) /dev/null | \
                    sed -e 's/%%% requires //;s/\\input{\(.*\)}$$/\1/' \
		        -e 's/[[:space:]]*\(%.*\)*$$//;s/:/ /')
@

After the dependencies are found, they are separated into files which depend
on others ([[NOWEB_UPPER]]), and files which are depended on
([[NOWEB_LOWER]]).   The files whilch are depended on may be specified
without the \texttt{.nw} extension, so the filesystem is checked for the
file, and \texttt{.nw} is added if the file does not exist.  If it still
does not exist after tacking on \texttt{.nw}, the missing file is an error.

<<Build variables for makefile includes>>=
# return rest of words of parm 2, starting at parm 1
rest = $(wordlist $1,$(words $2),$2)

# return every other word of parm, starting with first
ret_other = $(word 1,$1) $(if $(word 3,$1),$(call ret_other,$(call rest,3,$1)))

NOWEB_UPPER:=$(call ret_other,$(NOWEB_DEPS))

NOWEB_LOWER_BASE:=$(call ret_other,$(wordlist 2,$(words $(NOWEB_DEPS)),$(NOWEB_DEPS)))

NOWEB_LOWER:=$(foreach f,$(NOWEB_LOWER_BASE),$(if $(wildcard $f),$f,\
                                           $(if $(wildcard $f.nw),$f.nw, \
					   $(error $f and $f.nw not found))))
@

Next, the tree is traversed, from top to bottom.  The top is simply the list
of files on which no other files depend.  There must be at least one file at
the top, or nothing will work correctly.  Then, each file in the list is
checked for as-yet unfulfilled dependencies to tack on.  No dependency may
appear before files upon which it depends, so the dependecies are repeatedly
tacked onto the start of the list and stripped from the rest until the tree
settles.

<<Build variables for makefile includes>>=
NOWEB_HIGHEST:=$(filter-out $(NOWEB_LOWER),$(NOWEB))

$(if $(NOWEB_HIGHEST),,$(error Invalid dependency tree))

# return words from parm #3 in positions of parm #2 which match parm #1
match_words = $(if $(filter $1,$(word 1,$2)),$(word 1,$3)) \
  $(if $(word 2,$2),$(call match_words,$1,$(call rest,2,$2),$(call rest,2,$3)))

# return only unique words in parm, keeping only first occurrence
uniq = $(if $1,\
  $(firstword $1) $(call uniq,$(filter-out $(firstword $1),$(call rest,2,$1))))

# tack dependencies to left
predeps = $(call uniq, $(foreach f,$1,\
	        $(call match_words,$f,$(NOWEB_UPPER),$(NOWEB_LOWER))) $1)

# true if lists not equal
listne = $(strip $(if $1,$(if $2,$(if $(filter $(word 1,$1),$(word 1,$2)), \
                $(call listne,$(call rest,2,$1),$(call rest,2,$2)), \
	          y1), y2), $(if $2, y3)))

# expand dependencies until there are no more
tree = $(if $(call listne,$1,$(call predeps,$1)), \
           $(call tree,$(call predeps,$1)), $1)

NOWEB_ORDER:=$(call tree,$(NOWEB_HIGHEST))

ifeq ($(PROJECT_NAME),)
PROJECT_NAME:=$(subst .nw,,$(firstword $(NOWEB_HIGHEST)))
endif
@

<<makefile.config>>=
#Set to override the automatically determined project name
#PROJECT_NAME=

@

<<makefile.rules>>=
prtree:
	@echo Project: $(PROJECT_NAME)
	@echo Deps: $(NOWEB_DEPS)
	@echo Highest: $(NOWEB_HIGHEST)
	@echo Upper: $(NOWEB_UPPER)
	@echo Lower: $(NOWEB_LOWER)
	@echo Order: $(NOWEB_ORDER)
@

So, to generate a file, all of these noweb files are concatenated, in
reverse order, and passed into [[notangle]].  The makefile components in
particular need to be checked for errors in mostly the same way as the main
makefile.

<<Build rules for makefile includes>>=
makefile.config: $(NOWEB_ORDER)
	notangle -t8 -R$@ $^ 2>/dev/null | grep -v '^$$' >/dev/null
ifeq ($(MAKEPP_VERSION),)
	notangle -t8 -R$@ $^ 2>/dev/null | \
	    grep -v '^-include' | \
	      env -i PATH="$${PATH}" $(MAKE) -n -f- /dev/null >/dev/null
endif
	-notangle -t8 -R$@ $^ > $@

makefile.vars: makefile.config
	notangle -t8 -R$@ $(NOWEB_ORDER) 2>/dev/null | grep -v '^$$' >/dev/null
ifeq ($(MAKEPP_VERSION),)
	notangle -t8 -Rmakefile.config -R$@ $(NOWEB_ORDER) 2>/dev/null | \
	    grep -v '^-include' | \
	      env -i PATH="$${PATH}" $(MAKE) -n -f- /dev/null >/dev/null
endif
	-notangle -t8 -R$@ $(NOWEB_ORDER) > $@

makefile.rules: makefile.vars
	notangle -t8 -R$@ $(NOWEB_ORDER) 2>/dev/null | grep -v '^$$' >/dev/null
ifeq ($(MAKEPP_VERSION),)
	notangle -t8 -Rmakefile.config -Rmakefile.vars -R$@ \
	                                     $(NOWEB_ORDER) 2>/dev/null | \
	    grep -v '^-include' | \
	      env -i PATH="$${PATH}" $(MAKE) -n -f- /dev/null >/dev/null
endif
	-notangle -t8 -R$@ $(NOWEB_ORDER) > $@
@

Building plain files is done the same way, but without the complicated
checks, assuming that no additional processing needs to be done.  For files
where additional processing is necessary, additional dependencies on the
[[misc]] target, as well as [[<<Install other files>>]] can be used to add
files with special build rules.  Another way to add additional
processing steps is to modify the tangling pipeline (by adding
[[NOTANGLE_OPTS]] or replacing [[NOTANGLE]]) or to add a
post-processing pipeline (using [[NOTANGLE_POSTPROC]]).  GNU make
allows setting these variables per-target, making adding processing to
just one or two files relatively easy.  In order to allow for tangle
postprocessing to use locally built items, [[NOTANGLE_DEP]] can be
altered as well.  To make using this pipeline less error-prone (and
shorter), a single-argument function is provided to extract a given
named chunk ([[NOTANGLE_CHUNK]]).  Almost every usage I envision
uses [[$@]] as that argument, and redirects output to [[$@]].  No
special variable is set to that usage pattern, though.

Note that [[<<Plain Files>>]] is intended for installable targets.
For plain files generated as part of the build, use
[[<<Plain Build Files>>]] instead.  For a subtle change, any files
generated by means other than [[notangle]] can be added to
[[<<Plain Built Files>>]] in order to save making a clean rule for it.

<<makefile.vars>>=
NOTANGLE=notangle
NOTANGLE_DEP=
NOTANGLE_OPTS=
NOTANGLE_POSTPROC=
NOTANGLE_CHUNK=$(NOTANGLE) $(NOTANGLE_OPTS) -R$1 $(NOWEB_ORDER) $(NOTANGLE_POSTPROC)

MISC_TEMP_FILES = <<Plain Build Files>>

GENERATED_TEMP_FILES = <<Plain Built Files>>

@

<<Plain Build Files>>=
\
@

<<Plain Built Files>>=
\
@

<<makefile.rules>>=
$(MISC_FILES) $(MISC_TEMP_FILES): $(NOWEB_ORDER) $(NOTANGLE_DEP)
	-$(call NOTANGLE_CHUNK,$@) >$@
@

<<Clean built files>>=
rm -f $(MISC_FILES)
@

<<Clean temporary files>>=
rm -f $(MISC_TEMP_FILES) $(GENERATED_TEMP_FILES)
@

\subsection{Support Files}

Since [[noroots]] may not be on the target system, for example when using
the tar file, a close equivalent is provided.  This does not properly
filter out non-root nodes, but most build operations require specific
node names that are not likely to be used anywhere but the root level.

Makepp has trouble parsing this, so this is disabled for now.

<<Defaults for [[makefile.config]]>>=
NOROOTS=noroots
@

<<makefile.config>>=
# The noroots command is used to extract file names from $(NOWEB)
# The following works well enough in most cases if noweb is missing
# Note: due to Makepp restrictions, this can only be set on the command line
# or in makefile.config.local
#NOROOTS=sh -c "sed -n '/^@<<.*@>>=\$$/{s/=\$$//;p;}' \$$* \
#		                 | sort -u" /dev/null
NOROOTS=noroots

@

In fact, when building just the code, it is likely that the only part
of noweb required is [[notangle]].  However, with the extensions of
the next section (in particular parameterized chunks), a simple
replacement (as provided in previous versions of this document) is not
possible.

\subsection{Additional Features}

It may also be useful to build a tar file for building on systems where
noweb is not present.  This is meant to be a convenience, for building
binaries only, and not for distribution.  That means neither the source
documentation nor any means to build the source documentation will be
included.  Since it is not possible to distinguish between soft links
created for building and soft links to other noweb files, no attempt will be
made to force link dereferencing, either.

<<makefile.rules>>=
$(PROJECT_NAME)-src.tar.gz: $(BUILD_SOURCE)
	@# needs GNU tar
	tar czf $(PROJECT_NAME)-src.tar.gz $(BUILD_SOURCE)
@

<<makefile.vars>>=
BUILD_SOURCE=$(NOWEB) $(MAKEFILES) $(MISC_FILES) <<Build Source>>

@

<<Clean built files>>=
rm -f $(PROJECT_NAME)-src.tar.gz
@

It may also be useful to get some source code statistics.  Code counts
include all code chunks, but do not include the chunk name/start line or the
terminating [[@]].  Hidden sections are not included; they are delimited by
\texttt{<!-\,->} and \texttt{<-\,->} on their own line.

<<makefile.rules>>=
count: $(NOWEB) $(MISC_FILES) <<Files to Count>>

	@for x in $(NOWEB); do \
	   echo $${x}:; \
	   <<Count lines in [[$$x]]>> \
         done
	@echo "Tangled output:"
	@wc -l $(MISC_FILES) <<Files to Count>>
	      </dev/null | sort -k1n
@

<<Count lines in [[$$x]]>>=
tl=`cat $$x | wc -l`; \
bl=`grep -c '^[[:space:]]*$$' $$x`; \
hl=`sed -n -e '/^<!-->$$/{:a p;n;/^<-->$$/!ba;}' < $$x | wc -l`; \
cl=`sed -n -e '/^@<<.*>>=/{:a n;/^@[^@]/b;/^@$$/b;/^[[:space:]]*$$/!p;ba;}' \
           -e '/^<!-->$$/{:b n;/^<-->$$/!bb;}' < $$x | wc -l` ; \
echo " Lines: $$tl:"; \
echo "  $$cl code, $$((tl-bl-hl-cl)) doc, $$hl hidden, $$bl blank";
@

Finally, there is a rule to check the consistency of the noweb source.
Checking the list of chunks which are not specifically referenced
requires human intervention, so all root chunks are printed out for
review.  The list of missing chunks should always be accurate, though.
The terminating [[@]] for code chunks isn't required by the noweb
syntax when writing multiple consecutive code chunks, but it is
required by some of the document converters.  In addition, if it is
not followed by a blank line, the weaver will likely produce invalid
\LaTeX{}.  The (obsolescent, but still necessary) [[%def]] directive is
broken in that if the next non-blank text is another code chunk, it
corrupts the header, so that is flagged as well.  The reinsertion code
is stupid about unbalanced doc reinsertions, but instead of checking
correctly there, it's flagged here.

Because this check is a separate rule, target-specific altering of the
tangling process is not supported.  Making drastic changes to source
code using the tangle process alterations should not be done anyway,
and rules like this make some kinds of alterations impossible.

<<makefile.rules>>=
.PHONY: check
check: $(NOWEB) $(NOTANGLE_DEP)
	@# Print all roots to find misnamed/unreferenced chunks
	@# At same time, untangle each to find undefined chunks
	@ <<[[noroots]]>> $(NOWEB) | sed 's/@<<//;s/@>>//;' | sort | while read -r x; do \
          echo "Root: $$x"; \
          $(call NOTANGLE_CHUNK,"$$x") >/dev/null; \
        done
	@# Check for proper @-usage and %def-usage
	@for f in $(NOWEB); do \
	   lno=1 gotat=y gotdef=; \
           while IFS= read -r x; do \
	     test -n "$${gotat#y}" -a -n "$$x" && echo "$$f: $${lno}: no blank line after @"; \
	     gotat=$${gotat%x}; \
             case "$$x" in \
                "@<<"*"@>>"=) \
		   test "$$gotdef" && echo "$$f: $${lno}: %def w/o following text"; \
                   test "$$gotat" || echo "$$f: $$prev not terminated by @"; \
                   gotat=; \
                   prev="$${lno}: $$x";; \
		"@@<<"*|"@@>>"*|"@[["|"@]]") ;; \
                @|@[^@]*) \
		   test "$$gotat" && echo "$$f: $${lno}: extra @"; \
                   gotat=yx; \
		   case "$$x" in \
		     "@ %def "*) gotdef=y ;; \
		   esac;; \
		?*) gotdef= ;; \
             esac; \
             lno=$$((lno+1)); \
           done < "$$f"; \
	 done
	<<Other source file consistency checks>>
@

\section{Binaries}

Building the binaries is pretty simple, using automatic rules. 
Support is provided here for plain text executable scripts and C and
C++ executables.  In order to ensure a consistent build rule for all C
and C++ files, the default C/C++-to-executable rules are blanked out.
Executables are assumed to have one mainline source, named the same as
the executable but with a \texttt{.c} extension for C and
\texttt{.c++} for C++.  All local libraries built from local sources
are linked into every executable.

All source files share the ability to have per-file post-processing
using [[NOTANGLE_POSTPROC]].  Previous versions of this build system
only supported per-file post-processing for C files, using the
[[C_POSTPROCESS]] hook.  The C-specific hook was a function call
rather than a plain variable expansion; i.e., instead of using [[$@]]
to refer to the target, it was expected that [[$1]] would be used.
Using [[NOTANGLE_POSTPROC]] is more consistent, but some old documents
still use the old method.  The old method is supported, but
deprecated.

<<Executables>>=
$(C_EXEC) $(SCRIPT_EXEC) \
@

<<makefile.vars>>=
SCRIPT_EXEC=<<Script Executables>>

C_EXEC=<<C Executables>>

NOWEB_CFILES:=$(shell $(NOROOTS) $(NOWEB) | sed -n '/\.c>>/{s/@<<//;s/@>>//;p;}')
CFILES=<<C Files>>

NOWEB_RAWCXXFILES:=$(shell $(NOROOTS) $(NOWEB) | sed -n '/\.c++>>/{s/@<<//;s/@>>//;p;}')
<<Adjust [[NOWEB_CXXFILES]]>>
CXXFILES=<<C++ Files>>

NOWEB_HEADERS:=$(shell $(NOROOTS) $(NOWEB) | sed -n '/\.h>>/{s/@<<//;s/@>>//;p;}')
CHEADERS=<<C Headers>>

CXXHEADERS=<<C++ Headers>>

COFILES=$(patsubst %.c, %.o, $(CFILES))
CXXOFILES=$(patsubst %.c++, %.o, $(CXXFILES))
@

<<C Files>>=
$(NOWEB_CFILES) \
@

<<C++ Files>>=
$(NOWEB_CXXFILES) \
@

<<C Headers>>=
$(filter-out $(CXXHEADERS),$(NOWEB_HEADERS)) \
@

<<C++ Headers>>=
\
@

<<Build Source>>=
$(SCRIPT_EXEC) \
@

<<makefile.config>>=
# Set to -L for #line in C/C++ files, but lose code indentation
USE_LINE:=-L

@

<<makefile.vars>>=
$(NOWEB_HEADERS) $(NOWEB_CFILES) $(NOWEB_CXXFILES): NOTANGLE_OPTS += $(USE_LINE)
$(NOWEB_HEADERS) $(NOWEB_CFILES): NOTANGLE_POSTPROC += $(call C_POSTPROCESS,$@)
@

<<makefile.rules>>=
$(SCRIPT_EXEC): $(NOWEB) $(NOTANGLE_DEP)
	-$(call NOTANGLE_CHUNK,$@) >$@
	chmod +x $@

$(NOWEB_HEADERS) $(NOWEB_CFILES) $(NOWEB_CXXFILES): $(NOWEB) $(NOTANGLE_DEP)
	-$(call NOTANGLE_CHUNK,$@) >$@

# disable gmake built-in .c/.c++->exe rules
ifeq ($(MAKEPP_VERSION),)
%: %.c
%: %.c++
endif

%.o: %.c
	$(CC) $(CFLAGS) $(EXTRA_CFLAGS) -c -o $@ $<

%.o: %.c++
	$(CXX) $(CXXFLAGS) $(EXTRA_CXXFLAGS) -c -o $@ $<

%: %.o
	$(CXX) -o $@ $< $(LDFLAGS) -L. $(LOCAL_LIBS) $(EXTRA_LDFLAGS)
@

<<Files to Count>>=
$(CFILES) $(CXXFILES) $(CHEADERS) $(CXXHEADERS) \
@

<<Clean temporary files>>=
rm -f $(CFILES) $(CXXFILES) $(CHEADERS) $(CXXHEADERS)
rm -f $(COFILES) $(CXXOFILES)
@

<<Clean built files>>=
rm -f $(EXEC_FILES)
@

<<Script Executables>>=
\
@

<<C Executables>>=
\
@

<<Build Source>>=
$(CFILES) $(CXXFILES) $(CHEADERS) $(CXXHEADERS) \
@

Scripts and C/C++ programs may also be generated as part of the build
process, for example to create machine-generated code.

<<makefile.vars>>=
BUILD_SCRIPT_EXEC=<<Build Script Executables>>

BUILD_C_EXEC=<<C Build Executables>>

@

<<Build Source>>=
$(BUILD_SCRIPT_EXEC) \
@

<<makefile.rules>>=
$(BUILD_SCRIPT_EXEC): $(NOWEB) $(NOTANGLE_DEP)
	-$(call NOTANGLE_CHUNK,$@) >$@
	chmod +x $@
@

<<Clean temporary files>>=
rm -f $(BUILD_SCRIPT_EXEC) $(BUILD_C_EXEC)
@

<<Build Script Executables>>=
\
@

<<C Build Executables>>=
\
@

The local libraries are built using a chunk naming convention.
[[<<Library [[name]] Members>>]] chunks contain a plain listing of included
object files.  No support for shared libraries is provided at this time.
Ideally, this should ensure that libraries dependent on others are listed
earlier in the library order.  Instead, the list is printed in reverse
order, so the least dependent libraries are printed first.

Since \texttt{tac} may not be available everywhere, an alternative may
be specified in [[makefile.config]].

<<Defaults for [[makefile.config]]>>=
TACCMD=tac
@

<<makefile.config>>=
# Where to find the non-standard tac command (GNU coreutils)
# Note: due to Makepp restrictions, this can only be set on the command line
# or in makefile.config.local
#TACCMD:=sed -n -e '1!G;h;$$p'

@

<<Build variables for makefile includes>>=
LOCAL_LIBS_BASE:=$(shell $(NOROOTS) $(NOWEB_ORDER) | \
  sed -n 's/@<<Library \[\[\(.*\)]] Members@>>/\1/p' | \
    $(TACCMD))
LOCAL_LIBS:=$(LOCAL_LIBS_BASE:%=-l%)
LOCAL_LIB_FILES:=$(LOCAL_LIBS_BASE:%=lib%.a)
@

<<makefile.rules>>=
$(C_EXEC): $(LOCAL_LIB_FILES)
@

<<Clean temporary files>>=
rm -f $(LOCAL_LIB_FILES)
@

While it would be nice to not have to generate yet another support file for
this, notangle is required for this to work.  One of the goals of this
system is to be able to generate a source tarball that does not depend on
notangle.

<<Defaults for [[makefile.config]]>>=
AR=ar
AR_EXTRA=
RANLIB=ranlib
@

<<makefile.config>>=
# The ar program
AR=ar

# Extra $(AR) options (e.g. s to perform ranlib automatically)
AR_EXTRA=

# The ranlib program
RANLIB=ranlib

@

<<makefile.rules>>=
#define build-lib
#lib$(1).a: $(2)
#	rm -f $$@
#	$$(AR) cr$$(AR_EXTRA) $$@ $$^
#	$$(RANLIB) $$@
#LIB_OBJ += $(filter-out $(COFILES),$(2))
#endef
#
#$(foreach l,$(LOCAL_LIBS_BASE),$(eval $(call build-lib,$l, \
#     $(shell notangle -R'Library [[$l]] Members' $(NOWEB_ORDER) 2>/dev/null))))
@

Instead, [[makefile.libs]] is generated with the macros expanded.
Since the chunks are simple, and this is for makefile building, there
is no reason to support anything but the basic [[notangle]].

<<makefile>>=
MAKEFILES+=makefile.libs
makefile.libs: makefile.rules
	for x in $(LOCAL_LIBS_BASE); do \
	   printf %s lib$$x.a:; \
	   notangle -R"Library [[$$x]] Members" $(NOWEB_ORDER) | tr '\n' ' '; \
	   printf '\n\trm -f $$@\n'; \
	   printf '\t$$(AR) cr$$(AR_EXTRA) $$@ $$^\n'; \
	   printf '\t$$(RANLIB) $$@\n'; \
	   printf 'LIB_OBJ += $$(filter-out $$(COFILES) $$(CXXOFILES),'; \
	   notangle -R"Library [[$$x]] Members" $(NOWEB_ORDER) | tr '\n' ' '; \
	   printf ')\n'; \
	done > $@
-include makefile.libs
@

<<Clean temporary files>>=
rm -f $(LIB_OBJ)
@

Of course it might be useful to also provide installation rules for
selected libraries, as well as their support files (header files and
API documentation), but that is left for a future revision.  For now,
a simplistic method is provided to add headers to an exported header
list, and libraries to an exported library list.  Configuration files
can be installed using an exported misc. file list.  If the libraries
are converted into shared objects, they are not exported using this
method.  As a special hack, a subdirectory may be prefixed to the
include file and misc file names, and they may be exported either from
the named subdirectory or from the current directory.  Also, the
library names only need the base name; if this convention is followed,
it may be easier to install shared libraries in the future.  In any
case, no additional dependencies should be required for the
[[install]] target, as everything here should have been built by the
time [[bin]] and [[misc]] are made.

<<Libraries>>=
$(LOCAL_LIB_FILES) \
@

<<Installation Directory Configuration Options>>=
# Installation directory for include files
INCLUDE_DIR:=$(prefix)/include

# Installation directory for libraries
LIB_DIR:=$(prefix)/lib

# Installation directory for other configuration files
DATA_DIR:=$(prefix)/share

@

<<makefile.vars>>=
INSTALLED_LIBS:=<<Libraries to Install>>

INSTALLED_INCLUDES:=<<Headers to Install>>

INSTALLED_MISC:=<<Plain Files to Install>>

@

<<Install other files>>=
@@for x in $(INSTALLED_LIBS); do set -e; \
  mkdir -p $(DESTDIR)$(LIB_DIR); \
  test -f $$x || x=lib$${x}.a; \
  echo "install $(DESTDIR)$(LIB_DIR)/$$x"; \
  cp -p $$x $(DESTDIR)$(LIB_DIR); \
done
@@for x in $(INSTALLED_INCLUDES); do set -e; \
  echo "install $(DESTDIR)$(INCLUDE_DIR)/$$x"; \
  mkdir -p $(DESTDIR)$(INCLUDE_DIR); \
  y=$$x; test -f $$y || y=$${y##*/}; \
  test $${x%/*} != $$x && mkdir -p $(DESTDIR)$(INCLUDE_DIR)/$${x%/*}; \
  cp $$y $(DESTDIR)$(INCLUDE_DIR)/$$x; \
done
@@for x in $(INSTALLED_MISC); do set -e; \
  echo "install $(DESTDIR)$(DATA_DIR)/$$x"; \
  mkdir -p $(DESTDIR)$(DATA_DIR); \
  y=$$x; test -f $$y || y=$${y##*/}; \
  test $${x%/*} != $$x && mkdir -p $(DESTDIR)$(DATA_DIR)/$${x%/*}; \
  cp $$y $(DESTDIR)$(DATA_DIR)/$$x; \
done
@

<<Libraries to Install>>=
\
@

<<Headers to Install>>=
\
@

<<Plain Files to Install>>=
\
@

In addition to distributed binaries, test binaries may be built and
executed.  Other than checking return codes, no interpretation of test
results is supported.  It is intended primarily for manual testing.  Sample
binaries may be placed under these rules, as well.

<<makefile.rules>>=
.PHONY: test test-bin
test-bin: $(TEST_EXEC) $(TEST_EXEC_SUPPORT)

$(TEST_EXEC):  $(LIB_FILES)

test: test-bin
	@set -e; for f in $(TEST_EXEC); do echo running $$f:; ./$$f; done
	<<Additional Tests>>
@

<<makefile.vars>>=
TEST_EXEC=<<Test Executables>>

TEST_C_EXEC=<<C Test Executables>>

TEST_SCRIPT_EXEC=<<Test Scripts>>

TEST_EXEC_SUPPORT=<<Test Support Executables>>

TEST_C_EXEC_SUPPORT=<<C Test Support Executables>>

TEST_SCRIPT_EXEC_SUPPORT=<<Test Support Scripts>>

@

<<Test Executables>>=
$(TEST_C_EXEC) $(TEST_SCRIPT_EXEC) \
@

<<C Test Executables>>=
\
@

<<Test Scripts>>=
\
@

<<Test Support Executables>>=
$(TEST_C_EXEC_SUPPORT) $(TEST_SCRIPT_EXEC_SUPPORT) \
@

<<C Test Support Executables>>=
\
@

<<Test Support Scripts>>=
\
@

<<makefile.rules>>=
$(TEST_C_EXEC_SUPPORT) $(TEST_C_EXEC): $(LOCAL_LIB_FILES)
$(TEST_C_EXEC): $(TEST_C_EXEC_SUPPORT)

$(TEST_SCRIPT_EXEC) $(TEST_SCRIPT_EXEC_SUPPORT): $(NOWEB) $(NOTANGLE_DEP)
	-$(call NOTANGLE_CHUNK,$@) >$@
	chmod +x $@

@

<<Additional Tests>>=
@

<<Clean temporary files>>=
rm -f $(TEST_EXEC) $(TEST_C_EXEC:%=%.c{,++}) $(TEST_EXEC_SUPPORT) \
        $(TEST_C_EXEC_SUPPORT:%=%.c{,++})
@

Note that the above rule, as well as many others from here on in,
assume that the shell being used by [[make]] expands curly brace
expressions.  If it doesn't (e.g. Debian's [[dash]] or many other UNIX
systems' Bourne Shell), invoke make with an additional
\texttt{SHELL=}\textit{/path/to/better/shell} argument.  Or, set it in
[[makefile.config.local]]:

<<makefile.config>>=
# The shell to use.  This cannot be overridden in the enviornment, and
# is inconvenient to remember specifying every time you do a "make".
# This shell must support POSIX shell variable expansion.
# This shell must support curly brace expansion
# Examples:
#   /bin/ksh (default POSIX shell on most commercial UNIX systems)
#     (some systems, like AIX, make this the default /bin/sh as well)
#   /bin/bash (default /bin/sh on most free UNIX systems)
#   /bin/zsh (my preferred shell for interactive use)
# Counterexamples:
#   Bourne Shell - used as /bin/sh on many commercial UNIX systems
#   busybox - used as /bin/sh on many embedded systems
#   dash - used as /bin/sh on Debian systems by default
# GNU make selects /bin/sh by default.
#SHELL:=/bin/sh

@

\subsection{Support Files}

In order to avoid having to duplicate function prototypes in the main code
and a header, the prototypes are generated automatically using
cproto%
\footnote{\url{http://invisible-island.net/cproto/}}%
.  Prototypes allow functions to be defined in any order, so
prototypes are generated for static functions as well, and added to
each file.  This uses the GNU make method of generating rules:
[[$(eval)]]. Makepp does not support this.

The Makepp documentation claims that [[$(eval)]] can be replaced by
[[$[...]]], but my own experiments have shown me that this does not
work: variables are not expanded correctly, and only one rule may be
generated at a time (i.e., newlines are stripped out, even if they are
explicitly added via perl code).  In fact, using [[$(...)]] works fine
for one rule, but is broken for multiple rules because, once again,
newlines are stripped out.  So, instead, it copies Makepp's code for
adding a dependency directly.  This is liable to break with different
versions of Makepp, but it at least works with version 2.0.

Another Makepp issue triggered here was the original method of
creating a temporary file, and then moving it to [[cproto.h]].  Makepp
decided to add a rule dependency on the temporary file's name, which
could not be removed or overridden.  Now it just makes sure it got
deleted on errors.

An issue with using [[cproto]] is that it does not support C++.  I was
originally not going to support C++ here, anyway, but now that I have,
I use ``universal'' [[ctags]]%
\footnote{\url{https://ctags.io}}
for C++.  It seems to work for a few samples, but it is unreliable
enough that I will continue using [[cproto]] for C.

Of course there are times when blind prototype creation is
undesirable.  For example, libraries tend to place prototypes in
headers.  Some of them may use datatypes which are not defined in
every C file.  To allow this, any prototypes detected in a header with
the exact same syntax as automatically generated, up to the arguments'
opening parenthesis, is removed from [[cproto.h]].  One which tends to
vary from mainline to mainline is [[main]], so it is always removed.

This uses the GNU [[-i]] extension for [[sed]], which may not be
supported on all UNIX systems.  If it doesn't, obtaining and using GNU
[[sed]] is free and relatively painless.  GNU [[sed]] also does not
have a hidden line length limit or other misfeatures of other [[sed]]
implementations, so I recommend using it all the time.

\lstset{language=make}
<<makefile.rules>>=
$(COFILES): cproto.h

cproto.h: $(CFILES) $(CHEADERS)
	echo >$@
	test -z "$(strip $(CFILES))" || ( \
	   if cproto -E "$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $(CPROTO_CFLAGS) -E" \
	               $(CFILES) >$@.$$$$.h; then \
	    sed -i <<Remove protos from cproto.h>>
	              $@.$$$$.h; \
	    grep -vn '^/\*' $@.$$$$.h | $(TACCMD) | \
	      sed -e '<<Strip down C prototype>>' | \
	        while IFS=: read -r l p; do \
		  test -z "$$p" && continue; \
		  fgrep "$$p" $(CHEADERS) | fgrep ":$$p" >/dev/null && \
		    sed -i -e "$${l}d" $@.$$$$.h; \
		done; \
	    mv $@.$$$$.h $@; \
	  else \
	    rm -f $@ $@.$$$$.h; false; \
	  fi )

ifeq ($(MAKEPP_VERSION),)
static_proto_rule=$1: $(1:%.o=%.$2.static_proto)
$(foreach f,$(COFILES),$(eval $(call static_proto_rule,$f,c)))
$(foreach f,$(CXXOFILES),$(eval $(call static_proto_rule,$f,c++)))
else
sub f_adddep {
  my ($targ, $dep) = &args;
  my (undef, $mkfile, $mkline) = @_;
  my $tinfo = Mpp::File::file_info $targ, $mkfile->{CWD};
  Mpp::File::set_additional_dependencies($tinfo, $dep, $mkfile, $mkline);
  return '';
}
$(foreach f,$(COFILES),$(adddep $f,$(f:%.o=%.c.static_proto)))
$(foreach f,$(CXXOFILES),$(adddep $f,$(f:%.o=%.c++.static_proto)))
endif
$(COFILES:%.o=%.c.static_proto): $(CHEADERS)
cproto.h: $(CFILES:%=%.static_proto)
$(CXXOFILES:%.o=%.c++.static_proto): $(CXXHEADERS) $(CHEADERS)
%.c.static_proto: %.c
	( \
	  notangle -R$<.static_proto $(NOWEB_ORDER) 2>/dev/null || :; \
	  mkdir cp.$$$$; touch cp.$$$$/cproto.h; trap "rm -rf cp.$$$$" 0; \
	  cproto -S -E "$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $(CPROTO_CFLAGS) -Icp.$$$$ -E" $<; \
	) >$@ || (rm -f $@; false)
%.c++.static_proto: %.c++
	( \
	  notangle -R$<.static_proto $(NOWEB_ORDER) 2>/dev/null || :; \
	  mkdir cp.$$$$; touch cp.$$$$/cproto.h; trap "rm -rf cp.$$$$" 0; \
	  $(CXX) $(CXXFLAGS) $(EXTRA_CXXFLAGS) $(CPROTO_CXXFLAGS) -Icp.$$$$ \
	     -E $< >cp.$$$$/$*.c++; \
	    ctags -x --kinds-C++=f --language-force=C++ --line-directives=yes \
	          --_xformat='%F %N%S~ %{C++.properties} %t %N%S;' cp.$$$$/$*.c++ | \
	      sed -n "s%^cp\\.$$$$/[^ ]* %%;T;/ static /{s/typename://g;p;}" | \
	        while read -r p; do \
		  fgrep "$${p%%\~*}" $(CHEADERS) $(CXXHEADERS) >/dev/null || echo "$$p"; \
		done | sed 's/.*~ *//' \
	) >$@ || (rm -f $@; false)
@

<<Generate static proto>>=
@

<<Remove protos from cproto.h>>=
-e '/^int main(/d' \
@

<<Build Source>>=
cproto.h \
@

<<Clean temporary files>>=
rm -f cproto.h{,.*.h} $(CFILES:%=%.static_proto) $(CXXFILES:%=%.static_proto)
@

<<Build Source>>=
$(CFILES:%=%.static_proto) \
@

Also, local prototypes tend to rely on locally defined data types.  To
support this, the entire include of the static prototype file can be
moved down to a location specified by a line containing only the
comment [[// static_proto]].  This is done by adding a tangle
post-processing filter to automatically generated C and C++ files.
Care must be taken when overriding [[NOTANGLE_POSTPROC]] to not
clobber this.

An old hook was to insert the chunk named after the generated header
file at the top, but that is deprecated and undocumented.  That is why
the code above tangles [[$<.static_proto]], and why no effort is made
to convert that to using [[$(TANGLE)]].

<<makefile.vars>>=
# stupid GNU make does not support setting this directly (cuts off value)
CFILE_POSTPROC = | \
  (tf="/tmp/.$$$$"; trap "rm $$tf" 0; cat >$$tf; \
   if grep '^[ \t]*// static_proto$$' "$$tf" >/dev/null; then \
     p='^[ \t]*// static_proto$$'; \
   else \
     p='^\#include "cproto.h"'; \
   fi; \
   sed -e "s%$$p%"'&\n\#include "$@.static_proto"%' $$tf)
$(NOWEB_CXXFILES) $(NOWEB_CFILES): NOTANGLE_POSTPROC += $(CFILE_POSTPROC)
@

Stripping down the C prototype requires proper handling of nested
parentheses in the prameter list.  This is done by repeatedly trying
to strip the parameter list as if there were no nested parentheses,
and if that fails, find the last pair of parentheses with no nested
parentheses and remove them. The loop should eventually clear the
parameter list of any nested parentheses, allowing the first
substitution to succeed.

\lstset{language=sed}
<<Strip down C prototype>>=
/(/{:a s/([^)]*);/(/;t;s/\(.*\)([^)]*)/\1/;ta;}
@

\subsection{Support Chunks}

In addition, C and C++ files can all take advantage of the fact that
most headers are harmless to include, so it's easier to just include
everything everywhere.  Note that some C++ headers take inordinately
long to include, though, due to extensive use of templates.  Some sort
of precompiled header strategy may help with that.  The prototypes
cannot be included in this list, because they really need to come
last.

\lstset{language=C}
<<Common C Includes>>=
#include <stdio.h>
#include <stdlib.h>
#include <stddef.h>
#include <unistd.h>
#include <string.h>
#include <ctype.h>
#include <errno.h>
@

\lstset{language=C++}
<<Common C++ Includes>>=
#include <algorithm>
#include <vector>
#include <string>
#include <iostream>
#include <fstream>
#include <map>
#include <set>
using namespace std;
extern "C" {
<<Common C Includes>>
}
#if __cplusplus < 201100L
#define unordered_map map
#define unordered_set set
#else
#include <unordered_map>
#include <unordered_set>
#endif
typedef unordered_map<string, string> string_map;
@

\lstset{language=C}
<<Common C Warning>>=
/*
  <<Common noweb Warning>>
*/
@

<<Common C Header>>=
<<Common C Warning>>
<<Common C Includes>>
#include "cproto.h"
@

<<Common C++ Header>>=
<<Common C Warning>>
<<Common C++ Includes>>
@

Some of those prototypes are hard to enable, but here are a few extra flags
to help.

\lstset{language=make}
<<makefile.vars>>=
EXTRA_CFLAGS += -D_LARGEFILE_SOURCE -D_XOPEN_SOURCE=600 \
                -D_XOPEN_SOURCE_EXTENDED=1
EXTRA_CXXFLAGS += -D_LARGEFILE_SOURCE -D_XOPEN_SOURCE=600 \
                -D_XOPEN_SOURCE_EXTENDED=1
@

I perfer to have each file indicate its source version number(s).  The
easy way to do this is with [[#ident]], but that is not supported
everywhere, and places it in a common section with other things that
many prefer stripped out (e.g. gcc places system info in the same
section, which most don't care about).  There is no real portable way
to do this, so I'll just stick it in the main data section unless
using gcc and Linux, in which case it's placed in an arbitrarily named
custom section.  Placing it in a custom section prevents the data from
being loaded into memory.

A further complication is that static variables are optimized out.
Making it global will cause link errors unless a unique global name is
used, and there is no way to generate a unique global name.
[[__FILE__]] would be the obvious choice, but it has a dot (usually)
and maybe even more garbage, and in any case it is impossible to turn
into an identifier token.  One way to prevent this is to reference the
variable elsewhere, but that runs into the same issues.  On the other
hand, gcc does provide a psuedo-reference:  the [[used]] attribute.
For now, this is only enabled for Linux as well.  The proper test
would be ``systems which support arbitrarily named sections which are
normally ignored''.  This includes, at minimum, all ELF-based systems
(so this hack doesn't even work with a.out Linux, but that hasn't been
in common use in ages).

An alternate for Microsoft-compatible compilers (including gcc on
Microsoft Windows targets) would be to reference the static string
with a global pointer declared with the same name in every file, each
using the attribute [[__declspec(selectany)]].  Only one of the
pointers will be emitted, without generating an error or warning, but
all static strings should still be present.

\lstset{language=C}
<<Static C Version ID>>=
static const char version_id[]
#if defined(__GNUC__) && defined(__linux__)
  __attribute__(( section(".note.id"), used ))
#endif
   = <<Version Strings>>;
#ifdef _WIN32 /* not that build.nw really supports win32... */
__declspec(selectany) const char *global_version_id_string = version_id;
#endif
@

<<Common C Header>>=
<<Static C Version ID>>
@

<<Common C++ Header>>=
<<Static C Version ID>>
@

<<Version Strings>>=
"$Id$\n"
@

\section{noweb Enhancements}

noweb's tangler has a simple syntax and usage, but suffers from the
lack of a few features.  Sometimes, file should not end in a newline,
but notangle always adds one.  Also, sometimes, consecutive chunks
should not be separated by newlines, but again, notangle always adds
one.

I like to consolidate code as much as possible.  I do this in C using
either subroutines or preprocessor macros.  The macros are more useful
when the number of inputs and/or outputs are large, or using a macro
is significantly faster.  The subroutines cannot even be used when
doing something like defining a large number of similar symbols, or
providing a more convenient way to define data tables.  The
preprocessor can usually accomplish what I need, but the price is that
the debugger cannot debug code defined by the preprocessor (although
gdb tries very hard to support it more recently).  In addition, there
are some languages which lack a preprocessor.  As such, the original
Web system had macros with arguments, and I would like something
similar.

I used to provide both enhancements here, in the form of a tangler, a
weaving filter, and a [[noroots]] replacement.  I then adapted the
tangler as a filter, as well, and distributed that part separately.  I
have made some major changes, and do not wish to develop this in two
different places and slightly differently in each any more, so I now
defer to [[parms.nw]] for documentation and development of the
parameterized chunks, and only develop the rather simple no-newline
filter here.

% Begin-doc nw-enh-syntax
My proposed syntax for suppressing newlines at the end of a chunk
expansion is to prefix the chunk name with a star (*).  This of course
excludes a star by itself (the default chunk).
% End-doc nw-enh-syntax

\subsection{Tangling}

The newline removal filter is simple: look for definitions with
leading stars, strip the star, and then also strip the final [[@nl]]
directive just before the [[@end code]] which terminates the chunk
definition.  In order to keep line numbers accurate, the [[@nl]] is
re-inserted after the chunk.  Since it's so simple, I just use [[sed]]
for now.

\lstset{language=sh}
<<nt-nonl>>=
#!/bin/sh
<<Common noweb Warning>>
sed -e '
  /^@defn \*./ {
    s/\*//
   :a
    n
    /^@end code$/b
    /^@nl$/!ba
   :b
    N
    /\n@nl$/bb
    /\n@end code /!ba
    s/@nl\n//;s/$/\n@nl/
  }'
@

To use these with the build system everywhere except building the
makefiles and tangling within scripts, [[$(NOTANGLE)]] should be set
globally, except of course when building itself.  Note that the
[[nt-nonl]] rule must be explicit; per-target variables do not work on
dependencies.

\lstset{language=make}
<<Build Source>>=
nt-nonl <<Filters from [[parm.nw]]>>
@

<<Filters from [[parm.nw]]>>=
nt-parm \
@

<<Adjust [[NOWEB_CXXFILES]]>>=
NOWEB_CXXFILES:=$(filter-out $(patsubst %,%.c++,<<Filters from [[parm.nw]]>>
),$(NOWEB_RAWCXXFILES))
@

<<makefile.vars>>=
NOTANGLE:=notangle -filter "./nt-nonl|./nt-parm"
NOTANGLE_DEP:=nt-parm nt-nonl
nt-nonl nt-parm: NOTANGLE:=notangle
#nt-nonl nt-parm: NOTANGLE_DEP:=
PARM_NOWEB = $(if $(wildcard parm.nw),parm.nw,\
               $(patsubst %/$(BUILD_NOWEB), %/parm.nw, \
	          $(realpath $(BUILD_NOWEB))))
PARM_EXEC = <<Filters from [[parm.nw]]>>

PARM_CXX = $(patsubst %, %.c++, $(PARM_EXEC))
@

<<Clean temporary files>>=
rm -f $(patsubst %,%.o,$(PARM_EXEC))
rm -f nt-nonl $(PARM_EXEC) $(patsubst %,%.c++,$(PARM_EXEC))
@

<<makefile.rules>>=
nt-nonl: $(NOWEB)
	$(call NOTANGLE_CHUNK,$@) >$@
	chmod +x $@
$(PARM_CXX): %.c++: $(PARM_NOWEB)
	notangle -R$@ $< >$@
@

\subsection{Weaving}

The star prefix requires stripping the star before indexing, and
re-adding it afterwards.  These tasks can be accomplished by two
filters: one for before indexing, and one for after.  It hides the
star using a new keyword.  A similar set of filters is used for
parameter processing.

<<Filters from [[parm.nw]]>>=
nw-parm-preidx nw-parm-postidx \
@

<<Build Script Executables>>=
nw-nonl-preidx nw-nonl-postidx \
@

\lstset{language=sh}
<<nw-nonl-preidx>>=
#!/bin/sh
<<Common noweb Warning>>
sed -e '
  /^@defn \*./ {
    s/\*//
    s/^/@nonlstar\n/
  }'
@

<<nw-nonl-postidx>>=
#!/bin/sh
<<Common noweb Warning>>
sed -e '
  /^@nonlstar/ {
    d
    s/ / */
  }'
@

\subsection{Other}

This is not the end of it:  [[noroots]] (noweb-2 only) needs changes
as well.  Rather than re-implement this from scratch, as I did in
previous versions of this document, I simply copy the standard script
and insert the tangling filters into the pipeline.

<<Build Source>>=
noroots-enh \
@

<<makefile.rules>>=
noroots-enh: $(BUILD_NOWEB) nt-parm nt-nonl
	sed 's%| awk%| ./nt-nonl|./nt-parm -r &%' <`which noroots` >./noroots-enh
	chmod +x ./noroots-enh
@

<<Clean temporary files>>=
rm -f noroots-enh
@

The use of [[noroots]] within the build system is pretty much limited
to finding files to extract.  For these, a special [[noroots]] is
probably unnecessary.  The [[make check]] function should be a little
more accurate, though.

\lstset{language=make}
<<makefile.rules>>=
check: noroots-enh
@

<<[[noroots]]>>=
./noroots-enh
@

\section{PDF Code Documentation}

Building the PDF code documentation is simple: just convert to \LaTeX{}
using [[noweave]], and then convert to PDF using [[pdflatex]] or
[[xelatex]].  The conversion needs to be invoked repeatedly until the
cross references (and other things, such as [[longtable]]
measurements) settle down.  Of course the figures need to be 
converted to the appropriate format, too. Rules are provided for EPS,
xfig, and dia diagrams, since they can be stored in-line in the noweb
source; additional rules can be provided for other image formats.
Once again, the use of [[$(eval)]] requires special processing for
Makepp.

In order to support attachments, the [[xelatex]] backend may need to
be informed of the file location.  This should include the current
directory by default, but a common ``security'' measure is to remove
the current directory from the default configuration file search path,
which removes it from the attachment search path as a side effect.
Rather than hard-code the backend and configuration path, a note is
added to the config file so the user can set the environment variable
appropriately (which also coincidentally overrides the security
feature, as well).

Since [[xelatex]] is intended primarily for UTF-8 support, it can also
be selected using a global UTF-8 flag.  This automatically converts
[[pdflatex]] to [[xelatex]], but as mentioned in the previous
paragraph, this can't just be a direct replacement.  Thus the note is
moved to a new configuration variable, which sets the [[xelatex]]
command only.  The default value of that variable is what works on my
system, and may work on any TeX distribution which uses kpathsea.

\lstset{language=make}
<<makefile.config>>=
# Set to non-blank if noweb files are UTF-8.
NW_UTF8=

# The LaTeX-to-PDF command to use: must be pdflatex or xelatex.
# If this is pdflatex, it will automatically be switched to
# $(UTF8LATEXCMD) when NW_UTF8 is non-blank.
LATEXCMD=pdflatex

# The UTF-8 LaTeX-to-PDF command to use (xelatex).
# To use attachments, the search path may need to be adjusted as follows:
#<<[[xelatex]] command>>
UTF8LATEXCMD:=<<[[xelatex]] command>>

@

<<[[xelatex]] command>>=
env DVIPDFMXINPUTS=.:\$$TEXMF/dvipdfmx/ xelatex
@

<<makefile.vars>>=
NOWEB_NINCL:=$(shell \
  for x in $(NOWEB); do \
    grep '^\\input{'"$$x"'}$$' $(NOWEB) >/dev/null && continue; \
    echo "$$x"; \
  done)
latexcmd = $(if $(and $(NW_UTF8),$(findstring pdflatex,$(LATEXCMD))), \
                $(UTF8LATEXCMD),$(LATEXCMD))
latexargs=
@

<<Source Code Documentation Files>>=
$(patsubst %.nw,%.pdf,$(NOWEB_NINCL)) \
@

<<makefile.rules>>=
# returns names of \input'd files
inc_f=$(shell sed -n -e 's/^\\input{\(.*\.nw\)}$$/\1/p' $1)
# returns self and any files \input'd, recursively
inc_all=$1 $(foreach f,$(call inc_f,$1),$(call inc_all,$f))
figs=$(shell sed -n -e 's/^\\includegraphics.*{\(.*\)}$$/\1/p' \
       $(call inc_all,$1))
ifeq ($(MAKEPP_VERSION),)
pdf_figrule=$(if $2,$(basename $1).pdf: $(patsubst %,%.pdf,$2))
$(foreach f,$(NOWEB_NINCL),$(eval $(call pdf_figrule,$f,$(call figs,$f))))
else
$((foreach f,$(NOWEB_NINCL),
   $(foreach d,$(call figs,$f),$(adddep $(basename $f).pdf,$d.pdf))))
endif

# attempt to pull in multi-line messages, each line starts with same char
# but remove xparse/.*define-command messages, even though they
# are the only messages known to look like that
LMSCMD = /xparse\/.*define-command/b; \
         s/LaTeX $1/&/i;T; \
	 p; \
	 s/^\([^ ]\) LaTeX.*/\1/;T; \
	 h; \
      :a N; \
         s/\(.\)\n\1/\1/;T; \
	 p;g;ba;

define run_latex
  set -x; \
  f='$<'; d=.; case '$<' in */*) d="$${f%/*}";; esac; f="$${f##*/}"; \
  cd "$$d"; \
  while $(latexcmd) -interaction batchmode $(latexargs) "$$f" >/dev/null; do \
    rerun=; \
    <<Mangle aux files>>
    test -n "$$rerun" && continue; \
    egrep $(LATEX_REDO_MSG) $*.log >/dev/null || break; \
  done
  @sed -n -e '$(call LMSCMD,info)' $*.log
  @fgrep -i overfull $*.log || :
  @fgrep -i underfull $*.log | fgrep -v vbox || :
  @sed -n -e '$(call LMSCMD,warning)' $*.log
  @# maybe removing output file is too drastic, but manual run of
  @# pdflatex is not that hard
  @if sed -n -e '/^!.*Error/,/^l./p' $*.log | grep .; then \
    rm -f $@; \
    false; \
  else \
    :; \
  fi
endef

%.pdf: %.tex
	$(run_latex)
@

<<makefile.vars>>=
# picks up changes in references or long tables
LATEX_REDO_MSG:=" ha(s|ve) changed. Rerun[ .]"

FIGS:=$(shell sed -n -e 's/^\\includegraphics.*{\(.*\)}$$/\1/p' $(NOWEB))
FIGS_EPS:=$(patsubst %, %.eps, $(FIGS))
FIGS_PDF:=$(patsubst %, %.pdf, $(FIGS))
FIGS_DIA:=$(shell $(NOROOTS) $(NOWEB) | sed -n '/\.dia>>/{s/@<<//;s/@>>//;p;}')
FIGS_XFIG:=$(shell $(NOROOTS) $(NOWEB) | sed -n '/\.fig>>/{s/@<<//;s/@>>//;p;}')
FIGS_EPS_RAW:=$(shell $(NOROOTS) $(NOWEB) | sed -n '/\.eps>>/{s/@<<//;s/@>>//;p;}')
@

<<makefile.rules>>=
$(FIGS_DIA): $(NOWEB) $(NOTANGLE_DEP)
	$(call NOTANGLE_CHUNK,$@) | gzip >$@

%.eps: %.dia
	dia -e $@ $<

$(FIGS_EPS_RAW): $(NOWEB) $(NOTANGLE_DEP)
	$(call NOTANGLE_CHUNK,$@) >$@

$(FIGS_XFIG): $(NOWEB) $(NOTANGLE_DEP)
	$(call NOTANGLE_CHUNK,$@) >$@

%.eps: %.fig
	fig2dev -L eps $< $@

%.pdf: %.eps
	epspdf $< 2>/dev/null || epstopdf $<
	# epstopdf gives no return code on errors, so:
	pdfinfo $@ >/dev/null
@

<<Clean temporary files>>=
rm -f $(NOWEB:%.nw=%.{log,aux,out,toc,lot,lof,tex,idx,ind,ilg,bbl})
rm -f $(FIGS_XFIG) $(FIGS_DIA) $(FIGS_EPS_RAW) $(FIGS_PDF)
@

During the \LaTeX{} processing loop, steps may need to be taken to
update auxiliary files for the next run.  Since they are common,
[[bibtex]] and [[makeindex]] are run automatically if needed.  In
addition, a variable can be set to run things before these, and the
code chunk can be extended to run things afterwards.  Setting
[[$rerun]] to non-empty will cause a delayed rerun, and [[continue]]
will cause an immediate re-run.

<<Mangle aux files>>=
<<Mangle aux files before bbl and ind>>
fgrep '\citation' $*.aux && fgrep '\bibdata' $*.aux && bibtex $*; \
if [ -f $*.idx ]; then \
  makeindex $(INDEX_FLAGS) -i < $*.idx > $*.ind.$$$$; \
  cmp $*.ind $*.ind.$$$$ >/dev/null 2>&1; res=$$?; \
  mv $*.ind.$$$$ $*.ind; \
  test $$res -eq 0 || rerun=y; \
fi; \
@

<<Mangle aux files before bbl and ind>>=
\
@

\subsection{Pretty Printing Wrapper}

Or at least it should be simple, but some additional adjustments of the
standard noweb process need to be made.  For one thing, noweb does not
pretty-print code chunks natively, so pretty-printing the code chunks
requires a bit of work.  A generic code chunk formatter filter for noweb is
used for this.  Like all noweb filters, it just sucks in the marked up form
of the noweb source and spits out the same, with modifications.  The
language is tracked so that the formatter can behave differently depending
on language.

This used to be perl, but then perl, as I should probably have
expected, decided to declare all my code illegal.  I have switched to
C++, reusing some of the code from \texttt{parm.nw}.  I suppose one of
these days, I should switch all of the sed code as well.  There really
isn't any advantage to using scripting languages.

<<C Build Executables>>=
nwweavefilt \
@

\lstset{language=C++}
<<nwweavefilt.c++>>=
<<Common C++ Header>>
<<Extra [[nwweavefilt]] Includes>>

int main(int argc, char **argv)
{
  string line;
  <<Initialize [[nwweavefilt]]>>

  while(getline(cin, line).good() || line.size()) {
    <<Filter lines from noweb token stream>>
    cout @<< line @<< '\n'; // print most lines
    <<Process lines from noweb token stream>>
  }
  return 0;
}
@

Language is determined by scanning for language settings from the listings
package, which may be commented out if the listings package isn't currently
being used.

<<Common C Includes>>=
#include <regex.h>
@

<<nwbuild Regex Macros>>=
#define ec_regcomp(re, pat, flag) do { \
  int re_err = regcomp(&re, pat, flag|REG_EXTENDED); \
  if(re_err) { \
    char ebuf[80]; \
    regerror(re_err, &re, ebuf, sizeof(ebuf)); \
    perror(ebuf); \
    exit(1); \
  } \
} while(0)
#define line_match(rm) line.substr(rm.rm_so, rm.rm_eo - rm.rm_so)
@

<<Initialize [[nwweavefilt]]>>=
string lang = "txt";
regex_t lang_re;
regmatch_t lang_rem[3];
<<nwbuild Regex Macros>>
// [^{] is to prevent self-match
ec_regcomp(lang_re, "lstset\\{language=(.*\\])?([^}]*)[^{]*", 0);
@

<<Process lines from noweb token stream>>=
if(!regexec(&lang_re, line.c_str(), 3, lang_rem, 0)) {
  lang = line_match(lang_rem[2]);
  for(auto c = lang.begin(); c != lang.end(); c++)
    if(isupper(*c)) // will not work on Unicode, even with <locale>
      *c = tolower(*c);
}
@

A code chunk begins with the chunk name, which can be ignored.  After that,
all lines are sucked in as part of the code, until the end of the chunk.

<<Process lines from noweb token stream>>=
if(!line.compare(0, 5, "@defn")) {
  // skip @nl
  if(!getline(cin, line).good())
    break;
  <<Filter lines from noweb token stream>>
  cout @<< line @<< '\n';
  // accumulate code here
  string code;
  <<Accumulate code for highlighting>>
  <<Highlight accumulated code>>
}
@

Chunk references ([[@use]]) are replaced with a marker in the code that
should work with most dumb code formatters.  The marker is later replaced by
the [[@use]] in the output stream.  The marker can be any unique string that
isn't likely to appear in code.  No matter what is chosen, some bits of code
will probably need to be modified to ensure the pattern never appears.  For
now, this uses tripled caret ([[^]]) characters.  Other characters that
might've worked are doubled or tripled at signs ([[@]]), tildes ([[~]]), or
backticks ([[`]]).  Index definitions are moved to the bottom of the chunk
rather than being marked and replaced like [[@use]]; this removes the
immediate context, but also eliminates highlighter confusion.

<<Accumulate code for highlighting>>=
vector<string> use; // accumulate @use here
string end; // accumulate index defns here
while(getline(cin, line).good() || line.size()) {
  <<Filter lines from noweb token stream>>
  if(!line.compare(0, 6, "@text "))
    code += line.substr(6);
  else if(line == "@nl")
    code += '\n';
@

<<Initialize [[nwweavefilt]]>>=
regex_t index_re;
ec_regcomp(index_re, "^@index ((local|)defn|nl)", REG_NOSUB);
@

<<Accumulate code for highlighting>>=
  else if(!regexec(&index_re, line.c_str(), 0, NULL, 0))
    end += line + '\n';
@

<<Initialize [[nwweavefilt]]>>=
const char marker[] = "^""^^";
regex_t ref_re;
ec_regcomp(ref_re, "^@((nwparm|)use|index|xref) ", REG_NOSUB);
@

<<Accumulate code for highlighting>>=
  else if(!regexec(&ref_re, line.c_str(), 0, NULL, 0)) {
    code += marker;
    use.push_back(line);
  } else if(!line.compare(0, 4, "@end")) {
    use.push_back(end + line);
    break;
  } else if(!line.compare(0, 12, "@nwparmcall ")) {
    code += marker;
    string line2; // next line is always @use
    getline(cin, line2);
    use.push_back(line + '\n' + line2);
  } else
    cerr << " ????? " << line << " ??????\n";
}
@

The prettifier is passed in on the command line; it takes one (additional)
argument - the name of the current language.  It takes raw code (with
markers for chunk references), and returns pretty code.  Steps must be taken
by the prettifier to ensure that the markers remain intact.  The output of
the prettifier is placed in the noweb stream as [[@literal]]
lines%
\footnote{Ramsey says I should write a new back end for this purpose,
and has deprecated the [[@literal]] directive for this reason. 
Writing an entire back end just to reformat the code chunks a bit
seems extreme, though.  As of 2.11b, the directive still works.}%
; markers are replaced by the appropriate [[@use]] line from [[use]].

Bidirectional pipes are not supported natively by C++ (unlike perl),
but it's not that hard to implement (non-portably, but I don't care
right now).

<<Initialize [[nwweavefilt]]>>=
// first arg is name of highlighting helper
if(argc-- < 2)
  exit(1);
const char *filt = *++argv;
@

<<Highlight accumulated code>>=
int pfdi[2], pfdo[2];
pid_t pid;
if(pipe(pfdi) || pipe(pfdo) || (pid = fork()) < 0) {
  perror("filter pipe");
  exit(1);
}
if(!pid) {
  close(pfdi[1]);
  close(pfdo[0]);
  dup2(pfdi[0], 0);
  dup2(pfdo[1], 1);
  execl(filt, filt, lang.c_str(), NULL);
  perror("pipe exec");
  exit(1);
}
close(pfdi[0]);
close(pfdo[1]);
<<Adjust accumulated code>>
if(write(pfdi[1], code.data(), code.size()) != (int)code.size() || close(pfdi[1])) {
  perror("highlighter pipe write");
  exit(1);
}
// As if using POSIX file descriptors wasn't bad enough, there is no
// portable way to use them in C++, even via fdopen().  Seems like a
// major stupidity on the part of the C++ language designers.
// Since I don't want to pull in my mfgets(), this has to work for
// your C++ environment.  Currently, I only support GNU libstdc++.
__gnu_cxx::stdio_filebuf<char> fbuf(pfdo[0], std::ios::in);
istream reader(&fbuf);
int useno = 0;
while(getline(reader, line).good() || line.size()) {
  unsigned off = 0, poff = 0;
  while((off = line.find(marker, poff)) < line.size()) {
    if(poff < off)
      cout << "@literal " << line.substr(poff, off - poff) << '\n';
    cout << use[useno++] << '\n';
    poff = off + sizeof(marker) - 1;
  }
  if(line.size() > poff)
    cout << "@literal " << line.substr(poff) << '\n';
  cout << "@nl\n";
}
fbuf.close();
cout << use[useno] << '\n'; // the end
waitpid(pid, NULL, 0);
@

<<Extra [[nwweavefilt]] Includes>>=
#include <ext/stdio_filebuf.h>
extern "C" {
#include <sys/wait.h>
}
@

For UTF-8 input, tabs are once again stripped (incorrectly) by the
final weave stage.  In order to prevent this, at least for code
chunks, they are stripped here, instead.  The main text cannot be
processed this way without reconstructing the text from the pipeline
as above, so verbatim text in the documentation will have to be
changed manually.  Note that correction here only means cosideration
of UTF-8 code points as single characters (rather than up to 3), and
does not take grapheme clusters or double width characters into
consideration.   Doing the latter would require full Unicode table
integration.

<<Initialize [[nwweavefilt]]>>=
const char *utf8_mode = getenv("NW_UTF8");
if(utf8_mode && !*utf8_mode)
  utf8_mode = NULL;
@

<<Adjust accumulated code>>=
if(utf8_mode) {
  unsigned off = 0, last_cr = 0;
  while((off = code.find_first_of("\n\t", off)) < line.size()) {
    if(line[off] == '\n')
      last_cr = off + 1;
    else {
      unsigned len;
      for(len = 0, off = last_cr; line[off] != '\t'; off++)
        len += (uint8_t)line[off] < 0x80 || (uint8_t)line[off] >= 0xc0;
      line[off] = ' ';
      if((len + 1) % 8) {
        line.insert(off + 1, "       ", 8 - (len + 1) % 8);
	off += (len + 1) % 8;
      }
    }
    off++;
  }
}
@

\subsection{Pretty Printing with a highlighter and [[framed]]}

One method of highlighting is to use an external highlighting program.
Two related programs which work are [[highlight]]%
\footnote{\url{http://www.andre-simon.de}}
and GNU [[source-highlight]]%
\footnote{\url{http://www.gnu.org/software/src-highlite/}}%
.  An additional wrapper script needs to be made to invoke the
highlighter and massage the output a bit.

\lstset{language=sh}
<<Build Script Executables>>=
latexhl \
@

<<latexhl>>=
#!/bin/sh
<<Common noweb Warning>>
@

Since multiple highlighters are supported, the first thing to do is
figure out which one to use.  In previous versions of this system,
only [[highlight]] version 2 was supported, so it is the preferred
highlighter, if available.  Since this needs to take advantage of a
few low-level features, compatibility is not guaranteed.  For
[[highlight]] in particular, a check is made for major version 2 or 3
(early versions of 2 are not supported, but that is not checked).  For
either highlighter, it is difficult to come up with tests that ensure
that this will actually work.  If it doesn't, changes will need to be
made here.

Rather than checking this for every chunk, the overall wrapper sets an
environment variable to indicate which version to use.  It also does
any setup steps that do not need to be repeated for every chunk.

If the user wants, the check can be skipped entirely.  Of course if
the user setting does not match what the system actually has, there
will be plenty of error messages while this runs.

\lstset{language=make}
<<makefile.config>>=
# if non-blank, specify which highlighter to use:
#  highlight-2  named highlight, acts like highlight-2.x
#  highlight-3  named highlight, acts like highlight-3.x
#  source-highlight named source-highlight (probably 3.x)
# if blank, autodetect
HLPROG_TYPE=

@

\lstset{language=sh}
<<Prepare for weave>>=
<<Check highlighter>>
@

<<Check highlighter>>=
if [ -z "$HLPROG_TYPE" ]; then
  if type highlight >/dev/null 2>&1; then
    v=`highlight --version 2>&1 | fgrep highlight`
    v="${v##*version }"
    v="${v%%.*}"
    export HLPROG_TYPE=highlight-$v
    if [ $HLPROG_TYPE != highlight-2 -a $HLPROG_TYPE != highlight-3 ]; then
      # echo "Only version 2 or 3 of highlight supported." >&2
      # exit 1
      HLPROG_TYPE=
    fi
  elif type source-highlight >/dev/null 2>&1; then
    export HLPROG_TYPE=source-highlight
  else
    export HLPROG_TYPE=
  fi
else
  case "$HLPROG_TYPE" in
    highlight-[23]) type highlight >/dev/null 2>&1 || HLPROG_TYPE= ;;
    source-highlight) type source-highlight >/dev/null 2>&1 || HLPROG_TYPE= ;;
    *) HLPROG_TYPE= ;;
  esac
  if [ -z "$HLPROG_TYPE" ]; then
    echo "Invalid highlighter specified in HLPROG_TYPE" >&2
    exit 1
  fi
fi
@

Next, any preparations before running the source highlighter should be
made, and the highlighter itself should be called.

<<latexhl>>=
<<Do per-chunk highlighting preparation for [[latexhl]]>>
case $HLPROG_TYPE in
  highlight-2)
    highlight <<[[highlight]]-2 options for [[latexhl]]>> | \
      sed -n -e '<<Adjust [[highlight]]-2 \LaTeX{}>>' ;;
  highlight-3)
    highlight <<[[highlight]]-3 options for [[latexhl]]>> | \
      sed -n -e '<<Adjust [[highlight]]-3 \LaTeX{}>>' ;;
  source-highlight)
    <<Enter [[source-highlight]] data dir>>
    source-highlight <<[[source-highlight]] options for [[latexhl]]>> | \
      sed -n -e '<<Adjust [[source-highlight]] \LaTeX{}>>' ;;
esac
@

To make the highlighter do its job, changing the block's background
color needs to be supported as well.  To do this, a box-type
environment that sets background colors and can span pages is needed.
The [[framed]] package comes close, but requires a bit of work to get
the chunk to associate strongly with its header line.  Without moving
the code margin outside of the frame, it also extends the color box
too far to the left.  In the interest of simplicity, though, it will
be used instead of writing an equivalent replacement.  The background
color itself is the named color [[bgcolor]].

One problem to watch out for with [[framed]] is that if a code chunk
starts at the bottom of a page, it is sometimes not broken at
subsequent page boundaries.  These chunks can be detected by scanning
the \LaTeX{} log for \texttt{Overfull $\backslash$vbox} messages.
This can be cured by inserting an explicit [[\break]] before the
affected code chunk.  All [[\break]] commands should be revisited
regularly to make sure they are still necessary as the document
changes.  Of course this could also be taken as a sign that it is time
to break the code chunk into smaller chunks, but sometimes breaking
things up just for the sake of making them smaller just makes things
messier.

<<Do per-chunk highlighting preparation for [[latexhl]]>>=
case "$HLPROG_TYPE" in
  highlight-[23]) echo '\begin{framed}\advance\leftskip-\codemargin%' ;;
esac
@

<<latexhl>>=
case "$HLPROG_TYPE" in
  highlight-[23]) echo '\end{framed}%' ;;
esac
@

<<Prepare for [[latexhl]]>>=
if [ $HLPROG_TYPE = source-highlight ]; then
printf %s\\n \
  'include "latexcolor.outlang"
   nodoctemplate
     "\begin{framed}\advance\leftskip-\codemargin%
"
     "
\end{framed}%
"
   end' >$HLDIR/mylatex.outlang
fi
@

\lstset{language=TeX}
<<Framed Code Preamble>>=
\usepackage{framed}
\setlength\FrameSep{0pt}
\setlength\FrameRule{0pt}
% nwdoit is the code header & flag indicating pass <= 2
% nwdoita is a flag indicating pass #1
\let\nwdoit\relax\let\nwdoita\relax
\def\FrameCommand#1{%
  \ifx\nwdoit\relax%
    % 3rd+ pass is subsequent pages - no code header
    \hskip\codemargin\colorbox{bgcolor}{#1}%
  \else%
    % 1st pass is invisible sizing pass, but only on older TeX.  How to tell?
    % 2nd pass is 1st page
    \vbox{\vbox{\nwdoit}\colorbox{bgcolor}{#1}}%
    \ifx\nwdoita\relax%
      \global\let\nwdoit\relax%
     % comment/uncomment next line if TeX newer/older
%    \else%
      \global\let\nwdoita\relax%
    \fi%
  \fi}
\renewenvironment{framed}%
  {\MakeFramed {\advance\hsize-2\fboxsep\advance\hsize-\codemargin%
   \FrameRestore}}%
  {\endMakeFramed}
@

\lstset{language=sed}
<<Insert code header into frame>>=
/\\nwbegincode/s/\(sublabel{[^}]*}\)\(.*\)/\1\\def\\nwdoita{}\\def\\nwdoit{\2}%/
@

The actual colors come from a style file in [[highlight]]; there is
actually no need to worry about which style to use when highlighting
the chunks.  This style file can be extracted by highlighting a dummy
file, and parsing the output.  If the background color is white, it is
converted to a very light gray to ensure that code always looks a
little different.

\lstset{language=sh}
<<Prepare for [[latexhl]]>>=
case "$HLPROG_TYPE" in
  highlight-[23])
    highlight ${HL_THEME:+-s $HL_THEME} -S c --out-format=latex -I </dev/null | \
      egrep 'newcommand|definecolor' | \
        sed '/bgcolor/s/{1,1,1}/{.98,.98,.98}/' > latexhl.sty.$$
    mv latexhl.sty.$$ latexhl.sty
esac
@

<<Clean temporary files>>=
rm -f latexhl.sty{,.*}
@

On the other hand, [[source-highlight]] inserts the colors directly.
However, since the colors don't change from chunk to chunk, it is
better to extract them once, and use color names instead.  This is
done by creating an appropriate language definition and source file to
get all possible entity types. In addition, [[source-highlight]] uses
color names directly from HTML CSS files, which include invalid
hexadecimal color specifiers.  These need to be converted to valid
decimal RGB triples.  Some of the color names are not directly
provided either.  This is more difficult to correct, since there are
probably hundreds of color names that work in HTML but not in
\LaTeX{}, but a few known ones can be corrected.

<<Prepare for [[latexhl]]>>=
if [ $HLPROG_TYPE = source-highlight ]; then
  echo 'onestyle "\hl$style{$text}"' >> $HLDIR/mylatex.outlang
  <<Get all [[source-highlight]] color names>>
  # outlang def for background color only, in same format as latexcolor
  echo 'nodoctemplate
        "\definecolor{bgcolor}\textcolor{$docbgcolor}"
	""
	end' > $HLDIR/mybg.outlang
  (
    cd $HLDIR
    source-highlight ${HL_THEME:+--style-css-file=$HL_THEME.css} -f latexcolor \
                     --lang-def=source-hl-elts.lang < source-hl-elts | \
      sed -n -e '/^\\mbox{}/{
                   s/\\mbox{}//;s/ \\\\//;s/\$\\_\$//g;
                   s/\(.*{\)\([^}]*\)\(}*\)$/\\newcommand{\\hl\2}[1]{\1#1\3}/p;
		   s/^[^\\].*/\\newcommand{\\hl&}[1]{#1}/p;}'
    # fa is ff * .98
    source-highlight --lang-def=/dev/null --outlang-def=mybg.outlang \
                     ${HL_THEME:+--style-css-file=$HL_THEME.css} </dev/null | \
      sed 's/#ffffff/#fafafa/i;s/white/#fafafa/i'
  ) | <<Convert HTML color codes to \LaTeX{}>> | \
     sed -e '/^\\definecolor/{
               s/\\textcolor//
	       # convert textcolor rgb to definecolor rgb
	       s/\[rgb\]/{rgb}/;t
	       # convert textcolor named to definecolor named
	       s/{/{named}{/
	     }' > latexhl.sty.$$
  mv latexhl.sty.$$ latexhl.sty
fi
@

<<Get all [[source-highlight]] color names>>=
# need to sort in reverse order because source-highlight uses first match
# rather than longest match
for x in $(source-highlight --lang-list | cut -d\  -f3 | sort -u); do
  source-highlight --show-lang-elements=$x
done | sort -ur | fgrep -v 'named subexps' > $HLDIR/source-hl-elts
# lang def simply makes a type its name
sed 's/.*/& = "&"/' < $HLDIR/source-hl-elts > $HLDIR/source-hl-elts.lang
@

There are two types of colors in the style files: numeric colors and
named colors.  The named colors are unpredictable, so only a few
select colors are converted.

<<Convert HTML color codes to \LaTeX{}>>=
sed -e '<<Convert HTML color names to [[xcolor]] names>>' | \
@

<<Convert HTML color names to [[xcolor]] names>>=
s/navy/Navy/;s/silver/Silver/;s/maroon/Maroon/
@

In order to convert HTML colors to \LaTeX{} colors, the HTML color
needs to be parsed and converted to decimal.  Awk can do that, as long
as it is POSIX awk.  GNU awk does not work in non-POSIX mode because
it does not support converting hexadecimal strings to numbers.  To
enable POSIX mode, either the non-portable [[--posix]] option can be
given, or the slightly more portable [[POSIXLY_CORRECT]] environment
variable can be set.

<<Convert HTML color codes to \LaTeX{}>>=
POSIXLY_CORRECT=1 awk '<<Convert HTML color codes to \LaTeX{} using [[awk]]>>'
@

\lstset{language=awk}
<<Convert HTML color codes to \LaTeX{} using [[awk]]>>=
# change color styles from HTML color IDs to rgb color triples
# save RE in variable to avoid having to repeat
# note that this assumes 6-digit hex string, not 3-digit.
BEGIN { re1 = "\\\\textcolor\\{#[0-9a-fA-F]{6}\\}"
        re2 = "\\\\colorbox\\{#[0-9a-fA-F]{6}\\}" }
# print non-matching lines first
$0 !~ re1 "|" re2 { print }
# helper to convert to decimal triple
function todec3(start,   val, rv) {
    val = "0x" substr($0, RSTART + start, 2);
    rv = (val / 255) ",";
    val = "0x" substr($0, RSTART + start + 2, 2);
    rv = rv (val / 255) ",";
    val = "0x" substr($0, RSTART + start + 4, 2);
    rv = rv (val / 255);
    return rv
}
$0 ~ re1 "|" re2 {
  # loop to match all unconverted colors in this line
  while(match($0, re1)) {
    # capture groups would be real handy now
    # but instead, we extract each digit pair manually and append to
    # replacement string (ns)
    ns = "\\textcolor[rgb]{" todec3(12) "}";
    # convert string to be replaced to a regex by escaping \ { }
    nsre = "\\" substr($0, RSTART, RLENGTH);
    gsub("[{}]", "\\\\&", nsre);
    # globally replace all occurrences of this color
    gsub(nsre, ns);
  }
  while(match($0, re2)) {
    ns = "\\begingroup\\definecolor{boxcolor}{rgb}{" todec3(11) "}" \
         "\\colorbox{boxcolor}";
    # match the entire box so \endgroup can be placed after it
    match($0, re2 "\\{([^}\\\\]|\\\\[^a-zA-Z]|\\\\[a-z]+\\{\\})*\\}");
    $0 = substr($0, 1, RSTART - 1) ns \
         substr($0, RSTART + 18, RLENGTH - 18) "\\endgroup" \
	 substr($0, RSTART + RLENGTH);
  }
  print
}
@

A few syntax changes need to be made to the default language
definitions.  First, a language definition for highlighting sed
comments is provided.  As a special hack, [[#include]] is detected and
not considered a comment (this is due to my use of HDF in a particular
project, which uses this format).  Second, the C language gets
multi-line macros highlighted like regular source (already there in
[[source-highlight]]), and support for 64-bit integer constants (already
there in [[highlight]]-3).  Finally, data types are being highlighted
using the listings package's [[moreemph]] directive, so these
directives are extracted and inserted into the keyword highlight list.
This is actually done by using the name of the chunk containing all of
these data types, so they are properly combined ahead of time.

\lstset{language=sh}
<<Prepare for weave>>=
case "$HLPROG_TYPE" in
  source-highlight)
    # sed language only supports comments
    # #include supported for HDF
    echo "comment start '#(?!include\\W)'" > $HLDIR/sed.lang
    # c/c++ language needs a lot of help
    cat <<"EOF" >$HLDIR/cmods.lang
      subst number =
'\<[+-]?((0x[[:xdigit:]]+)|(([[:digit:]]*\.)?
[[:digit:]]+([eE][+-]?[[:digit:]]+)?))[uUlL]*\>'
      type =
EOF
    printf %s '<<Known Data Types>>' | egrep -v '^[[:space:]]*(%|$)' | \
      sed -e 's/[[:space:]]//g;s/,/|/g;s/%.*//;s/^/"/;s/|$/"/;$!s/$/,/' \
           >>$HLDIR/cmods.lang
    for x in c cpp; do
      cat <<EOF >$HLDIR/my$x.lang
        include "$x.lang"
	include "cmods.lang"
EOF
    done
    # don't override keywords & types with this broken regex
    touch $HLDIR/clike_vardeclaration.lang
    # don't highlight \x differently in strings
    cat <<"EOF" >$HLDIR/c_string.lang
      environment string delim "\"" "\"" begin
        string = '\\.'
      end
      environment string delim "'" "'" begin
        string = '\\.'
      end
EOF
    ;;
  highlight-2)
    mkdir -p $HLDIR/langDefs
    # sed language only supports comments
    # #include supported for HDF
    echo '$SL_COMMENT=regex(^\s*(#(?!include\W)))' > $HLDIR/langDefs/sed.lang
    # c language needs a lot of help
    hl_kw="`printf %s '<<Known Data Types>>' | egrep -v '^[[:space:]]*(%|$)' | \
            sed -e 's/[[:space:]]//g;s/,/ /g;s/%.*//;\$!s/\$/\\\\/'`"
    langdir=`highlight --print-config | sed -n '/Language def/s/.*: *//p'`
    sed < ${langdir:-/usr/share/highlight/langDefs}/c.lang -e '
      # remove cont to fully highlight multi-line macros
      s/.*CONTINUATIONSYMBOL.*//
      # fix preproc expression
      /DIRECTIVE/s/#/regex(^\\s*(#))/
      # add some kws to kwb using old listings package moreemph decls
      /KW_LIST(kwb)/a\'"${hl_kw}"'
      # same thing, but newer versions of highlight
      /KEYWORDS(kwb)/a\'"${hl_kw}"'
      # change number RE to include UL, LL, ULL, etc.
      $a\$DIGIT=regex((?:0x|0X)[0-9a-fA-F]+|\\d*[.]?\\d+(?:[eE][\\-+]\\d+)?[lLuU]*)
      ' > $HLDIR/langDefs/c.lang
    ;;
  highlight-3)
    # sed language only supports comments
    # #include supported for HDF
    echo 'Description="sed"
          Keywords={}
          Comments={
            {Block=false, Delimiter={[[^\s*(#(?!include\W))]]}}
	  }' > $HLDIR/sed.lang
    # c/c++ language needs a lot of help
    # note that hl_lang_dir (actually HL_LANG_DIR) just points
    # to the dir the config file is in (useless)
    langdir=`highlight --print-config | sed -n '/Language def/s/.*: *//p'`
    echo "dofile('${langdir}c.lang')" > $HLDIR/myc.lang
    cat @<<"EOF" @>> $HLDIR/myc.lang
      -- fix preproc expression and remove cont to highlight multi-line macros
      PreProcessor = { Prefix=[[^\s*(#)]] }
      --add some kws to ID=2 using old listings package moreemph decls
      for i, v in ipairs({
EOF
    printf %s '<<Known Data Types>>' | egrep -v '^[[:space:]]*(%|$)' | \
      sed -e 's/[[:space:]]//g;s/,/","/g;s/%.*//;s/$/"/;s/^/"/;s/""//;' \
         >> $HLDIR/myc.lang
    cat @<<"EOF" @>>$HLDIR/myc.lang
                         }) do
        table.insert(Keywords[2].List, v)
      end
EOF
    ;;
esac
@

Note that in order to suppress [[source-highlight]]'s
[[clike_vardeclaration.lang]], the data library directory needs to be
changed to [[$HLDIR]], and the current directory needs to be the old
data dirctory.  There is no way to actually determine the latter, so
I'll just guess for now.  I should probably add a configuration
parameter for that.

<<Enter [[source-highlight]] data dir>>=
d="`which source-highlight`"
d="${d%/source-highlight}"
d="${d%/bin}"
cd "$d"/share/source-highlight 2>/dev/null || \
cd /usr/local/share/source-highlight 2>/dev/null || \
cd /usr/share/source-highlight
@

<<[[source-highlight]] options for [[latexhl]]>>=
--data-dir=$HLDIR \
@

To actually invoke the highlighter, the language specified on the
command line must be passed in.  If it is one of the overrides, the
override should be used instead.  For [[highlight]]-2, this is done using
the [[-E]] option.  For the other two, it is done by specifying the
language definition file directly.  Since these require checking the
language argument anyway, they can also change the language to a
different one if necessary.  I have not tested all languages available
to the [[listings]] package, which is used to determine legal language
names, but any which mismatch what is supported by the external
highlighter should be listed here.

<<Do per-chunk highlighting preparation for [[latexhl]]>>=
<<Do generic per-chunk highlighting preparation>>
@

<<Do generic per-chunk highlighting preparation>>=
if [ source-highlight = $HLPROG_TYPE ]; then
  case $1 in
    <<Translate listings source type to source-highlight source type>>
    *) larg="-s $1"
  esac
fi
@

<<Translate listings source type to source-highlight source type>>=
[cC]) larg="--lang-def=myc.lang" ;;
[cC]++) larg="--lang-def=mycpp.lang" ;;
sed) larg="--lang-def=sed.lang" ;;
make) larg="-s makefile" ;;
@

<<[[source-highlight]] options for [[latexhl]]>>=
--outlang-def=mylatex.outlang --failsafe $larg
@

<<[[highlight]]-2 options for [[latexhl]]>>=
--out-format=latex -f -S $1 -E $HLDIR
@

<<Do generic per-chunk highlighting preparation>>=
if [ highlight-3 = $HLPROG_TYPE ]; then
  case $1 in
    <<Translate listings source type to highlight-3 source type>>
    *) larg="-S $1"
  esac
fi
@

<<Translate listings source type to highlight-3 source type>>=
[cC]|C++) larg="--config-file=$HLDIR/myc.lang" ;;
sed) larg="--config-file=$HLDIR/sed.lang" ;;
@

<<[[highlight]]-3 options for [[latexhl]]>>=
--out-format=latex -f $larg
@

Finally, the output needs to be adjusted.  All three need a strut on
the first line to ensure consistent appearance.  All three generate a
final blank line, which is just excess whitespace.  All three encode
the triple-hat sequence which is used to indicate chunk references as
\LaTeX{}, which must be decoded.  All three also add unnecessary
garbage at the end of each line.  The [[highlight]] program also
terminages \LaTeX{} commands with a space rather than curly braces,
which has caused me problems in some places.  The last line should
also be terminated by a percent sign to remove excess whitespace.

\lstset{language=sed}
<<Adjust [[source-highlight]] \LaTeX{}>>=
# remove blank line at end
/^\\mbox{}$/d
# detexify triple-^
s/\(\\textasciicircum{}\)\1\1/^''^^/g
# remove unnecessary EOL stuff
s/ \\\\$//
# fix _ in command names
s/\(\\hl[a-z]*\)_/\1/g
# add strut to first line
2s/$/\\strut{}/
# add a % to 2nd-to-last line
x;${s/$/%/;G};1!p
@

<<Adjust [[highlight]]-2 \LaTeX{}>>=
# removes 1st 2 lines (setting font & spacing - already done)
1,2d
# remove last few lines (just extra whitespace & restoring font/spacing)
/^\\mbox{}$/,$d
# since multiple lines were deleted, just start sed over so $ works right
p' | sed -e '
# detexify triple-^
s/\(\\textasciicircum \)\1\1/^''^^/g
# remove unnecessary fill on every line
s/\\hspace\*{\\fill}\\\\$//
# use {} instead of single space to terminate TeX cmds
s/\(\\[a-z][a-z]*\) /\1{}/g
# add strut to first line
1s/$/\\strut{}/
# remove last CR by tacking on % to last line
$s/$/%/
@

<<Adjust [[highlight]]-3 \LaTeX{}>>=
<<Adjust [[highlight]]-2 \LaTeX{}>>
@

\subsection{Pretty Printing with [[listings]]}

Or, since the commands from the listings package are used anyway, the
listings package could be used instead.  To do this, [[nwbegincode]] and
[[nwendcode]] need to be changed to the listings package equivalents. Then,
all commands within the code are escaped using ctrl-G, a character not
likely in text and easily specified within sed.  The escape character must
be activated using [[\lstset{escapechar=^^G}]] in the preamble.

<<Change [[nwcode]] to [[listings]]>>=
s/^\(.*\)\(\\nwbegincode{[0-9]*}\)\(.*\)$/\1\\begin{lstlisting}\n\a\3\a/
/\\def\\nwend/!s/^\(.*\)\(\\nwendcode\)\(.*\)$/\\end{lstlisting}\1\3/
@

\lstset{language=sh}
<<Build Script Executables>>=
addlistings \
@

<<addlistings>>=
#!/bin/sh
<<Common noweb Warning>>

# use the ctrl-g escape (\a = ann = bell)
sed -e 's/\^\^\^/\a&\a/g
        # insert dummy escapes into potential listing ender tokens
        s/\\end{lstlisting}/\\\a\aend{lstlisting}/g
        s/\\nwbegincode/\\\a\anwbegincode/g
        s/\\nwendcode/\\\a\anwendcode/g'
@

\lstset{language=TeX}
<<Preamble Adjustments for [[listings]]>>=
% Use ^G for escape char - not likely in document
\catcode`\^^G=13
\lstset{escapechar=^^G}
@

The appropriate language definitions need to be loaded at the end of the
\LaTeX{} preamble.  This includes comment highlighting for [[sed]] and
selecting the appropriate default sublanguages.

<<Preamble Adjustments for [[listings]]>>=
% Stuff for the listings package
% note: lstloadlanguages seems broken in some versions
%\lstloadlanguages{C,sh,[gnu]make,[LaTeX]TeX} %,sed
% Most of sed can't be expressed in the listings package - no regexes
\lstdefinelanguage{sed}{morecomment=[f]\#}
\lstdefinelanguage{txt}{}
\lstset{defaultdialect=[gnu]make,defaultdialect=[LaTeX]TeX}
%The supplied HTML def has no dialect, so setting this creates infinite loop
%\lstset{defaultdialect=[ext]HTML}
\lstset{language=C}
@

A few additional adjustments need to be made for those languages.  In
particular, some keywords are missing, and some standard data types are
missing.  Also, the data types provided by the used libraries need to be
highlighted.  The data type adjustments are applied when using [[highlight]]
as well, since the [[moreemph]] directives are parsed out.

<<Preamble Adjustments for [[listings]]>>=
% HTML doesn't bold enctype
\lstdefinelanguage[ext]{HTML}[]{HTML}{morekeywords={enctype,public,xml}}
% Known data types
\lstset{moreemph={<<Known Data Types>>
}}
@

<<Known Data Types>>=
% C
regex_t,regmatch_t,FILE,va_list,pollfd,pthread_t,size_t,ssize_t,%
pthread_mutex_t,pthread_cond_t,uid_t,gid_t,mode_t,pid_t,time_t,socklen_t,%
sockaddr_in,sockaddr,off_t,utimbuf,%
@

Since there are not really any styles for this package, the
highlighter program's styles can be used instead.  If they are
present, they need to be converted to the appropriate directives.  If
there is no highlighter program available, a dummy [[latexhl.sty]]
needs to be created as well.  The format of a listings style command
is a declaration rather than a command, so the style file needs to be
converted as well.

\lstset{language=sh}
<<Prepare for [[latexhl]]>>=
if [ -z "$HLPROG_TYPE" ]; then
  echo >latexhl.sty
fi
@

\lstset{language=sed}
<<Change [[highlight]] style to [[listings]] style>>=
s/\[1\]//;s/#1//g
s/^[^{]*{[^}]*}$/&{}/
# colorbox is not supported, so silently delete
s/\\colorbox{[^}]*}{\(.*\)}/\1/
s/\\text\(color[^}]*}\){\(.*\)}/\\\1\2/
s/\\bf{\(.*\)}/\\bfseries\1/
s/\\textbf{\(.*\)}/\\bfseries\1/
s/\\it{\(.*\)}/\\itshape\1/
s/\\textit{\(.*\)}/\\itshape\1/
# underline is not supported, so silently turn into sans-serif
s/\\underline{\(.*\)}/\\sffamily\1/
s/\\texttt{\(.*\)}/\\ttfamily\1/
@

\lstset{language=TeX}
<<Preamble Adjustments for [[listings]]>>=
\lstset{backgroundcolor=\color{bgcolor}}
\ifx\hlstd\undefined\else
\lstset{basicstyle=\hlstd\footnotesize,keywordstyle=\hlkwa,emphstyle=\hlkwb,%
        commentstyle=\hlcom,stringstyle=\hlstr,directivestyle=\hldir}
\fi
\ifx\hlnormal\undefined\else
\lstset{basicstyle=\hlnormal\footnotesize,keywordstyle=\hlkeyword,%
        emphstyle=\hltype,commentstyle=\hlcomment,stringstyle=\hlstring,%
	directivestyle=\hlpreproc}
\fi
@

Finally, some general appearance adjustments are made.  The entire listing
is shifted over a bit, and the font layout is set for minimum mangling.

<<Preamble Adjustments for [[listings]]>>=
\lstset{columns=[l]fixed,framexleftmargin=1.5em,framexrightmargin=1em}
@

\subsection{Additional Output Adjustments}

Now that code chunks can be pretty-printed, a few more adjustments need to
be made.  Underscores in chunk names (such as when the chunk name is a file
name) need to be escaped.  The easiest place to do this is in a
filter, which has the chunk names separated out already.  For maximum
flexibility, this is done in C++.

\lstset{language=C++}
<<Filter lines from noweb token stream>>=
if(!line.compare(0, 5, "@use ") || !line.compare(0, 6, "@defn ") ||
   !line.compare(0, 6, "@nwenh"))
  if(line.find("[[") >= line.size())
    for(unsigned pos = 0; (pos = line.find('_', pos)) < line.size(); pos++)
      line.insert(pos++, 1, '\\');
@

Chunk names referring to chunks in other files should be indicated as
such.  For now, this is done by prepending the chunk name with the
source file in parentheses.  The parentheses are to help force the
index entry to the top of the list.  The printed source file is the
top-most file which includes this definition; in other words, if a
file is included by another, its definitions are owned by the
includer.
% Begin-doc extref-dis
This can be disabled for user-level documentation by adding
the commment:

\begin{quote}
\verb|%%% no-ext-ref|
\end{quote}
% End-doc extref-dis

<<Initialize [[nwweavefilt]]>>=
regex_t input_re;
regmatch_t irm[2];
ec_regcomp(input_re, "^\\\\input\\{(.*\\.nw)\\}$", 0);
regex_t bdef_re;
regmatch_t brm[2];
ec_regcomp(bdef_re, "^@<<(.*)@>>=$", 0);

string_map ch;
// final args are noweb_order
string_map feqv;
bool disext = false;
// find included files
for(int a = 1; a < argc; a++) {
  ifstream A(argv[a]);
  while(getline(A, line).good() || line.size()) {
    if((disext = line == "%%% no-ext-refs"))
      break;
    if(!regexec(&input_re, line.c_str(), 2, irm, 0))
      feqv[line_match(irm[1])] = argv[a];
  }
}
if(!disext) {
  const char *myf = argv[--argc]; // my name is always last; skip it
  // now loop over files, making included files look like includer
  while(argc-- > 1) {
    const char *a = *++argv;
    const char *f = a;
    while(1) {
      auto ff = feqv.find(f);
      if(ff == feqv.end())
        break;
      f = ff->second.c_str();
    }
    if(!strcmp(f, myf))
      continue;
    ifstream A(a);
    while(getline(A, line).good() || line.size())
      if(!regexec(&bdef_re, line.c_str(), 2, brm, 0))
        ch[line_match(brm[1])] = f;
  }
}
regex_t xref_re;
regmatch_t xrm[3];
ec_regcomp(xref_re, "^(@use |@defn )(.*)", 0);
@

<<Filter lines from noweb token stream>>=
if(!disext && !regexec(&xref_re, line.c_str(), 3, xrm, 0)) {
  auto chf = ch.find(line_match(xrm[2]));
  if(chf != ch.end())
    line = line_match(xrm[1]) + '(' + chf->second + ") " +
           line_match(xrm[2]);
}
@

There is little point in separating consecutive code chunks with a lot of
white space, so the excess whitespace added by noweb needs to be removed.

\lstset{language=sed}
<<Reduce interchunk whitespace>>=
/\\nwendcode{}\\nwbegindocs/{
  N;N
  /\\lstset/{s/$/%/;N;}
  /\\nwenddocs/{s/\\nwbegindocs.*docspar\n\n//;s/\\nwenddocs{}//;}
}
@

Then, the \LaTeX{} preamble needs to be added.  Except for the overall
document style, this is mostly the same for all documents.  This means that
the first line, the document style, must come before the common preamble.
Rather than looking for that particular line, a special comment is replaced:

\begin{quote}
\begin{verbatim}
 %%% latex preamble
\end{verbatim}
\end{quote}

The following script does this, and brings all of the above together.
In order to merge sources correctly, the dependency order (tree) needs
to be passed in as well as the input and output file names.  The
highlight information is selected via environment variables, normally
passed in via exports from the makefile.  Selection of the listings
package versus an external highlighter is done by prefixing the
highlight theme name with [[list:]].

While most processing is identical for Latin-1 encoding and UTF-8
encoding, noweb's markup processor strips tabs from input, converting
them to spaces.  This does not work in UTF-8 when there are multi-byte
characters, so it is disabled.

Note that the [[-index]] option automatically marks unhighlighted code
(currently code using the double-square-bracket notation) with cross
references to identifiers it knows about.  This is broken for
languages that its [[finduses]] filter doesn't understand, such as
Forth.  For now, I've just disabled the feature entirely by tricking that
filter into thinking there are no identifiers ([[-indexfrom /dev/null]]).
I've already decided that automatic identifier extraction is broken by
design as well, and such highlighting never did work with highlighted
text, so nothing is really lost here.  The gain is that [[%def]] can
be used to manually add a definition-only index.

\lstset{language=make}
<<makefile.config>>=
# Highlight theme for PDF/HTML formatted source code
# Blank uses default highlight style; prefix w/ list: to use listings package
HL_THEME:=

@

<<makefile.vars>>=
export HL_THEME HLPROG_TYPE NW_UTF8
@

<<makefile.rules>>=
%.tex: %.nw nw2latex $(call tree,%.nw)
	./nw2latex $< $@ "$(call tree,$<)"
@

<<Build Script Executables>>=
nw2latex \
@

<<makefile.rules>>=
nw2latex: <<[[nw2latex]] deps>>

@

<<[[nw2latex]] deps>>=
latexhl nwweavefilt addlistings nw-nonl-preidx nw-nonl-postidx \
nw-parm-preidx nw-parm-postidx \
@

\lstset{language=sh}
<<nw2latex>>=
#!/bin/sh
<<Common noweb Warning>>

noweb="$1"
outf="$2"
noweb_order="$3"

# prefix theme with "list:" to enable lstlistings
uselst="${HL_THEME%%:*}"
if [ "list" = "$uselst" ]; then
  HL_THEME="${HL_THEME#list:}"
else
  uselst=
fi

# place to stash highlighter temporaries
trap "rm -rf /tmp/latexhl.*.$$" 0
export HLDIR=/tmp/latexhl.hl.$$
mkdir -p $HLDIR

<<Prepare for weave>>
<<Prepare for [[latexhl]]>>
(
  ln=`grep -n '^%%% latex preamble' "$noweb" | cut -d: -f1 | head -n 1`
  head -n $ln "$noweb"
  cat <<"EOF"
<<preamble.tex>>
EOF
  tail -n +$((ln+1)) "$noweb"
) | <<Pre-process before weave>>
  noweave -filter "./nw-parm-preidx $noweb_order | ./nw-nonl-preidx |
                   ./nwweavefilt $filt $noweb_order" -delay -indexfrom /dev/null \
	  -filter "./nw-nonl-postidx | ./nw-parm-postidx" ${NW_UTF8:+-t} - | \
	    <<Post-process latex after weave>>
            sed -e '<<Reduce interchunk whitespace>>' -e "$postproc" >"$outf"
@

<<Prepare for [[latexhl]]>>=
if [ -z "$uselst" -a -n "$HLPROG_TYPE" ]; then
  filt=./latexhl
  postproc='<<Insert code header into frame>>'
else
  filt=./addlistings
  postproc='<<Change [[nwcode]] to [[listings]]>>'
  sed -e '<<Change [[highlight]] style to [[listings]] style>>' \
     < latexhl.sty > latexhl.sty.$$
  mv latexhl.sty.$$ latexhl.sty
fi
@

The \LaTeX{} preamble is fairly straightforward.  The [[fontenc]],
[[inputenc]], [[fontspec]] and [[babel]] packages are pretty much
required anywhere.  The input encoding is forced to latin1 for
[[pdflatex]]; it is expected that [[xelatex]] be used for Unicode. 
The [[noweb]] package is obviously needed for the noweb output.  The
[[rcs]] package is included to easily incorporate those nasty \$-heavy
RCS/CVS/SVN keywords.  The [[ifpdf]] and [[ifxetex]] packages are
needed for using this \LaTeX{} source with non-PDF output and
non-Unicode input.  The standard PDF font packages are used to reduce
the PDF size and hopefully look smoother on a greater number of
operating systems.  The [[graphicx]] package is the easiest way to
include figures; the PDF output styles should use scalable PDF for
figure sources (most other types can be converted to this).  The
[[listings]], [[color]], and [[xcolor]] packages and the [[latexhl]]
highlight theme file are required for color syntax highlighting.  The
[[headings]] package gives more informative headers and footers by
default.  The [[hyperref]] package gives hyperlinks in the PDF file.
Links are made less distracting by replacing the red outline box with
dark blue-colored text, and URL links are colored the same way.  The
[[multitoc]] package provides multi-column tables of contents, which
once again is to reduce the number of pages a bit.

\lstset{language=TeX}
<<preamble.tex>>=
\usepackage[T1]{fontenc}
\usepackage{ifxetex}
\ifxetex
% Use Latin Modern; selecting standard PDF fonts does not work
% Latin Modern doesn't look that great, though (it's apparently not an
% exact duplicate of Computer Modern), but it's easy to set fonts after
% the standard preamble.
\usepackage{fontspec}
% \documentclass[twoside,english]{..} doesn't seem to work right with
% xetex.  Using polyglossia seems to help, with a language setting
% after the standard preamble instead (using \setdefaultlanguage)
\usepackage{polyglossia}
\else
\usepackage[latin1]{inputenc}
\usepackage{babel}
\fi
\usepackage{rcs}
\usepackage{ifpdf}
\ifpdf
% Use standard fonts for PDF output
\usepackage{courier}
\usepackage[scaled]{helvet}
\usepackage{mathptmx}
\fi
\usepackage{noweb}
\usepackage{graphicx}
\ifpdf
% Use pdf figs by default (hack)
\makeatletter
\def\Gin@extensions{.pdf}
\makeatother
\fi
\ifxetex
% Use pdf figs by default (hack)
\makeatletter
\def\Gin@extensions{.pdf}
\makeatother
\fi
\usepackage{listings}
\usepackage{color}
\usepackage[usenames,dvipsnames,svgnames,x11names,table]{xcolor}
\usepackage{latexhl}
%
\pagestyle{headings}
% new since 2009: rerunfilecheck (automatic include by hyperref)
\usepackage[aux]{rerunfilecheck}
%
% Hyperlink it!
\definecolor{darkblue}{rgb}{0,0,0.3}
\usepackage[colorlinks=true,linkcolor=darkblue,urlcolor=darkblue]{hyperref}
% And set the title/author properly (taken from hyperref slides.pdf)
\makeatletter
\newcommand{\org@maketitle}{}% ensure not defined
\let\org@maketitle\maketitle
\def\maketitle{\hypersetup{pdftitle={\@title}, pdfauthor={\@author}}%
\org@maketitle}
%
% Multi-column table of contents
% due to bug, individual tocs cannot actually be disabled (they'll be
% completely cleared)
% must disable all by \let\@starttoc\@multitoc@starttoc
\usepackage[toc,lof,lot]{multitoc}
@

% there were some issues on older TeX; not sure any more:
%% Get "Bad space factor(0)" if this isn't here (very strange)
%\makeatletter
%\@savsf=1
%\makeatother

Then, some adjustments need to be made for noweb.  noweb's [[\setupcode]]
does some adjustments to \LaTeX{} layout that override the code
highlighting, so that is disabled.  noweb tries too hard to keep code chunks
on one page, so some adjustments are made to those parameters.  Finally, the
noweb cross reference listings are adjusted a bit and the code size is made
smaller.  Also, the ``never defined'' message should probably be
changed for imported symbols, and in fact all imported symbols should
be shown identically.  It is not currently possible to remove the
``never defined'' message completely, so it is changed to
``(imported)'' even if it is not imported.

<<preamble.tex>>=
% Code formatting is done by highlight/lstlisting, not noweb
\def\setupcode{\Tt}

% This is from the noweb FAQ
% Allow code chunks to span pages
\def\nwendcode{\endtrivlist \endgroup}
%\def\nwendcode{\endtrivlist \endgroup \vfil\penalty10\vfilneg}
%
% Allow docs to be split from code chunks
\let\nwdocspar=\smallbreak
%\let\nwdocspar=\par
%
% try a little harder to keep code chunks on one page
\nwcodepenalty=1000
%
% Code can be huge, so make it smaller (smallcode, scriptsizecode, footnotesizecode)
% Make chunk index useful (longchunks)
% Remove xrefs from chunks themselves (noidentxref)
\noweboptions{footnotesizecode,longchunks,noidentxref}

% change "never defined" to "imported"
\makeatletter
\def\@nwlangdepnvd{imported}
\makeatother
@

One other thing I usually do for code is to allow line breaks after
underscores.

<<preamble.tex>>=
% break lines after _
\let\oldunderscore\_
\def\_{\oldunderscore\discretionary{}{}{}}
@

Finally, the syntax highlighter adjustments need to be made.

<<preamble.tex>>=
<<Preamble Adjustments for [[listings]]>>
% for highlight
\ifpdf
<<Framed Code Preamble>>
\fi
\ifxetex
<<Framed Code Preamble>>
\fi
@

\subsection{Attaching Files}

In addition to the adjustments for appearance, the source files need
to be attached.  Attachments are done using the \LaTeX{} [[navigator]]
package.  The [[embedfile]] package used by previous versions of this
build sytem does not support [[xelatex]].  The old command is
retained, though; any files in addition to the source files must be
attached using the [[\embedfile]] command in order for other portions
of the build system to pick up the files properly.

<<preamble.tex>>=
\ifpdf
\usepackage{embedfile}
\fi
\ifxetex
\usepackage{navigator}
\def\embedfile#1{\embeddedfile{#1}{#1}}
\fi
@

\lstset{language=sh}
<<Pre-process before weave>>=
(att_nwo="`echo $noweb_order | \
                 sed -e 's/\\([-_a-zA-Z0-9.]*\\) */\\\\\\\\embedfile{\\1}%\\\\n/g'`"
 sed -e 's/^\\begin{document}/'"$att_nwo&/") | \
@

\subsection{User Documentation}

One challenge with literate programming is to provide separate user-level
documentation that keeps track of the source code as well as the design
documentation.  For this, a few code reinsertion tricks are used.
First, excerpts from the documentation can be reinserted at a chosen
location.  These excerpts are surrounded with begin and end comments, which
must be at the beginning of the line:

\begin{quote}{\ttfamily
\% Begin-doc \emph{chunk-name}\\*
\ldots\\*
\% End-doc \emph{chunk-name}
}\end{quote}

The chunk name should consist only of alphanumeric characters, minus signs,
and underscores.  Insertion is done using a special input command:

\begin{quote}{\ttfamily
$\backslash$input\{\emph{chunk-name}.tex\} \%\%\% doc
}\end{quote}

Since this introduces a new macro language, it needs to be checked for
consistency along with the rest of the source.  This includes checking
for a documentation chunk nested within itself (although no check for
a documentation reference nested within the same chunk), mismatched
terminators, or missing terminators.  No checks are made for undefined
references or recursive insertions; these will show up in the main
error log (and cause an error abort) upon weaving.

<<Other source file consistency checks>>=
@@grep '^% Begin-doc ' $(NOWEB) | cut -d: -f2- | sort -u
@@for f in $(NOWEB); do \
  lno=1 std=0; \
  while IFS= read -r x; do \
    case "$$x" in \
      "% Begin-doc "*) \
        std=$$((std+1)); eval bd$$std='"$${x#* * }"' bl$$std=$$lno; \
	stp=1; while [ $$stp -lt $$std ]; do \
	  eval bd='"$$bd'$$stp\" bl=\$$bl$$stp; \
	  if [ "$${x#* * }" = "$$bd" ]; then \
	    echo "Self-nested Begin-doc $$bd at $$bl/$$lno"; break; \
	  fi; \
	  stp=$$((stp+1)); \
	done;; \
      "% End-doc "*) \
        if [ $$std -eq 0 ]; then \
	  echo "$$f: unexpected end-doc at $$lno"; \
	else \
	  eval bd='"$$bd'$$std\" bl=\$$bl$$std; \
	  if [ "$${x#* * }" != "$$bd" ]; then \
	    echo "$$f: Begin-doc $$bd at $$bl" \
	         "ended by $${x#% } at $$lno"; break; \
	  fi; \
	  std=$$((std-1)); \
	fi;; \
    esac; \
    lno=$$((lno+1)); \
  done < "$$f"; \
  while [ $$std -gt 0 ]; do \
    eval bd='"$$bd'$$std\" bl=\$$bl$$std; \
    echo "$$f: unended Begin-doc $$bd at $$bl"; \
    std=$$((std-1)); \
  done; \
done
@

The reinsertion is done before weaving so that noweb formatting can be
done on this, as well.  A sed script is generated to convert all of
the [[\input]] commands to their associated text, and then that script
is applied in the pre-processing pipeline.  It requires multiple
passes in order to resolve nested references.  The first pass simply
extracts all explicitly referenced chunks.  Then, while references
remain in the substitution text, they are replaced within the script
by the script that would have been generated.  The second pass is done
by generating a second script to replace just that reference in the
main script being generated.

<<Prepare for weave>>=
# use eval to expand $noweb_order properly
eval sed -n -e \''s/^\\input{\(.*\)\.tex} %%% doc$/\1/p'\' $noweb_order | \
  sort -u | while read c; do
    # on the input line, replace the line with % and append the text
    # /^\\input{blah\.tex} %%% doc$/{s/.*/%/;a\%\
    printf %s\\n "/^\\\\input{${c}\\.tex} %%% doc\$/{s/.*/%/;a\\%\\"
    eval sed -n "'/^% Begin-doc $c\$/,/^% End-doc $c\$/{
                  /^% Begin-doc $c\$/!{/^% End-doc $c\$/!"'{
                    s/\\/\\\\/g;s/$/\\/;p;};};}'\' $noweb_order | sed '$s/.$//'
    echo '}'
  done > /tmp/latexhl.doc.$$
touch /tmp/latexhl.docr.$$
while grep '^\\\\input{.*\.tex} %%% doc\\$' /tmp/latexhl.doc.$$ >/tmp/latexhl.docn.$$; do
  # check for recursion
  diff /tmp/latexhl.doc[rn].$$ >/dev/null && \
    echo "Recursive documentation reinsertion detected:" && \
    sed 's/^\\\\input{\(.*\)\.tex}.*/\1/' /tmp/latexhl.docr.$$ && \
    exit 1
  mv /tmp/latexhl.doc{n,r}.$$
  sed -e '
     # chunk header: replace script input line instead of doc input line
     # /^\\\\input{blah\\\.tex} %%% doc\\$/{s/.*/%\\/;a
     s/\^/^\\\\/;s/doc\$/doc\\\\\$/;s^/%/^/%\\\\/^;h;s/$/\\\\/
     # remainder of chunk:
     :a
       # until there is no terminating \ (which is not an explicit \)
       x;s/\\\\//g;/\\$/!{x;s/$/\\/;n;b}
       # double backslashes and add an extra backslash at end
       x;n;h;s/\\/\\\\/g;s/$/\\/
       # repeat
       ba' < /tmp/latexhl.doc.$$ >/tmp/latexhl.doc2.$$
  sed -f /tmp/latexhl.doc2.$$ /tmp/latexhl.doc.$$ >/tmp/latexhl.doc3.$$
  mv /tmp/latexhl.doc3.$$ /tmp/latexhl.doc.$$
done
@

<<Pre-process before weave>>=
sed -f /tmp/latexhl.doc.$$ | \
@

One problem with reinsertion is that labels are duplicated.  While
leaving the labels out manually works, it is cumbersome and may not be
what is wanted.  For example, one way to split the user documentation
into a separate file is to reinsert it into an otherwise empty noweb
source file.  Stripping the labels out manually means that the labels
will be undefined in the fresh document.  A better way is to have them
automatically filtered out.

One problem with filtering them out is deciding which one to filter
out.  Ideally, both would be kept, and the reinserted one would be
given a new, unique name, and all references to that label within
reinserted text would be updated as well.  This is a task for a future
revision, though.  In fact, even deciding that the main text or the
reinserted text should be preferred is too complex of a question for
now.  Instead, the first label after all reinsertions are complete is
kept, and all others are removed.  As an additional level of
stupidity, label commands may not span lines or contain curly braces.
They may also not appear after a percent-sign character on the same
line, even if that percent sign is escaped for \LaTeX{}.

Another issue is that labels defined in macros may have repeated
formats, but actually define different labels.  It is not really
possible to detect the macros being defined and filter out the
duplicates as is done for explicit labels, so these are just left
alone for now.  This is detected by checking only for the \#
character, which is necessary for macro arguments, but may also
indicate the \# chracter itself.  The only way to fix this would be to
detect the preamble, and require that such macros be defined there.
This is left for a future revision.

<<Pre-process before weave>>=
awk '<<Remove duplicate labels from \LaTeX{} source>>
     { print }' | \
@

\lstset{language=awk}
<<Remove duplicate labels from \LaTeX{} source>>=
/(^|[^\\])\\label{/ {
  s = $0
  os = ""
  while(match(s, "\\\\label\\{[^}]*\\}")) {
    os = os substr(s, 1, RSTART - 1)
    l = substr(s, RSTART, RLENGTH)
    s = substr(s, RSTART + RLENGTH)
    if(l in gotit)
      continue;
    os = os l
    if(!match(s, "#"))
      gotit[l] = 1
  }
  print(os s)
  next
}
@

Another problem is that there is more than one way to create a label.
For example, the sectioning commands may be redefined to automatically
create a label based on the section name.  This document defines no
such things%
\footnote{Actually, the hyperref package, the noweb HTML filter, and
noweb code chunks all create automatic labels, but they are based on
file location rather than the text at the label location, so they will
never conflict.}%
, but as implemented above, more awk code can be added to
[[<<Remove duplicate labels from \LaTeX{} source>>]].

One other problem peculiar to noweb text is that chunk references are
reformatted in a way that assumes the chunk is defined in the same
document.  When documenting chunk names themselves, like this document
does, this will clutter up the text with useless undefined chunk
reference labels.  Setting the undefined reference text to empty helps
a little, but it leaves a blank space between the chunk name and the
terminating bracket.  Since this is a feature of [[noweave]], it can't
really be filtered out for all files.  As a kludge, if no chunks are
defined in a document at all, the reference markers are filtered out.
Nothing is done for HTML output, since this only results in links that
go nowhere.

\lstset{language=sh}
<<Post-process latex after weave>>=
(if grep '^@<<.*@>>=$' "$noweb" >/dev/null; then cat; else \
 sed 's/~{\\nwtagstyle{}[^}]*}}//g'; fi) | \
@

Second, plain text chunks can be extracted, highlighted, and
reinserted using a special input command as well.  This has only one
percent (vs. three) to distinguish it from the directive above:

\begin{quote}{\ttfamily
$\backslash$input\{\emph{file}.tex\} \% \emph{language}
}\end{quote}

Rather than actually generate an additional file, the input directive is
replaced with what would have been the contents of that file.  This also
allows the file to only contain contents given the source and any
dependencies, rather than also including contents from sources which extend
the contents.  This is done after weaving, since the text is always literal,
but a preprocessing step is required to avoid noweb substitutions during the
weave.  The sed script to generate the sed script cannot be done in a
backtick expression, because the argument list may overflow.  Generating the
script to a file also reduces the number of backslash escapes required.

It seems pointless to highlight plain text files, but it may still be
desirable to give it a colored background.  For cases where this is
not so, a special language \texttt{verbatim} can be used to instead
insert the code as verbatim quoted plain text.

<<Pre-process before weave>>=
sed '/^\\input{.*\.tex} % [^ ]*$/{s/\[\[/@&/g;s/<</@&/g;}' | \
@

<<For each reinserted code chunk>>=
sed -n -e 's/^\\input{\(.*\)\.tex} % \([^ ]*\)$/\2 \1/p' $noweb_order | sort -u | \
    while read -r lang f; do
      fesc=$(printf %s "$f" | sed 's/[][\\.*?^$]/\\&/g')
@

<<Prepare for weave>>=
<<For each reinserted code chunk>>
  test verbatim = $lang || continue
  printf %s\\n "/^\\\\input{${fesc}\\.tex} % $lang\$/{s/.*/%/;a\\%\\"
  eval notangle '-R"$f"' $noweb_order 2>/dev/null | (
    echo '\begin{quote}\footnotesize\begin{verbatim}'
    sed 's/@<</@@<</g;s/@>>/@@>>/g;s/^@/@@/g'
    echo '\end{verbatim}\end{quote}'
    ) | sed 's/\\/\\\\/g;s/$/\\/' | sed '$s/.$//'
  echo '}'
done > /tmp/latexhl.verb.$$
@

<<Prepare for [[latexhl]]>>=
<<For each reinserted code chunk>>
  test verbatim = $lang && continue
  printf %s\\n "/^\\\\input{${fesc}\\.tex} % $lang\$/{s/.*/%/;a\\%\\"
  eval notangle '-R"$f"' $noweb_order 2>/dev/null | \
    if [ -z "$uselst" ]; then
      ./latexhl $lang | <<Insert latexhl output directly>>
    else
      printf %s\\n '\begin{lstlisting}'
      ./addlistings $lang
      printf %s\\n '\end{lstlisting}'
    fi | sed 's/\\/\\\\/g;s/$/\\/' | sed '$s/.$//'
  echo '}'
done > /tmp/latexhl.fmt.$$
@

<<Pre-process before weave>>=
sed -f /tmp/latexhl.verb.$$ | \
@

<<Insert latexhl output directly>>=
sed 's/^\\begin{framed}[^%]*/\\begin{quote}\\codemargin=0pt&\\begin{webcode}/;
     s/^\\end{framed}/\\end{webcode}&\\end{quote}/'
@

<<Post-process latex after weave>>=
sed -f /tmp/latexhl.fmt.$$ | \
@

C API documentation requires reinsertion of the function prototype,
formatted as C code.  Integration into a full API documentation package is
beyond the scope of the current document (previous revisions had their own
implementation of an API documentation program, but that was for Ada).
Instead, only the prototype reinsertion is supported.  To insert a
prototype, reference it using a specially formatted comment:

\begin{quote}{\ttfamily
\% \emph{function} prototype
}\end{quote}

This is done by gently extracting the prototype from [[cproto.h]]
using sed.  Since prototypes which appear in [[$(CHEADERS)]] are
removed from [[cproto.h]], they are allowed here as well. For
prototypes which do not appear in either place, the chunk
[[<<C Prototypes>>]] may be expanded as well.  Since the C preprocessor
might transform functions before they become prototypes, the
[[<<For each reinserted C prototype>>]] chunk may be extended,
transforming [[$mf]] as needed.  The sed script separates arguments by
comma and terminates by close parenthesis (which means that callbacks
require typedefs).  The code is reinserted with each argument on a
separate line, aligned by the first argument, i.e. after the left
parenthesis.  This is done by converting the text before the
parenthesis into spaces, and stripping off the text not to be printed
every pass until there is nothing left to print.

\lstset{language=sed}
<<Format a prototype from [[cproto.h]]>>=
# strip trailing semicolon to make pattern simpler (never gets re-added)
# also strip any trailing comment(s)
s/;.*//;
# strip whitespace preceeding , and ) (usually due to chunk insertion)
s/[ \n\t]*\([,)]\)/\1/g;
# strip whitespace following ( (usualy due to chunk insertion, again)
s/([ \n\t]*/(/g
# create spaces out of all text up to first paren and append full proto
h;s/[^(]/ /g;s/(.*/ /;G;
# remove up to first paren plus up to end of first parameter
s/[^ ][^(]*([^),]*[),] *//;
# strip off 2nd+ parameters & print first line
x;s/\(([^,)]*[,)]\).*/\1/;p;x;
# loop until no more saved text to print
:a
/^ *$/b;
# save, strip off 2nd+ paramters & print
h;s/\(^ *[^,)]*[,)]\).*/\1/;p;
# retrieve, strip off 1st parameter, and loop
g;s/[^ ][^),]*[),] *//;
ba;
@

\lstset{language=make}
<<makefile.rules>>=
ifneq ($(CFILES),)
nw2latex: cproto.h $(CHEADERS)
endif
nw2latex: $(CXXHEADERS)
@

\lstset{language=sh}
<<For each reinserted C prototype>>=
sed -f /tmp/latexhl.doc.$$ "$noweb" | sed -n -e 's/^% \(..*\) prototype$/\1/p' | \
 while read f; do
  mf="$f"
@

Note that to extract the full prototype, the first encountered string
which looks like a prototype, up to its terminating semicolon is
extracted, using an exteded regular expression.  This requires GNU
sed.  I suppose I could have just used [[[A-Za-z0-9]]] instead of
[[[[:alnum:]]]] and not needed GNU sed, but this way makes more sense.
The improved use of sed rather than egrep allows for multi-line
prototypes and comments after prototypes, but still does not allow
comments within prototypes, and picks up prototype-looking code that
isn't really prototypes.  In order to at least avoid function calls
from being picked up, only type definition-type characters must appear
before the function name, and the first character on the line must be
such a character.  Of course [[(void)]] may still look like a type
definition for this, but this at least filters out most calls.

Since this is not being done in the makefile, special tangler
overrides are ignored.  However, the enhanced tangler is used instead
of plain [[notangle]].

<<[[nw2latex]] deps>>=
nt-parm nt-nonl \
@

<<makefile.vars>>=
export HEADERS
@

<<Extract and format a C prototype from [[cproto.h]]>>=
notangle -filter "./nt-nonl|./nt-parm" -R"C Prototypes" $noweb_order 2>/dev/null | \
  eval "cat - cproto.h $HEADERS"  | grep -v '^#line' | \
     sed -n -r -e "/<<C Prototype Pattern>>/{<<Extract multi-line prototype>>;q;}" | \
     sed -n -e '<<Format a prototype from [[cproto.h]]>>'
@

\lstset{language=sed}
<<C Prototype Pattern>>=
^[_[:alnum:]][_[:alnum:][:space:]()*&]*[^_[:alnum:]]$mf\\(
@

<<Extract multi-line prototype>>=
:a N;/;/!ba; s/[\n\t]/ /g;s/  */ /g;p
@

\lstset{language=sh}
<<Prepare for [[latexhl]]>>=
<<For each reinserted C prototype>>
  printf %s\\n "/^% $f prototype\$/{a\\%\\"
  <<Extract and format a C prototype from [[cproto.h]]>> | \
    if [ -z "$uselst" ]; then
      ./latexhl C | <<Insert latexhl output directly>>
    else
      printf %s\\n '\begin{lstlisting}'
      ./addlistings C
      printf %s\\n '\end{lstlisting}'
    fi | sed 's/\\/\\\\/g;s/$/\\/' | sed '$s/.$//'
  echo '}'
done >> /tmp/latexhl.fmt.$$
@

<<C Prototypes>>=
@

\subsection{Include Directive Processing}

Finally, a method to include an arbitrary noweb file is provided.  The only
way to weave and tangle such files correctly is to insert the contents of
the file in place of the include directive.  Since [[\include]] has other
side effects (it forces a page break), only [[\input]] is supported.  The
syntax is very specifically defined in order to avoid conflicts with other
[[\input]] directives.  Namely, it must not have any following text, and the
file name must end in \texttt{.nw}.

<<Insert included noweb files>>=
awk '<<Insert included noweb files with [[awk]]>>'
@

\lstset{language=awk}
<<Insert included noweb files with [[awk]]>>=
function readfile(f) {
  while(getline < f) {
    if($0 ~ /^\\input{[^}]*\.nw}$/) {
      sub(".*{", ""); sub("}", "");
      fname = $0;
      if(index(fname, "/") != 0 && index(f, "/") >= 0) {
        fdir = f;
        sub("[^/]*$", "", fdir);
        fname = fdir fname;
      }
      readfile(fname);
      close(fname);
    } else
      print;
  }
}
BEGIN { readfile("-") }
@

\lstset{language=sh}
<<Pre-process before weave>>=
<<Insert included noweb files>> | \
@

%tangle needs more thought - need to incorporate into docs
%< <Pre-process before tangle> >=
%< <Insert included noweb files> >
%@

\section{HTML Code Documentation}

The noweb file is intended to be written in \LaTeX{}, and converted to HTML
after and/or during the weave process.

\subsection{HTML from TeX4ht}

One way to generate HTML would be to use TeX4ht%
\footnote{\url{http://www.cse.ohio-state.edu/~gurari/TeX4ht/}}%
.  It is supposed to produce output very similar to what [[pdflatex]]
produces, because it uses \LaTeX{} as its underlying engine.  Once
again a wrapper script ([[nwtex2html]]) is needed.

\lstset{language=make}
<<Source Code Documentation Files>>=
$(patsubst %.nw,%.html,$(NOWEB_NINCL)) \
@

<<makefile.rules tex4ht>>=
%.html: %.tex $(FIGS_EPS) nwtex2html
	./nwtex2html '$(latexcmd)' $< $@ '$(call tree,$*.nw)'
@

A configuration file is needed to set extended options.  In particular, the
[[\color]] command behaves oddly, so an attempt is made to treat it mostly
like [[\textcolor]] instead.  Even then, the background color doesn't show
through, so it's forced by placing all code the class [[lstlisting]] and
setting the color in the style sheet.  Overriding the code environments and
doing some other fine tuning for TeX4ht is done in the preamble.

\lstset{language=TeX}
<<myhtml.cfg>>=
%doesn't produce a nice index as is; I probably need to support xindy
%\RequirePackage{indexing4ht}
\Preamble{xhtml,uni-html4,css-in,$splitstyle,graphics-,Gin-dim+}
% attempted bug fix for tex4ht color
\HAssign\textcolornest=0
\makeatletter
\def\reset@color{
  \ifnum\textcolornest>0\gHAdvance\textcolornest by -1\HCode{</span>}\fi
  \special{color pop}
}
\makeatother
\Configure{color}{
   \gHAdvance\textcolorN by 1
   \HCode{<span id="textcolor\textcolorN">}
   \Configure{SetHColor}{%
   \gHAdvance\textcolornest by 1%
   \Css{span\#textcolor\textcolorN{color:\HColor}}}
}
% end bug fix
\begin{document}
  \Css{.lstlisting {$(<<Extract HTML CSS for current theme's background>>)
                    margin-left: 10pt; margin-right: 10pt;}}
\EndPreamble
@

\lstset{language=sh}
<<Extract HTML CSS for current theme's background>>=
case "$HLPROG_TYPE" in
  highlight-[23])
    highlight -S c ${HL_THEME:+-s $HL_THEME} -I </dev/null | \
      sed -n -e '/pre.hl/{s/.*{//;s/}.*//;
                 s/\(background-color:#\)ffffff/\1fafafa/;
                 s/#/\\#/g;
                 s/font-size.*/font-size: smaller;/
                 p;}'
    ;;
  source-highlight)
    echo 'nodoctemplate
          "background-color:$docbgcolor; font-size: smaller;"
	  end' > mybg.outlang
    source-highlight --lang-def=/dev/null --outlang-def=mybg.outlang \
                     ${HL_THEME:+--style-css-file=$HL_THEME.css} </dev/null | \
      sed 's/#ffffff/#fafafa/i;s/white/#f6f6f6/i'
    ;;
  *) echo "background-color:#fafafa; font-size: smaller;" ;;
esac
@

\lstset{language=TeX}
<<preamble.tex>>=
\ifpdf
\else
\ifxetex
\else
% Following is for tex4ht only
\def\embedfile#1{}
\newenvironment{framed}{\HCode{<div class="lstlisting">}}{\HCode{</div>}}
%\renewenvironment{lstlisting}{\HCode{<div class="lstlisting">}}{\HCode{</div>}}
\noweboptions{webnumbering,nomargintag}

% Minimize mangling by tex4ht:
\lstset{columns=flexible}
% make spaces in chunk names into nbsp
\let\oldsetupmodname\setupmodname
\def\setupmodname{\oldsetupmodname\catcode`\ =13}
\fi
\fi
@

The script just calls [[htlatex]] in its own directory (because it generates
a lot of the same aux files as [[pdflatex]]), cleans up the HTML a bit, and
tacks on the noweb source in compressed, uuencoded form.  It depends on a
working TeX4ht configuration, which is unfortunately mostly undocumented and
easily broken.  Configuring TeX4ht is beyond the scope of this document.

\lstset{language=make}
<<makefile.rules>>=
nwtex2html: tex4ht_postproc
%.dvi: %.tex
	$(run_latex)
@

\lstset{language=sh}
<<Build Script Executables>>=
nwtex2html \
@

<<nwtex2html>>=
#!/bin/sh
<<Common noweb Warning>>

tcmd="$1"
tex="$2"
base="${2%.tex}"
noweb="$base.nw"
outf="$3"
noweb_order="$4"
HL_THEME="${HL_THEME#*:}"

<<Check highlighter>>

figs=`sed -n 's/^\\\\includegraphics.*{\\(.*\\)}\$/\\1.eps/p' $tex`
texi=`sed -n -e 's/^\\\\input{\\(.*\\.tex\\)} % .*/\\1/p' "$tex"`

# Exit on error
set -e

# Done in its own dir to avoid stomping on .pdf aux files
dir=/tmp/html.$$
rm -rf $dir
trap "rm -rf $dir" 0
mkdir -p $dir
tdir="`pwd`"
for x in $noweb $tex ${tex%.tex}.ind $figs $texi *.sty; do ln -s "$tdir/$x" $dir; done
cd $dir

if [ y = "$HTML_SPLIT" ]; then
  splitstyle=frames,3
else
  splitstyle=fn-in
fi
cat <<EOF > myhtml.cfg
<<myhtml.cfg>>
EOF

# convert pdflatex/xelatex to htlatex/htxelatex
type -p "ht${tcmd}" >/dev/null && tcmd="ht${tcmd}"
type -p "ht${tcmd#pdf}" >/dev/null && tcmd="ht${tcmd#pdf}"
case "$tcmd" in
  ht*) tcmd="$(which "$tcmd")" ;;
  *)   echo "LATEXCMD $tmd not supported for tex4ht"; exit 1
esac

# runs latex 3x, regardless
#$tcmd "$tex" myhtml >&2
# so use my own re-running code, instead
# by extracting the rather complex latex command from the shell script:
fcmd() {
  local res="$(egrep " $1" "$tcmd" | head -n 1)"
  res="${res//\$1/$tex}"
  res="${res//\$2/myhtml}"
  res="${res//\$[345]}"
  printf %s "$res"
}
lcmd="$(fcmd "${tcmd##*/ht} .*\\\$1")"
lcmd="${lcmd//$tex}"
largs="${lcmd#*[^ ] }"
lcmd="$(echo "$lcmd" | while read -r a b; do echo "$a"; done)"
tex4htcmd="$(fcmd 'tex4ht -f')"
t4htcmd="$(fcmd 't4ht -f')"
if [ -z "$lcmd" -o -z "$tex4htcmd" -o -z "$t4htcmd" ]; then
  echo "Unable to extract commands from $tcmd"
  exit 1
fi
# And running make:
# I don't pass down $(MAKE).  Sorry.
make -C"$tdir" latexcmd="$lcmd" latexargs="$largs" "$dir/${tex%.tex}.dvi"
# and the rest:
eval "$tex4htcmd"; eval "$t4htcmd"

imgs=
rm -rf "$tdir/${1%.tex}_img"
for x in *.png *.svg; do
  imgs="s~src=\"$x\"~src=\"${1%.tex}_img/$x\"~g;$imgs"
  mkdir -p "$tdir/${1%.tex}_img"
  mv -f "$x" "$tdir/${1%.tex}_img" 2>/dev/null || :
done
for x in *.html; do
  case "$x" in
    "$outf") is_outf=y ;;
    *) is_outf=n ;;
  esac
  "$tdir"/tex4ht_postproc "$base" $is_outf <"$x" | \
    sed -e "$imgs" >"$tdir/$x"
  
done

(
  <<Print attachments as HTML comment>>
) >>"$tdir/$outf"
@

<<Print attachments as HTML comment>>=
# Attach original file as comment
# adds 64 to <>- to avoid HTML comment escapes
echo "<!--"
at="`sed -n -e 's/^\\\\embedfile.*{\([^}]*\)}%*$/\1/p' $noweb | sort -u`"
echo begin 644 ${noweb%.*}.tar.gz
eval tar chf - $noweb_order $at | gzip | \
  uuencode "${noweb%.*}".tar.gz | tail -n +2 | tr '<>-' '|~m'
echo "-->"
@

<<Clean built files>>=
rm -rf $(patsubst %.nw,%_img,$(NOWEB_NINCL))
@

The HTML cleanup is done using a C++ program, once again because awk
and sed are too slow for at least one of the tasks.

<<C Build Executables>>=
tex4ht_postproc \
@

\lstset{language=C++}
<<tex4ht_postproc.c++>>=
<<Common C++ Header>>

int main(int argc, char **argv)
{
  <<Initialize TeX4ht post-processor>>

  string line;
  while(getline(cin, line).good() || line.size()) {
    <<Post-process TeX4ht line>>
    cout << line << '\n';
  }
  return 0;
}
@

<<Initialize TeX4ht post-processor>>=
if(argc != 3)
  exit(1);
const char *base = argv[1];
bool is_outf = !strcmp(argv[2], "y");
@

<<Post-process TeX4ht line>>=
if(!line.compare(0, 6, "<html ")) {
  puts("<!--\n"
       "Generated using noweb+tex4ht;"
       " original .nw source attached in comment at end");
  if(is_outf)
    printf("\n"
	   "Extract with uudecode, or, if uudecode chokes, use:\n"
	   "   tr '|~m`' '<>- ' | uudecode -o '%s.nw.gz'\n"
	   "-->\n", base);
  else
    puts(" of top-level HTML (see that for details on how to extract). -->");
}
@

That task, in particular, is to compress the number of color styles.  TeX4ht
produces a new color style for every use of color, even if it is the same
color as a previous use.  To fix this, the color styles are extracted from
the CSS file, and a hash array is built to associate each with the first
style of the same color.  Then, all duplicates are filtered from the CSS
file and converted to the first version in the HTML.

<<Initialize TeX4ht post-processor>>=
regex_t col_re;
regmatch_t col_rem[3];
<<nwbuild Regex Macros>>
ec_regcomp(col_re, "^span#textcolor([0-9]+)(\\{.*\\})", 0);
string_map textcolor, rootcolor;
@

<<Copy TeX4ht CSS to stdout>>=
// way too many color styles are generated: merge them
ifstream CSS(string(base) + ".css");
while(getline(CSS, line).good() || line.size()) {
  if(!regexec(&col_re, line.c_str(), 3, col_rem, 0)) {
    string colorno = line_match(col_rem[1]),
           colordef = line_match(col_rem[2]);
    auto rcolor = rootcolor.find(colordef);
    if(rcolor != rootcolor.end())
      textcolor[colorno] = rcolor->second;
    else {
      rootcolor[colordef] = textcolor[colorno] = colorno;
      cout << line << '\n';
    }
  } else
    cout << line << '\n';
}
CSS.close();
@

<<Initialize TeX4ht post-processor>>=
const char spn[] = "<span id=\"textcolor";
@

<<Post-process TeX4ht line>>=
for(unsigned x = 0; (x = line.find(spn, x)) < line.size(); ) {
  x += sizeof(spn) - 1;
  unsigned xe;
  for(xe = x; xe < line.size() && isdigit(line[xe]); xe++);
  if(xe == x)
    break;
  string ox = line.substr(x, xe - x);
  string &nx = textcolor[ox];
  if(nx != ox) {
    line.replace(x, xe - x, nx);
    x += nx.size() + 1;
  } else
    x = xe + 1;
}
@

The next post-processing task is to reduce the number of files to one, if
possible.  This is done by in-lining the CSS.  TeX4ht will do this with the
right option, but only if run twice, which effectively runs \LaTeX{} six
times.  Instead, it is just generated directly into the output file during
hash table generation.

<<Post-process TeX4ht line>>=
if(line.find("<meta name=\"date") < line.size()) {
  cout << line << "\n<style type=\"text/css\">\n<!--\n";
  <<Copy TeX4ht CSS to stdout>>
  cout << "//-->\n</style>\n";
  continue;
}
@

The remainder of the adjustments are minor tweaks to the HTML output.

<<Post-process TeX4ht line>>=
// tex4ht inserts a newline before each chunk
// no easy way to deal right now, so leave it
@

<<Post-process TeX4ht line>>=
// tex4ht splits lines mid-tag; rejoin at least the span tags
while(line.size() > 6 && line.substr(line.size() - 6) == "<span ") {
  string line2;
  getline(cin, line2);
  line += line2;
}
@

<<Post-process TeX4ht line>>=
// Merge consecutive spans on same line w/ same class into one span
for(unsigned off = 0; (off = line.find("<span class\"", off)) < line.size(); ) {
  unsigned name = off, ename;
  for(ename = off; ename < line.size() && line[ename] != '"'; ename++);
  unsigned espan = line.find("</span>", ename);
  if(espan >= line.size())
    break;
  if(line.compare(espan + 7, 13, "<span class=\"") ||
     line.compare(espan + 20, ename - name, line.substr(name)) ||
     line[espan + 20 + ename - name] != '"') {
    off = espan + 7;
    continue;
  }
  off = espan;
  line.erase(espan, 22 + ename - name);
}
@

<<Post-process TeX4ht line>>=
// Remove plain text spans
for(unsigned off = 0; (off = line.find("<span class\"ecrm-0900\">", off)) < line.size(); ) {
  unsigned eoff = line.find("</span>", off + 23);
  if(eoff < line.size()) {
    line.erase(eoff, 7);
    line.erase(off, 23);
  }
}
@

<<Post-process TeX4ht line>>=
// I prefer mostly plain text for the web, so:
// remove ligatures
for(unsigned off = 0; (off = line.find("&#xFB0", off)) < line.size(); ) {
  if(off + 7 >= line.size() || line[off + 7] != ';' ||
     (line[off + 6] != '0' && line[off + 6] != '1')) {
    off += 6;
    continue;
  }
  line[off] = 'f';
  line[off + 1] = line[off + 6] == '0' ? 'i' : 'l';
  line.erase(off += 2, 6);
}
// remove hex nbsp
for(unsigned off = 0; (off = line.find("&#x00A0;", off)) < line.size(); )
  line.replace(off, 8, "&nbsp;");
// remove unicode quotes
for(unsigned off = 0; (off = line.find("&#x821", off)) < line.size(); ) {
  if(off + 7 >= line.size() || line[off + 7] != ';' ||
     (line[off + 6] != '7' && line[off + 6] != '6')) {
    off += 6;
    continue;
  }
  line[off] = line[off + 6] == '7' ? '\'' : '`';
  line.erase(++off, 7);
}
@

<<Post-process TeX4ht line>>=
// remove trailing space
while(line.size() > 0 && line[line.size() - 1] == ' ')
  line.erase(line.size() - 1, 1);
@

<<Post-process TeX4ht line>>=
// bytefield images need <br> between lines
if(!regexec(&img_re1, line.c_str(), 0, NULL, 0))
  line.append("<br>");
// got a timing diagram with unescaped <.."..> in alt text
if(!regexec(&img_re2, line.c_str(), 2, img_rem, 0))
  line = line_match(img_rem[1]);
@

<<Initialize TeX4ht post-processor>>=
regex_t img_re1, img_re2;
regmatch_t img_rem[2];
ec_regcomp(img_re1, "src=\"[^>]*alt=\"PICT\"[^>]*>$", REG_NOSUB);
ec_regcomp(img_re2, "(src=\"[^>]*alt=\"[^<>\"]*)<", 0);
@

\subsection{HTML from [[l2h]]}

Another way is to use noweb's included [[l2h]] filter.  This once again
requires some filtering, which is complicated enough that it's done in a
separate script ([[nw2html]]) instead of inline in the makefile.  In
addition, highlighting of chunks must produce HTML, so a small conversion
script ([[htmlhl]]) is used.  This uses the ordered list output, as the
ordered list CSS can be adjusted more easily to look like the \LaTeX{}
output.

\lstset{language=sh}
<<Build Script Executables>>=
nw2html \
htmlhl \
@

<<htmlhl>>=
#!/bin/sh
<<Common noweb Warning>>

case $HLPROG_TYPE in
  highlight-[23]) echo '<ol class="code">' ;;
esac
<<Do generic per-chunk highlighting preparation>>
case $HLPROG_TYPE in
  highlight-2)
    highlight -l --ordered-list -f -S $1 -E $HLDIR | \
      <<Filter [[highlight]] HTML output>> ;;
  highlight-3)
    highlight -l --ordered-list -f $larg | \
      <<Filter [[highlight]] HTML output>> ;;
  source-highlight)
    <<Enter [[source-highlight]] data dir>>
    # remove last line and move /pre up
    # should probably retain one line if empty
    source-highlight --data-dir=$HLDIR -f xhtml-css $larg | \
      sed -n -e 's/<pre><tt>/<pre class="code">/;x;$s%$%</pre>%;1!p'
esac
case $HLPROG_TYPE in
  highlight-[23]) echo '</ol>'
esac
@

<<Filter [[highlight]] HTML output>>=
sed 's%^</li>$%<li></li>%;s/<li[^>]*>/<li><pre>/;s%</li>%</pre></li>%'
@

\lstset{language=make}
<<makefile.vars>>=
FIGS_PNG:=$(patsubst %, %.png, $(FIGS))
@

<<makefile.rules l2h>>=
nw2html:  htmlhl nwweavefilt nt-nonl nt-parm nw-nonl-preidx nw-nonl-postidx \
          nw-parm-preidx nw-parm-postidx

%.html: %.nw $(FIGS_PNG) nw2html
	./nw2html $< $@ "$(call tree,$<)"
@

<<Clean built files>>=
rm -f $(FIGS_PNG)
@

The wrapper script tacks on a similar header to what the TeX4ht script
produces, but using the direct HTML output of [[highlight]] for the CSS, and
making adjustments to make the HTML appear at least vaguely similar to the
PDF.

\lstset{language=sh}
<<nw2html>>=
#!/bin/sh
<<Common noweb Warning>>

noweb="$1"
outf="$2"
noweb_order="$3"

# tex version allows list: prefix - ignore if there
HL_THEME="${HL_THEME#list:}"

# place to stash highlighter temporaries
trap "rm -rf /tmp/latexhl.*.$$" 0
export HLDIR=/tmp/latexhl.hl.$$
mkdir -p $HLDIR

<<Prepare for weave>>

(
# This replaces l2h's header
echo "<html><!--
Generated using noweb; original .nw source attached as tar.gz in comment at end
Extract with uudecode, or, if uudecode chokes, use:
   tr '|~m\`' '<>- ' | uudecode -o '${noweb%.*}.tar.gz'
-->
 <head>
<meta name=\"generator\" content=\"noweave -html -filter l2h, $HLPROG_TYPE\">
  <style type=\"text/css\">
  <!--"
case "$HLPROG_TYPE" in
  highlight-[23])
    highlight ${HL_THEME:+-s $HL_THEME} -S c -I </dev/null | sed -n -e '
      # replace white bg with very light gray to distinguish from text
      s/\(background-color:#\)ffffff/\1fafafa/g
      # remove extra junk from block style
      /pre.hl/{
        s/pre.hl/ .code/
        s/font-size:.*;/font-size: smaller;/
        p
        # make links look like normal text
        s/.code/& a/;s/margin.*;//
        p
      }
      # fix class names
      /^\./{s/^/.code /;s/\.hl\././;p;}'
    # option: line #s or not (leave both in HTML for user to see)
    echo ".code li {display: block;}"
    echo "/* either above line to remove line #s, or make space for line #s with: */"
    echo "/* .code {margin-left: 2em;} */"
    ;;
  source-highlight)
    <<Get all [[source-highlight]] color names>>
    cd $HLDIR
    source-highlight ${HL_THEME:+--style-css-file=$HL_THEME.css} --doc -f xhtml \
                     --lang-def=source-hl-elts.lang < source-hl-elts | \
      sed -n -e '
        # replace white bg with very light gray to distinguish from text
        s/\(background-color: *#\)ffffff/\1fafafa/ig
	s/\(background-color: *\)white/\1#fafafa/ig
	s/<body style="\(.*\)">/.code { \1; font-size: smaller }/p
	ta; :a s%</span>%%g;T
	s/\(.*>\)\([^>]*\)$/.code .\2 { \1 }/
	s/<span style="//g; s/">/; /g;p'
    ;;
esac
# option: border or not (leave both in HTML for user to see)
echo "/* optional: add border around code */"
echo "/* .code {border: thin solid black;} */"
# display code within chunk names correctly
echo "code {font-style: normal;}"
# display code chunk immediately below chunk name
echo "pre.defn {margin-bottom: 0;}"
# use same margins as LaTeX
echo ".code {margin-left: 10pt; margin-right: 10pt; margin-top: 0; padding: 1px}"
# squeeze lines together
echo ".code li,pre {margin-top: 0; margin-bottom: 0;}"
# make chunk refs more legible by removing underline
echo ".code a {text-decoration: none;}"
echo "pre.dfn a {text-decoration: none;}"
echo "//-->"
echo "</style>"
) >"$outf"
@

Next, it runs [[noweave]] with the [[l2h]] filter.  This filter runs
after the highlighter.  The index generator tries to make switching to
a code chunk also show the documentation by passing [[-docanchor 10]]
to [[noidx]].  This places anchors 10 lines above the code chunk,
regardless of what is actually there.  This can be very bad if that is
almost anything but plain text, so instead, [[@xref]] is removed using
a filter after [[-index]], and then [[nodix]] is called directly using
another filter, but without that [[-docanchor]] option.  If readers
want to read the documentation for a code chunk, they will need to
just scroll up manually.  Doing this eliminates the chunk reference
without keeping a placeholder where the index should go, causing the
reference to always appear at the end.  To fix this, l2h must emit a
different tag, which can then be replaced in the post-filter.  In any
case, post processing of the HTML is required.  After the document is
generated, the source is once again tacked on as an HTML comment.

<<nw2html>>=
( (
  ln=`grep -n '^%%% latex preamble' "$noweb" | cut -d: -f1 | head -n 1`
  head -n $ln "$noweb"
  cat <<"EOF"
<<preamble.l2h>>
% l2h substitution nowebchunks </nowebchunks>
% l2h substitution nowebindex </nowebindex>
<<preamble.tex>>
EOF
  <<Further l2h preamble>>
  tail -n +$((ln+1)) "$noweb"
) | <<Pre-process before weave>>
  noweave ${NW_UTF8:+-t} -html \
          -filter "./nw-parm-preidx $noweb_order |
	     ./nw-nonl-preidx $noweb_order |
             ./nwweavefilt ./htmlhl $noweb_order | l2h" -indexfrom /dev/null \
          -filter 'sed -n "<<Filter out noidx results>>" | noidx |
	           ./nw-nonl-postidx | ./nw-parm-postidx' \
	       | <<Post-process HTML after weave>>
         cat

<<Print attachments as HTML comment>>
) >>"$outf"
@

\lstset{language=sed}
<<Filter out noidx results>>=
/^@nl/{
  h;
:a
  n;
  /^@nl/{H;ba};
  /^@index begin/{
    :b
      n;
      /^@index end/b;
      bb;
  };
  x;p;g;
};
/^@index begin/{
  :b
    n;
    /^@index end/b;
    bb;
};
s%</nowebindex>%<nowebindex>%;
s%</nowebchunks>%<nowebchunks>%;
/^@xref/!p
@

The \LaTeX{} preamble is supplemented with directives for [[l2h]], which are
comments in the form \texttt{\% l2h \emph{directive} \emph{token}
\emph{arguments}}.  A new command, [[\hypertxt]], is introduced as well,
because doing hyperrefs with custom text requires an optional argument, and
optional arguments are not supported by [[l2h]] directives.  The RCS keyword
commands must be inserted by extracting them from the original file, or they
will end up coming from the build document every time.

\lstset{language=TeX}
<<preamble.tex>>=
\usepackage{comment}
\excludecomment{rawhtml}
% for l2h, since it doesn't understand opt args
\def\hypertxt#1#2{\hyperref[#1]{#2}}
@

<<preamble.l2h>>=
% Overrides for noweb's l2h
%
% Things it doesn't understand
% template is {, A=arg, [=optarg, C=optarg/save, ==assign, +=white, (=..)
% l2h macro hypertxt 2 <a href="###$1">#2</a>
% l2h macro url 1 <a href="#$1">#$1</a>
% l2h envblock centering center
% l2h ignore discretionary {{{
% l2h ignore nwcodepenalty =
% l2h ignore edef A{
% l2h ignore excludecomment {
% l2h ignore the
% l2h ignore toks A={
% l2h ignore lstset {
% l2h ignore lstloadlanguages {
% l2h ignore lstdefinelanguage [{[{{
% l2h ignore definecolor {{{
% l2h ignore ifpdf
% l2h ignore ifxetex
% l2h ignore bkframefalse
% l2h ignore bkcounttrue
% l2h ignore ,
% l2h ignore break
% l2h substitution textbar |
% l2h substitution textasciitilde ~
% l2h substitution textasciicircum ^
% l2h substitution sim ~
% l2h substitution textbackslash \
% l2h substitution backslash \
% l2h ignoreenv webcode
% l2h macro input 1 <!-- input #$1 -->
% l2h ignore embedfile [{
% \RCS still needs to go into header to get stripped out properly
% l2h ignore RCS
% l2h ignore catcode
%
% doesn't understand extensionless image names, or that I want png
% l2h argblock includegraphics <img#src=" .png"><br/> [
%
% I prefer my footnotes shrunken and italicized
% l2h argblock footnote <font#size=2><b>[</b><em> </em><b>]</b></font>
%
% l2h ignore select@language {
@

\lstset{language=sh}
<<Further l2h preamble>>=
sed -f /tmp/latexhl.doc.$$ "$noweb" | grep '^\\RCS \$' | while IFS= read -r l; do
  l="${l#*\$}"
  l="${l%\$*}"
  echo "% l2h macro RCS${l%%:*} 0 ${l#*:}"
done
echo "% l2h substitution jobname ${noweb%.*}"
@

The first required filter is one supplied with [[l2h]]:  generating the
hyperlinked table of contents.

<<Post-process HTML after weave>>=
htmltoc -1234 | \
@

Next, the hidden sections must be removed.  The top-level table of contents
entries may as well be removed at the same time.  These are always the
document title and the abstract, which are not in the \LaTeX{} table of
contents, either.

<<Post-process HTML after weave>>=
sed -n -e '<<Filter noweave output with removal>>' | \
@

\lstset{language=sed}
<<Filter noweave output with removal>>=
# strip out all before first header (title)
/<h1>/,${
  #strip out comment blocks delimited by <!--> & <--> on their own line
  # this is sort of unsafe, but it works for this document
  /^<!-->$/{
   :a
    n
    /^<-->$/bb
    ba
   :b
    n
  }
  #strip out the first top-level TOC - always title + abstract
  /^<tableofcontents>/{
   a\<h2>Table of Contents</h2>
   # find the document title
   :c p;n;/^ *<li>/!bc
   # skip to first actual TOC item
   :d n;/^ *<li>/!bd
   # there is no way to detect at this point whether or not an abstract
   # is there so hard-code the word "Abstract", which is very unsafe,
   # since it is trivial to rename the abstract and chpater one could
   # very well contain that word.
   /Abstract/bd
  }
  p
}
@

Then the title needs to actually be reinserted as the HTML document title.

\lstset{language=sh}
<<Post-process HTML after weave>>=
sed -e '<<Filter noweave output without removal>>' | \
@

\lstset{language=sed}
<<Filter noweave output without removal>>=
# finish up head, copying 1st header as title
1{
  h
  s/h1>/title>/g
  s/<a name=[^>]*>//
  s%</a>%%
  s%/title>%&\n </head>\n<body>%
  p
  x
}
@

Some adjustments need to be made to code chunk headers.  This allows them to
be inset and gives them similar appearance to the \LaTeX{} output.

<<Filter noweave output without removal>>=
# Give style to pre directives around defn start, and limit pre to 1 line
/<dfn>/{
  s/<pre>/<pre class="dfn">/;s%$%</pre>%
 :a
  # newer versions of highlight added "hl " to the class names
  s/class="hl /class="/g
  # Use same bracket style as LaTeX (<sigh> w3m/links/IE hate this)
  # mathematical seems lowered, so use discouraged non-mathematical
  #s/<i>&lt;/<i>\&#x2329;/g; s%&gt;</i>%\&#x232A;</i>%g
  #s/<dfn>&lt;/<dfn>\&#x3008;/; s%&gt;\(.*</dfn>\)%\&#x3009;\1%
  n;/^<\/pre>/!ba
  s%^</pre>%%
}
@

Finally, there are some miscellaneous minor tweaks.

<<Filter noweave output without removal>>=
# shrink & italicize "defined here" bits (from %def directives or autodefs)
s%\(<blockquote>\)\(Defines\)%\1<font size=1><em>\2%g
# it is hard to determine end of above group, so make HTML a litte bad
s%\(</blockquote>\)%</em></font>\1%g
@

<<Filter noweave output without removal>>=
# strip out diagrams from index
# actually, it would better to strip out all hidden section chunks
s%^<li>.*\.dia&gt;</i></a>:.*%%
@

<<Filter noweave output without removal>>=
# strip out *s on their own line - not sure what that is all about
s%<a name=[^>]*>\*</a>$%%
s/<br>\*$/<br>/
@

<<Filter noweave output without removal>>=
# Yet another useful subst, but w3m/links/IE hate it
#s/---/&mdash;/g
@

<<Filter noweave output without removal>>=
# l2h puts marks where hrefs point to, but that is not really necessary
s%<b>\[\*\]</b>%%g;s/\[\*\]//g
@

<<Filter noweave output without removal>>=
# l2h does not handle quotes at all, so just remove duplicate
#s/``/\&#8216;/g;s/'\'\''/\&#8217;/g
s/``/`/g;s/'\'\'/\''/g
@

One of those minor tweaks is to move the footnotes into their own section at
the bottom of the document.  Since [[sed]] is not up to the task, [[awk]] is
used.

\lstset{language=sh}
<<Post-process HTML after weave>>=
awk '<<Move noweave filtering with awk>>' | \
@

\lstset{language=awk}
<<Move noweave filtering with awk>>=
# May as well just move footnotes to the bottom
BEGIN {
  # uses FS to extract footnote text
  FS="<font size=2><b>.</b><em>|</em><b>.</b></font>";
  # odd is 1 if previous footnote spanned lines
  odd=0;
  # fn is the footnote counter
  fn=0;
  # fnt is the complete text of all footnotes
  fnt="";
}
# if NF > 1, a footnote has been detected
# odd is set to 1 if a footnote spans lines
NF > 1 || odd {
  # accumulate non-footnote text into ln
  ln="";
  i=0;
  if(odd) ft=ft "\n";
  while(i < NF) {
    i++;
    if(i > 1 && i % 2 == odd) {
      fn++;
      ln=ln "<sup><a href=\"#fn" fn "\">" fn "</a></sup>";
      ft=ft "\n<hr/>\n<a name=\"fn" fn "\"><b>" fn "</b>.</a> "; 
    }
    if(i % 2 == odd) ft = ft $i; else ln = ln $i;
  }
  if(!odd || NF > 1) print ln;
  if(NF % 2 == 0) odd = 1 - odd;
}
# dump all footnotes at the end
$0 ~ "</body>" {
  if(fn > 0) print "<hr/><h3>Footnotes</h3>" ft
  print
}
# print everything that is not a footnote
NF <= 1 && !odd && ($0 !~ "</body>")
@

Copying in the highlighted code must be done after [[noweave]] is finished,
or else it would need raw HTML sentinels.  On the other hand, verbatim
code should be copied in before weaving, and in fact it already has
above.

\lstset{language=sh}
<<nw2html>>=
# insert highlighted extracted chunks/files
<<For each reinserted code chunk>>
  test $lang = verbatim && continue
  printf '%s\\\n' '/<!-- input '"$fesc"'.tex --> <!-- '$lang'-->/{
       s/<!--.*-->//;i'
  eval notangle '-R"$f"' $noweb_order 2>/dev/null | ./htmlhl $lang | \
    sed 's/\\/\\\\/g;$!s/$/\\/'
  printf '}\n'
done >/tmp/latexhl.fmt.$$
# insert highlighted C prototypes
<<For each reinserted C prototype>>
  printf '%s\\\n' "/<!-- $f prototype-->/{
       s/<!--.*-->//;i"
  <<Extract and format a C prototype from [[cproto.h]]>> | ./htmlhl C | \
    sed '$!s/$/\\/'
  echo '}'
done >> /tmp/latexhl.fmt.$$
sed -i -f /tmp/latexhl.fmt.$$ "$outf"  # -i is GNU sed only
@

\lstset{language=make}
<<makefile.rules>>=
nw2html: cproto.h $(CHEADERS) $(CXXHEADERS)
@

Since the sed scripts and simplicity of [[l2h]] can cause the HTML to not be
standards compliant any more, HTML Tidy%
\footnote{\url{http://tidy.sourceforge.net}}
is used to clean up the output.  When switching to [[xelatex]] for
Unicode input, HTML Tidy needs to switch to UTF-8 input as well.

<<makefile.vars>>=
export TIDY_FLAGS
@

\lstset{language=sed}
<<Filter noweave output without removal>>=
# tidy gives errors on unknown tags
s%</*tableofcontents>%%
@

<<Filter noweave output without removal>>=
# rather than trying to put TOC /li in right place, just remove them all
/href="#toc/s%</li>%%
@

\lstset{language=sh}
<<nw2html>>=
tidy -q -wrap 0 -i -asxhtml -m ${NW_UTF8:+-utf8} "$outf" 2>tidy.log
# report unexpected errors
fgrep -ivf- tidy.log <<EOF >/dev/null || :
missing <!DOCTYPE>
unexpected </em>
unexpected </font>
replacing unexpected em
missing </a> before <a>
discarding unexpected </a>
lacks "summary" attribute
lacks "alt" attribute
column 18 - Warning: <a> anchor "NW
trimming empty
EOF
@

<<Clean temporary files>>=
rm -f tidy.log
@

Generation of the figures depends on
ImageMagick%
\footnote{\url{http://www.imagemagick.org}}
with GhostScript support to convert them to PNG format, even though using GhostScript
directly would have worked as well.

\lstset{language=make}
<<makefile.rules>>=
%.png: %.eps
	convert -density 144x144 -comment '' $< -transparent white $@
@

\subsection{Method Selection}

Unfortunately, neither formatter is obviously better.  TeX4ht mangles its
output a bit (color in particular) and takes forever to run.  noweb's l2h
filter is quicker, but is pretty stupid and unconfigurable for things like
tables and footnotes.  For now, I prefer the speed and accuracy of l2h.

<<makefile.config>>=
# Program to use for generating HTML formatted source code
# Set to tex4ht or l2h
HTML_CONV:=l2h

@

<<makefile.rules>>=
ifeq ($(HTML_CONV),l2h)
<<makefile.rules l2h>>
else
ifeq ($(HTML_CONV),tex4ht)
<<makefile.rules tex4ht>>
else
$(error Unknown HTML_CONV conversion method; use l2h or tex4ht)
endif
endif
@

\section{Usage}

% Begin-doc build-usage-1
Users of this document fall into at least two categories: ones who wish to
create documents that use this build system, and ones who wish to build
packages which were created using this build system.  This section is for
the former.
% End-doc build-usage-1
Instructions for the latter were included on the first page,
but should also be included in any document using this system:

% Begin-doc build-usage
\begin{quote}
\verb!\input{build-doc.tex} %%% doc!
\end{quote}

Of course the instructions for document users are required to build and
test the project for document creators as well.  After creating the
makefile, you can use \texttt{make} to build and test.  The standard
targets are \texttt{all} (the default), \texttt{bin},
\texttt{install}, \texttt{doc}, \texttt{misc}, \texttt{clean},
\texttt{distclean}, \texttt{count}, \texttt{check}, \texttt{test-bin}, and
\texttt{test}.  In particular, the \texttt{check} target can help
ensure that changes to a source file will not result in weird errors
due to lost chunks or mistyped chunk names.

The first few lines of a noweb file are \LaTeX{} code.  This must at least
include the \verb|\documentclass| directive.  It can also include the build
system's preamble in the place marked by \verb!%%% latex preamble!. Copying
the beginning of this document to start with would not be a bad idea:

\begin{quote}
\verb!% Build with noweb:!\\*
\verb!%  notangle -t8 build.nw > makefile!\\*
\verb!%  make!\\*
\verb!\documentclass[twoside,english]{article}!\\*
\verb!\usepackage[letterpaper,rmargin=1.5in,bmargin=1in]{geometry}!\\*
\verb!%%% latex preamble!\\*
\verb!\RCS $!\verb!Id$!\\*
\verb!\RCS $!\verb!Revision$!\\*
\\*
\verb!\begin{document}!\\*
\\*
\verb!\title{!\ldots\}\\*
\verb!\author{!\ldots\}\\*
\verb!\date{Revision \RCSRevision}!\\*
\\*
\verb!\maketitle!
\end{quote}

Dependencies on other files should be listed somewhere (usually near the
top), as well.  They are specified by comments of the form 
\verb!%%% requires !\textit{name} on lines by themselves.  Only direct
dependencies need to be listed, as a full tree will be generated.  The
\texttt{.nw} extension is optional.  For example, somewhere in your project,
you will need to depend on this file:

\begin{quote}
\verb!%%% requires build!
\end{quote}

Chunks defined in a document extend chunks defined in documents on which it
depends.  Chunks defined in other documents can be used, as well.  In both
cases, they will not be weaved correctly.  Chunks which are extended
will be indicated with the original source file in parentheses at the
start of the chunk name; this format is used mainly to move external
referneces to the top of the chunk index and sort them by source name.
However, the first extension will have no ``previous chunk'' link,
rather than pointing to the source document's definition.  Uses are
also indicated with the original file name, but they always claim to
be undefined.  This is somewhat alleviated by the fact that undefined
references are labeled with the word imported rather than undefined,
but the only way to remove that indicator is to define the chunk
elsewhere for extension (e.g. in an appendix) as neutral as possible
(blank or a comment in the source language).
\input{extref-dis.tex} %%% doc

Within the document, several extensions can be used:

\begin{itemize}
\item Language definitions: the listing package's
\verb!\lstset{language=!\textit{lang}\verb!}! directive is used to set the
language for any subsequent code chunks.  The default language is C.

\item Hidden sections: any figures or other files which are normally
included in the document in a processed form can be stored in-line in the
noweb document in their raw form in hidden sections.  Each hidden section is
delimited by \verb|<!-->| on its own line at the start, and \verb!<-->! on
its own line at the end.  This is intended to be used as follows:

\begin{quote}
\footnotesize
\verb!% hidden section - notangle chunks that shouldn't appear in printed output!\\*
\verb!% mainly for diagrams, which are extracted and included in visual form.!\\*
\verb!\begin{rawhtml}!\\*
\verb|<!-->|\\*
\verb!\end{rawhtml}!\\*
\verb!\iffalse!\\*

\ldots{} hidden material \ldots{}\\

\verb!\fi!\\*
\verb!\begin{rawhtml}!\\*
\verb!<-->!\\*
\verb!\end{rawhtml}!\\*

\end{quote}
These are also provided as reinsertable documentation chunks, so the
following is equivalent:
\begin{quote}
\verb!\input{begin-hidden.tex} %%% doc!\\*

\ldots{} hidden material \ldots{}\\

\verb!\input{end-hidden.tex} %%% doc!\\*
\end{quote}
% Begin-doc begin-hidden
% hidden section - notangle chunks that shouldn't appear in printed output
% mainly for diagrams, which are extracted and included in visual form.
\begin{rawhtml}
<!-->
\end{rawhtml}
\iffalse
% End-doc begin-hidden
% Begin-doc end-hidden
\fi
\begin{rawhtml}
<-->
\end{rawhtml}
% End-doc end-hidden

noweb doesn't actually strip these out --- iffalse does that.  Because
noweb generates cross-reference information and the chunk index after
the last chunk, the last chunk may not be commented out in this way.
If you see a message like ``LaTeX Warning: The are no
\textbackslash{}nowebchunks on input line \ldots{}'', there are eiher
no code chunks at all, or the last code chunk was commented out.

\item Syntax highlighted, fully extracted code chunks: a code chunk can be
repeated in e.g. a users' guide appendix, fully extracted and syntax
highlighted, by using a special [[\input]] directive:
\verb!\input{!\emph{chunk-name}\verb!.tex} % !\emph{language}.  This
directive will be replaced by the contents of the named chunk.  For example,
the configurable variables are included here using:

\begin{quote}
\verb!\input{makefile.config.tex} % make!
\end{quote}

% End-doc build-usage
\input{makefile.config.tex} % make

% Begin-doc build-usage
One use of this feature which should be in every document is to list the
RCS/CVS/subversion ID tag of all source files from which the document was
built:
\begin{quote}
This document was generated from the following sources, all of which are
attached to the original electronic forms of this document:\\
\verb!\input{Sources.tex} % txt!
\end{quote}

\item noweb extensions:  by default, filters are used to enhance the
noweb tools, collectively known as noweb-enhanced.  The parameter
substitution filters are described and implemented in a separate
document, [[parm.nw]].  The enhancements consist of the following
executables, the last of which requires noweb 2.x:
\begin{itemize}
\item [[nt-parm]] and [[nt-monl]] are filters to be added to
[[notangle]] using its [[-filter]] option.
\item [[nw-parm-preidx]] and [[nw-nonl-preidx]] are filters to be
added to [[noweave]] using its [[-filter]] option before indexing
options.  They take an optional list of noweb files as their
arguments; this should be the full list of files (optionally excluding
the one being weaved) used for tangling.
\item [[nw-parm-postidx]] and [[nw-nonl-postidx]] are filters to be
added to [[noweave]] using its [[-filter]] option after indexing options.
\item [[noroots-enh]] is a replacement for [[noroots]] which supports
the extensions.  This requires noweb 2.x, as it is a copy of that
version's [[noroots]] which simply inserts the [[nt-parm]] and
[[nt-nonl]] filters into the pipeline.
\end{itemize}

These provide the following extensions:
\input{nw-enh-syntax.tex} %%% doc

\item Customized tangling:  most rules use variables to allow
overriding the tangling process.  The primary makefiles are excluded,
as are a few other documented instances.  As with any other variables,
these may be set on a per-target basis in GNU make.

\begin{itemize}
\item [[NOTANGLE]] sets the name of the executable.  By default, this
is the enhanced version described above.
\item [[NOTANGLE_OPTS]] sets additional command-line options.
Normally, this would be used to set [[-t8]] for files needing tabs.
This can also be used to add more filters to the noweb pipeline.
\item [[NOTANGLE_POSTPROC]] is a shell pipeline to be appended to the
tangle process.  It must begin with the pipe symbol (|).  Generally,
assignment should be done using [[+=]] rather than [[=]] in order to
allow previously defined filters to work.  For example:
\begin{quote}
[[<<makefile.vars>>=]]\\
[[$(MY_FILES): NOTANGLE_POSTPROC+=|sed -e '1i\copyright blurb']]\\
[[@]]
\end{quote}
\item [[NOTANGLE_DEP]] sets additional dependencies before tangling,
in case generated files are used in [[NOTANGLE]] or
[[NOTANGLE_POSTPROC]]. By default, this depends on the enhanced
tangler.  This should only be set for global effect, since variables
in dependencies cannot be target-specific.  If a target-specific
dependency is needed, simply add the dependency manually while
setting the other target-specific variables.
\item [[NOTANGLE_CHUNK]] is a single-argument function which
implements the above variables.  This allows convenient creation of
rules to create special files, without having to remember the long
variable reference-heavy command line.  For example:
\begin{quote}
[[<<makefile.rules>>=
$(MY_SUID_SCRIPTS): $(NOWEB) $(NOTANGLE_DEP)
	-$(call NOTANGLE_CHUNK,$@) >$@
	chmod +xs $@
@@]]
\end{quote}
\end{itemize}
\item Repeated documentation: a separate noweb-like macro facility which
only does substitutions can be used to repeat documentation in e.g. a users'
guide in an appendix.  Such documentation chunks are delimited by
\verb!% Begin-doc !\emph{chunk-name} and \verb!% End-doc !\emph{chunk-name}
at the beginning of a line.  They are inserted in place of any occurence of
\verb!\input {!\textit{chunk-name}\verb!.tex} %%% doc! at the beginning of a
line.

\item Simple input directives: a file can be included verbatim using
\verb!\input{!\emph{file}\verb!.nw}!.  This actually substitutes the file in
place rather than doing an actual include so that weaving and tangling
operate correctly.

\item Documentation attachments: any files which should be attached to the
PDF or HTML output can be attached using the [[embedfile]] package's
[[\embedfile]] directive.

\end{itemize}

Somewhere in the document, the code chunk index should be added.
Right now, an identifier index is not supported.

\begin{quote}
\footnotesize
\begin{verbatim}
\section{Code Index}
\nowebchunks

% note: no identifier indexing is done right now

\begin{rawhtml}
<!-->
\end{rawhtml}
%\vspace{1ex}
%\hrule
%\vspace{1ex}
\begin{rawhtml}
<-->
\end{rawhtml}

%\nowebindex

\end{verbatim}
\end{quote}

Of course the document must be ended with \verb|\end{document}|, like
any \LaTeX{} document.

\begin{quote}
\footnotesize
\begin{verbatim}
\end{document}
\end{verbatim}
\end{quote}

Once the documentation has been built, the PDF output needs to be checked
for overflows.  Horizontal overflows in code chunks are hard to detect, as
they do not always generate [[Overfull \hbox]] messages.  They can be
corrected by reformatting the source code.  Vertical overflows as the result
of problems with the [[framed]] package can be detected by
[[Overfull \vbox]] messages, and can be corrected by prefixing chunks which
are not broken at the end of a page with a [[\break]] directive.  Naturally,
any edits to the document will require rechecking all [[\break]] directives
to see if they are still necessary, and possibly to add more.

Since I write a lot of C, a few special tricks are available for C
programs.  In addition to the C-specific chunks described below, a
special header is generated, [[cproto.h]], which contains all exported
prototypes, except for prototypes already explicitly included in
headers.  This is always included after [[<<Common C Includes>>]] in
the [[<<Common C Header>>]].  In order to facilitate writing static
functions out of order, a static prototype file is generated for each
C file as well.  This is included right after [[cproto.h]], unless the
comment \texttt{//~static\_proto} is found on a line by itself, in
which case it is placed after that comment.  That way, the static
prototypes can be placed after any private data type declarations.

In order to assist in documenting C functions, the prototype of a C
function can be repeated as a highlighted code chunk using a special
comment:

\begin{quote}
\verb!% !\emph{function-name}\verb! prototype!
\end{quote}

This prints a reformatted version of the prototype extracted from
[[cproto.h]], described above.  Any prototypes which would not be
placed there can be manually added using the [[<<C Prototypes>>]]
chunk.  In paritcular, macro prototypes should be added to this chunk.
[[<<C Prototypes>>]] is always extracted using standard [[notangle]],
regardless of [[$(NOTANGLE)]]'s setting.

Several internal variables are meant to be set or expanded in
extensions to [[<<makefile.vars>>]]:

\begin{itemize}
\item [[C_POSTPROCESS]] is a deprecated method of adding
post-procesing to C files while they are being generated.  Its value
should be a post-processing pipeline; see [[NOTANGLE_POSTPROC]] for
details.  Unlike [[NOTANGLE_POSTPROC]], this is a function which takes
the target name as its single argument.   The argument can, of course,
be ignored, and there is no difference between a simple variable
assignment and a function ignoring all arguments.
\item The [[EXTRA_CFLAGS]] variable should be extended with required
options; that way, the user can replace [[CFLAGS]] with just
optimization and debugging flags.
\item The [[CPROTO_CFLAGS]] variable should be extended with flags
required to make [[cproto]] work.
\item The [[EXTRA_LDFLAGS]] variable should be extended with required
options; that way, the user can replace [[LDFLAGS]] with just
optimization and debugging flags.
\end{itemize}

Finally, extensions to the build system can be made by adding code chunks;
they will be appended to existing values in the order of the tree, from the
most depended on file to the least depended on file (i.e., anything
depending on \texttt{build.nw} will append values \emph{after} the values set
in \texttt{build.nw}):

\paragraph{Basic build rules:}
\begin{itemize}
\item [[<<makefile.config>>]]: add user-configurable variables to the
makefile.  [[<<Installation Directory Configuration Options>>]] is the
first chunk expanded within this file in order to clump such options
together.
\item [[<<makefile.vars>>]]: add non-configurable variables to the makefile.
\item [[<<makefile.rules>>]]: add rules to the makefile.
\item [[<<Install other files>>]]: add installation commands.
\item [[<<Clean temporary files>>]]: add commands to clean temporary files.
\item [[<<Clean built files>>]]: add commands to clean built files.
\item [[<<Plain Files>>]]: add plain files to build; always add a backslash
to the end of each added line.  If additional processing other than simple
extraction is required for any file, it should instead be added to
[[<<makefile.rules>>]] as a separate rule and a dependency for the
\texttt{misc} target.  If any plain files are not in the noweb source, but
instead are distributed separately, they should instead be added to the
[[ATTACH_EXTRA]] variable, to be attached to the documentation.  That
variable can also be a make function taking the name of the file being
attached to as an argument.
\item [[<<Plain Build Files>>]]: add plain text files to be built when
required as a dependency; always add a backslash to the end of each
added line.  This is not added to any explicit target.  It is merely a
shorthand to create a rule for extracting it from noweb and cleaning
it up afterwards.  It is intended for build support files that are not
meant to be distributed.
\item [[<<Plain Files to Install>>]]: add plain text files to install.
These should already be added to [[<<Plain Files>>]].  They may be
prefixed with an installation subdirectory, which will be stripped off
to determine the source file name.  Each line must end in a backslash.
\item [[<<Script Executables>>]]: add plain text files to build as
executables; always add a backslash to the end of each added line.  Like
[[<<Plain Files>>]], no processing is done by the default rules.
\item [[<<Build Script Executables>>]]: add plain text files to build as
executables when required; always add a backslash to the end of each
added line.  This is not added to any explicit target.  It is merely a
shorthand to create a rule for extracting it from noweb and cleaning
it up afterwards.  It is intended for build support files that are not
meant to be distributed.
\item [[<<Test Scripts>>]]: add plain text files to build as executables,
but only for the [[test]] and [[test-bin]] targets.  The [[test]] target
will execute these scripts without any interpretation other than
return code checks.
\item [[<<Test Support Scripts>>]]:  names of executable scripts to
build for the [[test-bin]] target.  These scripts are not run, but are
guaranteed to be built before running any tests.
\item [[<<Additional Tests>>]]: tests that are more complicated to run
than simple scripts.  These are makefile actions.
\item [[<<Build Source>>]]: add files to the noweb-free tar distribution;
always add a backslash to the end of each added line.  In particular, any
specially processed plain files with their own rules should be added to this
list.
\item [[<<Files to Count>>]]: files to count as part of the \texttt{count}
target; always add a backslash to the end of each added line.  Anything
added to the [[<<Build Source>>]] should probably be added here, as well.
\item [[<<Executables>>]]: additional specially-built execuatables, other
than scripts and C executables.
\end{itemize}
\paragraph{C-Specific rules}
\begin{itemize}
\item [[<<C Files>>]], [[<<C Headers>>]]: C files which are noweb
roots are automatically found, but generated files are not.  Add these
with these chunks, terminated by a backslash.
\item [[<<C Executables>>]]: C files are either the name of an executable
with the \texttt{.c} extension, or the name of a library member, with
\texttt{.o} replaced by \texttt{.c}.  The executables are listed here,
without the \texttt{.c} extension, but with a trailing backslash on every
line.  Technically, this could be considered a list of generated
executables instead, since there is no rule requiring that these be C.
\item [[<<C Test Executables>>]]:  names of executables to build for the
[[test]] and [[test-bin]] targets.  See [[<<C Executables>>]] and
[[<<Test Scripts>>]] for details.
\item [[<<C Test Support Executables>>]]:  names of executables to
build for the [[test-bin]] target, as explained by [[<<C Executables>>]]
and [[<<Test Support Scripts>>]] for details.
\item [[<<C Build Executables>>]]:  names of executables to build when
required by other dependencies.  See [[<<C Executables>>]] and
[[<<Build Script Executables>>]] for details.
\item [[<<Library [[name]] Members>>]]:  A library named [[name]] will be
built using objects listed in these chunks.  [[name]] can of course be
anything appropriate.  Unlike the chunks which expand into specific [[make]]
variables, no backslashes are required or allowed at the end of each line.
\item [[<<Libraries to Install>>]]: A list of libraries to install.
These must be built using the [[bin]] rule (e.g. via the Members
chunks above).  Either [[name]] or [[libname.a]] may be used, although
the former is recommended.  Unlike the Members chunks, backslashes are
once again required after every line.
\item [[<<Headers to Install>>]]: A list of header files to install.
These may be prefixed with an installation subdirectory, which will be
stripped to find the source file location.  End each line with a
backslash.
\item [[<<Remove protos from cproto.h>>]]: sed command-line options to
strip prototypes from cproto.h.  End each line with a backslash.
For example:
\begin{quote}
[[<<Remove protos from cproto.h>>=]]\\
[[-e '/generated_functions/d' \]]\\
[[@]]
\end{quote}
\item [[<<For each reinserted C prototype>>]]:  add a transformation
for the modified function name ([[$mf]]).  Typically, this will
involve invoking the C preprocessor with a few standard include files,
followed by the function name.  For example:
\begin{quote}
[[<<makefile.vars>>=]]\\
[[export MY_CPP=$(CC) $(CFLAGS) $(EXTRA_CFLAGS) -E]]\\
[[@]]\\\\
[[<<For each reinserted C prototype>>=]]\\
\hspace*{0pt}[[    mf="`printf '<<Common C Includes>>\\n%s' \"\$mf\" |]]\\
\hspace*{0pt}[[           eval \"\$MY_CPP -\" | tail -n 1`"]]\\
[[@]]\\
\end{quote}
\end{itemize}
\paragraph{Boilerplate}
\begin{itemize}
\item [[<<Common noweb Warning>>]]: a boilerplate comment for the top of
script files; extend with [[# $]][[Id$]].
\item [[<<Version Strings>>]]: A C string containing the version ID of all
contributing source files; extend with [["$]][[Id$\n"]].
\item [[<<Sources>>]]: A simple text chunk containing the version ID of all
contributing source files; extend with just [[$]][[Id$]].
\item [[<<Common C Warning>>]]: the same comment enclosed in a C
comment; not meant to be extended
\item [[<<Common C Includes>>]]: a list of [[#include]] directives for
files safe to be included by all C source
\item [[<<Common C Header>>]]: stuff safe for the top of every C file,
including the warning, common includes, static version string, and
automatically generated prototypes.  Prototypes for static functions
for the file are included as well, unless a comment of the form
[[// static_proto]] is found on a line by itself; in that case, the
static prototypes will be included there instead.
\end{itemize}
\paragraph{Highlighting Assistance}
\begin{itemize}
\item [[<<Known Data Types>>]]: comma-separated list of data types to
emphasize; end each line (including the last) with a comma and a
percent-sign.
\item [[<<Translate listings source type to source-highlight source type>>]],
[[<<Translate listings source type to highlight-3 source type>>]]: add
new cases to a shell case switch to convert listings language names to
highlighter language names.
\item [[<<Mangle aux files>>]] can be extended to further process the
output from the previous \LaTeX{} run.  It is makefile action text for
a single command, so semicolons and backslashes need to be after every
line, and dollar signs need to be doubled.  The make variable [[$*]]
is the base name of the file being processed.  Setting the shell
variable [[$rerun]] to non-empty will force \LaTeX{} to be rerun
without checking the log file.  These actions are run after the index
and bibliography have been generated. To run something before that,
extend [[<<Mangle aux files before bbl and ind>>]] instead.
\end{itemize}

% End-doc build-usage

\section{Code Index}
\nowebchunks

% note: no identifier indexing is done right now

\begin{rawhtml}
<!-->
\end{rawhtml}
%\vspace{1ex}
%\hrule
%\vspace{1ex}
\begin{rawhtml}
<-->
\end{rawhtml}

%\nowebindex

\section{TODO}

This is an unorganized TODO list for my own amusement.  It in no way
represents a promise to get anything done; sometimes I completely
reverse my position on these items, and some have been lingering on my
TODO list for years.

\subsection{Not missing}

\begin{itemize}

\item Reintegration with LyX:  LyX has its own document format, only
editable with LyX.  It is \emph{not} a \LaTeX{} editor, even though it
is often promoted as such (even by me).  Great care would need to be
taken to ensure conversion between its native format and more portable
formats is sane (and in my experience, it very rarely is).  Making it
more programmer-friendly would likely require either modification of
LyX or abusing external insets somehow (which would require all code
chunks to be stored in separate files, so probably not).  Also, LyX
was moving to distance itself even more from \LaTeX{} last time I
abandoned it.

\item LibreOffice/OpenOffice integration:  Like LyX, LibreOffice and
OpenOffice aren't really programmer's editors.  However, at least
their document format is standardized and editable with other
products, as well.  It might be possible to make a plugin to behave
more programmer-editor-like in code blocks (likewise with LyX,
actually), but that would require a lot of effort and still wouldn't
take away the bulkiness, slowness, and inability to edit in a
text-only terminal.  Existing tools to use this format:  nano-lp
(tangle only; uses specially named style to tag code blocks:
\url{http://code.google.com/p/nano-lp/}), odtWeave/StatWeave (weave
only; primarily for result reinsertions; not really literate despite
claims:
\url{http://www.stat.uiowa.edu/~rlenth/StatWeave/}).

\item Plain \TeX{}: not really an option, even though noweb supports
it.  Porting framed and listings from \LaTeX{} is beyond the scope of
this project.  Most of the highlighters I find also expect to be used
with \LaTeX{}.

\item texinfo: Way too dependent on plain \TeX{}; see above item about
that.  It's not really as output-independent as claimed; much of its
independence comes from having to use output-specific escapes
everywhere.  Most output independence I need is HTML vs. PDF, and I
get that with latex-to-HTML converters with about as much utility as
texinfo provides.  And the texinfo format itself (having to repeat
section titles in node definitions and manually insert TOC entries as
menus) is not that appealing (although I could automate some of that).

\item docbook: probably not possible, but should look at it since it's
popular.  It's definitely a pain in the ass raw; like all XML formats,
it's not meant to be edited by a plain text editor or read/understood
by humans (even though the XML movement started with the exact
opposite claims).

\item asciidoc: docbook in a way humans can edit it, but still can't
produce PDF easily or correctly.

\end{itemize}

\subsection{Missing}

\begin{itemize}

\item bibtex, makeindex for l2h.  bibtex requires running latex and
bibtex once, and makeindex requires running latex and makeindex and
also providing support for reading the .ind and formatting it correctly.

\item optional automatic symbol index (requires a lot of work; the
symbol extractor for noweb is way too simplistic and in fact anything
short of a compiler can't deal with scoping, name spaces, preprocessor
name mangling and other issues).

\item highlighting of in-line code.  In addition to just intercepting
text between @quote and @endquote, the system would need to handle
line wrapping somehow.  Currently, quoted code may wrap, and that's
usually a good thing.

Placing the quoted code in a box in order to set its background color
will likely screw everything up.  I have found two solutions to this
problem, both of which split the text into potentially splittable
chunks, and then apply the background color to each chunk separately.
Neither solution works:  one uses only a specific character to split
into chunks
(\url{http://tex.stackexchange.com/questions/94965/wrapping-inline-long-lines-and-adding-background-color})
and one is smarter, but still can't handle basic syntax-highlighted
text as its argument (the soul package).  The soulpos package still
depends on the soul package to do its parsing, so even though its
chunks are bigger, they still don't work.  Supposedly soul can skip
``font-setting'' commands, but registering all of the commands defined
in latexhl.sty does not help, either.  The former solution may be
adaptable somehow using syntax highlighting boundaries as break points
(and perhaps also inserting such boundaries after underscores and
spaces), but even that may not work.  It seems the only solution will
require luatex, with its hooking mechanism.  A command can introduce
sentinels, which are later used to insert the background color using a
post-processing hook.

On the other hand, in-line code could just get no background.
However, it is common to use background color only to highlight
in-line code, so that would be a major shortcoming.

\item Integration of automatically generated documentation.  I used to
support this for Ada in a limited way, but more is needed.  In
particular, it should be possible to generate the help files used by
programmers' editors, and maybe man pages (I haven't written a man
page by hand in years).  Right now, doxygen seems like the most
promising in that regard, although I absolutely hate its default PDF
and HTML output formats.  From what I've seen, gtk-doc looks better,
but only supports C (and GObject) and only produces help files for
Gnome (via devhelp).  All of these will require automatic conversion
of \LaTeX{} to whatever the tool decided was the best markup language,
and some way of determining documentation boundaries.

\item A symbol index would be nice, even if it is not automatically
generated by parsing the code.  There are several issues.  First, API
documentation should have an index pointing to the API documentation,
and code usually has an index pointing to code chunk(s).  Both types
should be supported.  Second, the anchors should point to the
beginning of the documentation, not the code chunk, prototype, or
whatever.  noweb's rule of placing them 10 lines up is error-prone and
insufficient.  Implementing this makes the previous item less
pressing, although really they are separate issues.

\end{itemize}

\subsection{Needs testing}

\begin{itemize}

\item tex4ht is an undocumented piece of crap (well, I guess the
original author being dead may have something to do with that, but it
has always been that way, so I doubt the situation would be any
different were that not the case) that is flaky and in fact the last
release I tested does not work any more and I don't have the energy to
try and fix it.

\item I don't have access to commercial UNIX any more, so I don't know
what does or does not work.  In particular, I use GNU sed's [[-i]]
option too much now.  I should probably just convert all that to C or
perl or something like that that people regularly install on systems.

\end{itemize}

\subsection{Improvements}

\begin{itemize}

\item reStructuredText: slightly better than asciidoc; may be worth a
look some day.  Fernando G\'omez did some work on producing HTML
output with reST and noweb using sphinx.  Basically, a noweb weaver
plugin must generate special paragraph types for code, and a sphinx
plugin must be created to handle that and generate the appropriate
HTML and \LaTeX{}.  The plain docutils would probably be harder to
hook into, but many of sphinx's features are not really needed for
this build system (and in fact may get in the way).  Note that I said
\LaTeX{}: there is a native PDF converter (rst2pdf), but its output
looks rather unprofessional. Also, on my system, installing it with
hyphenation support is a pain.

\item Speaking of reStructuredText: it uses the pygments
(\url{http://pygments.org}) syntax highlighter.  Pygments claims to
support a large number of languages for input and HTML and \LaTeX{} for
output.  Maybe I should support it.  However, I will still substitute
code blocks with direct highlighter output as always, and not rely on
the built-in code block highlighting in docutils.

\item Replace hodge-podge of sed, awk, grep, and perl with C.  The
only non-portable dependency would be POSIX regex and GNU make
(maybe).  I'd probably leave the perl notangle, though, since it's
nice to be able to copy and paste a notangle replacement out of the
document.  On the other hand, having the variety of languages is a
great demonstration of the multi-language syntax highlighting feature.

\item Switch to something other than GNU make.  GNU make shows bugs:
sometimes dependencies are just ignored, especially in parallel builds
where they are most important.  Makepp is a little better in some
respects, but is only stopgap.  Something like Odin would be best, but
Odin needs a lot of work (and in particular making a simple bootstrap
like this system uses may be impossible).  On the other hand, this
document could just implement its own simple build system (aka the old
v1.x shell script method, but with parallel build and dependency
support).  Of course that loses some of the advantages of make:
predefined rules for things I don't otherwise support, well-known
variable names for various overrides (e.g. CFLAGS), and well-known
usage (e.g. make -j4 doc).

\begin{itemize}
\item Global customization variables (e.g. CC, CFLAGS).  All implicit
rules for make use this, but explicit rules need to know these and add
them manually.  Often, C++ code accidentally uses CFLAGS causing me
excess warning headaches (there is an option for C which just raises
warnings for C++).  Odin sort of supports this, but only allows
changing the variables when creating a new cache.  Since Odin
automatically checks for command changes anyway, this should not be a
requirement.
\item Implicit rules (e.g. producing obj from C).  Both GNU make and
Odin have implicit rule support, and may built-in implicit rules.
However, GNU make implicit rules are all based on file name patterns,
which is annoying.  Odin rules are also based on file name patterns,
but a file or product can be explicitly ``cast'' to another type.  The
main deficiency of Odin is that all implicit rules must be set up
during cache creation, like the variables above.  Unlike the
variables, allowing changes after the fact has some serious
consequences with respect to the next point.
\item The rule for a file product should be relatively easy to find.
This is a weird way of saying that the rule for a creating a file
should be in the same directory as the file.  Odin enforces this
somewhat (although an Odinfile can include files from other
directories, which may actually contain the relevant rule), making it
possible to e.g. ``odin x/y/z'' and be assured that Odin will find the
correct rules.  Makefiles, on the other hand, often contain rules for
targets in other directories. Recursive makefiles make this job even
more difficult, making it nearly impossible to figure out which
directory to start in, and often making it impossible to request a
specific target.  For example, the Linux makefiles support making a
specific module, but always require starting make in the top level
directory, and using a special variable to indicate which target to
build.
\item Complex rules.  GNU make's language is not really up to the
task.  Other make clones add powerful languages directly to the
makefile syntax.  Odin's implicit rules are run by external commands,
which use a complex API that both transfers all relevant information
to and from the command, and also ensures that Odin knows when the
command needs to be re-run due to environment changes.  Odinfile rules
can sometimes be complex as well.
\item Platform independence.  Having a sufficiently powerful scripting
language eliminates the need for using the shell everywhere (or
anywhere, as in the case of my odin-lua project).  Portability is
further improved by providing functions that deal with any platform
specifics.  I actually don't care about non-UNIX systems any more, but
if anyone other than me ever uses my tools, they might appreciate
cross-platform support.
\item Simple limited-target rule override.  GNU make has per-target
variable assignments for this.  Odin sort of supports inserting more
options in rules, but only one rule at a time, and sometimes things
need to be repeated.
\item Intermediate file removal.  Intermediate files clutter up the
directory.  Odin by default keeps intermediate files in the cache.
GNU make does not build intermediate files if they have been deleted,
and a rule inference can be made to show that a target is newer than
all its root sources.  However, make still requires explicit rules for
cleaning.  Due to Odin's nature, there is generally not a ``clean''
target, so that should probably be added as well.  An automatic
``clean'' target could probably be built by removing all targets which
have been derived from other sources, but which are not marked as final.
\item Installation targets.  GNU autoconf or CMake are generally used
to add automatic rules for installation targets to make.  Odin has no
such facility.  It would be nice to have automatic install and
uninstall rules, like CMake.
\item Catching all results of an operation.  \TeX{} in particular has
directives to create auxiliary files of any name, and these are used
by numerous packages to create their own special file.  Odin catches
these by running an action in its own directory.  GNU make has no such
facility, and in fact does not support multiple outputs from a single
rule at all.  Makepp does catch newly created files and warns, at least,
and supports multiple outputs from a single rule, but still requires
explicit mention of each output.  Makepp also parses the command line
a bit, but its notion of shell syntax is too simplistic for anything
but very common tasks.
\item Automatic dependency calculation.  This has been the focus of
most make replacements.  There is a fine art in making this
calculation quick and reliable.  As with everything else, Odin uses
external commands to do this in a relatively simple way.  How fast and
reliable it is depends on how complex the script is.  A clever
filtering mechanism makes Odin pretty good.  Makepp, GNU autoconf, and
CMake all enhance makefiles with some automatic dependency checking as
well, but the latter mostly only support C, and all require some work
to do anything unusual.
\item Manual dependency calculation.  Some make replacements don't
really support this properly, Odin included.
\item Change detection other than just timestamps.  If a file changed,
it shouldn't matter if it got older (e.g. reverting to an older
version).  If it didn't, it shouldn't matter if it got newer.  In more
advanced detection, if a comment changed, there should be no need to
recompile (unless debugging is enabled, in which case debug info may
change).  GNU make only supports timestamps.  Odin at least detects
file changes, and may support custom intermediate products that detect
code-only changes.  Makepp tries to do it all, but fails sometimes,
since it stops short of compiling the code to find out if anything
really changes.
\end{itemize}

\item While I'm at it, replace noweb as well.  The main deficiency of
noweb is the lack of macro support (but I guess I've got a partially
working solution above).  Other deficiencies include no method of
having chunks which do not end in a newline (again, solved above), and
no method to have indented code with [[#line]] directives.  Of course
this could be added using a filter, but too many filters just makes
everything slow and awkward.

How (see above for implementation of macros and no-newline):

\begin{itemize}

\item For the lack of properly indented [[#line]] directives, there is
little that can be done.  That deficiency might need to stay.  One
possible minor remedy would be to add a \%-directive such as \%t to
add the current indentation level after this and before each
subsequent line.  That would in fact cure most of the problem.

\item Speaking of [[#line]] directives, insertion into a C macro
definition is impossible, due to macros needing specific formatting.
In general, it might be nice to have a feature similar to no-newline
to suppress [[#line]] processing on an expansion.  In fact, the syntax
could be the same:  a leading star in a reference means [[#line]]
suppression.  For now, I always force parameter insertions to be
[[#line]]-free so that C-style token pasting works.

\item The fangle (\url{www.nongnu.org/fangle}) tool only supports
TeXmacs and LyX file formats, but supports macros and automatic
escaping.  The automatic escaping bit in particular would be useful.
For example, shell code inserted into make code has dollar signs
doubled, and any code inserted into a C string gets properly escaped
for C strings.  This requires a lot of language-specific coding,
though.

\end{itemize}

\item Go back to explicit import/export: maybe just require a prefix
like ! for all globals.

How:
\begin{itemize}

\item In tangle, prepend source name to all nodes that don't have !
prefix in a filter:
\begin{quote}
\begin{verbatim}
 @file <fn> -> set file
 @defn <x> -> convert <x> if not !-prefixed
 @use <x> -> convert <x> if not !-prefixed
\end{verbatim}
\end{quote}

\item In weave, remove !, but add "(exported)" to node name if first
defined in this file, or "(source-file.nw)" to node name if defined
elsewhere.  Note that r77 already implemented the latter, but could
not implement the former because exports are not specially flagged.
Perhaps printing the first file that defines the symbol is not so
good:  the last file before this would be better, as any extensions
would be included, and presumably that file then links to the previous
file in the chain anyway.  On the other hand, documentation for the
symbol is likely only in the first.

\item Would need backend work to support hyperlinking to other
documents, adding a label in front of the first definition of
an exported symbol and then modifying the chunk index and the first
chunk's previous reference somehow.  That would be nearly impossible
with split HTML, though, unless the split HTML were generated first.

\end{itemize}

\item Related to above:  it might be possible to come up with
alternate methods of combining globals:
\begin{quote}
\begin{verbatim}
Global import/export requires prefix char:
 @<<+...@>>= -> append to inherited global
 @<<-...@>>= -> prepend to inherited global
 @<<=...@>>= -> replace inherited global
 @<<&...@>>= -> same as @<<+, but disallow replacement
 @<<!...@>>= -> reference to a global (import only)
\end{verbatim}
\end{quote}
How:
\begin{itemize}

\item Actual symbol name is renamed to
<sourcename>+<node\_without\_prefix> in tangle

\item If an inherited global is not defined in a file, the symbol is
inherited from parent by defining a single chunk with the parent's
chunk as contents.  In other words, as if an empty @<<+...@>>= chunk
defined.

\item In one file, can only use one prefix.  Same node name with two
different prefixes raises error and refuses to tangle/weave.

\item If any file declares !, and later one decalres =, raises error and
refuses to tangle/weave.

\item in tangle, insert parent's global after first +. 

\item in tangle, insert parent's global after end of last -.

\item in tangle, ignore parent's global if =.

\item in tangle, replace all !-references with top-level source file's
version

\end{itemize}

\item Automatically set CreationDate and ModDate for PDF to RCSDate.
Need to actually find all dates of all relevant files and use most
recent.  svn-multi will do that automatically in \LaTeX{}, but I need to
do it manually for l2h.  Even for svn-multi, I need to tack all svnid
commands from depenencies.  Probably not that useful, since even if
ModDate and CreationDate are both static, multiple runs will still
always produce different results.  It is apparently impossible to
stabilize the output or filter out just the time-sensitive parts.

\verb|\hypersetup{pdfinfo={CreationDate=D:\svnpdfdate}}|\\
\verb|\hypersetup{pdfinfo={CreationDate=D:YYYYMMDDHHMMSS-hh'mm')}|

\item Generate [[\RCSDate]] correctly for HTML output

\item At least support isolation if not explicit import/export.  Build
separate trees for each present file.  When doing tangling (and
noroots), consider each tree separately until a result is found.  For
weaving, use the tree containing the file being weaved.

\item Generate [[<<Common noweb Warning>>]] and [[<<Version Strings>>]]
from [[<<Sources>>]].

\item Generate exported prototypes into an exportable header
automatically.  Perhaps a sepcially formated comment?

\item Root nodes should be specially indicated in the chunk index.  An
option should be provided to move them to the top and/or move them to
a different section.

\item Chunk languages should be set automatically by default or some
other way (e.g. a specially formatted comment).  The old listings
package directives can simply be added automatically as needed.

\item Perhaps do simple continuation, such as @<<@>>= to continue last
chunk, or elipses following chunk names to abbreviate.  However, both
are prone to accidental misuse, so probably avoid.

\item Perhaps conditional tangling/weaving (leew's nocond looks OK)

\end{itemize}

\end{document}