[docs] Add some notes on BuildSystem internals.

 - This also incorporates recommonmark so that we can start writing
   documentation in Markdown instead of ReST.

docs/buildsystem-internals.md

@@ -0,0 +1,74 @@
+# Build System Internals
+
+This document is complementary to the documentation on the external
+``BuildSystem`` interface, and describes how the build system internally maps
+the client facing model to the underlying ``Core`` build engine.
+
+## Overview
+
+Internally, the build system models all of its keys using the `BuildKey` type
+which is a union over all possible keys, and which implements a simple binary
+encoding to and from the raw data used by the low-level build engine.
+
+Similarly, all of the values which can be produced are modelled as via
+`BuildValue` type which is a similar union with binary coding support.
+
+Even though the input graph is bipartite, we model all keys and values in the
+same namespace to simplify encoding issues, and because the low-level build
+engine does not yet provide any facilities for enforcing the type safety of the
+keys and values with regard to changes in the key space (for example, should a
+build node's name ever change to become that of a build command, the engine has
+no facility to automatically invalidate any prior `BuildValue` associated with
+that key).
+
+
+## Directory Handling
+
+Directory handling by the build system component is split across several
+coordinating tasks:
+
+### Directory Contents 
+
+The directory contents task transforms `DirectoryContents` key requests into a
+list of the filenames contained within that directory (if present, and a
+directory) as well as the ``FileInfo`` for the directory itself which is used as
+a proxy to validate its contents on incremental rebuilds.
+
+### Directory Tree Signature
+
+The directory tree signature task is responsible for computing the signature for
+an entire recursive directory tree structure. It does so incrementally and
+dynamically by recursively requesting the information for each subpath it
+discovers, as follows:
+
+1. The contents of the input path are requested.
+
+2. The build node for each path contained within the directory is requested,
+   based on the information in #1.
+
+   An important side effect of this is that it establishes a strong dependency
+   between the signature and any producers of the nodes contained in the
+   directory, thus effectively dynamically discovering the commands which
+   contribute content to the directory.
+
+   ```eval_rst
+   .. note::
+   
+     NOTE: The signature task does *NOT* attempt to perform a global scan of the
+     dependency graph to find nodes which **are not** currently present in the
+     directory, but which **are** the output of some task. The client is
+     currently responsible for ensuring that any commands which produce output
+     in the directory should be strongly ordered before the signature node.
+   ```
+
+3. Recursively request directory signatures for any directories discovered as
+   part of #2.
+
+The task accumulates the results from all of the dependency requests made as
+part of #2 and #3, and stores them in a deterministic order (based on the sorted
+names of the directory subpaths).
+
+Finally, the signature is computed as a hash of all of the accumulated
+values. Since the raw encoded value is deterministic, stable, and a complete
+representation of the node, we simply compute the aggregate hash for the all of
+the encoded results.

docs/buildsystem.rst

@@ -21,8 +21,9 @@ and requirements so that the process of executing a typical software build is
 enough information to be able to perform sophisticated tasks like automatic
 distributed compilation or making use of a in-process compiler design.
 
-Build Graph
-===========
+
+Build Description
+=================
 
 Conceptually, the ``BuildSystem`` is organized around a bipartite graph of
 commands and nodes. A ``Command`` represent a units of computation that need to
@@ -33,6 +34,11 @@ is to be used to create it. The tools themselves are integrated into the
 allows them to provided customizable features for use in the build file, and to
 be deeply integrated with *llbuild*.
 
+The build graph is supplied to the ``BuildSystem`` via a ``BuildDescription``
+which can either be loaded from a build file (see below), or can be directly
+constructed by clients. The latter facility is currently only used for
+constructing unit tests, and is not exposed via the public llbuild API.
+
 Nodes
 -----
 

docs/conf.py

@@ -11,6 +11,9 @@
 # serve to show the default.
 
 import datetime, sys, os
+import recommonmark
+from recommonmark.parser import CommonMarkParser
+from recommonmark.transform import AutoStructify
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
@@ -34,7 +37,10 @@
 templates_path = ['_templates']
 
 # The suffix of source filenames.
-source_suffix = '.rst'
+source_parsers = {
+    '.md': CommonMarkParser,
+}
+source_suffix = ['.rst', '.md']
 
 # The encoding of source files.
 #source_encoding = 'utf-8-sig'
@@ -66,7 +72,7 @@
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
-exclude_patterns = ['_build']
+exclude_patterns = ['README.md', '_build']
 
 # The reST default role (used for this markup: `text`) to use for all documents.
 #default_role = None
@@ -238,3 +244,16 @@
 
 # The depth of the table of contents in toc.ncx.
 #epub_tocdepth = 3
+
+
+# -- Options for AutoStructify output ------------------------------------------
+
+# At the bottom of conf.py
+github_doc_root = 'https://github.com/apple/swift-llbuild/tree/master/docs/'
+def setup(app):
+    app.add_config_value('recommonmark_config',
+                         {
+                             'url_resolver': lambda url: github_doc_root + url,
+                             'enable_auto_doc_ref': True,
+                         }, True)
+    app.add_transform(AutoStructify)

docs/contents.rst

@@ -9,6 +9,7 @@ Contents
    development
    buildengine
    buildsystem
+   buildsystem-internals
    TODO
 
 

utils/Xcode/install-user-sphinx.sh

@@ -11,7 +11,7 @@ if [ -z "${SPHINX_MODULE_PATH}" ]; then
     mkdir -p "${BUILT_PRODUCTS_DIR}"
     INSTALL_LOG_PATH="${BUILT_PRODUCTS_DIR}/sphinx.install.log"
     echo "note: attempting automatic 'sphinx' install, see log at: '${INSTALL_LOG_PATH}'"
-    if ( ! easy_install --user sphinx &> ${INSTALL_LOG_PATH} ); then
+    if ( ! easy_install --user sphinx recommonmark &> ${INSTALL_LOG_PATH} ); then
         echo "error: unable to automatically install, please consult \
 log or install manually."
           exit 1