Document the build system of YARP

Author: eregon

docs/build_system.md

@@ -0,0 +1,74 @@
+# Build System
+
+There are many ways to build YARP, which means the build system is a bit more complicated than usual.
+
+## General Design
+
+1. Templates are generated by `templates/template.rb`
+2. `autoconf` creates `./configure` and `autoheader` creates `config.h.in` (both files are platform-independent)
+3. `./configure` creates `include/yarp/config.h` (which contains `HAVE_*` macros, platform-specific) and the `Makefile`
+4. The `Makefile` compiles both `librubyparser.a` and `librubyparser.so/dylib/dll` from the `*.c` and `*.h` files
+5. The `Rakefile` `:compile` task ensures the above prerequisites are done, then calls `make`,
+  and uses `Rake::ExtensionTask` to compile the C extension (using its `extconf.rb`), which uses `librubyparser.a`
+
+This way there is minimal duplication, and each layer builds on the previous one and has its own responsibilities.
+
+The static library exports no symbols, to avoid any conflict.
+The shared library exports some symbols, and this is fine since there should only be one librubyparser shared library
+loaded per process (i.e., at most one version of the yarp *gem* loaded in a process, only the gem uses the shared library).
+
+## The various ways to build YARP
+
+### Building from ruby/yarp repository with `bundle exec rake`
+
+`rake` calls `make` and then uses `Rake::ExtensionTask` to compile the C extension (see above).
+
+### Building the yarp gem by `gem install/bundle install`
+
+The gem contains the pre-generated templates, as well as `configure` and `config.h.in`
+When installing the gem, `extconf.rb` is used and that:
+* runs `./configure` which creates the `Makefile` and `include/yarp/config.h`
+* runs `make build/librubyparser.a`
+* compiles the C extension with mkmf
+
+When installing the gem on JRuby and TruffleRuby, no C extension is built, so instead of the last step,
+there is Ruby code using Fiddle which uses `librubyparser.so/dylib/dll`
+to implement the same methods as the C extension, but using serialization instead of many native calls/accesses
+(JRuby does not support C extensions, serialization is faster on TruffleRuby than the C extension).
+
+### Building the yarp gem from git, e.g. `gem 'yarp', github: 'ruby/yarp'`
+
+The same as above, except the `extconf.rb` additionally runs first:
+* `templates/template.rb` to generate the templates
+* `autoconf` and `autoheader` to generate `configure` and `config.h.in`
+
+Because of course those files are not part of the git repository.
+
+### Building YARP as part of CRuby
+
+[This script](https://github.com/ruby/ruby/blob/32e828bb4a6c65a392b2300f3bdf93008c7b6f25/tool/sync_default_gems.rb#L399-L426) imports YARP sources in CRuby.
+
+The script generates the templates when importing.
+
+`include/yarp/config.h` is replaced by `#include "ruby/config.h"`.
+It is assumed that CRuby's `./configure` is a superset of YARP's configure checks.
+
+YARP's `autotools` is not used at all in CRuby and in fact YARP's `Makefile` is not used either.
+Instead, CRuby's `autotools` setup is used, and `CRuby`'s Makefiles are used.
+
+### Building YARP as part of TruffleRuby
+
+[This script](https://github.com/oracle/truffleruby/blob/master/tool/import-yarp.sh) imports YARP sources in TruffleRuby.
+The script generates the templates when importing.
+It also generates `configure` and `config.h.in` (to avoid needing `autotools` on every machine building TruffleRuby).
+
+Then when `mx build` builds TruffleRuby and the `yarp` mx project inside, it:
+* runs `./configure`
+* runs `make`
+
+Then the `yarp bindings` mx project is built, which contains the [bindings](https://github.com/oracle/truffleruby/blob/master/src/main/c/yarp_bindings/src/yarp_bindings.c)
+and links to `librubyparser.a` (to avoid exporting symbols, so no conflict when installing the yarp gem).
+
+### Building YARP as part of JRuby
+
+TODO, probably similar to TruffleRuby.

yarp.gemspec

@@ -20,6 +20,7 @@ Gem::Specification.new do |spec|
     "config.h.in",
     "config.yml",
     "configure",
+    "docs/build_system.md",
     "docs/building.md",
     "docs/configuration.md",
     "docs/design.md",