HACKING

Instructions for hacking on Xapian

Table of contents

Tip

Before reading the contents of this file, you should look at the Xapian developer guide. If you want, you can grab it from github and build it locally.

The bulk of the previous contents of this file now live in the developer guide, and it is likely that the rest will follow in due course.

Snapshots

If you want to try unreleased Xapian code, you can fetch it from our git repository. For convenience, we also provide bootstrapped tarballs (much like the sourcecode download for any release version) which get built every 20 minutes if there have been any changes checked in. These tarballs need to pass "make distcheck" to be automatically uploaded, so using them will help to assure that you don't pick a "bad" version. The snapshots are available from the "Bleeding Edge" page of the Xapian website.

Building from git

When building from a git checkout, we strongly recommend that you use the bootstrap script in the top level directory to set up the tree ready for building. This script will check which directories you have checked out, so you can bootstrap a partial tree. You can also touch .nobootstrap in a subdirectory to tell bootstrap to ignore it.

You will need the following tools installed to build from git:

• GNU m4 >= 4.6 (for autoconf)
• perl >= 5.6 (for automake; also for various maintainer scripts)
• python >= 3.3 (for generating the Python bindings)
• GNU make (or another make which support VPATH for explicit rules)
• GNU bison (for building SWIG, used for generating the bindings)
• Tcl (to generate unicode/unicode-data.cc)

For a recent version of Debian or Ubuntu, this command should ensure you have all the necessary tools and libraries:

apt-get install build-essential m4 perl python3 zlib1g-dev uuid-dev wget bison tcl

If you want to build Omega, you'll also need:

apt-get install libpcre2-dev libmagic-dev

On Fedora, the uuid library can be installed by doing:

yum install libuuid-devel

On macOS, if you're using macports you'll want the following:

• file (magic.h in configure)

If you're using homebrew you'll want the following:

brew install libmagic pcre2

If you're doing much development work, you'll probably also want the following tools installed:

• valgrind for better testsuite error finding
• ccache for faster rebuilds
• eatmydata for faster testsuite runs

The repository does not contain any automatically generated files (such as configure, Makefile.in, Snowball-generated stemmers, Lemon-generated parsers, SWIG-generated code, etc) because experience shows it's best to keep these out of version control. To avoid requiring you to install the correct versions of the tools required, we either include the source to these tools in the repo directly (in the case of Snowball and Lemon), or the bootstrap script will download them as tarballs (autoconf, automake, libtool) or from git (SWIG), build them, and install them within the source tree.

To download source tarballs, bootstrap will use wget, curl or lwp-request if installed. If not, it will give an error telling you the URL to download from by hand and where to copy the file to.

Bootstrap will then run autoreconf on each of the checked-out subdirectories, and generate a top-level configure script. This configure script allows you to configure xapian-core and any other modules you've checked out with single simple command, such that the other modules link against the uninstalled xapian-core (which is very handy for development work and a bit fiddly to set up by hand). It automatically passes --enable-maintainer-mode to the subprojects so that the autotools will be rerun if configure.ac, Makefile.am, etc are modified.

The bootstrap script doesn't care what the current directory is. The top-level configure script generated by it supports building in a separate directory to the sources: simply create the directory you want to build in, and then run the configure script from inside that directory. For example, to build in a directory called "build" (starting in the top level source directory):

./bootstrap
mkdir build
cd build
../configure

When running bootstrap, if you need to add any extra macro directories to the path searched by aclocal (which is part of automake), you can do this by specifying these in the ACLOCAL_FLAGS environment variable, e.g.:

ACLOCAL_FLAGS=-I/extra/macro/directory ./bootstrap

If you wish to prevent bootstrap from downloading and building the autotools pass the --without-autotools option. You can force it to delete the downloaded and installed versions by passing --clean.

If you are tracking development in git, there will sometimes be changes to the build system sources which require regeneration of the generated makefiles and associated machinery. We aim to make the build system automatically regenerate the necessary files, but in the event that a build fails after an update, it may be worth re-running the bootstrap script to regenerate the build system from scratch, before looking for the cause of the error elsewhere.

Tools required to build documentation

If you want to be able to build distribution tarballs (with "make dist") then you'll also need some further tools. If you don't want to have to install all these tools, then pass --disable-documentation to configure to disable these rules (the default state of this follows the setting of --enable-maintainer-mode, so in a non-maintainer-mode tree, you can pass --enable-documentation to enable these rules). Without the documentation, "make dist" will fail (to prevent accidentally distributing tarballs without documentation), but you can configure and build.

The documentation tools are:

• doxygen (v1.8.8 is used for 1.3.x snapshots and releases; 1.7.6.1 fails to process trunk after PL2Weight was added).
• dot (part of Graphviz. Doxygen's DOT_MULTI_TARGETS option apparently needs ">1.8.10")
• help2man
• rst2html or rst2html.py (in python3-docutils on Debian/Ubuntu)
• pngcrush (optional - used to reduce the size of PNG files in the HTML apidocs)
• sphinx-doc (in python3-sphinx on Debian/Ubuntu, or as sphinx via pip install)

For a recent version of Debian or Ubuntu, this command should install all the required documentation tools:

apt-get install doxygen graphviz help2man python-docutils pngcrush python-sphinx python3-sphinx

Documentation builds on macOS

On macOS, if you're using homebrew, you'll want the following:

brew install doxygen help2man graphviz pngcrush

(Ensure you're up to date with brew, as earlier packaging of graphviz didn't properly install dot.)

You also need sphinx and docutils, which are python packages; you can install them via pip:

pip3 install sphinx docutils

You may find it easier to use homebrew to install python first, so these packages are separate from the system python:

brew install python

On macOS, if you're using macports you'll want the following:

• texlive (pdflatex during build)
• texlive-basic (for makeindex in configure)
• texlive-latex-extra (latex style)

The homebrew texlive package only supports 32 bit systems, so even if you're using homebrew, you'll probably want to install MacTeX from https://www.tug.org/mactex/ instead.

As of 1.3.2, we no longer build PDF versions of the API docs by default, but you can build them yourself with:

make -C docs apidoc.pdf

Autotools versions

• autoconf 2.69 is used to generate snapshots and releases.

  autoconf 2.64 is a hard minimum requirement.

  autoconf 2.60 is required for docdir support and AC_TYPE_SSIZE_T.

  autoconf 2.62 generates faster configure scripts and warns about unrecognised options passed to configure.

  autoconf 2.63 fixes a regression in AC_C_BIGENDIAN introduced in 2.62 (Omega uses this macro).

  autoconf 2.64 generates smaller configure scripts by using shell functions.

• automake 1.16.1 is used to generate snapshots and releases.

  automake 1.13 is a hard minimum requirement, needed for AC_CONFIG_MACRO_DIRS.

• libtool 2.4.6 is used to generate snapshots and releases.

  libtool 2.2.8 is the current hard minimum requirement.

  libtool 2.2 is required for us to be able to override link_all_deplibs_CXX and sys_lib_dlsearch_path_spec in configure. It also fixes some long-standing issues and is significantly faster.

Please tell us if you find that newer versions of any of these tools work or fail to work.

There is a good GNU autotools tutorial at <https://www.lrde.epita.fr/~adl/autotools.html>.

Building from git on Windows with MSVC

Building using MSVC is now supported by the autotools build system. You need to install a set of Unix-like tools first - we recommended MSYS2: https://www.msys2.org/

For details of how to specify MSVC to configure see the "INSTALL" document.

When building from git, by default you'll need some extra tools to generate Unicode tables (Tcl) and build documentation (doxygen, help2man, sphinx-doc). We don't currently have detailed advice on how to do this (if you can provide some then please send a patch).

You can avoid needing Tcl by copying xapian-core/unicode/unicode-data.cc from another platform or a release which uses the same Unicode version. You can avoid needing most of the documentation tools by running configure with the --disable-documentation option.

Submitting Patches:

If you have a patch to fix a problem in Xapian, or to add a new feature, please send it to us for inclusion. Any major changes should be discussed on the xapian-devel mailing list first: <https://xapian.org/lists>

Also, please read the following section on licensing of patches before submitting a patch.

If you're working from a tarball, you can unpack a second clean copy of the files and compare the two versions with "diff -pruN" (-p reports the function name for each chunk, -r acts recursively, -u does a unified diff, and -N shows new files in the diff). Alternatively "ptardiff" (which comes with perl, at least on Debian and Ubuntu) can diff against the original tarball, unpacking it on the fly.

Please try to follow the style of the existing code.

We will do our best to give credit where credit is due - if we have used patches from you, or received helpful reports or advice, we will add your name to the AUTHORS file (unless you specifically request us not to). If you see we have forgotten to do this, please draw it to our attention so that we can address the omission.

Tips for Submitting a Good Patch

Make sure the tests are right

• If you're adding a feature, also add feature tests for it. These both ensure that the feature isn't broken to start with and detect if later changes stop it working as intended.
• If you've fixed a bug, make sure there's a regression test which fails on the existing code and succeeds after your changes.

• If you're adding a new testcase to demonstrate an existing bug, and not checking a fix in at the same time, mark the testcase as a known failure (by calling XFAIL("explanatory message") at the start of your testcase (if necessary this can be conditional on backend or other factors - the backend case has explicit support via XFAIL_FOR_BACKEND("backend", "message")).

  This will mean that this testcase failing will be reported as "XFAIL" which won't cause the test run to fail. If such a testcase in fact passes, that gets reported as "XPASS" and will cause the test run to fail. A testcase should not be flagged as "XFAIL" for a long time, but it can be useful to be able to add such testcases during development. It also allows a patch series which fixes a bug to first demonstrate the bug via a new testcase marked as "XFAIL", then fix the bug and remove the "XFAIL" - this makes it clear that the regression test actually failed before the fix.

  Note that failures which are due to valgrind errors or leaked fds are not affected by this macro - such errors are inherently not suitable for "XFAIL" as they go away when the testsuite is run without valgrind or on a platform where our fd leak detector code isn't supported.

• Make sure all existing tests continue to pass.

If you don't know how to write tests using the Xapian test rig, then ask. It's reasonably simple once you've done it once. There is a brief introduction to the Xapian test system in docs/tests.html.

Update trac

If there's a related trac ticket, update it (if the issue is completely addressed by the changes you've made, then close it).

Update the release notes for the most recent release with a copy of the patch. If the commit from git applies cleanly, you can just link to it. If it fails to apply, please attach an adjusted patch which does. If there are conflicts in test cases which aren't easy to resolve, it is acceptable to just drop those changes from the patch if we can still be confident that the issue is actually fixed by the patch.

Commit message

If there's a trac ticket or other reference for the bug, mention it in the commit message - it's a great help to future developers trying to work out why a change was made.