Building sites that use multiple git repositories

Some of the Linaro web sites (e.g. 96Boards) require more than one git repository to be combined to produce the correct source material from which Jekyll will build the site.

The approach taken is that each of the repositories contains a manifest file which informs the build process how the combination needs to be constructed. Here is the manifest file for the 96Boards web site:

{
    "site_dirs": [
        "staging.96boards.org",
        "production.96boards.org"
    ],
    "repos": [
        {
            "tag": "NSBREPO1",
            "url": "https://github.com/96boards/website.git",
            "dir": "/"
        },
        {
            "tag": "NSBREPO2",
            "url": "https://github.com/96boards/documentation.git",
            "dir": "/_documentation"
        }
    ]
}

The JSON file consists of two parts:

• site_dirs: this lists the different directory names that can be used when building the site. These are needed so that the build process can ignore those directories when merging the sources together.

• repos: this lists the details of the git repositories required to build the site. Each repo has the following information included:

  • tag: The name of an environment variable which, if set, points to a directory on your computer where a local copy of this repository can be found, e.g.:

    NSBREPO2=/home/96b-documentation ./build-site.sh

  • url: The HTTPS URL that can be used to fetch the repository if you don't have a copy of it locally. It must not be the SSH URL.

  • dir: Where in the combined sources directory the files from this repository should be copied.

As a minimum, it is possible to work on just one of the repositories used in building a site then run the build-site.sh script and the build process will automatically take care of the rest, including downloading any missing repositories.

When referencing a copy of a repository already on your computer (as with the NSBREPO2 example above), the full path to the directory must be provided. It is not possible to use relative paths, e.g. ../96b-documentation. This is a limitation of Docker.

The built copy of the site is placed in the same directory that you are running the build-site.sh script from. So, for example, if you are in the 96Boards documentation directory and run the script, a directory called staging.96boards.org or production.96boards.org will be created there with the site inside it. There will also be a directory called merged_sources created which is where the combination of all of the repositories is placed.

Please note that if the build process downloads a missing repository from GitHub, it will use the default branch for that repo. If you need more control over the branches used, you must clone the repository onto your computer and then specify the location of the repo when you run the build-site script.

When merging the repos together, the build process takes advantage of anything pre-existing in merged_sources. If a built site is not as you would expect, you may want to consider deleting the merged_sources directory and trying to build the site again to see if that solves the problem you are seeing. PLEASE NOTE that if you delete anything from one of the source repos, it will not be deleted from merged_sources - you will need to delete the merged_sources directory under these circumstances.