The Buildbarn storage daemon

Translations: Chinese

The Buildbarn project provides an implementation of the Remote Execution protocol. This protocol is used by tools such as Bazel, BuildStream and recc to cache and optionally execute build actions remotely.

This repository provides Buildbarn's storage daemon. This daemon can be used to build a scalable build cache. On its own, it cannot be used to execute build actions remotely. When using only this storage daemon, build actions will still be executed on the local system. This daemon does, however, facilitate remote execution by allowing execution requests to be forwarded to a separate remote execution service.

This storage daemon can be configured to use a whole series of backends. Examples include Redis and S3. It also provides a local on-disk storage backend that writes data to a large file, using a hash table as an index. This storage backend is self-cleaning; no garbage collection is needed. The schema of the storage configuration file gives a good overview of which storage backends are available and how they can be configured.

Setting up the Buildbarn storage daemon

Run the following command to build the Buildbarn storage daemon from source, create container image and push it into the Docker daemon running on the current system:

$ bazel run --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64 //cmd/bb_storage:bb_storage_container
...
Tagging ... as bazel/cmd/bb_storage:bb_storage_container

This container image can then be launched using Docker as follows:

$ cat config/bb_storage.jsonnet
{
  contentAddressableStorage: {
    backend: {
      'local': {
        keyLocationMapOnBlockDevice: {
          file: {
            path: '/storage-cas/key_location_map',
            sizeBytes: 16 * 1024 * 1024,
          },
        },
        keyLocationMapMaximumGetAttempts: 8,
        keyLocationMapMaximumPutAttempts: 32,
        oldBlocks: 8,
        currentBlocks: 24,
        newBlocks: 3,
        blocksOnBlockDevice: {
          source: {
            file: {
              path: '/storage-cas/blocks',
              sizeBytes: 10 * 1024 * 1024 * 1024,
            },
          },
          spareBlocks: 3,
        },
        persistent: {
          stateDirectoryPath: '/storage-cas/persistent_state',
          minimumEpochInterval: '300s',
        },
      },
    },
    getAuthorizer: { allow: {} },
    putAuthorizer: { allow: {} },
    findMissingAuthorizer: { allow: {} },
  },
  actionCache: {
    backend: {
      completenessChecking: {
        'local': {
          keyLocationMapOnBlockDevice: {
            file: {
              path: '/storage-ac/key_location_map',
              sizeBytes: 1024 * 1024,
            },
          },
          keyLocationMapMaximumGetAttempts: 8,
          keyLocationMapMaximumPutAttempts: 32,
          oldBlocks: 8,
          currentBlocks: 24,
          newBlocks: 1,
          blocksOnBlockDevice: {
            source: {
              file: {
                path: '/storage-ac/blocks',
                sizeBytes: 100 * 1024 * 1024,
              },
            },
            spareBlocks: 3,
          },
          persistent: {
            stateDirectoryPath: '/storage-ac/persistent_state',
            minimumEpochInterval: '300s',
          },
        },
      },
    },
    getAuthorizer: { allow: {} },
    putAuthorizer: { instanceNamePrefix: {
      allowedInstanceNamePrefixes: ['foo'],
    } },
  },
  global: { diagnosticsHttpServer: {
    listenAddress: ':9980',
    enablePrometheus: true,
    enablePprof: true,
  } },
  grpcServers: [{
    listenAddresses: [':8980'],
    authenticationPolicy: { allow: {} },
  }],
  schedulers: {
    bar: { endpoint: { address: 'bar-scheduler:8981' } },
  },
  executeAuthorizer: { allow: {} },
  maximumMessageSizeBytes: 16 * 1024 * 1024,
}
$ mkdir -p storage-{ac,cas}/persistent_state
$ docker run \
      -p 8980:8980 \
      -p 9980:9980 \
      -v $(pwd)/config:/config \
      -v $(pwd)/storage-cas:/storage-cas \
      -v $(pwd)/storage-ac:/storage-ac \
      bazel/cmd/bb_storage:bb_storage_container \
      /config/bb_storage.jsonnet

In the example above, the daemon is configured to store a single on-disk CAS. Two ACs are made, corresponding with instance names foo and bar. The former is intended just for remote caching, which is why it's made client-writable through actionCacheAuthorizers in the configuration file. The latter is intended for remote execution, which is why schedulers is used to forward build action execution requests to a separate scheduler service at address bar-scheduler:8981. Please refer to the configuration schema for an overview of all available options.

Bazel can be configured to use the remote cache as follows:

$ bazel build --remote_cache=grpc://localhost:8980 --remote_instance_name=foo //...

Prebuilt binaries of the Buildbarn storage daemon may be obtained by choosing a build on the GitHub Actions page. Prebuilt container images may be found on Docker Hub. More examples of how the Buildbarn storage daemon may be deployed can be found in the Buildbarn deployments repository.

Join us on Slack!

There is a #buildbarn channel on buildteamworld.slack.com that you can join to get in touch with other people who use and hack on Buildbarn.

Commercial Support

Buildbarn has an active and enthusiastic community. Though we try to help and support those who have issues or questions, sometimes organisations need more dedicated support. The following is a list of community members who you can contact if you require commercial support. Please submit a PR if you wish to have your name listed here. Having a name listed is not necessarily an endorsement.

• Finn Ball - Freelance Consultant
• Fredrik Medley - Consultant

Commercial Hosting and Professional Services

Meroton - Cloud Hosted Buildbarn and Services

Buildbarn does not encourage commercial forks and is willing to engage with organisations to merge changes upstream in order to be maintained by the community.