Buildbarn Remote Asset - Prototype

N.B This repository provides tools which are in early development and may be subject to regular changes to functionality and/or configuration definitions.

This repository provides a service for the remote asset protocol. This protocol is used by tools such as bazel / buildstream to provide a mapping between URIs and qualifiers to digests which can be used by the remote execution (REv2) protocol.

The remote asset daemon can be configured with bb-storage blobstore backends to enable a scalable remote asset service which can be integrated with any REv2 compatible GRPC cache.

Setting up the Remote Asset daemon

With Action Cache

$ cat config/bb_remote_asset.jsonnet
{
  contentAddressableStorage: common.blobstore.contentAddressableStorage,
  assetCache: {
    actionCache: common.blobstore.actionCache,
  },
  fetcher: {
    http: {},
  },
  global: common.global,
  grpcServers: [{
    listenAddresses: [':8981'],
    authenticationPolicy: { allow: {} },
  }],
  allowUpdatesForInstances: ['foo'],
  maximumMessageSizeBytes: 16 * 1024 * 1024 * 1024,
  fetchAuthorizer: { allow: {} },
  pushAuthorizer: { allow: {} },
}

With Blob Access cache

$ cat config/bb_remote_asset.jsonnet
{
  contentAddressableStorage: common.blobstore.contentAddressableStorage,
  assetCache: {
    blobAccess: {
      'local': {
        keyLocationMapOnBlockDevice: {
          file: {
            path: '/storage/key_location_map',
            sizeBytes: 1024 * 1024,
          },
        },
        keyLocationMapMaximumGetAttempts: 8,
        keyLocationMapMaximumPutAttempts: 32,
        oldBlocks: 8,
        currentBlocks: 24,
        newBlocks: 1,
        blocksOnBlockDevice: {
          source: {
            file: {
              path: '/storage/blocks',
              sizeBytes: 100 * 1024 * 1024,
            },
          },
          spareBlocks: 3,
        },
        persistent: {
          stateDirectoryPath: '/storage/persistent_state',
          minimumEpochInterval: '300s',
        },
      },
    },
  },
  fetcher: {
    http: {},
  },
  global: common.global,
  grpcServers: [{
    listenAddresses: [':8981'],
    authenticationPolicy: { allow: {} },
  }],
  allowUpdatesForInstances: ['foo'],
  maximumMessageSizeBytes: 16 * 1024 * 1024 * 1024,
  fetchAuthorizer: { allow: {} },
  pushAuthorizer: { allow: {} },
}

Both of the above configs rely on there being a common.libsonnet file containing a definition of the bb-storage blobstore.

$ docker run \
    -p 8981:8981 \
    -v $(pwd)/config:/config \
    -v $(pwd)/storage-asset:/storage-asset \
    bazel/cmd/bb_remote_asset:bb_remote_asset_container \
    /config/bb_remote_asset.jsonnet

In the example above, the daemon is configured to store asset references within a disk backed circular storage backend. The fetcher is configured to support fetching via HTTP when a reference is not found matching the request URI/Qualifier criteria, these fetched blobs are placed into a REv2 compatible GRPC cache and the digest returned to the remote asset client. HTTP Fetched blobs are configured to be cached references to newly fetched blobs in the asset store for future fetches.

Bazel can be configured to use this service as a remote uploader as follows:

$ bazel build --remote_cache=grpc://<cache_address>:<cache grpc port> --remote_instance_name=foo --experimental_remote_downloader="grpc://localhost:8981" //...