Support for Buildah in kolla_builder

Buildah and Podman will replace Docker.
This patch will be used by tripleoclient when running:

  $ openstack overcloud container image build --use-buildah

When using Buildah, the kolla_builder will, in that order:

1) Generate container templates but not actually build the images.
   The directories are generated by kolla-build and containers files
   like Dockerfiles and such.
2) Generate container dependencies and build a dictionary, later used by
   the new BuildahBuilder.

In this patch, we introduce a Class for builders. For now, we only have
BuildahBuilder but later we will refactor kolla_builder.

The BuildahBuilder has in charge of:

1) Build containers using "buildah bud". This command is used because
   Kolla uses Dockerfiles to build images. Each image build is logged in
   the directory that contains the Dockerfile.
   During the build, logging displays the container that is being built
   and also the buildah command that is used.

   The image layers that don't have childs are multi-threaded to
   accelerate the build. We don't go over 8 builds at the same time
   otherwise Buildah struggles with the locks too hard.
   We also setup a timeout of 30 minutes for the workers to report back.

   For example:

   base
    └─openstack-base
       ├─nova-base
       │  ├─nova-api
       │  └─nova-conductor
       └─neutron-base
          └─neutron-dhcp
    └─multipathd
    └─crond

    The builder will first build "base" then:
    - build openstack-base, multipathd and crond in same time.
    - build nova-base and neutron-base in same time
    - build nova-api, nova-conductor in same time
    - etc

2) Push containers to a Docker registry. We'll support more than Docker
   registries, but later.

Note: All commands are executed using processutils from oslo_concurrency
      which is pretty and rock solid.

Note2: kolla_builder will be refactored to use the new Builder class.
      This patch is an initial support for Buildah, improvements will come
      later.

Co-Authored-By: Alex Schultz <aschultz@redhat.com>
Co-Authored-By: Christophe Fontaine <cfontain@redhat.com>

blueprint podman-support

Change-Id: Ieff41a5f84456530b4621218b01f3b546cd867bf

releasenotes/notes/buildah_build-727eb0f35f819731.yaml

@@ -0,0 +1,4 @@
+---
+features:
+  - |
+    kolla_builder now supports Buildah and not just Docker.

tripleo_common/constants.py

@@ -67,6 +67,9 @@
 # Swift via SwiftPlanStorageBackend to identify them from other containers
 TRIPLEO_META_USAGE_KEY = 'x-container-meta-usage-tripleo'
 
+# 30 minutes maximum to build the child layers at the same time.
+BUILD_TIMEOUT = 1800
+
 #: List of names of parameters that contain passwords
 PASSWORD_PARAMETER_NAMES = (
     'AdminPassword',

tripleo_common/image/builder/base.py

@@ -0,0 +1,25 @@
+#   Copyright 2019 Red Hat, Inc.
+#
+#   Licensed under the Apache License, Version 2.0 (the "License"); you may
+#   not use this file except in compliance with the License. You may obtain
+#   a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#   WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#   License for the specific language governing permissions and limitations
+#   under the License.
+#
+
+
+class BaseBuilder(object):
+    """Base Tripleo-Common Image Builder.
+
+    For now it does nothing but this interface will allow
+    to support multiple builders and not just buildah or docker.
+    """
+
+    def __init__(self):
+        pass

tripleo_common/image/builder/buildah.py

@@ -0,0 +1,172 @@
+#   Copyright 2019 Red Hat, Inc.
+#
+#   Licensed under the Apache License, Version 2.0 (the "License"); you may
+#   not use this file except in compliance with the License. You may obtain
+#   a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#   WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#   License for the specific language governing permissions and limitations
+#   under the License.
+#
+
+
+from concurrent import futures
+import os
+import six
+
+from oslo_concurrency import processutils
+from oslo_log import log as logging
+
+from tripleo_common import constants
+from tripleo_common.image.builder import base
+from tripleo_common.utils import process
+
+LOG = logging.getLogger(__name__)
+
+
+class BuildahBuilder(base.BaseBuilder):
+    """Builder to build container images with Buildah."""
+
+    def __init__(self, work_dir, deps, base='fedora', img_type='binary',
+                 tag='latest', namespace='master',
+                 registry_address='127.0.0.1:8787'):
+        """Setup the parameters to build with Buildah.
+
+        :params work_dir: Directory where the Dockerfiles
+            are generated by Kolla.
+        :params deps: Dictionary defining the container images
+            dependencies.
+        :params base: Base image on which the containers are built.
+            Default to fedora.
+        :params img_type: Method used to build the image. All TripleO images
+            are built from binary method.
+        :params tag: Tag used to identify the images that we build.
+            Default to latest.
+        :params namespace: Namespace used to build the containers.
+            Default to master.
+        :params registry_address: IP + port of the registry where we push
+            the images. Default is 127.0.0.1:8787.
+        """
+
+        super(BuildahBuilder, self).__init__()
+        self.build_timeout = constants.BUILD_TIMEOUT
+        self.work_dir = work_dir
+        self.deps = deps
+        self.base = base
+        self.img_type = img_type
+        self.tag = tag
+        self.namespace = namespace
+        self.registry_address = registry_address
+        # Each container image has a Dockerfile. Buildah needs to know
+        # the base directory later.
+        self.cont_map = {os.path.basename(root): root for root, dirs,
+                         fnames in os.walk(self.work_dir)
+                         if 'Dockerfile' in fnames}
+        # Building images with root so overlayfs is used, and not fuse-overlay
+        # from userspace, which would be slower.
+        self.buildah_cmd = ['sudo', 'buildah']
+
+    def _find_container_dir(self, container_name):
+        """Return the path of the Dockerfile directory.
+
+        :params container_name: Name of the container.
+        """
+
+        if container_name not in self.cont_map:
+            LOG.error('Container not found in Kolla '
+                      'deps: %s' % container_name)
+        return self.cont_map.get(container_name, '')
+
+    def _generate_container(self, container_name):
+        """Generate a container image by building and pushing the image.
+
+        :params container_name: Name of the container.
+        """
+
+        self.build(container_name, self._find_container_dir(container_name))
+        destination = "{}/{}/{}-{}-{}:{}".format(
+            self.registry_address,
+            self.namespace,
+            self.base,
+            self.img_type,
+            container_name,
+            self.tag
+        )
+        self.push(destination)
+
+    def build(self, container_name, container_build_path):
+        """Build an image from a given directory.
+
+        :params container_name: Name of the container.
+        :params container_build_path: Directory where the Dockerfile and other
+            files are located to build the image.
+        """
+
+        destination = "{}/{}/{}-{}-{}:{}".format(
+            self.registry_address,
+            self.namespace,
+            self.base,
+            self.img_type,
+            container_name,
+            self.tag
+        )
+        # 'buildah bud' is the command we want because Kolla uses Dockefile to
+        # build images.
+        # TODO(emilien): Stop ignoring TLS. The deployer should either secure
+        # the registry or add it to insecure_registries.
+        logfile = container_build_path + '/' + container_name + '-build.log'
+        args = self.buildah_cmd + ['bud', '--tls-verify=False', '--logfile',
+                                   logfile, '-t', destination,
+                                   container_build_path]
+        print("Building %s image with: %s" % (container_name, ' '.join(args)))
+        process.execute(*args, run_as_root=False, use_standard_locale=True)
+
+    def push(self, destination):
+        """Push an image to a container registry.
+
+        :params destination: URL to used to push the container. It contains
+            the registry address, namespace, base, img_type, container name
+            and tag.
+        """
+        # TODO(emilien): Stop ignoring TLS. The deployer should either secure
+        # the registry or add it to insecure_registries.
+        # TODO(emilien) We need to figure out how we can push to something
+        # else than a Docker registry.
+        args = self.buildah_cmd + ['push', '--tls-verify=False', destination,
+                                   'docker://' + destination]
+        print("Pushing %s image with: %s" % (destination, ' '.join(args)))
+        process.execute(*args, run_as_root=False, use_standard_locale=True)
+
+    def build_all(self, deps=None):
+        """Function that browse containers dependencies and build them.
+
+        :params deps: Dictionary defining the container images
+            dependencies.
+        """
+
+        if deps is None:
+            deps = self.deps
+        if isinstance(deps, (list,)):
+            # Only a list of images can be multi-processed because they
+            # are the last layer to build. Otherwise we could have issues
+            # to build multiple times the same layer.
+            # Number of workers will be based on CPU count with a min 2,
+            # max 8. Concurrency in Buildah isn't that great so it's not
+            # useful to go above 8.
+            workers = min(8, max(2, processutils.get_worker_count()))
+            with futures.ThreadPoolExecutor(max_workers=workers) as executor:
+                future_to_build = {executor.submit(self.build_all,
+                                   container): container for container in
+                                   deps}
+                futures.wait(future_to_build, timeout=self.build_timeout,
+                             return_when=futures.ALL_COMPLETED)
+        elif isinstance(deps, (dict,)):
+            for container in deps:
+                self._generate_container(container)
+                self.build_all(deps.get(container))
+        elif isinstance(deps, six.string_types):
+            self._generate_container(deps)

tripleo_common/image/kolla_builder.py

@@ -432,7 +432,8 @@ def container_images_from_template(self, filter=None, **kwargs):
                 result.append(entry)
         return result
 
-    def build_images(self, kolla_config_files=None, excludes=[]):
+    def build_images(self, kolla_config_files=None, excludes=[],
+                     template_only=False, kolla_tmp_dir=None):
 
         cmd = ['kolla-build']
         if kolla_config_files:
@@ -452,11 +453,32 @@ def build_images(self, kolla_config_files=None, excludes=[]):
             if image and image not in excludes:
                 cmd.append(image)
 
+        if template_only:
+            # build the dep list cmd line
+            cmd_deps = list(cmd)
+            cmd_deps.append('--list-dependencies')
+            # build the template only cmd line
+            cmd.append('--template-only')
+            cmd.append('--work-dir')
+            cmd.append(kolla_tmp_dir)
+
         self.logger.info('Running %s' % ' '.join(cmd))
         env = os.environ.copy()
         process = subprocess.Popen(cmd, env=env, stdout=subprocess.PIPE,
                                    universal_newlines=True)
         out, err = process.communicate()
         if process.returncode != 0:
             raise subprocess.CalledProcessError(process.returncode, cmd, err)
+
+        if template_only:
+            self.logger.info('Running %s' % ' '.join(cmd_deps))
+            env = os.environ.copy()
+            process = subprocess.Popen(cmd_deps, env=env,
+                                       stdout=subprocess.PIPE,
+                                       stderr=subprocess.PIPE,
+                                       universal_newlines=True)
+            out, err = process.communicate()
+            if process.returncode != 0:
+                raise subprocess.CalledProcessError(process.returncode,
+                                                    cmd_deps, err)
         return out

tripleo_common/tests/image/builder/test_buildah.py

@@ -0,0 +1,56 @@
+#   Copyright 2019 Red Hat, Inc.
+#
+#   Licensed under the Apache License, Version 2.0 (the "License"); you may
+#   not use this file except in compliance with the License. You may obtain
+#   a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#   WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#   License for the specific language governing permissions and limitations
+#   under the License.
+#
+"""Unit tests for image.builder.buildah"""
+
+import copy
+import mock
+
+from tripleo_common.image.builder.buildah import BuildahBuilder as bb
+from tripleo_common.tests import base
+from tripleo_common.utils import process
+
+
+BUILDAH_CMD_BASE = ['sudo', 'buildah']
+DEPS = {"base"}
+WORK_DIR = '/tmp/kolla'
+
+
+class TestBuildahBuilder(base.TestCase):
+
+    @mock.patch.object(process, 'execute', autospec=True)
+    def test_build(self, mock_process):
+        args = copy.copy(BUILDAH_CMD_BASE)
+        dest = '127.0.0.1:8787/master/fedora-binary-fedora-base:latest'
+        container_build_path = WORK_DIR + '/' + 'fedora-base'
+        logfile = '/tmp/kolla/fedora-base/fedora-base-build.log'
+        buildah_cmd_build = ['bud', '--tls-verify=False', '--logfile',
+                             logfile, '-t', dest, container_build_path]
+        args.extend(buildah_cmd_build)
+        bb(WORK_DIR, DEPS).build('fedora-base', container_build_path)
+        mock_process.assert_called_once_with(
+            *args, run_as_root=False, use_standard_locale=True
+        )
+
+    @mock.patch.object(process, 'execute', autospec=True)
+    def test_push(self, mock_process):
+        args = copy.copy(BUILDAH_CMD_BASE)
+        dest = '127.0.0.1:8787/master/fedora-binary-fedora-base:latest'
+        buildah_cmd_push = ['push', '--tls-verify=False', dest,
+                            'docker://' + dest]
+        args.extend(buildah_cmd_push)
+        bb(WORK_DIR, DEPS).push(dest)
+        mock_process.assert_called_once_with(
+            *args, run_as_root=False, use_standard_locale=True
+        )

tripleo_common/tests/image/test_kolla_builder.py

@@ -124,6 +124,45 @@ def test_build_images(self, mock_popen, mock_path):
             'image-with-missing-tag',
         ], env=env, stdout=-1, universal_newlines=True)
 
+    @mock.patch('tripleo_common.image.base.open',
+                mock.mock_open(read_data=filedata), create=True)
+    @mock.patch('os.path.isfile', return_value=True)
+    @mock.patch('subprocess.Popen')
+    def test_build_images_template_only(self, mock_popen, mock_path):
+        process = mock.Mock()
+        process.returncode = 0
+        process.communicate.return_value = 'done', ''
+        mock_popen.return_value = process
+
+        builder = kb.KollaImageBuilder(self.filelist)
+        self.assertEqual('done',
+                         builder.build_images(
+                             ['kolla-config.conf'], [], True, '/tmp/kolla'))
+        env = os.environ.copy()
+        call1 = mock.call([
+            'kolla-build',
+            '--config-file',
+            'kolla-config.conf',
+            'nova-compute',
+            'nova-libvirt',
+            'heat-docker-agents-centos',
+            'image-with-missing-tag',
+            '--template-only',
+            '--work-dir', '/tmp/kolla',
+        ], env=env, stdout=-1, universal_newlines=True)
+        call2 = mock.call([
+            'kolla-build',
+            '--config-file',
+            'kolla-config.conf',
+            'nova-compute',
+            'nova-libvirt',
+            'heat-docker-agents-centos',
+            'image-with-missing-tag',
+            '--list-dependencies',
+        ], env=env, stdout=-1, stderr=-1, universal_newlines=True)
+        calls = [call1, call2]
+        mock_popen.assert_has_calls(calls, any_order=True)
+
     @mock.patch('subprocess.Popen')
     def test_build_images_no_conf(self, mock_popen):
         process = mock.Mock()