Add pkg_sub_rpm rule for RPM subpackages (#824)

* rpm: Add support for sub packages to make_rpm.py script

Before we can enable support for sub RPM building as part of a single
`pkg_rpm()` rule we must add the underlying support to make_pkg.py
which is the underlying driver for `pkg_rpm()`.

This covers three pieces:

 * specifying `buildsubdir` rpm variable
 * capturing multiple RPM files as outputs
 * injecting sub RPM definitions into specfile

* rpm: Factor out rpm_ctx helper

The various processing functions pass around a bunch of collections
everywhere which is a bit fragile.  This collects them together into a
struct to make it a bit less messy.

* rpm: Factor out _process_dep() helper function

_process_dep() handles processing an individual dep and is currently
called from the processing loop.  We'll need to re-use this logic for
processing individual sub RPMs as well so we want it in a helper.

* rpm: Capture generated output RPM files in rpm_ctx

Currently we only generate one RPM file, but once we generate sub RPM
files we'll need to be able to capture those outputs as well.  This
prepares us for that step.

* rpm: Add args for make_rpm to rpm_ctx

We'll need to add additional arguments to make_rpm for sub RPM
building.  It's easier to capture this in our context object than to
try to shuttle these bits around.

* rpm: Pass correct `--name` argument to make_rpm

If we don't pass the correct RPM name to `make_rpm.py` than we won't be
able to correctly determine the subrpm names.  Currently, the name is
only used by `make_rpm` to generate some progress output, so this
shouldn't break anything internally.

* rpm: Implementation of `pkg_sub_rpm` rule

This introduces a `pkg_sub_rpm` rule that allows us to generate and
capture RPM subpackages and generate them as part of a single RPM
invocation in lieu of cobbling this together from multiple RPM rules.
This has a few benefits:

- faster execution due to single rpmbuild invocation
- sharing configuration between RPMs in the same fashion as vanilla
  RPM building from a specfile
- will enable the proper construction of debuginfo RPMs in a later PR

The current implementation *only* works with non-specfile based rules
and currently allows for a subset of the general RPM configuration.

Internally, the process is for `pkg_sub_rpm` to generate a
PackageSubRPMInfo provider that will subsequently get consumed by the
`pkg_rpm` rule that generates the actual RPMs.  We re-use the internal
dependency processing logic that's used by the top-level RPM to
generate the content related to the sub-RPM.

* rpm: Update entry points for RPM rules to include pkg_sub_rpm

This change updates `rpm.bzl` in two ways to account for sub RPMs:

- it adds an entrypoint for `pkg_sub_rpm`

- it adds a check in `pkg_rpm` to assert incompatibility with
  `spec_file`

* examples: Add an example of how to use the subrpm rule

This provides a basic example using the `pkg_sub_rpm` rule to generate
multiple RPMs.

* Fix buildifier noise

* Fix make_rpm failures

* doc: Clean up sub-RPM docstring and add to doc_build

The initial docstring for pkg_sub_rpm is not great, so this change
fixes that while adding this to the list of rules to have
documentation generated for them.

* doc: Additional documentation for subrpms in pkg_rpm

This clarifies the `subrpms` attribute usage as well as indicating the
incompatibility with `spec_file` mode.

* Fix issue in subrpm passthrough

When adding the check to verify if we can use subrpms, this pass
through wasn't added.

* Add a basic test for pkg_sub_rpm

This introduces a basic test with a single sub RPM and main RPM each
containing a single source file.

* Tweaks to test

* Test fixes for CentOS 7

The `rpm` version on CentOS 7 doesn't work exactly the same way as
newer versions of rpm and requires a `-p` parameter to inspect
non-installed RPMs.  CentOS 7 also inserts a `Relocations` field that
we didn't see on other platforms so we'll filter that out to make our
test a bit more portable.

* Move PackageSubRPMInfoProvider into rpm_pfg.bzl

This is private to the RPM rules so should probably live there.

doc_build/BUILD

@@ -53,6 +53,7 @@ ORDER = [
     ("common", None),
     ("pkg_deb", "//pkg/private/deb:deb.bzl"),
     ("pkg_deb_impl", "//pkg/private/deb:deb.bzl"),
+    ("pkg_sub_rpm", "//pkg:rpm_pfg.bzl"),
     ("pkg_rpm", "//pkg:rpm_pfg.bzl"),
     ("pkg_tar", "//pkg/private/tar:tar.bzl"),
     ("pkg_tar_impl", "//pkg/private/tar:tar.bzl"),

examples/rpm/subrpm/BUILD

@@ -0,0 +1,79 @@
+# Copyright 2024 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -*- coding: utf-8 -*-
+
+load("@rules_pkg//pkg:mappings.bzl", "pkg_files")
+load("@rules_pkg//pkg:rpm.bzl", "pkg_sub_rpm", "pkg_rpm")
+
+pkg_files(
+    name = "subrpm_files",
+    srcs = [
+        "BUILD",
+    ],
+)
+
+pkg_sub_rpm(
+    name = "subrpm",
+    package_name = "subrpm",
+    summary = "Test subrpm",
+    description = "Test subrpm description",
+    requires = [
+        "somerpm",
+    ],
+    provides = [
+        "someprovision",
+    ],
+    srcs = [
+        ":subrpm_files",
+    ],
+)
+
+pkg_files(
+    name = "rpm_files",
+    srcs = [
+        "MODULE.bazel",
+        "README.md",
+    ],
+)
+
+pkg_rpm(
+    name = "test-rpm",
+    srcs = [
+        ":rpm_files",
+    ],
+    release = "0",
+    version = "1",
+    summary = "rules_pkg example RPM",
+    description = "This is a package description.",
+    license = "Apache License, v2.0",
+    architecture = "x86_64",
+    requires = [
+        "somerpm",
+    ],
+    provides = [
+        "somefile",
+    ],
+    subrpms = [
+        ":subrpm",
+    ],
+)
+
+# If you have rpmbuild, you probably have rpm2cpio too.
+# Feature idea: Add rpm2cpio and cpio to the rpmbuild toolchain
+genrule(
+    name = "inspect_content",
+    srcs = [":test-rpm"],
+    outs = ["content.txt"],
+    cmd = "rpm2cpio $(locations :test-rpm) | cpio -ivt >$@",
+)

examples/rpm/subrpm/MODULE.bazel

@@ -0,0 +1,32 @@
+# Copyright 2024 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module(name = "rules_pkg_example_rpm_system_rpmbuild_bzlmod")
+
+bazel_dep(name = "rules_pkg")
+
+local_path_override(
+    module_name = "rules_pkg",
+    path = "../../..",
+)
+
+find_rpmbuild = use_extension(
+    "@rules_pkg//toolchains/rpm:rpmbuild_configure.bzl",
+    "find_system_rpmbuild_bzlmod",
+)
+use_repo(find_rpmbuild, "rules_pkg_rpmbuild")
+
+register_toolchains(
+    "@rules_pkg_rpmbuild//:all",
+)

examples/rpm/subrpm/README.md

@@ -0,0 +1,14 @@
+# Using system rpmbuild with bzlmod
+
+## Summary
+
+This example builds an RPM using the system `rpmbuild` from a pure bazel
+`pkg_rpm()` definition instead of using a separate specfile.
+
+## To use
+
+```
+bazel build :*
+rpm2cpio bazel-bin/test-rpm.rpm | cpio -ivt
+cat bazel-bin/content.txt
+```

pkg/make_rpm.py

@@ -81,10 +81,9 @@ def Cleanup():
 
 def FindOutputFile(log):
   """Find the written file from the log information."""
-
-  m = WROTE_FILE_RE.search(log)
+  m = WROTE_FILE_RE.findall(log)
   if m:
-    return m.group('rpm_path')
+    return m
   return None
 
 def SlurpFile(input_path):
@@ -187,7 +186,7 @@ def __init__(self, name, version, release, arch, rpmbuild_path,
     self.arch = arch
     self.files = []
     self.rpmbuild_path = FindRpmbuild(rpmbuild_path)
-    self.rpm_path = None
+    self.rpm_paths = None
     self.source_date_epoch = helpers.GetFlagValue(source_date_epoch)
     self.debug = debug
 
@@ -204,6 +203,7 @@ def __init__(self, name, version, release, arch, rpmbuild_path,
     self.post_scriptlet = None
     self.preun_scriptlet = None
     self.postun_scriptlet = None
+    self.subrpms = None
 
   def AddFiles(self, paths, root=''):
     """Add a set of files to the current RPM.
@@ -227,6 +227,7 @@ def SetupWorkdir(self,
                    preamble_file=None,
                    description_file=None,
                    install_script_file=None,
+                   subrpms_file=None,
                    pre_scriptlet_path=None,
                    post_scriptlet_path=None,
                    preun_scriptlet_path=None,
@@ -267,6 +268,8 @@ def SetupWorkdir(self,
       SlurpFile(os.path.join(original_dir, postun_scriptlet_path)) if postun_scriptlet_path is not None else ''
     self.posttrans_scriptlet = \
       SlurpFile(os.path.join(original_dir, posttrans_scriptlet_path)) if posttrans_scriptlet_path is not None else ''
+    self.subrpms = \
+      SlurpFile(os.path.join(original_dir, subrpms_file)) if subrpms_file is not None else ''
 
     # Then prepare for textual substitution.  This is typically only the case for the
     # experimental `pkg_rpm`.
@@ -276,6 +279,7 @@ def SetupWorkdir(self,
       'PREUN_SCRIPTLET': ("%preun\n" + self.preun_scriptlet) if self.preun_scriptlet else "",
       'POSTUN_SCRIPTLET': ("%postun\n" + self.postun_scriptlet) if self.postun_scriptlet else "",
       'POSTTRANS_SCRIPTLET': ("%posttrans\n" + self.posttrans_scriptlet) if self.posttrans_scriptlet else "",
+      'SUBRPMS' : (self.subrpms if self.subrpms else ""),
       'CHANGELOG': ""
     }
 
@@ -362,6 +366,7 @@ def CallRpmBuild(self, dirname, rpmbuild_args):
         '--define', '_topdir %s' % dirname,
         '--define', '_tmppath %s/TMP' % dirname,
         '--define', '_builddir %s/BUILD' % dirname,
+        '--define', 'buildsubdir .',
         '--bb',
         '--buildroot=%s' % buildroot,
     ]  # yapf: disable
@@ -405,9 +410,9 @@ def CallRpmBuild(self, dirname, rpmbuild_args):
 
     if p.returncode == 0:
       # Find the created file.
-      self.rpm_path = FindOutputFile(output)
+      self.rpm_paths = FindOutputFile(output)
 
-    if p.returncode != 0 or not self.rpm_path:
+    if p.returncode != 0 or not self.rpm_paths:
       print('Error calling rpmbuild:')
       print(output)
     elif self.debug:
@@ -416,20 +421,35 @@ def CallRpmBuild(self, dirname, rpmbuild_args):
     # Return the status.
     return p.returncode
 
-  def SaveResult(self, out_file):
+  def SaveResult(self, out_file, subrpm_out_files):
     """Save the result RPM out of the temporary working directory."""
-
-    if self.rpm_path:
-      shutil.copy(self.rpm_path, out_file)
-      if self.debug:
-        print('Saved RPM file to %s' % out_file)
+    if self.rpm_paths:
+      for p in self.rpm_paths:
+         is_subrpm = False
+
+         for subrpm_name, subrpm_out_file in subrpm_out_files:
+            subrpm_prefix = self.name + '-' + subrpm_name
+
+            if os.path.basename(p).startswith(subrpm_prefix):
+               shutil.copy(p, subrpm_out_file)
+               is_subrpm = True
+               if self.debug or True:
+                  print('Saved %s sub RPM file to %s' % (
+                     subrpm_name, subrpm_out_file))
+               break
+
+         if not is_subrpm:
+            shutil.copy(p, out_file)
+            if self.debug or True:
+               print('Saved RPM file to %s' % out_file)
     else:
       print('No RPM file created.')
 
-  def Build(self, spec_file, out_file,
+  def Build(self, spec_file, out_file, subrpm_out_files=None,
             preamble_file=None,
             description_file=None,
             install_script_file=None,
+            subrpms_file=None,
             pre_scriptlet_path=None,
             post_scriptlet_path=None,
             preun_scriptlet_path=None,
@@ -446,12 +466,21 @@ def Build(self, spec_file, out_file,
     original_dir = os.getcwd()
     spec_file = os.path.join(original_dir, spec_file)
     out_file = os.path.join(original_dir, out_file)
+
+    if subrpm_out_files is not None:
+      subrpm_out_files = (s.split(':') for s in subrpm_out_files)
+      subrpm_out_files = [
+         (s[0], os.path.join(original_dir, s[1])) for s in subrpm_out_files]
+    else:
+      subrpm_out_files = []
+
     with Tempdir() as dirname:
       self.SetupWorkdir(spec_file,
                         original_dir,
                         preamble_file=preamble_file,
                         description_file=description_file,
                         install_script_file=install_script_file,
+                        subrpms_file=subrpms_file,
                         file_list_path=file_list_path,
                         pre_scriptlet_path=pre_scriptlet_path,
                         post_scriptlet_path=post_scriptlet_path,
@@ -460,7 +489,7 @@ def Build(self, spec_file, out_file,
                         posttrans_scriptlet_path=posttrans_scriptlet_path,
                         changelog_file=changelog_file)
       status = self.CallRpmBuild(dirname, rpmbuild_args or [])
-      self.SaveResult(out_file)
+      self.SaveResult(out_file, subrpm_out_files)
 
     return status
 
@@ -483,6 +512,9 @@ def main(argv):
                       help='The file containing the RPM specification.')
   parser.add_argument('--out_file', required=True,
                       help='The destination to save the resulting RPM file to.')
+  parser.add_argument('--subrpm_out_file', action='append',
+                      help='List of destinations to save resulting ' +
+                      'Sub RPMs to in the form of name:destination')
   parser.add_argument('--rpmbuild', help='Path to rpmbuild executable.')
   parser.add_argument('--source_date_epoch',
                       help='Value for the SOURCE_DATE_EPOCH rpmbuild '
@@ -499,6 +531,8 @@ def main(argv):
                       help='File containing the RPM Preamble')
   parser.add_argument('--description',
                       help='File containing the RPM %description text')
+  parser.add_argument('--subrpms',
+                      help='File containing the RPM subrpm details')
   parser.add_argument('--pre_scriptlet',
                       help='File containing the RPM %pre scriptlet, if to be substituted')
   parser.add_argument('--post_scriptlet',
@@ -526,9 +560,11 @@ def main(argv):
                          debug=options.debug)
     builder.AddFiles(options.files)
     return builder.Build(options.spec_file, options.out_file,
+                         options.subrpm_out_file,
                          preamble_file=options.preamble,
                          description_file=options.description,
                          install_script_file=options.install_script,
+                         subrpms_file=options.subrpms,
                          file_list_path=options.file_list,
                          pre_scriptlet_path=options.pre_scriptlet,
                          post_scriptlet_path=options.post_scriptlet,

pkg/rpm.bzl

@@ -29,10 +29,12 @@ The mechanism for choosing between the two is documented in the function itself.
 
 """
 
-load("//pkg:rpm_pfg.bzl", pkg_rpm_pfg = "pkg_rpm")
+load("//pkg:rpm_pfg.bzl", _pkg_sub_rpm = "pkg_sub_rpm", pkg_rpm_pfg = "pkg_rpm")
 load("//pkg/legacy:rpm.bzl", pkg_rpm_legacy = "pkg_rpm")
 
-def pkg_rpm(name, srcs = None, spec_file = None, **kwargs):
+pkg_sub_rpm = _pkg_sub_rpm
+
+def pkg_rpm(name, srcs = None, spec_file = None, subrpms = None, **kwargs):
     """pkg_rpm wrapper
 
     This rule selects between the two implementations of pkg_rpm as described in
@@ -54,13 +56,17 @@ def pkg_rpm(name, srcs = None, spec_file = None, **kwargs):
     if srcs and spec_file:
         fail("Cannot determine which pkg_rpm rule to use.  `srcs` and `spec_file` are mutually exclusive")
 
+    if subrpms and spec_file:
+        fail("Cannot build sub RPMs with a specfile.  `subrpms` and `spec_file` are mutually exclusive")
+
     if not srcs and not spec_file:
         fail("Either `srcs` or `spec_file` must be provided.")
 
     if srcs:
         pkg_rpm_pfg(
             name = name,
             srcs = srcs,
+	    subrpms = subrpms,
             **kwargs
         )
     elif spec_file:

pkg/rpm/template.spec.tpl

@@ -21,4 +21,6 @@ ${POSTUN_SCRIPTLET}
 
 ${POSTTRANS_SCRIPTLET}
 
+${SUBRPMS}
+
 ${CHANGELOG}