Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Finalise interface between composer and image definitions #3444

Merged
merged 40 commits into from
May 31, 2023

Commits on May 30, 2023

  1. distro: inline initializeManifest

    Inline the initializeManifest() function so we can start simplifying the
    PackageSets() and Manifest() (the two callers) separately.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    1914131 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    abd68dc View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    730840c View commit details
    Browse the repository at this point in the history
  4. distro: simplify checkOptions() arguments

    Make checkOptions() take the whole blueprint and options.  There is no
    need to pass in the resolved containers separately since we only care
    whether there are any containers defined for image types that don't
    support them.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    c3a9654 View commit details
    Browse the repository at this point in the history
  5. subscription: new package for subscription options

    Move the subscription options from distro to its own package.
    Now we can import the manifest package into the distro package (instead
    of the other way around) so we can work with the manifest.Manifest type
    in distro.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    54ec501 View commit details
    Browse the repository at this point in the history
  6. distro: remove architecture names and use platform enum

    Removing the dependence of the manifest package on the distro package to
    import manifest into distro.
    Wherever arch names are needed, we use the enums from the platform
    package instead.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    2850288 View commit details
    Browse the repository at this point in the history
  7. manifest: serialize to OSBuildManifest instead of distro.Manifest

    Copy the Marshal and Unmarshal functions from distro.Manifest to
    manifest.OSBuildManifest to keep the same behaviour.
    
    The Version() function isn't used, so let's drop it.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    b624d36 View commit details
    Browse the repository at this point in the history
  8. distro: delete distro.Manifest type

    Delete the distro.Manifest type and its tests.
    Change all occurrences of distro.Manifest to manifest.OSBuildManifest.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    034a7bb View commit details
    Browse the repository at this point in the history
  9. manifest: expand the manifest package doc

    Expand the manifest package documentation and move it to the manifest.go
    file.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    3038403 View commit details
    Browse the repository at this point in the history
  10. container: create a SourceSpec for the resolve parameters

    Create a struct to conveniently hold the source parameters for a
    container resolve request and use it in the Add() function of the
    resolver.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    4af3641 View commit details
    Browse the repository at this point in the history
  11. ostree: rename RequestParams to SourceSpec

    Same as with the container SourceSpec, the struct specifies the required
    information to resolve an ostree commit from a source (URL, ref, and
    optional parent).
    Renaming for consistency.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    d7e6ad2 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    f6839f9 View commit details
    Browse the repository at this point in the history
  13. pipeline: expand interface to return source specs

    Add two new methods to the interface for returning container source
    specifications and ostree commit source specifications respectively.
    These are analogous to the package set chains; they are unresolved
    source specifications that can be used with the appropriate resolver to
    create a full specification for a manifest.
    
    Only the Base pipeline implements the base empty functions for these for
    now.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    53e2512 View commit details
    Browse the repository at this point in the history
  14. rhsm: move FactsImageOptions to the rhsm/facts package

    Move the FactsImageOptions from distro to the new rhsm/facts package.
    At the same time define the values we use as an enum, including the
    "test-manifest" value.
    Though the values don't really matter, the test value is defined first
    so it takes the 0 value, which feels nicer conceptually.
    
    The field in the distro.ImageOptions is changed to be a pointer to allow
    for nil values.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    2b2e10e View commit details
    Browse the repository at this point in the history
  15. ostree: move OSTreeImageOptions to the ostree package

    Move the ostree image options to the ostree package and rename the type
    to ImageOptions (ostree.ImageOptions).
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    658c62f View commit details
    Browse the repository at this point in the history
  16. Make Manifest() return manifest.Manifest

    Return manifest.Manifest from the Manifest() function without
    serializing.  The caller then has to call the manifest.Serialize()
    function using the depsolved packages.
    
    This moves towards changing the order of actions required to generate a
    manifest.  With this change, the manifest creation and depsolving can be
    done independently, but this still requires instantiating the manifest
    object twice (InstantiateManifest() is called in PackageSets() and
    Manifest()), which we don't want to have to do.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    f3da509 View commit details
    Browse the repository at this point in the history
  17. manifest: attach unresolved Content to Manifest struct

    Add Content to the Manifest struct to represent unresolved packages,
    container, and ostree commit specs that the caller should resolve before
    serialising the manifest.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    3fc6a11 View commit details
    Browse the repository at this point in the history
  18. manifest: document Pipeline interface

    Docstrings for the Manifest.pipelines field and the Pipeline type
    itself.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    388f866 View commit details
    Browse the repository at this point in the history
  19. distro: pass entire Blueprint to Manifest()

    Pass the entire Blueprint to Manifest() instead of just the
    Customizations.  The goal is to combine the functionality of the
    ImageType.PackageSets() and ImageType.Manifest() methods into one call.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    cfd0c85 View commit details
    Browse the repository at this point in the history
  20. distro: copy PackageSets() functionality into Manifest()

    Copy the functionality of the ImageType.PackageSets() methods into
    ImageType.Manifest() for each distro.
    
    The Manifest() method now collects all package sets and repositories
    from the blueprint and image type and after generating the Manifest
    instance, calls the GetPackageSetChains() method to attach the computed
    package sets to the Manifest before returning it.
    
    The package sets in the call are now renamed to staticPackageSets to
    differentiate from the dynamic (computed) package sets that are affected
    by the manifest generation.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    b2178f8 View commit details
    Browse the repository at this point in the history
  21. cmd: remove usage of ImageType.PackageSets()

    Use the new manifest generation procedure in the cmd line tools.  The
    new procedure doesn't rely on ImageType.PackageSets() to compute the
    packages for the depsolving.  Instead, it calls Manifest() and depsolves
    the packages attached to the returned object
    (manifest.Content.PackageSets).
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    c0c9d27 View commit details
    Browse the repository at this point in the history
  22. weldr: remove usage of ImageType.PackageSets()

    Use the new manifest generation procedure in the Weldr API.
    
    Updated test distro to include the same packages from the PackageSets()
    method in the Manifest.Content.PackageSets.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    51a6d44 View commit details
    Browse the repository at this point in the history
  23. distro: attach payload repositories to workload package sets

    The merging of payload repositories into the os pipeline had the
    unwanted side effect of using the payload repos for the first depsolve
    in the os chain when instead they should only be used for the second
    (the depsolve for the blueprint or workload packages).  This wasn't an
    issue before because we didn't do the merging in the PackageSets()
    function, but now we rely on the Manifest() function for that
    functionality instead.
    
    Before the merging of the two functions, the PackageSets() function did
    not merge repositories and the repository-to-package-set mapping was
    maintained correctly, but the merging was needed in the Manifest()
    function so that rpm stage options were generated for all repositories.
    With this change, we are removing the merging so that the mapping is
    maintained, and will fix the rpm stage option generation in the pipeline
    generator.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    1a4618a View commit details
    Browse the repository at this point in the history
  24. manifest: fix repository collection in the os pipeline

    In getPackageSetChain(), the workload repositories did not include the
    ExtraBaseRepos.
    
    In serialize(), when creating the rpm stage options (which collects
    repository GPG keys), only the base repos were used, which is why we
    previously had to merge repositories.  Instead of merging repositories
    in the calling function in distro, we should keep them separated so that
    we can easily distinguish which repositories are only meant for the
    blueprint or workload when we need to.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    30860c5 View commit details
    Browse the repository at this point in the history
  25. distro: move the checkOptions() call to the top of Manifest()

    There's no need to do anything if the options fail to validate, so do
    that first.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    2a55e34 View commit details
    Browse the repository at this point in the history
  26. distro: remove usage of ImageType.PackageSets() from tests

    Use the new manifest generation procedure in the distro tests.
    
    Use assert instead of require in TestImageTypePipelineNames to continue
    running the rest of the subtests after a failure.
    
    Some initialisations (image options and blueprint customizations) had to
    be adjusted to work with the ImageType.Manifest() function.
    
    Some helper functions in distro_test_common are no longer necessary and
    have been removed.
    
    The TestPipelineRepositories and TestImageTypePipelineNames tests must
    be (partially) skipped for image types which specify a workload
    (currently only azure-eap7-rhui), because they ignore payload
    repositories.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    b8b8d5b View commit details
    Browse the repository at this point in the history
  27. distro/fedora: remove test for empty ostree ref

    The ImageType.PackageSets() function is going away and instead we will
    rely on the ImageType.Manifest() function to both prepare the manifest
    and return the package sets.  The Manifest() function should never be
    called without an ostree ref for ostree type images.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    92d9c7e View commit details
    Browse the repository at this point in the history
  28. manifest: collect container and ostree source specs

    Much like the GetPackageSetChains() manifest method, these two new
    methods collect the container and ostree source specifications from the
    pipelines that support them.  Currently, only one pipeline per manifest
    contains references to containers or ostree commits, but we collect them
    in a map, keyed by the pipeline name, both for consistency with the
    package sets and for any potential future changes that may require
    differentiating which pipeline a content source belongs to.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    a8afe6f View commit details
    Browse the repository at this point in the history
  29. manifest: use container SourceSpec instead of Spec

    When creating a Manifest object, collect container SourceSpecs instead
    of resolved Specs.
    
    This is the same way we handle packages: The blueprint option is
    converted to source specs and attached to the Manifest object during
    creation.  Later, the SourceSpecs will be resolved to full container
    Specs and used during serialization.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    052082c View commit details
    Browse the repository at this point in the history
  30. manifest: container specs added during serialization

    Add a second argument, map[string][]container.Spec, during
    serialization, which serves the same purpose as the depsolved package
    sets.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    19a5ee3 View commit details
    Browse the repository at this point in the history
  31. gen-manifests: resolve containers after Manifest()

    Demonstrate the new workflow for resolving containers.
    
    1. First call Manifest().
    2. Get container SourceSpecs from manifest struct.
    3. Resolve them.
    4. Serialize() with resolved container specs.
    
    The changes in the test manifests are just the information about the
    container sources (was a slice but is now a map) and the actual manifest
    object isn't affected.
    
    The TestDistro_Manifest test in distro_test_common is adapted
    accordingly as well.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    e81aca0 View commit details
    Browse the repository at this point in the history
  32. Configuration menu
    Copy the full SHA
    81b30dd View commit details
    Browse the repository at this point in the history
  33. weldr: resolve containers after Manifest()

    Use the new workflow for generating the manifest before resolving
    containers.
    
    The resolver function is adjusted to handle a map of container slices,
    but with all current use cases, the map should only ever have one key
    for the payload (os) pipeline.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    3b4d51e View commit details
    Browse the repository at this point in the history
  34. distro: pass containers to Serialize() in test

    This has no effect since we don't use any containers in the test.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    7176ec8 View commit details
    Browse the repository at this point in the history
  35. distro: remove packageSpecSets and containers from Manifest() args

    The arguments aren't used in the function anymore.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    aba9ba0 View commit details
    Browse the repository at this point in the history
  36. cloudapi: remove redundant type from composite literal

    Silence linter warning.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    1f04220 View commit details
    Browse the repository at this point in the history
  37. cloudapi: remove usage of ImageType.PackageSets()

    Still not using the new process for generating the manifest exactly.
    This commit only replaces the call to PackageSets() with a call to
    Manifest() to get the Content.PackageSets.  This is essentially the same
    as before, when we were initialising the manifest object twice.
    
    The Manifest() function does a tiny bit more work than PackageSets(),
    but it's minimal and we gain the benefit of only having a single code
    path and, although it's run twice, it's always run in the same way.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    e6b7df0 View commit details
    Browse the repository at this point in the history
  38. cloudapi: add container specs to Serialize()

    To add the container specs to Serialize(), we need to map them to the
    payload (OS) pipeline.  We assume the first name in the image type's
    PayloadPipelines() list is the OS pipeline, which is true of all image
    types right now but might not be necessarily in the future.
    
    This is a temporary workaround.  Eventually, the mapping will be set by
    the image type itself when we use the container source specs attached to
    the Manifest object.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    ee7a2cd View commit details
    Browse the repository at this point in the history
  39. distro: remove PackageSets() from the ImageType interface

    Drop the PackageSets() function completely since it's no longer needed.
    achilleas-k committed May 30, 2023
    Configuration menu
    Copy the full SHA
    f321275 View commit details
    Browse the repository at this point in the history

Commits on May 31, 2023

  1. cloudapi: set the default ostree ref for package selection

    Before, this was done in the PackageSets() function.
    The reason for this is that having an ostree ref affects package
    selection (for example, it adds rpm-ostree).  At the package selection
    phase, it doesn't matter what the ostree ref is; it is just used to
    determine if a pipeline is for an ostree-based image type and it doesn't
    affect non-ostree-based image types because the image functions ignore
    it.
    
    This is only needed in the cloudapi now because other places have
    switched to using the new order of operations, where the manifest is
    generated after the ostree commit is resolved, so it's always added when
    needed.
    achilleas-k committed May 31, 2023
    Configuration menu
    Copy the full SHA
    ce10100 View commit details
    Browse the repository at this point in the history