From 0095171190643abe2ab11b2fca11ae211b4ab27a Mon Sep 17 00:00:00 2001 From: Michael Greaves Date: Thu, 17 Jul 2025 10:47:48 +0200 Subject: [PATCH 1/2] Package Orchestration proofreading Signed-off-by: Michael Greaves --- .../en/docs/porch/package-orchestration.md | 576 ++++++++++-------- 1 file changed, 316 insertions(+), 260 deletions(-) diff --git a/content/en/docs/porch/package-orchestration.md b/content/en/docs/porch/package-orchestration.md index bb4ea923..c886ab33 100644 --- a/content/en/docs/porch/package-orchestration.md +++ b/content/en/docs/porch/package-orchestration.md @@ -5,381 +5,437 @@ weight: 2 description: --- -## Why +Customers who want to take advantage of the benefits of [Configuration as Data](config-as-data.md) +can do so today using a [kpt](https://kpt.dev) CLI and the kpt function ecosystem, including its +[functions catalog](https://catalog.kpt.dev/). Package authoring is possible using a variety of +editors with [YAML](https://yaml.org/) support. That said, a UI experience of +what-you-see-is-what-you-get (WYSIWYG) package authoring which supports a broader package lifecycle, +including package authoring with *guardrails*, approval workflows, package deployment, and more, is +not yet available. -People who want to take advantage of the benefits of [Configuration as Data](config-as-data.md) can do so today using -a [kpt](https://kpt.dev) CLI and the kpt function ecosystem, including its [functions catalog](https://catalog.kpt.dev/). -Package authoring is possible using a variety of editors with [YAML](https://yaml.org/) support. That said, a delightful -UI experience of WYSIWYG package authoring which supports broader package lifecycle, including package authoring with -*guardrails*, approval workflow, package deployment, and more, is not yet available. +The *Package Orchestration* (Porch) service is a part of the Nephio implementation of the +Configuration as Data approach. It offers an API and a CLI that enable you to build the UI +experience for supporting the configuration lifecycle. -Porch *Package Orchestration* (Porch) is part of the Nephio implementation of a Configuration as Data approach. It offers an API and -a CLI that enables building that delightful UI experience for supporting the configuration lifecycle. - -## Core Concepts +## Core concepts This section briefly describes core concepts of package orchestration: -***Package***: Package is a collection of related configuration files containing configuration of [KRM][krm] -**resources**. Specifically, configuration packages are [kpt packages](https://kpt.dev/). - -***Repository***: Repositories store packages. For example [git][] or [OCI][oci]. ([more details](#repositories)) - -Packages are sequentially ***versioned***; multiple versions of the same package may exist in a repository. -([more details](#package-versioning)) - -A package may have a link (URL) to an ***upstream package*** (a specific version) from which it was cloned. -([more details](#package-relationships)) +***Package***: A package is a collection of related configuration files containing configurations +of [KRM][krm] **resources**. Specifically, configuration packages are [kpt packages](https://kpt.dev/). +Packages are sequentially ***versioned***. Multiple versions of the same package may exist in a +repository. ([more details](#package-versioning)) A package may have a link (URL) to an +***upstream package*** (a specific version) from which it was cloned. ([more details](#package-relationships)) Packages go through three lifecycle stages: ***Draft***, ***Proposed***, and ***Published***: -Package may be in one of several lifecycle stages: + * ***Draft***: The package is being created or edited. The contents of the package can be + modified; however, the package is not ready to be used (or deployed). + * ***Proposed***: The author of the package has proposed that the package be published. + * ***Published***: The changes to the package have been approved and the package is ready to be + used. Published packages can be deployed or cloned. -* ***Draft*** - package is being created or edited. The package contents can be modified but package is not ready to be - used (i.e. deployed) -* ***Proposed*** - author of the package proposed that the package be published -* ***Published*** - the changes to the package have been approved and the package is ready to be used. Published - packages can be deployed or cloned +***Repository***: The repository stores packages. [git][] and [OCI][oci] are two examples of a +repository. ([more details](#repositories)) A repository can be designated as a +***deployment repository***. *Published* packages in a deployment repository are considered to be +deployment-ready. ([more details](#deployment)) +***Functions***: Functions (specifically, [KRM functions][krm functions]) can be applied to +packages to mutate or validate resources the resources within them. Functions can be applied to a +package to create specific package mutations while editing a package draft. Functions can be added +to a package's Kptfile [pipeline][]. -***Functions*** (specifically, [KRM functions][krm functions]) can be applied to packages to mutate or validate resources -within them. Functions can be applied to a package to create specific package mutation while editing a package draft, -functions can be added to package's Kptfile [pipeline][]. +## Core components of the Configuration as Data implementation -A repository can be designated as ***deployment repository***. *Published* packages in a deployment repository are -considered deployment-ready. ([more details](#deployment)) +The core implementation of Configuration as Data, or *CaD Core*, is a set of components and APIs +which collectively enable the following: -## Core Components of Configuration as Data Implementation +* Registration of the repositories (Git, OCI) containing kpt packages and the discovery of packages. +* Management of package lifecycles. This includes the authoring, versioning, deletion, creation, +and mutations of a package draft, the process of proposing the package draft, and the publishing of +the approved package. +* Package lifecycle operations, such as the following: -The Core implementation of Configuration as Data, *CaD Core*, is a set of components and APIs which collectively enable: + * The assisted or automated rollout of a package upgrade when a new version of the upstream + package version becomes available (the three-way merge). + * The rollback of a package to its previous version. -* Registration of repositories (Git, OCI) containing kpt packages and the discovery of packages -* Management of package lifecycles, including authoring, versioning, deletion, creation and mutations of a package draft, - process of proposing the package draft, and publishing of the approved package -* Package lifecycle operations such as: +* The deployment of the packages from the deployment repositories, and the observability of their +deployment status. +* A permission model that allows role-based access control. - * assisted or automated rollout of package upgrade when a new version of the upstream package version becomes - available (3 way merge) - * rollback of a package to previous version -* Deployment of packages from deployment repositories and observability of their deployment status -* Permission model that allows role-based access control +### High-level architecture -### High-Level Architecture +At the high level, the Core CaD functionality consists of the following components: -At the high level, the Core CaD functionality comprises: - -* a generic (i.e. not task-specific) package orchestration service implementing +* A generic (that is, not task-specific) package orchestration service implementing the following: * package repository management - * package discovery, authoring and lifecycle management + * package discovery, authoring, and lifecycle management -* [porchctl](user-guides/porchctl-cli-guide.md) - a Git-native, schema-aware, extensible client-side tool for managing KRM packages -* a GitOps-based deployment mechanism (for example [configsync][]), which distributes and deploys configuration, and - provides observability of the status of deployed resources -* a task-specific UI supporting repository management, package discovery, authoring, and lifecycle +* The Porch CLI tool [porchctl](user-guides/porchctl-cli-guide.md): this is a Git-native, +schema-aware, extensible client-side tool for managing KRM packages. +* A GitOps-based deployment mechanism (for example [configsync][]), which distributes and deploys +configurations, and provides observability of the status of the deployed resources. +* A task-specific UI supporting repository management, package discovery, authoring, and lifecycle. ![CaD Core Architecture](/static/images/porch/CaD-Core-Architecture.svg) -## CaD Concepts Elaborated +## CaD concepts elaborated -Concepts briefly introduced above are elaborated in more detail in this section. +The concepts that were briefly introduced in **High-level architecture** are elaborated in more +detail in this section. ### Repositories -Porch and [configsync][] currently integrate with [git][] repositories, and there is an existing design to add OCI -support to kpt. Initially, the Package Orchestration service will prioritize integration with [git][], and support for -additional repository types may be added in the future as required. +Porch and [configsync][] currently integrate with [git][] repositories. There is an existing design +that adds OCI support to kpt. Initially, the Package Orchestration service will prioritize +integration with [git][]. Support for additional repository types may be added in the future, as +required. -Requirements applicable to all repositories include: ability to store packages, their versions, and sufficient metadata -associated with package to capture: +Requirements applicable to all repositories include the ability to store the packages and their +versions, and sufficient metadata associated with the packages to capture the following: * package dependency relationships (upstream - downstream) * package lifecycle state (draft, proposed, published) * package purpose (base package) -* (optionally) customer-defined attributes +* customer-defined attributes (optional) -At repository registration, customers must be able to specify details needed to store packages in appropriate locations -in the repository. For example, registration of a Git repository must accept a branch and a directory. +At repository registration, the customers must be able to specify the details needed to store the +packages in appropriate locations in the repository. For example, registration of a Git repository +must accept a branch and a directory. {{% alert title="Note" color="primary" %}} -A user role with sufficient permissions can register a package or function repository, including repositories -containing functions authored by the customer, or other providers. Since the functions in the registered repositories -become discoverable, customers must be aware of the implications of registering function repositories and trust the -contents thereof. +A user role with sufficient permissions can register a package or a function repository, including +repositories containing functions authored by the customer, or by other providers. Since the +functions in the registered repositories become discoverable, customers must be aware of the +implications of registering function repositories and trust the contents thereof. {{% /alert %}} -### Package Versioning +### Package versioning -Packages are sequentially versioned. The important requirements are: +Packages are versioned sequentially. The requirements are as follows: -* ability to compare any 2 versions of a package to be either "newer than", equal, or "older than" relationship -* ability to support automatic assignment of versions -* ability to support [optimistic concurrency][optimistic-concurrency] of package changes via version numbers -* a simple model which easily supports automation +* The ability to compare any two versions of a package as "newer than", "equal to", or "older than" + the other. +* The ability to support the automatic assignment of versions. +* The ability to support the [optimistic concurrency][optimistic-concurrency] of package changes + via version numbers. +* A simple model that easily supports automation. -We use a simple integer sequence to represent package versions. +A simple integer sequence is used to represent the package versions. -### Package Relationships +### Package relationships -Kpt packages support the concept of ***upstream***. When a package is cloned from another, the new package -(called the ***downstream*** package) maintains an upstream link to the specific version of the package from which it was -cloned. If a new version of the upstream package becomes available, the upstream link can be used to update the downstream package. +The Kpt packages support the concept of ***upstream***. When one package is cloned from another, +the new package, known as the ***downstream*** package, maintains an upstream link to the version +of the package from which it was cloned. If a new version of the upstream package becomes available, +then the upstream link can be used to update the downstream package. ### Deployment -The deployment mechanism is responsible for deploying configuration packages from a repository and affecting the live -state. Because the configuration is stored in standard repositories (Git, and in the future OCI), the deployment -component is pluggable. By default, [configsync][] is the deployment mechanism used by CaD Core implementation but -others can be used as well. - -Here we highlight some key attributes of the deployment mechanism and its integration within the CaD Core: - -* _Published_ packages in a deployment repository are considered ready to be deployed -* configsync supports deploying individual packages and whole repositories. For Git specifically that translates to a - requirement to be able to specify repository, branch/tag/ref, and directory when instructing configsync to deploy a +The deployment mechanism is responsible for deploying the configuration packages from a repository +and affecting the live state. Because the configuration is stored in standard repositories (Git, +and in the future OCI), the deployment component is pluggable. By default, [configsync][] is the +deployment mechanism used by CaD Core implementation. However, other deployment mechanisms can be +also used. + +Some of the key attributes of the deployment mechanism and its integration within the CaD Core are +highlighted here: + +* _Published_ packages in a deployment repository are considered to be ready to be deployed. +* configsync supports the deployment of individual packages and whole repositories. For Git + specifically, that translates to a requirement to be able to specify the repository, + branch/tag/ref, and directory when instructing configsync to deploy a package. +* _Draft_ packages need to be identified in such a way that configsync can easily avoid deploying + them. +* configsync needs to be able to pin to specific versions of deployable packages, in order to + orchestrate rollouts and rollbacks. This means it must be possible to get a specific version of a package. -* _Draft_ packages need to be identified in such a way that configsync can easily avoid deploying them. -* configsync needs to be able to pin to specific versions of deployable packages in order to orchestrate rollouts and - rollbacks. This means it must be possible to GET a specific version of a package. * configsync needs to be able to discover when new versions are available for deployment. -## Package Orchestration - Porch +## Package Orchestration (Porch) -Having established the context of the CaD Core components and the overall architecture, the remainder of the document -will focus on **Porch** - Package Orchestration service. +Having established the context of the CaD Core components and the overall architecture, the +remainder of the document will focus on the Package Orchestration service, or **Porch** for short. -To reiterate the role of Package Orchestration service among the CaD Core components, it is: +The role of the Package Orchestration service among the CaD Core components covers the following +areas: * [Repository Management](#repository-management) * [Package Discovery](#package-discovery) * [Package Authoring](#package-authoring) and Lifecycle -In the following section we'll expand more on each of these areas. The term _client_ used in these sections can be -either a person interacting with the UI such as a web application or a command-line tool, or an automated agent or -process. +In the next sections we will expand on each of these areas. The term _client_ used in these +sections can be either a person interacting with the user interface, such as a web application or a +command-line tool, or an automated agent or process. -### Repository Management +### Repository management -The repository management functionality of Package Orchestration service enables the client to: +The repository management functionality of the Package Orchestration service enables the client to +do the following: -* register, unregister, update registration of repositories, and discover registered repositories. Git repository - integration will be available first, with OCI and possibly more delivered in the subsequent releases. -* manage repository-wide upstream/downstream relationships, i.e. designate default upstream repository from which - packages will be cloned. -* annotate repository with metadata such as whether repository contains deployment ready packages or not; metadata can - be application or customer specific +* Register, unregister, and update the registration of the repositories, and discover registered + repositories. Git repository integration will be available first, with OCI and possibly more + delivered in the subsequent releases. +* Manage repository-wide upstream/downstream relationships, that is, designate the default upstream + repositories from which the packages will be cloned. +* Annotate the repositories with metadata, such as whether or not each repository contains + deployment-ready packages. Metadata can be application- or customer-specific. -### Package Discovery +### Package discovery -The package discovery functionality of Package Orchestration service enables the client to: +The package discovery functionality of the Package Orchestration service enables the client to do +the following: -* browse packages in a repository -* discover configuration packages in registered repositories and sort/filter based on the repository containing the - package, package metadata, version, package lifecycle stage (draft, proposed, published) -* retrieve resources and metadata of an individual package, including latest version or any specific version or draft - of a package, for the purpose of introspection of a single package or for comparison of contents of multiple - versions of a package, or related packages -* enumerate _upstream_ packages available for creating (cloning) a _downstream_ package -* identify downstream packages that need to be upgraded after a change is made to an upstream package -* identify all deployment-ready packages in a deployment repository that are ready to be synced to a deployment target - by configsync -* identify new versions of packages in a deployment repository that can be rolled out to a deployment target by configsync +* Browse the packages in a repository. +* Discover the configuration packages in the registered repositories, and sort and/or filter them + based on the repository containing the package, package metadata, version, and package lifecycle + stage (draft, proposed, and published). +* Retrieve the resources and metadata of an individual package, including the latest version, or + any specific version or draft of a package, for the purpose of introspection of a single package, + or for comparison of the contents of multiple versions of a package or related packages. +* Enumerate the _upstream_ packages that are available for creating (cloning) a _downstream_ + package. +* Identify the downstream packages that need to be upgraded after a change has been made to an + upstream package. +* Identify all the deployment-ready packages in a deployment repository that are ready to be synced + to a deployment target by configsync. +* Identify new versions of packages in a deployment repository that can be rolled out to a + deployment target by configsync. -### Package Authoring +### Package authoring -The package authoring and lifecycle functionality of the package Orchestration service enables the client to: +The package authoring and lifecycle functionality of the package Orchestration service enables the +client to do the following: * Create a package _draft_ via one of the following means: - * an empty draft 'from scratch' (`porchctl rpkg init`) - * clone of an upstream package (`porchctl rpkg clone`) from either a - registered upstream repository or from another accessible, unregistered, repository - * edit an existing package (`porchctl rpkg pull`) - * roll back / restore a package to any of its previous versions - (`porchctl rpkg pull` of a previous version) - -* Push changes to a package _draft_. In general, mutations include adding/modifying/deleting any part of the package's - contents. Some specific examples include: - - * add/change/delete package metadata (i.e. some properties in the `Kptfile`) - * add/change/delete resources in the package - * add function mutators/validators to the package's pipeline - * add/change/delete sub-package - * retrieve the contents of the package for arbitrary client-side mutations (`porchctl rpkg pull`) - * update/replace the package contents with new contents, for example results of a client-side mutations by a UI - (`porchctl rpkg push`) - -* Rebase a package onto another upstream base package or onto a newer version of the same package (to - aid with conflict resolution during the process of publishing a draft package) - -* Get feedback during package authoring, and assistance in recovery from merge conflicts, invalid package changes, guardrail violations - -* Propose for a _draft_ package be _published_. -* Apply an arbitrary decision criteria, and by a manual or automated action, approve (or reject) proposal of a _draft_ - package to be _published_. -* Perform bulk operations such as: - - * Assisted/automated update (upgrade, rollback) of groups of packages matching specific criteria (i.e. base package - has new version or specific base package version has a vulnerability and should be rolled back) - * Proposed change validation (pre-validating change that adds a validator function to a base package) + * An empty draft from scratch (`porchctl rpkg init`). + * A clone of an upstream package (`porchctl rpkg clone`) from a registered upstream repository or + from another accessible, unregistered repository. + * Editing an existing package (`porchctl rpkg pull`). + * Rolling back or restoring a package to any of its previous versions + (`porchctl rpkg pull` of a previous version). + +* Push changes to a package _draft_. In general, mutations include adding, modifying, and deleting + any part of the package's contents. Specific examples include the following: + + * Adding, changing, or deleting package metadata (that is, some properties in the `Kptfile`). + * Adding, changing, or deleting resources in the package. + * Adding function mutators/validators to the package's pipeline. + * Adding, changing, or deleting sub-packages. + * Retrieving the contents of the package for arbitrary client-side mutations + (`porchctl rpkg pull`). + * Updating or replacing the package contents with new contents, for example, the results of + client-side mutations by a UI (`porchctl rpkg push`). + +* Rebase a package onto another upstream base package or onto a newer version of the same package + (to assist with conflict resolution during the process of publishing a draft package). + +* Get feedback during package authoring, and assistance in recovery from merge conflicts, invalid + package changes, or guardrail violations. + +* Propose that a _draft_ package be _published_. +* Apply arbitrary decision criteria, and by a manual or an automated action, approve or reject a + proposal for _draft_ package to be _published_. +* Perform bulk operations, such as the following: + + * Assisted/automated updates (upgrades and rollbacks) of groups of packages matching specific + criteria (for example, if a base package has new version or a specific base package version has + a vulnerability and needs to be rolled back). + * Proposed change validation (prevalidating changes that add a validator function to a base + package). * Delete an existing package. -#### Authoring & Latency +#### Authoring and latency -An important goal of the Package Orchestration service is to support building of task-specific UIs. In order to deliver -low latency user experience acceptable to UI interactions, the innermost authoring loop (depicted below) will require: +An important aim of the Package Orchestration service is to support the building of task-specific +UIs. To deliver a low-latency user experience that is acceptable to UI interactions, the innermost +authoring loop depicted below requires the following: -* high performance access to the package store (load/save package) with caching -* low latency execution of mutations and transformations on the package contents -* low latency [KRM function][krm functions] evaluation and package rendering (evaluation of package's function - pipelines) +* high-performance access to the package store (loading or saving a package) with caching +* low-latency execution of mutations and transformations of the package contents +* low-latency [KRM function][krm functions] evaluation and package rendering (evaluation of a + package's function pipelines) ![Inner Loop](/static/images/porch/Porch-Inner-Loop.svg) -#### Authoring & Access Control +#### Authoring and access control -A client can assign actors (persons, service accounts) to roles that determine which operations they are allowed to -perform in order to satisfy requirements of the basic roles. For example, only permitted roles can: +A client can assign actors (for example, persons, service accounts, and so on) to roles that +determine which operations they are allowed to perform, in order to satisfy the requirements of the +basic roles. For example, only permitted roles can do the following: -* manipulate repository registration, enforcement of repository-wide invariants and guardrails -* create a draft of a package and propose the draft be published -* approve (or reject) the proposal to publish a draft package -* clone a package from a specific upstream repository -* perform bulk operations such as rollout upgrade of downstream packages, including rollouts across multiple downstream - repositories -* etc. +* Manipulate repository registration, and enforcement of repository-wide invariants and guardrails. +* Create a draft of a package and propose that the draft be published. +* Approve or reject a proposal to publish a draft package. +* Clone a package from a specific upstream repository. +* Perform bulk operations, such as rollout upgrade of downstream packages, including rollouts + across multiple downstream repositories. -### Porch Architecture +### Porch architecture -The Package Orchestration service, **Porch** is designed to be hosted in a [Kubernetes](https://kubernetes.io/) cluster. +The Package Orchestration (**Porch**) service is designed to be hosted in a +[Kubernetes](https://kubernetes.io/) cluster. -The overall architecture is shown below, and includes also existing components (k8s apiserver and configsync). +The overall architecture is shown in the following figure. It also includes existing components, +such as the k8s apiserver and configsync. ![Porch Architecture](/static/images/porch/Porch-Architecture.svg) -In addition to satisfying requirements highlighted above, the focus of the architecture was to: +In addition to satisfying the requirements highlighted above, the focus of the architecture was to +do the following: -* establish clear components and interfaces -* support a low-latency package authoring experience required by the UIs +* Establish clear components and interfaces. +* Support a low-latency package authoring experience required by the UIs. -The Porch components are: +The Porch architecture comprises three components: -#### Porch Server +* the Porch server +* the function runner +* the CaD Library -The Porch server is implemented as [Kubernetes extension API server][apiserver]. The benefits of using Kubernetes -extension API server are: +#### Porch server -* well-defined and familiar API style -* availability of generated clients -* integration with existing Kubernetes ecosystem and tools such as `kubectl` CLI, - [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) -* avoids requirement to open another network port to access a separate endpoint running inside k8s cluster (this is a - distinct advantage over GRPC which we considered as an alternative approach) +The Porch server is implemented as a [Kubernetes extension API server][apiserver]. The benefits of +using the Kubernetes extension API server are as follows: -Resources implemented by Porch include: +* A well-defined and familiar API style. +* The availability of generated clients. +* Integration with the existing Kubernetes ecosystem and tools, such as the `kubectl` CLI, + [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/). +* The Kubernetes extension API server removes the need to open another network port to access a + separate endpoint running inside the k8s cluster. This is a clear advantage over Google Remote + Procedure Calls (GRPC), which was considered as an alternative approach. -* `PackageRevision` - represents the _metadata_ of the configuration package revision stored in a _package_ repository. -* `PackageRevisionResources` - represents the _contents_ of the package revision +The resources implemented by Porch include the following: -Note that each configuration package revision is represented by a _pair_ of resources which each present a different -view (or [representation][] of the same underlying package revision. +* `PackageRevision`: This represents the _metadata_ of the configuration package revision stored in + a _package_ repository. +* `PackageRevisionResources`: This represents the _contents_ of the package revision. -Repository registration is supported by a `Repository` [custom resource][crds]. - -**Porch server** itself comprises several key components, including: - -* The *Porch aggregated apiserver* which implements the integration into the main Kubernetes apiserver, and directly - serves API requests for the `PackageRevision`, `PackageRevisionResources` resources. -* Package orchestration *engine* which implements the package lifecycle operations, and package mutation workflows -* *CaD Library* which implements specific package manipulation algorithms such as package rendering (evaluation of - package's function *pipeline*), initialization of a new package, etc. The CaD Library is shared with `kpt` - where it likewise provides the core package manipulation algorithms. -* *Package cache* which enables both local caching, as well as abstract manipulation of packages and their contents - irrespectively of the underlying storage mechanism (Git, or OCI) -* *Repository adapters* for Git and OCI which implement the specific logic of interacting with those types of package - repositories. -* *Function runtime* which implements support for evaluating [kpt functions][functions] and multi-tier cache of - functions to support low latency function evaluation - -#### Function Runner +{{% alert title="Note" color="primary"%}} -**Function runner** is a separate service responsible for evaluating [kpt functions][functions]. Function runner exposes -a [GRPC](https://grpc.io/) endpoint which enables evaluating a kpt function on the provided configuration package. +Each configuration package revision is represented by a _pair_ of resources, each of which presents +a different view, or a [representation][] of the same underlying package revision. -The GRPC technology was chosen for the function runner service because the [requirements](#grpc-api) that informed -choice of KRM API for the Package Orchestration service do not apply. The function runner is an internal microservice, -an implementation detail not exposed to external callers. This makes GRPC perfectly suitable. - -The function runner also maintains cache of functions to support low latency function evaluation. - -#### CaD Library - -The [kpt](https://kpt.dev/) CLI already implements foundational package manipulation algorithms in order to provide the -command line user experience, including: - -* [kpt pkg init](https://kpt.dev/reference/cli/pkg/init/) - create an empty, valid, KRM package -* [kpt pkg get](https://kpt.dev/reference/cli/pkg/get/) - create a downstream package by cloning an upstream package; - set up the upstream reference of the downstream package -* [kpt pkg update](https://kpt.dev/reference/cli/pkg/update/) - update the downstream package with changes from new - version of upstream, 3-way merge -* [kpt fn eval](https://kpt.dev/reference/cli/fn/eval/) - evaluate a kpt function on a package -* [kpt fn render](https://kpt.dev/reference/cli/fn/render/) - render the package by executing the function pipeline of - the package and its nested packages -* [kpt fn source](https://kpt.dev/reference/cli/fn/source/) and [kpt fn sink](https://kpt.dev/reference/cli/fn/sink/) - - read package from local disk as a `ResourceList` and write package represented as `ResourcesList` into local disk +{{% /alert %}} -The same set of primitives form the foundational building blocks of the package orchestration service. Further, the -package orchestration service combines these primitives into higher-level operations (for example, package orchestrator -renders packages automatically on changes, future versions will support bulk operations such as upgrade of multiple -packages, etc). +Repository registration is supported by a `Repository` [custom resource][crds]. -The implementation of the package manipulation primitives in kpt was refactored (with initial refactoring completed, and -more to be performed as needed) in order to: +The **Porch server** itself comprises several key components, including the following: + +* The *Porch aggregated apiserver* + The *Porch aggregated apiserver* implements the integration into the main Kubernetes apiserver, + and directly serves the API requests for the `PackageRevision`, `PackageRevisionResources` + resources. +* The Package Orchestration *engine* + The Package Orchestration *engine* implements the package lifecycle operations, and the package + mutation workflows. +* The *CaD Library* + The *CaD Library* implements specific package manipulation algorithms, such as package rendering + (the evaluation of a package's function *pipeline*), the initialization of a new package, and so + on. The CaD Library is shared with `kpt`, where it likewise provides the core package + manipulation algorithms. +* The *package cache* + The *package cache* enables both local caching, as well as the abstract manipulation of packages + and their contents, irrespective of the underlying storage mechanism, such as Git, or OCI. +* The *repository adapters* for Git and OCI + The *repository adapters* implement the specific logic of interacting with those types of package + repositories. +* The *function runtime* + The *function runtime* implements support for evaluating the [kpt functions][functions] and the + multitier cache of functions to support low-latency function evaluation. -* create a reusable CaD library, usable by both kpt CLI and Package Orchestration service -* create abstractions for dependencies which differ between CLI and Porch, most notable are dependency on Docker for - function evaluation, and dependency on the local file system for package rendering. +#### Function runner -Over time, the CaD Library will provide the package manipulation primitives: +The **function runner** is a separate service that is responsible for evaluating the +[kpt functions][functions]. The function runner exposes a Google Remote Procedure Calls +([GRPC](https://grpc.io/)) endpoint, which enables the evaluation of a kpt function on the provided +configuration package. -* create a valid empty package (init) -* update package upstream pointers (get) -* perform 3-way merge (update) -* render - core package rendering algorithm using a pluggable function evaluator to support: +The GRPC technology was chosen for the function runner service because the +[requirements](#grpc-api) that informed the choice of the KRM API for the Package Orchestration +service do not apply. The function runner is an internal microservice, an implementation detail not +exposed to external callers. This makes GRPC particularly suitable. - * function evaluation via Docker (used by kpt CLI) - * function evaluation via an RPC to a service or appropriate function sandbox - * high-performance evaluation of trusted, built-in, functions without sandbox +The function runner also maintains a cache of functions to support low latency function evaluation. -* heal configuration (restore comments after lossy transformation) +#### CaD Library -and both kpt CLI and Porch will consume the library. This approach will allow leveraging the investment already made -into the high quality package manipulation primitives, and enable functional parity between Kpt CLI and Package -Orchestration service. +The [kpt](https://kpt.dev/) CLI already implements foundational package manipulation algorithms, in +order to provide the command line user experience, including the following: + +* [kpt pkg init](https://kpt.dev/reference/cli/pkg/init/): this creates an empty, valid KRM package. +* [kpt pkg get](https://kpt.dev/reference/cli/pkg/get/): this creates a downstream package by + cloning an upstream package. It sets up the upstream reference of the downstream package. +* [kpt pkg update](https://kpt.dev/reference/cli/pkg/update/): this updates the downstream package + with changes from the new version of the upstream, three-way merge. +* [kpt fn eval](https://kpt.dev/reference/cli/fn/eval/): this evaluates a kpt function on a package. +* [kpt fn render](https://kpt.dev/reference/cli/fn/render/): this renders the package by executing + the function pipeline of the package and its nested packages. +* [kpt fn source](https://kpt.dev/reference/cli/fn/source/) and + [kpt fn sink](https://kpt.dev/reference/cli/fn/sink/): these read packages from a local disk as + a `ResourceList` and write the packages represented as a `ResourcesList` into the local disk. + +The same set of primitives form the building blocks of the package orchestration service. Further, +the Package Orchestration service combines these primitives into higher-level operations (for +example, package orchestrator renders the packages automatically on changes. Future versions will +support bulk operations, such as the upgrade of multiple packages, and so on). + +The implementation of the package manipulation primitives in the kpt was refactored (with the +initial refactoring completed, and more to be performed as needed), in order to do the following: + +* Create a reusable CaD library, usable by both the kpt CLI and the Package Orchestration service. +* Create abstractions for dependencies which differ between the CLI and Porch. Most notable are + the dependency on Docker for function evaluation, and the dependency on the local file system for + package rendering. + +Over time, the CaD Library will provide the package manipulation primitives, to perform the +following tasks: + +* Create a valid empty package (init). +* Update the package upstream pointers (get). +* Perform three-way merges (update). +* Render: using a core package rendering algorithm that uses a pluggable function evaluator, to + support the following: + + * Function evaluation via Docker (used by kpt CLI). + * Function evaluation via an RPC to a service or an appropriate function sandbox. + * High-performance evaluation of trusted, built-in functions without a sandbox. + +* Heal the configuration (restore comments after lossy transformation). + +Both the kpt CLI and Porch will consume the library. This approach will allow the leveraging of the +investment already made into the high-quality package manipulation primitives, and enable +functional parity between the kpt CLI and the Package Orchestration service. ## User Guide -Find the Porch User Guide in a dedicated +The Porch User Guide can be found in a dedicated document, via this link: [document](https://github.com/kptdev/kpt/blob/main/site/guides/porch-user-guide.md). -## Open Issues/Questions +## Open issues and questions -### Deployment Rollouts & Orchestration +### Deployment rollouts and orchestration __Not Yet Resolved__ -Cross-cluster rollouts and orchestration of deployment activity. For example, package deployed by configsync in cluster -A, and only on success, the same (or a different) package deployed by configsync in cluster B. +Cross-cluster rollouts and orchestration of deployment activity. For example, a package deployed by +configsync in cluster A, and only on success, the same (or a different) package deployed by +configsync in cluster B. -## Alternatives Considered +## Alternatives considered ### GRPC API -We considered the use of [GRPC]() for the Porch API. The primary advantages of implementing Porch as an extension -Kubernetes apiserver are: +The use of Google Remote Procedure Calls ([GRPC]()) was considered for the Porch API. The primary +advantages of implementing Porch as an extension of the Kubernetes apiserver are as follows: -* customers won't have to open another port to their Kubernetes cluster and can reuse their existing infrastructure -* customers can likewise reuse existing, familiar, Kubernetes tooling ecosystem +* Customers would not have to open another port to their Kubernetes cluster and would be able to + reuse their existing infrastructure. +* Customers could likewise reuse the existing Kubernetes tooling ecosystem. [krm]: https://github.com/kubernetes/design-proposals-archive/blob/main/architecture/resource-management.md From b38b63ec22503bb30210ab4a6a4e3e0895db3feb Mon Sep 17 00:00:00 2001 From: mpgreaves <103421482+mpgreaves@users.noreply.github.com> Date: Thu, 24 Jul 2025 11:18:03 +0200 Subject: [PATCH 2/2] Apply suggestions from code review Co-authored-by: Catalin Stratulat <159934629+Catalin-Stratulat-Ericsson@users.noreply.github.com> --- .../en/docs/porch/package-orchestration.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/content/en/docs/porch/package-orchestration.md b/content/en/docs/porch/package-orchestration.md index f9e5671c..608675f6 100644 --- a/content/en/docs/porch/package-orchestration.md +++ b/content/en/docs/porch/package-orchestration.md @@ -6,7 +6,7 @@ description: --- Customers who want to take advantage of the benefits of [Configuration as Data](config-as-data.md) -can do so today using a [kpt](https://kpt.dev) CLI and the kpt function ecosystem, including its +can do so today using the [kpt](https://kpt.dev) CLI and the kpt function ecosystem, including its [functions catalog](https://catalog.kpt.dev/). Package authoring is possible using a variety of editors with [YAML](https://yaml.org/) support. That said, a UI experience of what-you-see-is-what-you-get (WYSIWYG) package authoring which supports a broader package lifecycle, @@ -22,10 +22,10 @@ experience for supporting the configuration lifecycle. This section briefly describes core concepts of package orchestration: ***Package***: A package is a collection of related configuration files containing configurations -of [KRM][krm] **resources**. Specifically, configuration packages are [kpt packages](https://kpt.dev/). +of [KRM][krm] **resources**. Specifically, configuration packages are [kpt packages](https://kpt.dev/book/02-concepts/01-packages). Packages are sequentially ***versioned***. Multiple versions of the same package may exist in a -repository. ([more details](#package-versioning)) A package may have a link (URL) to an -***upstream package*** (a specific version) from which it was cloned. ([more details](#package-relationships)) Packages go through three lifecycle stages: ***Draft***, ***Proposed***, and ***Published***: +([repository](#package-versioning)). A package may have a link (URL) to an +***upstream package*** (a specific version) ([from which it was cloned](#package-relationships)) . Packages go through three lifecycle stages: ***Draft***, ***Proposed***, and ***Published***: * ***Draft***: The package is being created or edited. The contents of the package can be modified; however, the package is not ready to be used (or deployed). @@ -34,15 +34,15 @@ repository. ([more details](#package-versioning)) A package may have a link (URL used. Published packages can be deployed or cloned. ***Repository***: The repository stores packages. [git][] and [OCI][oci] are two examples of a -repository. ([more details](#repositories)) A repository can be designated as a +([repository](#repositories)). A repository can be designated as a ***deployment repository***. *Published* packages in a deployment repository are considered to be -deployment-ready. ([more details](#deployment)) +([deployment-ready](#deployment)). ***Functions***: Functions (specifically, [KRM functions][krm functions]) can be applied to -packages to mutate or validate resources the resources within them. Functions can be applied to a +packages to mutate or validate the resources within them. Functions can be applied to a package to create specific package mutations while editing a package draft. Functions can be added to a package's Kptfile [pipeline][]. -## Core components of the Configuration as Data implementation +## Core components of the Configuration as Data (CAD) implementation The core implementation of Configuration as Data, or *CaD Core*, is a set of components and APIs which collectively enable the following: @@ -59,7 +59,7 @@ the approved package. * The deployment of the packages from the deployment repositories, and the observability of their deployment status. -* A permission model that allows role-based access control. +* A permission model that allows role-based access control (RBAC). ### High-level architecture @@ -135,7 +135,7 @@ then the upstream link can be used to update the downstream package. The deployment mechanism is responsible for deploying the configuration packages from a repository and affecting the live state. Because the configuration is stored in standard repositories (Git, -and in the future OCI), the deployment component is pluggable. By default, [configsync][] is the +and in the future OCI), the deployment component is pluggable. By default, [configsync][https://cloud.google.com/kubernetes-engine/enterprise/config-sync/docs/overview] is the deployment mechanism used by CaD Core implementation. However, other deployment mechanisms can be also used. @@ -342,7 +342,7 @@ The **Porch server** itself comprises several key components, including the foll The *package cache* enables both local caching, as well as the abstract manipulation of packages and their contents, irrespective of the underlying storage mechanism, such as Git, or OCI. * The *repository adapters* for Git and OCI - The *repository adapters* implement the specific logic of interacting with those types of package + The *repository adapters* for Git and OCI implement the specific logic of interacting with those types of package repositories. * The *function runtime* The *function runtime* implements support for evaluating the [kpt functions][functions] and the