Porch Roadmap

[johnbelamaric]

As discussed in Nov 15, 2023 SIG Automation meeting, the kpt project is focusing on the CLI side of things, which means they are giving us the Porch pieces. I will be moving that code over to the nephio repository. This allows us to evolve it in the ways that are most useful to us. I want to document my current thinking in this area, and get the community feedback and ideas.

High-level Migration Plan

1. Wrap Up In-Progress Items. There is one PR I will wrap up in the existing repo: Revise package revision caching strategy kptdev/kpt#4094, and we will do one final release out of that repo.
2. Migrate Server Code to Nephio. After that, the first step is to get the server-side code building in Nephio. This will allow us to build all the server-side container images out of the Nephio repository. I actually have this working in my local environment, and should be able to get a PR shortly after updating it for the one last PR above.
3. Migrate the Server CI Tests to Nephio. The Porch unit and E2E tests will need to be run on changes to this code. We will also want to change our regular Nephio E2E tests to use the Porch images built from the PR. I could use help here.
4. Create a nephioctl Tool. The kpt alpha rpkg functions would move to this new tool, along with the tests associated with them.
5. Move Issues. If possible, it would be great to move existing issues from the https://github.com/kptdev/kpt repository. I know you can move issues to different repos within an org, not sure if you can move across orgs.
6. Remove the code from Kpt. Finally, once we have it all building and working in Nephio, we will work with the kpt team to remove the code from their repository. This is a code move, not a fork.

Porch Evolution In Nephio

Here's where it gets fun. There's a lot of stuff that we can do to improve Nephio functioning. The first step will be some clean up. There are a number of parts of Porch that we do not use, and therefore no longer need to maintain. The idea here is to simplify before making some of the other architectural changes that Wim, Tal, me, and others have been discussing.

Clean Up

1. Remove controllers we do not need or use: functiondiscovery, klippy, remotesyncsets, rootsyncdeployments, rootsyncrollouts, rootsyncsets. We may consider moving these to a nephio-experimental rather than deleting them entirely.
2. Move GCP-specific controllers to a GCP-specific repository instead of in the nephio repository: fleetsyncs, workloadidentitybindings.
3. Remove the "Task List" concept from PackageRevision. The primary purpose for this is smarter updates, but we are not using it, it is not fully tested and likely has some bugs, and the implementation is not scalable (it leads to very large PackageRevision objects), and that complicates moving PackageRevisions to pure CRs (see below).
4. Remove the Function CRD, as we are not using it.
5. Remove the "Package" CRD, as it is not fully implemented.
6. Consider removing OCI support. We don't use it today, and it is not as well exercised as the Git support.

Architectural Improvements

PackageRevision to CRD

Right now, PackageRevisions are served up by the Porch aggregated API server. They are discovered in the Git or OCI repository, and cached in memory by the Porch server. API requests to retrieve these package revisions are served out of that cache. Additionally, some metadata (e.g., labels) are associated with a packagerev CR. That is, the data about a package revision is split across the API server and git.

Porch is constantly polling to see if there have been out-of-band changes to the underlying repository, and refreshing the cache. This is expensive (as is maintaining it in memory). In the most recent few PRs I have improved this so that all mutations are done on the underlying repo, and then the cache refreshed. But it's really not the right architecture.

The plan is to move to PackageRevisions as ordinary custom resources. See kptdev/kpt#3630 for more discussion, this is an ongoing design discussion (please join in!). But the basic idea is that a PackageRevision becomoes a simple pointer into a repo - either to a commit, tag, or branch (for drafts in particular), and a directory. If we want it, we can have a package discovery controller that works something like what the Porch server does for package discovery today, but it will be asynchronously reconciling an ordinary CR rather than trying to maintain a consistent cache. In fact it probably just needs to populate rather than fully reconcile them (we can discuss what we need). It really is probably totally unnecessary. You will be able to create and manage PackageRevisions like any other resource (and the names will become more sensible).

With this change, the only thing served out of the aggregated API server would be the PackageRevisionResources.

Eliminate Hard Dependency on kpt

Requiring kpt has been a friction for Nephio from Day 1. We don't really need kpt, we need "config-as-data". The move of Porch to Nephio makes this viable. You can still use kpt for client-side mutations, but it won't be required (since the rpkg commands will be in nephioctl).

To complete this, we need to stop requiring a Kptfile in our packages. The Kptfile is used for a few things today:

1. To identify the directory as a "Package" and provide some meta-data/demographic info about it.
2. To identify the upstream package - that is, to hold a pointer to the original repository package from which this package was cloned (thus enabling us to discover and apply changes in those packages).
3. To hold the pipelines of functions to execute for mutation and for validation.
4. To hold the readiness gates and package conditions.

The change in how a PackageRevision is defined described above makes it possible to change all of this. If a PackageRevision is a pointer into a repo directory at a given tag/branch/commit, we don't need the first one at all. If we want to make a package discoverable, we could have some resource we add to the package and have controller that does package discovery. I am not sure we want to do that, and it won't be mandatory. We can now just create those PackageRevision like any other resource. There's also a lot of other stuff we might want to discover about a package, which would be used to feed the dependency management system. But that's all open for future discussion.

The upstream identity can be captured in an in-package resource, or in an in-cluster PackageRevision resource. As the current plan stands, a PackageRevision resource would still need an associated Repository, so if we want to be able to just point to a git or OCI instance as an upstream, we would need a little more flexibility here. I think it's still open to discussion whether this lives in-package or in-cluster (in-package things can be manipulated by CLIs locally, without server access).

The pipelines can just be another resource that Porch interprets when saving a package, as it does today. Wim also has some ideas and prototypes that would eliminate the need for the pipeline altogether.

Readiness gates and package conditions, should probably become part of the PackageRevision resource itself, like other K8s resources. For pipeline execution, we may want to project these down to the package resource list. That would be needed to support our existing function pipelines, for example. But it's not too hard to do.

Functional Enhancements

We have discussed bunch of functional enhancements. These probably should wait on the architectural changes. But here they are for discussion:

• Porch: PackageVariant Redux kptdev/kpt#3973
• CEL for function inputs kptdev/kpt#3964
• porch: consider Helm chart support as a package format kptdev/kpt#3847
• porch: live status of deployed package kptdev/kpt#3543
• Package dependencies: expressing ("my package requires") and fulfilling ("my package provides") kptdev/kpt#3448

As I mentioned, Wim has some additional prototypes that may replace some of these enhancements.

Logistics

I am working on the migration and plan to have it mostly done by the end of the year. I hope to have some help with the architectural improvements, and show some progress on those in January. The target would be to have all of those done around the end of 2024Q1; I would hope to get to some of the functional enhancements as well, certainly for R3.

[johnbelamaric]

cc @mortent @henderiw @tliron @s3wong @nephio-project/sig-automation

[tliron]

So much to discuss. :) Some very quick thoughts --

1. Calling the tool "nephioctl" might be assuming too much. I wonder if it would be better to call it "porchctl", at least at first, and then consider which features are truly "Nephio". After all, Porch right now is a generic orchestrator and has nothing per se to do with telco or Nephio. This can of course evolve now that Porch lives in Nephio, at which point we can consider a "nephioctl" tool that does various things other than just interacting with Porch.

2. Turning PackageRevision into CRs means that that aspect of scalability will be handled by etcd. We generally have a lot of unanswered questions about scalability, perhaps worth creating a special scalability task force to investigate them at some point. My own opinion, at least currently, as that etcd is not suitable for the orders of magnitude of scale we are intending. An aggregated API has the advantage that Porch can store all workload-cluster-related data in whatever data backend is appropriate for it. (My tko experiment has everything stored in PostreSQL.)

3. tko also uses an alternative approach to pipelines (also without Kptfiles).

[johnbelamaric]

So much to discuss. :) Some very quick thoughts --

1. Calling the tool "nephioctl" might be assuming too much. I wonder if it would be better to call it "porchctl", at least at first, and then consider which features are truly "Nephio". After all, Porch right now is a generic orchestrator and has nothing per se to do with telco or Nephio. This can of course evolve now that Porch lives in Nephio, at which point we can consider a "nephioctl" tool that does various things other than just interacting with Porch.

Sure, that works.

2. Turning PackageRevision into CRs means that that aspect of scalability will be handled by etcd. We generally have a lot of unanswered questions about scalability, perhaps worth creating a special scalability task force to investigate them at some point. My own opinion, at least currently, as that etcd is not suitable for the orders of magnitude of scale we are intending. An aggregated API has the advantage that Porch can store all workload-cluster-related data in whatever data backend is appropriate for it. (My tko experiment has everything stored in PostreSQL.)

Yeah, that's a good point. The Package Revision though is a "container" of other resources, though, and its size will be relatively constant. That is, it aggregates resources on average probably > 10:1 ratio. Still, that may still be an issue with etcd at high scale. We could consider backing it with something else. But it sure is nice to get all the free behavior from CRs - without that you have ti implement it all yourself (metadata, watches, etc. etc.). I wonder if anyone has created a generalized aggregated API server backed by SQL...?

But it is really important to get out of the business of maintaining a cache on top of git/oci the way we do today. I think we could probably do the PR-to-CR move, and then even later move to a new and better aggregated API server to back PRs in the future for scalability. I think trying to implement a new and different aggregated API server to back package revisions is ~ 10x the work. But I would love to be proven wrong.

3. tko also uses an alternative approach to pipelines (also without Kptfiles).

Yeah, we can think about how to make it flexible to support different options.

[kispaljr]

Taking over porch sounds exciting and definitely makes it easier for us (Nephio community) to contribute code to it. At the same time I am surprised that the kpt project is willing to give up control on porch and decided to focus on the CLI side of things instead. To be honest kpt seems to me quite hollow without porch (especially if they also remove ConfigSync, as well). But that's just me thinking out loud, however I have some actual questions, as well:

1. Does Step 2 "Migrate Server Code to Nephio" means to move to code to its own repo under the nephio-project organization, or will it live in a subdirectory inside https://github.com/nephio-project/nephio ?

2. Do we want to encourage people to use porch outside of the Nephio context, or will porch seize to exist as a sovereign project? (@tliron also addressed this above)

3. Regarding removing the kptfile from the package: it seems to me that all we are doing is splitting 1 in-package resource into 4:

   • "we could have some resource we add to the package and have controller that does package discovery"
   • "The upstream identity can be captured in an in-package resource"
   • "The pipelines can just be another resource"
   • "we may want to project these down to the package resource list"

   I agree that splitting the one kptfile into separate resources is a cleaner solution, and enables new features like having multiple pipelines in a package. At the same time I don't really see why we want to remove the existing in-package kptfile resource, if we just replace it with other ones.

[johnbelamaric]

Taking over porch sounds exciting and definitely makes it easier for us (Nephio community) to contribute code to it. At the same time I am surprised that the kpt project is willing to give up control on porch and decided to focus on the CLI side of things instead. To be honest kpt seems to me quite hollow without porch (especially if they also remove ConfigSync, as well). But that's just me thinking out loud, however I have some actual questions, as well:

1. Does Step 2 "Migrate Server Code to Nephio" means to move to code to its own repo under the nephio-project organization, or will it live in a subdirectory inside https://github.com/nephio-project/nephio ?

I plan to put it as a directory under nephio, unless folks have some objections. I think it's simpler that way.

2. Do we want to encourage people to use porch outside of the Nephio context, or will porch seize to exist as a sovereign project? (@tliron also addressed this above)

I think we do not want to encourage external users for now. Maybe someday, but we have enough to do already! So, yes, it would cease to be a sovereign project and just be part of Nephio.

3. Regarding removing the kptfile from the package: it seems to me that all we are doing is splitting 1 in-package resource into 4:

   • "we could have some resource we add to the package and have controller that does package discovery"
   • "The upstream identity can be captured in an in-package resource"
   • "The pipelines can just be another resource"
   • "we may want to project these down to the package resource list"

   I agree that splitting the one kptfile into separate resources is a cleaner solution, and enables new features like having multiple pipelines in a package. At the same time I don't really see why we want to remove the existing in-package kptfile resource, if we just replace it with other ones.

Correct. But I expect they will change a little as well, and that in fact we may have lots of packages without any of these resources. We will not need them if they in-cluster resources give us what we need and we do not need to reconcile those resources in the CLI. On the other hand, I could see a fully CLI-based version of PackageVariant as well...so...I think this part is much less clear, and we'll have to discuss it and decide what's best.

A concern here is really that it is a "Kptfile" resource, again outside of our control, and ties us directly to kpt as a hard dependency.

[rravindran123]

Quick question regarding removing Kptfile, would this cause incompatibility across domains, for e.g. Telco using the non-kpt file based packaging and Enterprises using the Kpt file based packaging, and in scenarios where you have to handle the two packaging formats for a specific use case scenario, specifically the KRM functions and controllers that act on them ?

[johnbelamaric]

KRM functions are a K8s standard, not a kpt thing. So we should be able to access those regardless. But you are right, if we move away from the Kptfile, then kpt itself won't be useful for those operations, unless we work with them to make those changes. That's not out-of-the question. The kpt is looking at decomposition too (I have discussed this there too, though not in great detail) - see for example kptdev/kpt#4093.

If we coordinate with them well, and with the K8s community, we may be able to upstream some of these resources in K8s and share them between Nephio, kpt, and (hopefully!) other projects. But it will require some real effort, and so far I have not had that level of coordination with the kpt team. These plans are evolving...let's try to evolve them so they work together.

[tliron]

1. Does Step 2 "Migrate Server Code to Nephio" means to move to code to its own repo under the nephio-project organization, or will it live in a subdirectory inside https://github.com/nephio-project/nephio ?

I plan to put it as a directory under nephio, unless folks have some objections. I think it's simpler that way.

I'm a "Unix philosophy" guy, so I have a personal preference to put it in a separate repo, called "porch". At least for now, it's a generic orchestrator, not Nephio-specific. This of course may evolve, perhaps some things would move to "nephio", some things remain in "porch".

[johnbelamaric]

I put a PR out. If you want to create the separate repo and all that entails, I am happy to move the PR to that repo. But I don't have time to do the overhead bits.

[adetalhouet]

I admit not to have engaged in all the meetings lately; but I'm surprised by this direction. I understand Porch is a key component of Nephio today.

Should Nephio be in the business of supplying orchestration related tooling?
Can't Nephio accommodate with what the industry has to offer without being distracted by maintaining more code?

Nephio scope is about infra and nf LCM.
I would encourage architecture discussion and identify what are the functionalities we need before keeping it as the go-to solution.
kpt getting rid of this isn't necessarily a good sign and I would be mindful of what to do, without jumping on it.

[johnbelamaric]

I admit not to have engaged in all the meetings lately; but I'm surprised by this direction. I understand Porch is a key component of Nephio today.

Should Nephio be in the business of supplying orchestration related tooling? Can't Nephio accommodate with what the industry has to offer without being distracted by maintaining more code?

Maybe, maybe not. But we have the dependency now, so tactically we need to continue its support since the original project is not planning to do so. To mitigate the risks you raise, the plan includes 1) removal of all code we don't actually need; 2) not promoting it for use outside of Nephio (for now at least). We can decide to migrate to something else later, but we need the continuity now.

Nephio scope is about infra and nf LCM. I would encourage architecture discussion and identify what are the functionalities we need before keeping it as the go-to solution. kpt getting rid of this isn't necessarily a good sign and I would be mindful of what to do, without jumping on it.

I actually disagree with that statement ("about infra and nf LCM") and others in the community also have a different opinion (and therefore we have the Enterprise discussion in the TSC, for example). In order to manage multiple domains, it's clear that a general purpose tool is needed; there is nothing super special about NF and NF infra - mostly it is the same stuff, just harder / more complex.

I agree though with the basic point - that we should identify exactly what we need and scope it to that. I also think it makes sense to see if we should transition away from Porch (this plan already reduces dependency on kpt). But those are strategic questions we will address over time. The first part of this plan is tactical, and the second, less developed, is strategic.

Tal has talked about abstracting this part into an interface and then Porch can just be a reference implementation. I think that's viable. A couple words of caution, though:

• An interface with a single implementation and no real prospects for more is just added complexity
• I do think CaD is unique and other ecosystem projects have not quite gotten there. But they could...but it would require evangelism in those separate communities.
• If we do not rely on the unique, opinionated nature of CaD in that interface, we will not have anything materially different than prior efforts, and will not be materially more successful.

Along the lines of the interface, though, I think the key thing to recognize is our K8s APIs. I think the core pieces we need are:

• Repository
• PackageRevision (or something like it), representing a particular instance of a package at a particular version. Important aspects are the concept of "Draft", and the ability to managed package conditions and readiness gates.
• Some way to access / tweak the package contents via K8s controllers (currently PackageRevisionResources - which is only a separate object because CRDS do not support arbitrary sub-resources)
• PackageVariant / PackageVariantSet

I think it should be possible to implement these concepts with different tooling. But it's a bigger question that we will need to answer later ... likely post-R3.