Adding NF CRDs in workload cluster package

[arora-sagar]

Currently, we are not installing CRDs via the workload package. It is being done via NF operators. Doing it as part of the workload cluster package is probably a good practice.

• NFConfig
• NFDeployment
• ConfigRef

If this is not a good place to put the CRDs then we can think of making a separate package.

[electrocucaracha]

@johnbelamaric I'm also not sure about the right place to allocate these CRDs, any idea?

[johnbelamaric]

Questions:

1. These CRDs live on the management cluster or the workload cluster?
2. These CRDs are OAI specific or general Nephio?

Assuming the answer to 1) is "management" and to 2) is "general", then these should be included with the Nephio controller manager package. In general, CRDs should be included with the reconciler that owns the corresponding CRs. If the CR may be managed by multiple controllers (say, depending on class), then they probably belong in their own package (or, for example, with the core Nephio controller manager).

I don't think we have built packages for "Nephio core" yet, outside of the R1 "nephio-example-packages" repository. The plan is that those packages would move to here: https://github.com/nephio-project/catalog/tree/main/nephio/core

[henderiw]

These CRDs are used in specialisation on mgmt cluster but are not actuated on the k8s API.
So this is needed on the workload cluster where they get actuated. So there are the nephio CRDs if you will for the workload cluster and are meta CRDs that are agnostic to the vendor.
So the scenario this solves is if you have multiple operators, they don't need to install them themselves or duplicate them, but they now have a dependency that they gets installed.

[arora-sagar]

These CRDs are needed on the workload cluster and in R2 both RAN and Core controllers need them. The core controllers and packages reside in the OAI GitHub repository and the RAN controller and packages reside in the catalog repository. So we need a common location from where they can be applied on the workload cluster.

[johnbelamaric]

A couple things:

1. This package doesn't make sense where it is. It should be in the infra/capi subdirectory and shared between the OAI and free5gc code. Is there something OAI specific about this?
2. This package is applied to the management cluster, so adding the CRDs here doesn't make sense if they are supposed to land on the workload cluster itself. I would create a separate package containing all the stuff we need on the workload cluster, then create a PV in this package to deliver that package to the workload cluster repository.

[arora-sagar]

A couple things:

1. This package doesn't make sense where it is. It should be in the infra/capi subdirectory and shared between the OAI and free5gc code. Is there something OAI specific about this?

There is nothing OAI specific we can move but I am not sure if infra/capi is a good place. We need to install these CRDs on all the workload clusters. At the moment we aren't using infra/capi.

2. This package is applied to the management cluster, so adding the CRDs here doesn't make sense if they are supposed to land on the workload cluster itself. I would create a separate package containing all the stuff we need on the workload cluster, then create a PV in this package to deliver that package to the workload cluster repository.

Ah okay now I somewhat understand your point. I will create a separate package and its PV. Where should that package and its PV should reside? infra/capi ?

[johnbelamaric]

The package for provisioning a workload cluster should live in infra/capi. The package that contains the various Nephio CRDs probably should live in nephio/core and be called something like "nephio-workload-crds" to indicate that it contains the CRDs that should be loaded on all workload clusters.

The "nephio-workload-cluster" package would then include a PV that installs that package on all new workload clusters.

[arora-sagar]

The package for provisioning a workload cluster should live in infra/capi. The package that contains the various Nephio CRDs probably should live in nephio/core and be called something like "nephio-workload-crds" to indicate that it contains the CRDs that should be loaded on all workload clusters.

The "nephio-workload-cluster" package would then include a PV that installs that package on all new workload clusters.

@johnbelamaric Do you mean something like this? Also, will we change the locations of other packages too? like kindnet, multus which are required by workload cluster? Right now they are in examples.

[johnbelamaric]

Yes, that's the right idea. The main thing is to separate out the platform from the use case. So:

• nephio/core directory contains all those things that should be in any given Nephio installation
• nephio/optional contains other things that may or may not be needed (for example, resource backend)
• distros contains individual opinionated distributions for different environments. In there we clone packages from nephio/* and create derivative packages that are customized to their specific environment. That is, they will will have an upstream in nephio/core, for example.
• infra contains packages for deploying particular kinds of infrastructure, such as a Kind cluster via CAPI, or a GKE cluster via Config Controller.
• workloads contains the packages for deploying workloads on top of those clusters

The "repo" for the various PVs won't be "catalog". Instead, we need to register the directories as individual repositories, in any particular opinionated installation. See for example the repo-*.yaml files here: https://github.com/nephio-project/catalog/tree/main/distros/gcp/nephio-mgmt for the ones I register for the GCP installation.

We may want to include a package of repositories in the "optional" directory, much like the "nephio-stock-repositories" package in examples. This would register a bunch of

The

[arora-sagar]

@johnbelamaric @henderiw @tliron @electrocucaracha so I am moving the packages from nephio-example-packages to catalog as per what @johnbelamaric mentioned in the previous comment.

Is everyone okay with this move?

.
├── distros
│   ├── gcp
│   │   ├── cc-rootsync
│   │   └── nephio-mgmt
│   │       ├── cert-manager
│   │       ├── nephio-controllers 
│   │       │   ├── app
│   │       │   │   └── controller
│   │       │   └── crd
│   │       │       └── bases
│   │       ├── nephio-webui
│   │       ├── network-config
│   │       │   ├── app
│   │       │   │   └── controller
│   │       │   └── crd
│   │       ├── porch
│   │       └── resource-backend
│   │           ├── app
│   │           │   └── controller
│   │           │       └── grpc
│   │           └── crd
│   │               └── bases
│   ├── openshift
│   └── sandbox
├── infra
│   ├── capi
│   │   ├── cert-manager
│   │   ├── cluster-capi
│   │   ├── cluster-capi-infrastructure-docker
│   │   ├── cluster-capi-kind
│   │   ├── cluster-capi-kind-docker-templates
│   │   ├── gitea
│   │   ├── kindnet
│   │   ├── local-path-provisioner
│   │   ├── metallb
│   │   ├── nephio-workload-cluster
│   │   ├── repository
│   │   └── vlanindex
│   └── gcp
│       ├── cc-cluster-gke-std-csr-cs
│       ├── cc-repo-csr
│       ├── nephio-blueprint-repo
│       └── nephio-workload-cluster-gke
├── nephio
│   ├── core
│   │   ├── configsync
│   │   ├── network-config
│   │   │   ├── app
│   │   │   │   └── controller
│   │   │   └── crd
│   │   ├── rootsync
│   │   └── workload-crds
│   └── optional
│       ├── nephio-stock-repos
│       ├── nephio-webui
│       └── resource-backend
│           ├── app
│           │   └── controller
│           │       └── grpc
│           └── crd
│               └── bases
└── workloads
    ├── free5gc
    └── oai
        ├── network
        ├── oai-ran-operator
        │   └── operator
        ├── package-variants
        ├── pkg-example-cucp-bp
        ├── pkg-example-cuup-bp
        ├── pkg-example-du-bp
        └── pkg-example-ue-bp
            └── oai-ue

nephio-controller and porch will be moved in nephio/core after merging #14 and #13

In nephio-stock-repo I will mention all package repositories individually for example cluster-capi, multus, kindnet etc.

I was thinking of mentioning the revisions in packages main because in the main branch all the packages should correspond to main right?

Is this okay? Did I understand it correctly? I just want to double-check before making the commit because once I committed I want to test and have to make changes in test-infra too.

[electrocucaracha]

Maybe we can do it in several PRs to facilitate the review

[johnbelamaric]

Generally this looks OK. Gitea and metal-lb are sandbox choices, and are distinct from "capi". Similarly while cert-manager is needed for CAPI it's needed for other things too, so I think it should be in distros. So:

The following should move from infra/capi to distros/sandbox: cert-manager, gitea, metallb, repository

The following should move from nephio/core to nephio/optional: network-config

The following should move from workloads/oai to distros/sandbox: network

I think "multus" also is needed in infra/capi. "vlanindex" probably is good in infra/capi, although maybe that belongs in distros/sandbox.

We will also I think eventually clone things from nephio/core and nephio/optional into distros/sandbox. This is how I did the distros/gcp, for example. Take the base Nephio package, and then customize it for the specific distro.

[electrocucaracha]

Generally this looks OK. Gitea and metal-lb are sandbox choices, and are distinct from "capi". Similarly while cert-manager is needed for CAPI it's needed for other things too, so I think it should be in distros. So:

The following should move from infra/capi to distros/sandbox: cert-manager, gitea, metallb, repository

The following should move from nephio/core to nephio/optional: network-config

The following should move from workloads/oai to distros/sandbox: network

I think "multus" also is needed in infra/capi. "vlanindex" probably is good in infra/capi, although maybe that belongs in distros/sandbox.

We will also I think eventually clone things from nephio/core and nephio/optional into distros/sandbox. This is how I did the distros/gcp, for example. Take the base Nephio package, and then customize it for the specific distro.

What about creating a workloads/common folder for multus package? It's used by free5gc and oai test scenarios and they don't require a specific version.

[johnbelamaric]

It's not a workload, it's a part of the cluster infrastructure, and applies specifically to CAPI clusters. It may be used by workloads, but for example it cannot be installed in GKE. And other cluster provisioners (e.g., OpenShift) probably have their own opinionated version and way to install it.

[electrocucaracha]

I think that we can merge this as it is and create another PRs for moving the rest of the packages

/lgtm

[electrocucaracha]

/lgtm

[electrocucaracha]

/approve

[johnbelamaric]

/approve
/lgtm

[nephio-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: arora-sagar, electrocucaracha, johnbelamaric

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [johnbelamaric]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment