Nephio Controller Package Update

[arora-sagar]

While using the latest image in the old deployment.yaml of nephio controller. The controller have some errors.

2023-12-15T10:09:00.028Z	INFO	cluster client not found, retry...	{"controller": "BootstrapPackageController", "controllerGroup": "[porch.kpt.dev](http://porch.kpt.dev/)", "controllerKind": "PackageRevision", "PackageRevision": {"name":"mgmt-staging-cb118c6f25c243371040ef6652faec917e20fc28","namespace":"default"}, "namespace": "default", "name": "mgmt-staging-cb118c6f25c243371040ef6652faec917e20fc28", "reconcileID": "910a522f-63d4-4be9-aa7c-38094b11bca9"}

We would need to update the controller and token deployment CR.

[johnbelamaric]

I'm confused. What needs to change in the manifest?

See nephio-project/catalog#14, where I have moved the package from examples to the catalog repo. I changed to point to :latest, but no other changes to the manifests. Is there something needed there?

[liamfallon]

I was looking at this as well to see if the unit test refactoring could have caused this but right now I can't see how they could. @arora-sagar could you give a little more context on how this popped up?

[arora-sagar]

Hi @liamfallon so I wanted to try the new nephio-controller image so I deployed using the test-infra scripts and later I changed the deployment.yaml (I suppose it is still taking from https://github.com/nephio-project/nephio-example-packages/blob/main/nephio-controllers/app/controller/deployment-controller.yaml) of nephio-controller to use latest image instead of v1.0.1. When I did that I had these errors. If you want the complete logs then I may have to run another VM and provide you the error.

[arora-sagar]

@johnbelamaric @henderiw I tried just now and I used latest images for nephio-operator and porch. I took the latest changes from John's PR. Though I can create cluster but I still see these INFO messages in the controller (kubectl logs nephio-controller-xxxx -c controller) logs

Cluster client not found, retry.... What does it signify?

2024-01-04T21:49:21.046Z	INFO	reconcile package revision	{"controller": "BootstrapPackageController", "controllerGroup": "porch.kpt.dev", "controllerKind": "PackageRevision", "PackageRevision": {"name":"mgmt-staging-ef26516a4836a44cc2da33640733dbd71a3b8e0f","namespace":"default"}, "namespace": "default", "name": "mgmt-staging-ef26516a4836a44cc2da33640733dbd71a3b8e0f", "reconcileID": "eb5d6c9d-4f8c-4a74-8241-ae8b0b457cb2"}
2024-01-04T21:49:21.063Z	INFO	cluster client not found, retry...	{"controller": "BootstrapPackageController", "controllerGroup": "porch.kpt.dev", "controllerKind": "PackageRevision", "PackageRevision": {"name":"mgmt-staging-ef26516a4836a44cc2da33640733dbd71a3b8e0f","namespace":"default"}, "namespace": "default", "name": "mgmt-staging-ef26516a4836a44cc2da33640733dbd71a3b8e0f", "reconcileID": "eb5d6c9d-4f8c-4a74-8241-ae8b0b457cb2"}
2024-01-04T21:49:21.368Z	INFO	reconcile package revision	{"controller": "BootstrapPackageController", "controllerGroup": "porch.kpt.dev", "controllerKind": "PackageRevision", "PackageRevision": {"name":"mgmt-staging-e136801fb3a3da92252d3e5801fbe3aec03072c8","namespace":"default"}, "namespace": "default", "name": "mgmt-staging-e136801fb3a3da92252d3e5801fbe3aec03072c8", "reconcileID": "f19690f5-943b-482d-949f-f4549f1185c9"}
2024-01-04T21:49:22.363Z	INFO	cluster client not found, retry...	{"controller": "BootstrapPackageController", "controllerGroup": "porch.kpt.dev", "controllerKind": "PackageRevision", "PackageRevision": {"name":"mgmt-staging-e136801fb3a3da92252d3e5801fbe3aec03072c8","namespace":"default"}, "namespace": "default", "name": "mgmt-staging-e136801fb3a3da92252d3e5801fbe3aec03072c8", "reconcileID": "f19690f5-943b-482d-949f-f4549f1185c9"}
2024-01-04T21:49:22.371Z	INFO	reconcile genericspecializer	{"controller": "GenericSpecializer", "controllerGroup": "porch.kpt.dev", "controllerKind": "PackageRevision", "PackageRevision": {"name":"mgmt-staging-1194f62910accd76e3d1eb393ecd642890c31be4","namespace":"default"}, "namespace": "default", "name": "mgmt-staging-1194f62910accd76e3d1eb393ecd642890c31be4", "reconcileID": "96fb8f9d-72da-47cb-b98f-64e7e11c8db6", "req": {"name":"mgmt-staging-1194f62910accd76e3d1eb393ecd642890c31be4","namespace":"default"}}
2024-01-04T21:49:22.373Z	INFO	reconcile approval	{"controller": "ApprovalController", "controllerGroup": "porch.kpt.dev", "controllerKind": "PackageRevision", "PackageRevision": {"name":"mgmt-staging-1157ff0d8dfade58b155988f1f66422740b86951","namespace":"default"}, "namespace": "default", "name": "mgmt-staging-1157ff0d8dfade58b155988f1f66422740b86951", "reconcileID": "b563c626-4411-4407-81fe-f5b4c5e30b4a", "req": {"name":"mgmt-staging-1157ff0d8dfade58b155988f1f66422740b86951","namespace":"default"}}
2024-01-04T21:49:22.373Z	DEBUG	events	all approval policies met	{"type": "Normal", "object": {"kind":"PackageRevision","namespace":"default","name":"mgmt-staging-1194f62910accd76e3d1eb393ecd642890c31be4","uid":"uid:edge-rootsync:packagevariant-1","apiVersion":"porch.kpt.dev/v1alpha1","resourceVersion":"4738b10494ef3ff1857798c7a71b791ce1286ee6"}, "reason": "Approved"}
2024-01-04T21:49:22.379Z	INFO	reconcile package revision	{"controller": "BootstrapPackageController", "controllerGroup": "porch.kpt.dev", "controllerKind": "PackageRevision", "PackageRevision": {"name":"mgmt-staging-1194f62910accd76e3d1eb393ecd642890c31be4","namespace":"default"}, "namespace": "default", "name": "mgmt-staging-1194f62910accd76e3d1eb393ecd642890c31be4", "reconcileID": "44f1d9db-116f-4abe-ad0c-0fa6095eddf5"}
2024-01-04T21:49:22.391Z	INFO	cluster client not found, retry...	{"controller": "BootstrapPackageController", "controllerGroup": "porch.kpt.dev", "controllerKind": "PackageRevision", "PackageRevision": {"name":"mgmt-staging-1194f62910accd76e3d1eb393ecd642890c31be4","namespace":"default"}, "namespace": "default", "name": "mgmt-staging-1194f62910accd76e3d1eb393ecd642890c31be4", "reconcileID": "44f1d9db-116f-4abe-ad0c-0fa6095eddf5"}
2024-01-04T21:49:22.478Z	INFO	reconcile package revision	{"controller": "BootstrapPackageController", "controllerGroup": "porch.kpt.dev", "controllerKind": "PackageRevision", "PackageRevision": {"name":"mgmt-staging-12537ddebde5d788a82aa07f0319065a35e2c430","namespace":"default"}, "namespace": "default", "name": "mgmt-staging-12537ddebde5d788a82aa07f0319065a35e2c430", "reconcileID": "9a861a89-cce8-44ec-a9e5-edee11faa729"}

[henderiw]

it means the cluster client to access the workload cluster was not found. We should have a secret with name -kubeconfig in the cluster. Dow e see this?

[arora-sagar]

it means the cluster client to access the workload cluster was not found. We should have a secret with name -kubeconfig in the cluster. Dow e see this?

Here are all the secrets and I do see kubeconfig secret

$ kubectl get secrets --all-namespaces
NAMESPACE                           NAME                                              TYPE                            DATA   AGE
capd-system                         capd-webhook-service-cert                         kubernetes.io/tls               3      98m
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-webhook-service-cert       kubernetes.io/tls               3      98m
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-webhook-service-cert   kubernetes.io/tls               3      98m
capi-system                         capi-webhook-service-cert                         kubernetes.io/tls               3      98m
cert-manager                        cert-manager-webhook-ca                           Opaque                          3      99m
config-management-system            mgmt-access-token-configsync                      kubernetes.io/basic-auth        3      94m
config-management-system            mgmt-staging-access-token-configsync              kubernetes.io/basic-auth        3      93m
default                             core-access-token-configsync                      kubernetes.io/basic-auth        3      23m
default                             core-access-token-porch                           kubernetes.io/basic-auth        3      23m
default                             core-ca                                           cluster.x-k8s.io/secret         2      23m
default                             core-etcd                                         cluster.x-k8s.io/secret         2      23m
default                             core-kubeconfig                                   cluster.x-k8s.io/secret         1      23m
default                             core-md-0-297bp-nj75g                             cluster.x-k8s.io/secret         2      22m
default                             core-proxy                                        cluster.x-k8s.io/secret         2      23m
default                             core-qggnv-strh5                                  cluster.x-k8s.io/secret         2      23m
default                             core-sa                                           cluster.x-k8s.io/secret         2      23m
default                             edge-5cr2z-ftwcf                                  cluster.x-k8s.io/secret         2      22m
default                             edge-access-token-configsync                      kubernetes.io/basic-auth        3      22m
default                             edge-access-token-porch                           kubernetes.io/basic-auth        3      22m
default                             edge-ca                                           cluster.x-k8s.io/secret         2      22m
default                             edge-etcd                                         cluster.x-k8s.io/secret         2      22m
default                             edge-kubeconfig                                   cluster.x-k8s.io/secret         1      22m
default                             edge-md-0-whpjq-4bsz2                             cluster.x-k8s.io/secret         2      21m
default                             edge-proxy                                        cluster.x-k8s.io/secret         2      22m
default                             edge-sa                                           cluster.x-k8s.io/secret         2      22m
default                             mgmt-access-token-porch                           kubernetes.io/basic-auth        3      94m
default                             mgmt-staging-access-token-porch                   kubernetes.io/basic-auth        3      93m
default                             regional-access-token-configsync                  kubernetes.io/basic-auth        3      22m
default                             regional-access-token-porch                       kubernetes.io/basic-auth        3      22m
default                             regional-ca                                       cluster.x-k8s.io/secret         2      22m
default                             regional-dxnbz-mrwln                              cluster.x-k8s.io/secret         2      22m
default                             regional-etcd                                     cluster.x-k8s.io/secret         2      22m
default                             regional-kubeconfig                               cluster.x-k8s.io/secret         1      22m
default                             regional-md-0-wkqkz-n9lg5                         cluster.x-k8s.io/secret         2      21m
default                             regional-proxy                                    cluster.x-k8s.io/secret         2      22m
default                             regional-sa                                       cluster.x-k8s.io/secret         2      22m
gitea                               git-user-secret                                   kubernetes.io/basic-auth        2      99m
gitea                               gitea                                             Opaque                          1      99m
gitea                               gitea-init                                        Opaque                          3      99m
gitea                               gitea-inline-config                               Opaque                          7      99m
gitea                               gitea-postgresql                                  Opaque                          2      99m
kube-system                         bootstrap-token-abcdef                            bootstrap.kubernetes.io/token   6      100m
metallb-system                      memberlist                                        Opaque                          1      99m
metallb-system                      webhook-server-cert                               Opaque                          4      99m
nephio-system                       git-user-secret                                   kubernetes.io/basic-auth        2      95m

Another thing @electrocucaracha the script 001.sh is getting timeout. It takes more than 20 mins to create the three cluster. Yesterday I tried even after 30 mins all the machinesets were not ready. I am testing with latest porch and controller versions.

[arora-sagar]

This issue was solved by 606c866