Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

porch: packagevariant creating extra revisions #3877

Open
johnbelamaric opened this issue Mar 13, 2023 · 15 comments
Open

porch: packagevariant creating extra revisions #3877

johnbelamaric opened this issue Mar 13, 2023 · 15 comments
Labels
area/porch bug Something isn't working triaged Issue has been triaged by adding an `area/` label

Comments

@johnbelamaric
Copy link
Contributor

Expected behavior

PackageVariant will create a single downstream.

Actual behavior

Usually it works. But I have this weirdness:

Repositories

jbelamaric@jbelamaric:~/proj/tmp/cachingdns-topology$ k get repositories
NAME                      TYPE   CONTENT   DEPLOYMENT   READY   ADDRESS
nephio-packages           git    Package                True    https://github.com/nephio-project/nephio-packages.git
nephio-testorg-packages   git    Package                True    https://github.com/johnbelamaric/nephio-testorg-packages.git
us-central1-edge-01       git    Package   true         True    https://github.com/nephio-test/jbelamaric-us-central1-edge-01.git
us-central1-edge-02       git    Package   true         True    https://github.com/nephio-test/jbelamaric-us-central1-edge-02.git
us-central1-edge-03       git    Package   true         True    https://github.com/nephio-test/jbelamaric-us-central1-edge-03.git
us-central1-edge-04       git    Package   true         True    https://github.com/nephio-test/jbelamaric-us-central1-edge-04.git
us-central1-regional      git    Package   true         True    https://github.com/nephio-test/jbelamaric-us-central1-regional.git
us-central2-regional      git    Package   true         True    https://github.com/nephio-test/jbelamaric-us-central2-regional.git
us-east1-edge-01          git    Package   true         True    https://github.com/nephio-test/jbelamaric-us-east1-edge-01.git
us-east1-edge-02          git    Package   true         True    https://github.com/nephio-test/jbelamaric-us-east1-edge-02.git
us-east1-regional-01      git    Package   true         True    https://github.com/nephio-test/jbelamaric-us-east1-regional-01.git
us-east1-regional-02      git    Package   true         True    https://github.com/nephio-test/jbelamaric-us-east1-regional-02.git
us-east2-regional         git    Package   true         True    https://github.com/nephio-test/jbelamaric-us-east2-regional.git

PackageVariant resources (created by PackageVariantSet controller):

NAME                                                               AGE
cachingdns-distribution-1e718cfbdb754e386474da9aa91bd159ed27c6d7   25m
cachingdns-distribution-52cb15c06dc9252e966897bde315bc6b2e06af4f   25m
cachingdns-distribution-54154c218b775b84e280258f12f684c813e76c49   25m
cachingdns-distribution-aa0bd84da557680b052811b54e528625c13907cf   25m
cachingdns-distribution-d1579cce150353b437bdaa9845702b33f0900cdc   25m
cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7           25m
cachingdns-edge-3cc7f3a7d7f8d27acc6375cba42b5cc168d70fa0           25m
cachingdns-edge-44f604043046c915ea89667128b276b9c03211b7           25m
cachingdns-edge-596ce7b3b682f6eb284d7c1aec74aaaf179ad834           25m
cachingdns-edge-6b3bac48e67223051e4e47b5c52095eda9c1ee35           25m
cachingdns-edge-edf34f330bd64333cb936090fb4682e1d84f0159           25m

Those both look fine. But, here are my rpkgs:

jbelamaric@jbelamaric:~/proj/tmp/cachingdns-topology$ kpt alpha rpkg get | grep packagevariant
us-central1-edge-01-fc057482816768663fef81c4e5252856a632751d       cachingdns-edge           packagevariant-1               false    Draft       us-central1-edge-01
us-central1-edge-02-06a45d5516402094e3150d7fd24ae5ab857a2cf2       cachingdns-distribution   packagevariant-4               false    Draft       us-central1-edge-02
us-central1-edge-02-3df1fdf1ccfd97e68b3bf998220c3a2948b10e01       cachingdns-distribution   packagevariant-6               false    Draft       us-central1-edge-02
us-central1-edge-02-93594975a009c248ddf9dd362d7cdd5e5ff6e0e5       cachingdns-distribution   packagevariant-2               false    Draft       us-central1-edge-02
us-central1-edge-02-9f4cb6e8866d95d3fe3d1ab30b2ddf70621005e8       cachingdns-distribution   packagevariant-5               false    Draft       us-central1-edge-02
us-central1-edge-02-d3b22ec4000657ed8ce88ffc6d7551ba4561c101       cachingdns-distribution   packagevariant-3               false    Draft       us-central1-edge-02
us-central1-edge-02-f399f895cabbe912a980f5428a88b4720d9b583e       cachingdns-distribution   packagevariant-1               false    Draft       us-central1-edge-02
us-central1-edge-02-0762cce1279792c52a4d42be1368ace3e3adc136       cachingdns-edge           packagevariant-33              false    Draft       us-central1-edge-02
us-central1-edge-02-09e084d02cdace5f4b8c91c440f5da93fcb95129       cachingdns-edge           packagevariant-6               false    Draft       us-central1-edge-02
us-central1-edge-02-0fb62c5023354f87b6f1b7479c5ca85a960509aa       cachingdns-edge           packagevariant-35              false    Draft       us-central1-edge-02
us-central1-edge-02-1016bd461c10fd57133b7332a0296311d035e2e3       cachingdns-edge           packagevariant-11              false    Draft       us-central1-edge-02
us-central1-edge-02-3784c881bbf0490480786935fa4524ade10ee087       cachingdns-edge           packagevariant-28              false    Draft       us-central1-edge-02
us-central1-edge-02-395852bc65dd70791213a6b6534973bcd1d8d8cd       cachingdns-edge           packagevariant-24              false    Draft       us-central1-edge-02
us-central1-edge-02-416da315c383ae26e33d3139e7dabd335fe04ecf       cachingdns-edge           packagevariant-4               false    Draft       us-central1-edge-02
us-central1-edge-02-487f5fd64471a08a0817f37843a5d9477a9fe33e       cachingdns-edge           packagevariant-19              false    Draft       us-central1-edge-02
us-central1-edge-02-49f0e8b9458b2f995544df5a4732dc0d95ff066a       cachingdns-edge           packagevariant-30              false    Draft       us-central1-edge-02
us-central1-edge-02-4bd6f30b30795031caff80f0e2fdc25a573701b2       cachingdns-edge           packagevariant-22              false    Draft       us-central1-edge-02
us-central1-edge-02-5624c548be4419e5e50820f346fe4b855b0b6691       cachingdns-edge           packagevariant-15              false    Draft       us-central1-edge-02
us-central1-edge-02-5f6a1c61c1e296922b457f099cbd893c1a1d9c04       cachingdns-edge           packagevariant-21              false    Draft       us-central1-edge-02
us-central1-edge-02-64641a2a553e9c40af9aa21fdfbf4cf531c7349f       cachingdns-edge           packagevariant-2               false    Draft       us-central1-edge-02
us-central1-edge-02-66ebded096fe9bc454b12b0ffed761f98f3a628f       cachingdns-edge           packagevariant-32              false    Draft       us-central1-edge-02
us-central1-edge-02-6e4f72848e5d0b609e72012161fa245b0312119d       cachingdns-edge           packagevariant-8               false    Draft       us-central1-edge-02
us-central1-edge-02-6e965eca522fd0a47eec31778b6736d70ef4c0a8       cachingdns-edge           packagevariant-5               false    Draft       us-central1-edge-02
us-central1-edge-02-7402304235c26cfccce8c78510dbcefd90a2e41d       cachingdns-edge           packagevariant-14              false    Draft       us-central1-edge-02
us-central1-edge-02-78e6ec06c813b6e5210015e7525df1546d122259       cachingdns-edge           packagevariant-1               false    Draft       us-central1-edge-02
us-central1-edge-02-84ac1ac174d3be6eb56f8954660663b0250d0683       cachingdns-edge           packagevariant-7               false    Draft       us-central1-edge-02
us-central1-edge-02-8eeb3ea6942da97ac354cbd71114f8c8cccfa2a6       cachingdns-edge           packagevariant-25              false    Draft       us-central1-edge-02
us-central1-edge-02-8f194f5ae63dac4946be76c15d69a1384980a061       cachingdns-edge           packagevariant-26              false    Draft       us-central1-edge-02
us-central1-edge-02-9ccf6de38e495c327372a04dabf63e3276a716a3       cachingdns-edge           packagevariant-29              false    Draft       us-central1-edge-02
us-central1-edge-02-a4f829aa9b7ed8816d327b1503c7336210cb1e13       cachingdns-edge           packagevariant-12              false    Draft       us-central1-edge-02
us-central1-edge-02-abcc12021c54e5a19d36e9be2fcda2e5558b063d       cachingdns-edge           packagevariant-3               false    Draft       us-central1-edge-02
us-central1-edge-02-b9da53a7ba942accc51520d5ad715a3459bf7d3e       cachingdns-edge           packagevariant-18              false    Draft       us-central1-edge-02
us-central1-edge-02-bc97ce6211f08e395e71e853fc2fdf57774da6f5       cachingdns-edge           packagevariant-10              false    Draft       us-central1-edge-02
us-central1-edge-02-c5d5e4d1d358cab4ea6aab52151ec2626bc3191c       cachingdns-edge           packagevariant-16              false    Draft       us-central1-edge-02
us-central1-edge-02-c8480450e11ed9d3eb82270606665ea0493de856       cachingdns-edge           packagevariant-23              false    Draft       us-central1-edge-02
us-central1-edge-02-d33d3cd9b59cbb3e959a03404a156b172f55d12d       cachingdns-edge           packagevariant-13              false    Draft       us-central1-edge-02
us-central1-edge-02-dbfc104305f952056bc2037d7ce0f8363d46baa6       cachingdns-edge           packagevariant-9               false    Draft       us-central1-edge-02
us-central1-edge-02-eec3ede48babddb5d98fad34d4422f0325e947bd       cachingdns-edge           packagevariant-27              false    Draft       us-central1-edge-02
us-central1-edge-02-ef67e0e9ec797aa76086e1bae7eac958efe3088d       cachingdns-edge           packagevariant-17              false    Draft       us-central1-edge-02
us-central1-edge-02-efc16abcf6b52ac234170799af0b30356024daca       cachingdns-edge           packagevariant-31              false    Draft       us-central1-edge-02
us-central1-edge-02-f1e71381e54f7850a73a43e538f8c47ad23c89c7       cachingdns-edge           packagevariant-34              false    Draft       us-central1-edge-02
us-central1-edge-02-fadd99b4f301abf29dd1cd7fd46e1e2121b872d6       cachingdns-edge           packagevariant-20              false    Draft       us-central1-edge-02
us-central1-edge-03-e01e67636b8d845b321a650a3423d07b9881b97c       cachingdns-edge           packagevariant-1               false    Draft       us-central1-edge-03
us-central1-edge-04-fc22f957113a122cf04158b8dfe079c9f7503e99       cachingdns-edge           packagevariant-1               false    Draft       us-central1-edge-04
us-central1-regional-92794a396633223eff36ed41701210b6e319d5f5      cachingdns-distribution   packagevariant-1               false    Draft       us-central1-regional
us-central2-regional-322bcb84ae8a8a6ddd40513f3a2bb55f99b3f75a      cachingdns-distribution   packagevariant-1               false    Draft       us-central2-regional
us-east1-edge-01-459f97554e7a1551072260a0bc616f6c4e9f7968          cachingdns-edge           packagevariant-1               false    Draft       us-east1-edge-01
us-east1-edge-02-2b0efaed7ac3998524a8b01e968a5c29843d5cf4          cachingdns-edge           packagevariant-1               false    Draft       us-east1-edge-02
us-east1-regional-01-72d397520036dd95fd49d60589b7a76edab5709e      cachingdns-distribution   packagevariant-1               false    Draft       us-east1-regional-01
us-east1-regional-02-bed387c8e383e4c20355bcb496811b300b3da653      cachingdns-distribution   packagevariant-1               false    Draft       us-east1-regional-02
us-east2-regional-183271f74d1feed2ffe5058c938945067b2c8559         cachingdns-distribution   packagevariant-1               false    Draft       us-east2-regional

The us-central1-edge-02 variant has gone crazy!

This is in the logs a lot:

I0313 18:33:54.170521       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0313 18:34:53.380233       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0313 18:35:53.774376       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0313 18:36:55.084324       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0313 18:37:54.785738       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0313 18:38:55.879202       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0313 18:39:55.783493       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0313 18:40:52.972026       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0313 18:41:57.681830       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0313 18:42:56.670062       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0313 18:43:57.176065       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0313 18:44:52.470967       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0313 18:45:57.877849       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0313 18:46:55.880729       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0313 18:47:52.878709       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""

Information

Steps to reproduce the behavior

I'll work on a cleaner repro.
@johnbelamaric johnbelamaric added the bug Something isn't working label Mar 13, 2023
@johnbelamaric johnbelamaric mentioned this issue Mar 13, 2023
Open
@johnbelamaric johnbelamaric mentioned this issue Mar 14, 2023
Open
@johnbelamaric
Copy link
Contributor Author

porch-controllers.log

Controller log. Seeing this a lot:

I0314 17:06:26.474898       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
E0314 17:07:00.871929       1 controller.go:326]  "msg"="Reconciler error" "error"="Timeout: request did not complete within requested timeout - context deadline exceeded" "PackageVariant"={"name":"cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7","namespace":"default"} "controller"="packagevariant" "controllerGroup"="config.porch.kpt.dev" "controllerKind"="PackageVariant" "name"="cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" "namespace"="default" "reconcileID"="63524b1d-6e1d-423f-87f9-2f37853e695d"
I0314 17:07:00.879073       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
E0314 17:07:35.081099       1 controller.go:326]  "msg"="Reconciler error" "error"="Timeout: request did not complete within requested timeout - context deadline exceeded" "PackageVariant"={"name":"cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7","namespace":"default"} "controller"="packagevariant" "controllerGroup"="config.porch.kpt.dev" "controllerKind"="PackageVariant" "name"="cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" "namespace"="default" "reconcileID"="11b1575b-25da-4a43-8ffa-b1ffee71b351"
I0314 17:08:23.772097       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
E0314 17:08:58.073898       1 controller.go:326]  "msg"="Reconciler error" "error"="Timeout: request did not complete within requested timeout - context deadline exceeded" "PackageVariant"={"name":"cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7","namespace":"default"} "controller"="packagevariant" "controllerGroup"="config.porch.kpt.dev" "controllerKind"="PackageVariant" "name"="cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" "namespace"="default" "reconcileID"="b07c51c5-fa62-47ef-a7fd-4d055616c857"
I0314 17:08:58.081976       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
E0314 17:09:32.371065       1 controller.go:326]  "msg"="Reconciler error" "error"="Timeout: request did not complete within requested timeout - context deadline exceeded" "PackageVariant"={"name":"cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7","namespace":"default"} "controller"="packagevariant" "controllerGroup"="config.porch.kpt.dev" "controllerKind"="PackageVariant" "name"="cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" "namespace"="default" "reconcileID"="62351605-0d6f-47a3-b804-6c56462c06c1"

@johnbelamaric
Copy link
Contributor Author

This is so weird. I deleted all the packagevariantsets; all the packagevariants were deleted (see though #3878). I manually deleted all the lingering branches for "edge-02" (see #3882). I used kpt alpha rpkg del to delete the remaining lingering Drafts in the other repos. So, in theory we are back to original state. But, I redeploy the PVSs, and I am seeing the same thing again, with the same repo:

us-central1-edge-01-fc057482816768663fef81c4e5252856a632751d       cachingdns-edge           packagevariant-1                false    Draft       us-central1-edge-01
us-central1-edge-02-037d2e76d7b3cfce0670a50cc580c4d3b7b950b4       cachingdns-edge           packagevariant-203              false    Draft       us-central1-edge-02
us-central1-edge-02-3a293d54ed1fc31baf0751cbbf8e3010509fc82a       cachingdns-edge           packagevariant-202              false    Draft       us-central1-edge-02
us-central1-edge-02-49585d06272feccda423ee21e0c8314fb8ea17d0       cachingdns-edge           packagevariant-208              false    Draft       us-central1-edge-02
us-central1-edge-02-50cb78cc4cffd97f4789c06341bee481d3b050d4       cachingdns-edge           packagevariant-200              false    Draft       us-central1-edge-02
us-central1-edge-02-698e57415b142775421c1a9e0f1136feebf93ba0       cachingdns-edge           packagevariant-206              false    Draft       us-central1-edge-02
us-central1-edge-02-749452573d6327c1e722e2ffe40e00077e1e68b2       cachingdns-edge           packagevariant-201              false    Draft       us-central1-edge-02
us-central1-edge-02-d27235098b919b529879725e6bf345382da7f888       cachingdns-edge           packagevariant-210              false    Draft       us-central1-edge-02
us-central1-edge-02-e1ab6665a67f553a9eeb96942de5a099c2d5c677       cachingdns-edge           packagevariant-211              false    Draft       us-central1-edge-02
us-central1-edge-02-f68df374d0e36061b345bdf89f98a43fcc72b2d7       cachingdns-edge           packagevariant-207              false    Draft       us-central1-edge-02
us-central1-edge-02-f9c0f18c7bacd9a5c8ae05442c07ff6d93fea57b       cachingdns-edge           packagevariant-205              false    Draft       us-central1-edge-02
us-central1-edge-02-fc0ea9d97bf4e174d7e1b3b427962c8b472d20c0       cachingdns-edge           packagevariant-204              false    Draft       us-central1-edge-02
us-central1-edge-02-fde0df04692f2ad2926ead7e72ee17561abbc1cd       cachingdns-edge           packagevariant-209              false    Draft       us-central1-edge-02

@johnbelamaric
Copy link
Contributor Author

More info:

jbelamaric@jbelamaric:~/proj/tmp/cachingdns-topology$ k get packagerevision us-central1-edge-02-3a293d54ed1fc31baf0751cbbf8e3010509fc82a -o yaml
apiVersion: porch.kpt.dev/v1alpha1
kind: PackageRevision
metadata:
  creationTimestamp: "2023-03-14T18:48:02Z"
  name: us-central1-edge-02-3a293d54ed1fc31baf0751cbbf8e3010509fc82a
  namespace: default
  resourceVersion: 854df9b1b3ffb03d18a93ea235fde92fb0a06f63
  uid: uid:cachingdns-edge:packagevariant-202
spec:
  lifecycle: Draft
  packageName: cachingdns-edge
  repository: us-central1-edge-02
  tasks:
  - clone:
      upstreamRef:
        upstreamRef:
          name: nephio-testorg-packages-4ad103ac302c07b54dd709719f6ef500d64424d9
    type: clone
  - eval:
      config: null
      image: render
      match: {}
    type: eval
  workspaceName: packagevariant-202
status:
  deployment: true
  publishTimestamp: null
  upstreamLock:
    git:
      commit: 3c8d8c0fb2081375ac31c01db07563dff00dc190
      directory: cachingdns-edge
      ref: cachingdns-edge/v2
      repo: https://github.com/johnbelamaric/nephio-testorg-packages.git
    type: git
jbelamaric@jbelamaric:~/proj/tmp/cachingdns-topology$ k get repository us-central1-edge-02 -o yaml
apiVersion: config.porch.kpt.dev/v1alpha1
kind: Repository
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"config.porch.kpt.dev/v1alpha1","kind":"Repository","metadata":{"annotations":{},"labels":{"kpt.dev/deployment-environment":"production","nephio.org/region":"us-central1","nephio.org/site":"us-central1-edge-02","nephio.org/site-type":"edge"},"name":"us-central1-edge-02","namespace":"default"},"spec":{"content":"Package","deployment":true,"description":"US Central 1 Edge Site 02 Cluster Workloads","git":{"branch":"main","createBranch":true,"directory":"/","repo":"https://github.com/nephio-test/jbelamaric-us-central1-edge-02.git","secretRef":{"name":"nephio-test-auth"}},"type":"git"}}
  creationTimestamp: "2023-03-09T22:47:01Z"
  generation: 2
  labels:
    kpt.dev/deployment-environment: production
    nephio.org/region: us-central1
    nephio.org/site: us-central1-edge-02
    nephio.org/site-type: edge
  name: us-central1-edge-02
  namespace: default
  resourceVersion: "5635626"
  uid: e7409f75-997e-41aa-bef2-33b324fdd341
spec:
  content: Package
  deployment: true
  description: US Central 1 Edge Site 02 Cluster Workloads
  git:
    branch: main
    createBranch: true
    directory: /
    repo: https://github.com/nephio-test/jbelamaric-us-central1-edge-02.git
    secretRef:
      name: nephio-test-auth
  type: git
status:
  conditions:
  - lastTransitionTime: "2023-03-09T23:06:36Z"
    message: ""
    observedGeneration: 2
    reason: Ready
    status: "True"
    type: Ready
jbelamaric@jbelamaric:~/proj/tmp/cachingdns-topology$ k get packagevariants
NAME                                                               AGE
cachingdns-distribution-1e718cfbdb754e386474da9aa91bd159ed27c6d7   16m
cachingdns-distribution-52cb15c06dc9252e966897bde315bc6b2e06af4f   16m
cachingdns-distribution-54154c218b775b84e280258f12f684c813e76c49   16m
cachingdns-distribution-aa0bd84da557680b052811b54e528625c13907cf   16m
cachingdns-distribution-d1579cce150353b437bdaa9845702b33f0900cdc   16m
cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7           16m
cachingdns-edge-3cc7f3a7d7f8d27acc6375cba42b5cc168d70fa0           16m
cachingdns-edge-44f604043046c915ea89667128b276b9c03211b7           16m
cachingdns-edge-596ce7b3b682f6eb284d7c1aec74aaaf179ad834           16m
cachingdns-edge-6b3bac48e67223051e4e47b5c52095eda9c1ee35           16m
cachingdns-edge-edf34f330bd64333cb936090fb4682e1d84f0159           16m
jbelamaric@jbelamaric:~/proj/tmp/cachingdns-topology$ k get packagevariant cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7 -o yaml
apiVersion: config.porch.kpt.dev/v1alpha1
kind: PackageVariant
metadata:
  creationTimestamp: "2023-03-14T18:45:45Z"
  finalizers:
  - config.porch.kpt.dev/packagevariants
  generation: 1
  labels:
    config.porch.kpt.dev/packagevariantset: 12a1914a-69e3-4050-acef-3bb97dc2683e
  name: cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7
  namespace: default
  ownerReferences:
  - apiVersion: config.porch.kpt.dev/v1alpha1
    controller: true
    kind: PackageVariantSet
    name: cachingdns-edge
    uid: 12a1914a-69e3-4050-acef-3bb97dc2683e
  resourceVersion: "7160488"
  uid: 8baa9b23-2511-4128-bbd5-2f659caa2943
spec:
  adoptionPolicy: adoptNone
  deletionPolicy: delete
  downstream:
    package: cachingdns-edge
    repo: us-central1-edge-02
  upstream:
    package: cachingdns-edge
    repo: nephio-testorg-packages
    revision: v2
jbelamaric@jbelamaric:~/proj/tmp/cachingdns-topology$

@johnbelamaric
Copy link
Contributor Author

From the logs, I see this over and over:

I0314 18:53:53.720819       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0314 18:54:51.903654       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0314 18:55:52.123161       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0314 18:57:10.288456       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0314 18:58:07.186308       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0314 18:58:52.597565       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""
I0314 18:59:56.185249       1 packagevariant_controller.go:253] package variant "cachingdns-edge-2795d42dbf61192c4c66795f7f35fe2589106ad7" is creating package revision ""

This is from the line:

        klog.Infoln(fmt.Sprintf("package variant %q is creating package revision %q", pv.Name, newPR.Name))

So, @natasha41575, we are not setting newPR.Name when contructing the PackageRevision. Is this intentional? Is the idea that Porch will create a name for us? That's fine - but for some reason, just with this package, even though there doesn't seem to be an error, we keep creating new PRs.

@johnbelamaric
Copy link
Contributor Author

johnbelamaric commented Mar 14, 2023 •
edited
Loading

Ok, so this may be part of the issue. The PackageRevision is not created with the ownerRef. Look at these two PRs, one is for "edge-02" and one is not:

jbelamaric@jbelamaric:~/proj/gh/johnbelamaric/kpt/porch/controllers$ k get packagerevisions us-central1-edge-01-fc057482816768663fef81c4e5252856a632751d us-central1-edge-02-0359e96849f5e498f5bf529bfa799f027596167b -o yaml
apiVersion: v1
items:
- apiVersion: porch.kpt.dev/v1alpha1
  kind: PackageRevision
  metadata:
    creationTimestamp: "2023-03-14T18:47:28Z"
    name: us-central1-edge-01-fc057482816768663fef81c4e5252856a632751d
    namespace: default
    ownerReferences:
    - apiVersion: config.porch.kpt.dev/v1alpha1
      controller: true
      kind: PackageVariant
      name: cachingdns-edge-596ce7b3b682f6eb284d7c1aec74aaaf179ad834
      uid: cb41117f-ae87-4b5f-8606-bc7e7552bbb9
    resourceVersion: 30952d6417924df47c6aa9f914062f998ec7cd4b
    uid: uid:cachingdns-edge:packagevariant-1
  spec:
    lifecycle: Draft
    packageName: cachingdns-edge
    repository: us-central1-edge-01
    tasks:
    - clone:
        upstreamRef:
          upstreamRef:
            name: nephio-testorg-packages-4ad103ac302c07b54dd709719f6ef500d64424d9
      type: clone
    - eval:
        config: null
        image: render
        match: {}
      type: eval
    workspaceName: packagevariant-1
  status:
    deployment: true
    publishTimestamp: null
    upstreamLock:
      git:
        commit: 3c8d8c0fb2081375ac31c01db07563dff00dc190
        directory: cachingdns-edge
        ref: cachingdns-edge/v2
        repo: https://github.com/johnbelamaric/nephio-testorg-packages.git
      type: git
- apiVersion: porch.kpt.dev/v1alpha1
  kind: PackageRevision
  metadata:
    creationTimestamp: "2023-03-14T19:23:39Z"
    name: us-central1-edge-02-0359e96849f5e498f5bf529bfa799f027596167b
    namespace: default
    resourceVersion: 044fd1e1c1478e47d82c147ee0ebf367b1ee8ea7
    uid: uid:cachingdns-edge:packagevariant-237
  spec:
    lifecycle: Draft
    packageName: cachingdns-edge
    repository: us-central1-edge-02
    tasks:
    - clone:
        upstreamRef:
          upstreamRef:
            name: nephio-testorg-packages-4ad103ac302c07b54dd709719f6ef500d64424d9
      type: clone
    - eval:
        config: null
        image: render
        match: {}
      type: eval
    workspaceName: packagevariant-237
  status:
    deployment: true
    publishTimestamp: null
    upstreamLock:
      git:
        commit: 3c8d8c0fb2081375ac31c01db07563dff00dc190
        directory: cachingdns-edge
        ref: cachingdns-edge/v2
        repo: https://github.com/johnbelamaric/nephio-testorg-packages.git
      type: git
kind: List
metadata:
  resourceVersion: ""
jbelamaric@jbelamaric:~/proj/gh/johnbelamaric/kpt/porch/controllers$

The one that is working has the ownerRef set, the other doesn't. I have no idea how that is happening, though, since it is set here:

https://github.com/GoogleContainerTools/kpt/blob/833194ed14b5a9cd31bc6ccab50a6c7a5e809100/porch/controllers/packagevariants/pkg/controllers/packagevariant/packagevariant_controller.go#L230

@johnbelamaric
Copy link
Contributor Author

Actually, one of the few dozen entries for edge-02 does have an ownerRef. Huh.

edge-02.txt

@johnbelamaric
Copy link
Contributor Author

Hmm. That's the most recent one.

@natasha41575 natasha41575 added the triaged Issue has been triaged by adding an `area/` label label Mar 14, 2023
@natasha41575
Copy link
Contributor

natasha41575 commented Mar 14, 2023 •
edited
Loading

I'm trying to understand why this is happening just with us-central1-edge-02. Is there anything obviously different about that one vs all your other ones?

Also, could you share what your PackageVariantSets look like? I'm wondering if it could possibly be creating multiple PackageVariants that are causing strange interactions with each other. In any case it will help me repro this. I assume the repo you are using is https://github.com/johnbelamaric/nephio-testorg-deploy, is that right?

@johnbelamaric
Copy link
Contributor Author

I don't think so. I am confused as well.

@johnbelamaric
Copy link
Contributor Author

No on that repo, because of the other bug. I am using http://github.com/nephio-test

@johnbelamaric
Copy link
Contributor Author

I am kpt live apply'ing this package: https://github.com/johnbelamaric/nephio-testorg-packages/tree/main/cachingdns-topology

Repos are setup like this the yamls here, using the repos.sh script in there (see, they no longer point to the directories): https://github.com/johnbelamaric/nephio-testorg-deploy

@johnbelamaric
Copy link
Contributor Author

Oddly I am not seeing the same behavior when I run the controller locally.

@johnbelamaric
Copy link
Contributor Author

The other weird thing as I mentioned before is that the UI does not show the bogus package revisions, only the CLI and of course looking directly at the repo.

@johnbelamaric
Copy link
Contributor Author

So, after restarting the porch-controllers and the porch-server having restarted itself:

  • I haven't seen any references to the old, deleted nephio-testorg-deploy repository
  • I haven't seen any new bogus packages be created

This makes me think that somehow the porch-server state got confused due to all the previous errors, and perhaps this is the source of the issue. I still can't explain it. But I will watch to see if there are more issues.

@natasha41575
Copy link
Contributor

So as I said offline, since this is (hopefully) an uncommon issue that will be difficult to repro, I think we can put it in the backlog for now. It's possible that porch was confused by all the previous errors, but I'm still not sure how that would affect the PackageVariant's ownerrefs and detection of existing packages. If we see any similar issues, we should investigate further.

This was referenced Apr 8, 2024
Closed
Closed
This was referenced Apr 23, 2024
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/porch bug Something isn't working triaged Issue has been triaged by adding an `area/` label
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@droot @johnbelamaric @natasha41575