Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Install Plan creation with permissions #3108

Closed
stevekuznetsov opened this issue Nov 13, 2023 · 7 comments · Fixed by #3129
Closed

Install Plan creation with permissions #3108

stevekuznetsov opened this issue Nov 13, 2023 · 7 comments · Fixed by #3129
Assignees
@tmshort
Labels
kind/flake Categorizes issue or PR as related to a flaky test.

Comments

@stevekuznetsov
Copy link
Member 

 Install Plan creation with permissions
/go/src/github.com/openshift/operator-framework-olm/staging/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go:2710
  created the install-plan-e2e-vg886 testing namespace
  Creating catalog source nginx-cataloglrblz in namespace install-plan-e2e-vg886...
  Catalog source nginx-cataloglrblz created
waiting 142.077223ms for catalog pod install-plan-e2e-vg886/nginx-cataloglrblz to be available (for sync) - NO_CONNECTION
waiting 98.120985ms for catalog pod install-plan-e2e-vg886/nginx-cataloglrblz to be available (for sync) - TRANSIENT_FAILURE
waiting 15.351299079s for catalog pod install-plan-e2e-vg886/nginx-cataloglrblz to be available (for sync) - READY
probing catalog nginx-cataloglrblz pod with address nginx-cataloglrblz.install-plan-e2e-vg886.svc:50051
skipping health check
  17:50:18.6422: subscription install-plan-e2e-vg886/sub-nginx-lkkkz state: UpgradePending (csv nginx9ld7b-stable): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"install-plan-e2e-vg886", Name:"install-ccrcg", UID:"df84614d-2e1f-47f7-acb5-9c3c15cff097", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"156380", FieldPath:""}
  waiting 803.038045ms for subscription install-plan-e2e-vg886/sub-nginx-lkkkz to have installplan ref: has ref &v1.ObjectReference{Kind:"InstallPlan", Namespace:"install-plan-e2e-vg886", Name:"install-ccrcg", UID:"df84614d-2e1f-47f7-acb5-9c3c15cff097", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"156380", FieldPath:""}
  waiting 197.898463ms for installplan install-plan-e2e-vg886/install-ccrcg to be phases [Failed Complete], in phase Installing
  waiting 4.196911018s for installplan install-plan-e2e-vg886/install-ccrcg to be phases [Failed Complete], in phase Complete
  {nginx9ld7b-stable ClusterServiceVersion }, {nginx9ld7b-stable ClusterRoleBinding }: true && false
  {nginx9ld7b-stable ClusterServiceVersion }, {insqcpnr.cluster.com CustomResourceDefinition }: false && false
  {nginx9ld7b-stable ClusterServiceVersion }, {nginx-sabgxrp ServiceAccount }: false && false
  {nginx9ld7b-stable ClusterServiceVersion }, {nginx9ld7b-stable Role }: true && false
  {nginx9ld7b-stable ClusterServiceVersion }, {nginx9ld7b-stable RoleBinding }: true && false
  {nginx9ld7b-stable ClusterServiceVersion }, {nginx9ld7b-stable ClusterRole }: true && false
  {insqcpnr.cluster.com CustomResourceDefinition }, {nginx9ld7b-stable ClusterRoleBinding }: false && false
  {insqcpnr.cluster.com CustomResourceDefinition }, {nginx-sabgxrp ServiceAccount }: false && false
  {insqcpnr.cluster.com CustomResourceDefinition }, {nginx9ld7b-stable Role }: false && false
  {insqcpnr.cluster.com CustomResourceDefinition }, {nginx9ld7b-stable RoleBinding }: false && false
  {insqcpnr.cluster.com CustomResourceDefinition }, {nginx9ld7b-stable ClusterRole }: false && false
  {nginx-sabgxrp ServiceAccount }, {nginx9ld7b-stable Role }: false && false
  {nginx-sabgxrp ServiceAccount }, {nginx9ld7b-stable RoleBinding }: false && false
  {nginx-sabgxrp ServiceAccount }, {nginx9ld7b-stable ClusterRole }: false && false
  {nginx-sabgxrp ServiceAccount }, {nginx9ld7b-stable ClusterRoleBinding }: false && false
  {nginx9ld7b-stable-nginx-sabgxrp-7c86848fc4 Role }, {nginx9ld7b-stable RoleBinding }: true && false
  {nginx9ld7b-stable-nginx-sabgxrp-7c86848fc4 Role }, {nginx9ld7b-stable ClusterRole }: true && false
  {nginx9ld7b-stable-nginx-sabgxrp-7c86848fc4 Role }, {nginx9ld7b-stable ClusterRoleBinding }: true && false
  {nginx9ld7b-stable-nginx-sabgxrp-7c86848fc4 RoleBinding }, {nginx9ld7b-stable ClusterRole }: true && false
  {nginx9ld7b-stable-nginx-sabgxrp-7c86848fc4 RoleBinding }, {nginx9ld7b-stable ClusterRoleBinding }: true && false
  {nginx9ld7b-stable-747c79bd4f ClusterRole }, {nginx9ld7b-stable ClusterRoleBinding }: true && false
  deleting csv install-plan-e2e-vg886/nginx9ld7b-stable
  Deleting config map nginx-cataloglrblz-configmap...
  Deleting catalog source nginx-cataloglrblz...
  waiting for the catalog source nginx-cataloglrblz-d9qpx pod to be deleted...
  [FAILED] in [It] - /go/src/github.com/openshift/operator-framework-olm/staging/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go:2905 @ 11/13/23 18:00:24.687
  collecting the install-plan-e2e-vg886 namespace artifacts as the 'creation with permissions' test case failed
  collecting logs in the /logs/artifacts artifacts directory
  unable to determine kubeconfig path so defaulting to the $KUBECONFIG value
Using the oc kubectl binary
Using the /logs/artifacts/install-plan-e2e-vg886 output directory
Storing the test artifact output in the /logs/artifacts/install-plan-e2e-vg886 directory
Collecting get catalogsources -o yaml output...
Collecting get subscriptions -o yaml output...
Collecting get operatorgroups -o yaml output...
Collecting get clusterserviceversions -o yaml output...
Collecting get installplans -o yaml output...
Collecting get pods -o wide output...
Collecting get events --sort-by .lastTimestamp output...
  tearing down the install-plan-e2e-vg886 namespace
  resetting e2e kube client
  deleting install-plan-e2e-vg886/install-plan-e2e-vg886-operatorgroup
  deleting <global>/install-plan-e2e-vg886
  garbage collecting CRDs
• [FAILED] [627.154 seconds]
Install Plan [It] creation with permissions
/go/src/github.com/openshift/operator-framework-olm/staging/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go:2710
  [FAILED] Timed out after 600.020s.
  Expected
      <bool>: false
  to be true
  In [It] at: /go/src/github.com/openshift/operator-framework-olm/staging/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go:2905 @ 11/13/23 18:00:24.687
  Full Stack Trace
    github.com/operator-framework/operator-lifecycle-manager/test/e2e.glob..func17.13()
    	/go/src/github.com/openshift/operator-framework-olm/staging/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go:2905 +0x33ca

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_operator-framework-olm/606/pull-ci-openshift-operator-framework-olm-master-e2e-gcp-olm/1724082802661527552
@stevekuznetsov stevekuznetsov added the kind/flake Categorizes issue or PR as related to a flaky test. label Nov 13, 2023
This was referenced Nov 13, 2023
Merged
Merged
@stevekuznetsov
Copy link
Member Author 

   created the install-plan-e2e-mr75q testing namespace
  STEP: Create manifests @ 11/13/23 20:20:24.998
  STEP: Create new CRDs @ 11/13/23 20:20:24.998
  STEP: Defer CRD clean up @ 11/13/23 20:20:24.998
  STEP: Generate permissions @ 11/13/23 20:20:24.998
  STEP: Generate permissions @ 11/13/23 20:20:24.998
  STEP: Create a new NamedInstallStrategy @ 11/13/23 20:20:24.998
  STEP: Create new CSVs @ 11/13/23 20:20:24.998
  STEP: Create CatalogSource @ 11/13/23 20:20:24.998
  Creating catalog source nginx-cataloghn47q in namespace install-plan-e2e-mr75q...
  Catalog source nginx-cataloghn47q created
  STEP: Attempt to get CatalogSource @ 11/13/23 20:20:25.006
  20:20:46.0519: subscription install-plan-e2e-mr75q/sub-nginx-sfpb8 state: UpgradePending (csv nginxm4gp9-stable): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"install-plan-e2e-mr75q", Name:"install-ldtl8", UID:"aaddfbf0-bdf0-47d8-94fe-6e8d456781fe", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"4876", FieldPath:""}
  waiting 1.397875896s for subscription install-plan-e2e-mr75q/sub-nginx-sfpb8 to have installplan ref: has ref &v1.ObjectReference{Kind:"InstallPlan", Namespace:"install-plan-e2e-mr75q", Name:"install-ldtl8", UID:"aaddfbf0-bdf0-47d8-94fe-6e8d456781fe", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"4876", FieldPath:""}
  STEP: Attempt to get InstallPlan @ 11/13/23 20:20:46.052
  waiting 200.799354ms for installplan install-plan-e2e-mr75q/install-ldtl8 to be phases [Failed Complete], in phase Installing
  waiting 3.999452781s for installplan install-plan-e2e-mr75q/install-ldtl8 to be phases [Failed Complete], in phase Complete
  STEP: Expect correct RBAC resources to be resolved and created @ 11/13/23 20:20:50.252
  Found unexpected step registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterServiceVersion", Namespace:""}, expected registry.ResourceKey{Name:"insg4xdq.cluster.com", Kind:"CustomResourceDefinition", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterServiceVersion", Namespace:""}, expected registry.ResourceKey{Name:"nginx-satvq9d", Kind:"ServiceAccount", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterServiceVersion", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"Role", Namespace:""}: name has prefix: true kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterServiceVersion", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"RoleBinding", Namespace:""}: name has prefix: true kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterServiceVersion", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterRole", Namespace:""}: name has prefix: true kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterServiceVersion", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterRoleBinding", Namespace:""}: name has prefix: true kinds match false
  STEP: This operator was installed into a global operator group, so the roles should have been lifted to clusterroles @ 11/13/23 20:20:50.252
  Found unexpected step registry.ResourceKey{Name:"insg4xdq.cluster.com", Kind:"CustomResourceDefinition", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterRole", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"insg4xdq.cluster.com", Kind:"CustomResourceDefinition", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterRoleBinding", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"insg4xdq.cluster.com", Kind:"CustomResourceDefinition", Namespace:""}, expected registry.ResourceKey{Name:"nginx-satvq9d", Kind:"ServiceAccount", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"insg4xdq.cluster.com", Kind:"CustomResourceDefinition", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"Role", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"insg4xdq.cluster.com", Kind:"CustomResourceDefinition", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"RoleBinding", Namespace:""}: name has prefix: false kinds match false
  STEP: This operator was installed into a global operator group, so the roles should have been lifted to clusterroles @ 11/13/23 20:20:50.252
  Found unexpected step registry.ResourceKey{Name:"nginx-satvq9d", Kind:"ServiceAccount", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterRole", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginx-satvq9d", Kind:"ServiceAccount", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterRoleBinding", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginx-satvq9d", Kind:"ServiceAccount", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"Role", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginx-satvq9d", Kind:"ServiceAccount", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"RoleBinding", Namespace:""}: name has prefix: false kinds match false
  STEP: This operator was installed into a global operator group, so the roles should have been lifted to clusterroles @ 11/13/23 20:20:50.252
  Found unexpected step registry.ResourceKey{Name:"nginxm4gp9-stable-nginx-satvq9d-787b844474", Kind:"Role", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterRole", Namespace:""}: name has prefix: true kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginxm4gp9-stable-nginx-satvq9d-787b844474", Kind:"Role", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterRoleBinding", Namespace:""}: name has prefix: true kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginxm4gp9-stable-nginx-satvq9d-787b844474", Kind:"Role", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"RoleBinding", Namespace:""}: name has prefix: true kinds match false
  STEP: This operator was installed into a global operator group, so the roles should have been lifted to clusterroles @ 11/13/23 20:20:50.252
  Found unexpected step registry.ResourceKey{Name:"nginxm4gp9-stable-nginx-satvq9d-787b844474", Kind:"RoleBinding", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterRole", Namespace:""}: name has prefix: true kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginxm4gp9-stable-nginx-satvq9d-787b844474", Kind:"RoleBinding", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterRoleBinding", Namespace:""}: name has prefix: true kinds match false
  STEP: This operator was installed into a global operator group, so the roles should have been lifted to clusterroles @ 11/13/23 20:20:50.355
  Found unexpected step registry.ResourceKey{Name:"nginxm4gp9-stable-b965c788b", Kind:"ClusterRole", Namespace:""}, expected registry.ResourceKey{Name:"nginxm4gp9-stable", Kind:"ClusterRoleBinding", Namespace:""}: name has prefix: true kinds match false
  STEP: This operator was installed into a global operator group, so the roles should have been lifted to clusterroles @ 11/13/23 20:20:50.458
  STEP: This operator was installed into a global operator group, so the roles should have been lifted to clusterroles @ 11/13/23 20:20:50.458
  STEP: Should have removed every matching step @ 11/13/23 20:20:50.458
  deleting csv install-plan-e2e-mr75q/nginxm4gp9-stable
  STEP: Explicitly delete the CSV @ 11/13/23 20:20:50.458
  STEP: Looking for no error OR IsNotFound error @ 11/13/23 20:20:50.462
  Deleting config map nginx-cataloghn47q-configmap...
  Deleting catalog source nginx-cataloghn47q...
  waiting for the catalog source nginx-cataloghn47q-vvdvt pod to be deleted...
  [FAILED] in [It] - /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go:2905 @ 11/13/23 20:30:51.579
  collecting the install-plan-e2e-mr75q namespace artifacts as the 'creation with permissions' test case failed
  collecting logs in the ./artifacts/ artifacts directory
  tearing down the install-plan-e2e-mr75q namespace
  resetting e2e kube client
  deleting install-plan-e2e-mr75q/install-plan-e2e-mr75q-operatorgroup
  deleting <global>/install-plan-e2e-mr75q
  garbage collecting CRDs
  << Timeline

  [FAILED] Timed out after 600.093s.
  Expected
      <bool>: false
  to be true
  In [It] at: /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go:2905 @ 11/13/23 20:30:51.579

  Full Stack Trace
    github.com/operator-framework/operator-lifecycle-manager/test/e2e.glob..func16.13()
    	/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go:2905 +0x2009

@stevekuznetsov
Copy link
Member Author 

 • [FAILED] [661.526 seconds]
Install Plan [It] creation with permissions
/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go:2710

  Captured StdOut/StdErr Output >>
  waiting 102.458009ms for catalog pod install-plan-e2e-nfd27/nginx-catalogcjc5j to be available (for sync) - NO_CONNECTION
  waiting 1.937648274s for catalog pod install-plan-e2e-nfd27/nginx-catalogcjc5j to be available (for sync) - IDLE
  waiting 198.193479ms for catalog pod install-plan-e2e-nfd27/nginx-catalogcjc5j to be available (for sync) - TRANSIENT_FAILURE
  waiting 1.000173811s for catalog pod install-plan-e2e-nfd27/nginx-catalogcjc5j to be available (for sync) - CONNECTING
  waiting 19.800086663s for catalog pod install-plan-e2e-nfd27/nginx-catalogcjc5j to be available (for sync) - TRANSIENT_FAILURE
  waiting 1.799857267s for catalog pod install-plan-e2e-nfd27/nginx-catalogcjc5j to be available (for sync) - CONNECTING
  waiting 19.999429058s for catalog pod install-plan-e2e-nfd27/nginx-catalogcjc5j to be available (for sync) - TRANSIENT_FAILURE
  waiting 2.601147351s for catalog pod install-plan-e2e-nfd27/nginx-catalogcjc5j to be available (for sync) - CONNECTING
  waiting 199.246548ms for catalog pod install-plan-e2e-nfd27/nginx-catalogcjc5j to be available (for sync) - READY
  probing catalog nginx-catalogcjc5j pod with address nginx-catalogcjc5j.install-plan-e2e-nfd27.svc:50051
  skipping health check
  Using the kubectl kubectl binary
  Using the artifacts/install-plan-e2e-nfd27 output directory
  Storing the test artifact output in the artifacts/install-plan-e2e-nfd27 directory
  Collecting get catalogsources -o yaml output...
  Collecting get subscriptions -o yaml output...
  Collecting get operatorgroups -o yaml output...
  Collecting get clusterserviceversions -o yaml output...
  Collecting get installplans -o yaml output...
  Collecting get pods -o wide output...
  Collecting get events --sort-by .lastTimestamp output...
  << Captured StdOut/StdErr Output

  Timeline >>
  created the install-plan-e2e-nfd27 testing namespace
  created the install-plan-e2e-nfd27/install-plan-e2e-nfd27-operatorgroup operator group
  STEP: Create manifests @ 12/01/23 15:57:28.984
  STEP: Create new CRDs @ 12/01/23 15:57:28.984
  STEP: Defer CRD clean up @ 12/01/23 15:57:28.984
  STEP: Generate permissions @ 12/01/23 15:57:28.984
  STEP: Generate permissions @ 12/01/23 15:57:28.984
  STEP: Create a new NamedInstallStrategy @ 12/01/23 15:57:28.984
  STEP: Create new CSVs @ 12/01/23 15:57:28.984
  STEP: Create CatalogSource @ 12/01/23 15:57:28.984
  Creating catalog source nginx-catalogcjc5j in namespace install-plan-e2e-nfd27...
  Catalog source nginx-catalogcjc5j created
  STEP: Attempt to get CatalogSource @ 12/01/23 15:57:28.992
  15:58:18.2332: subscription install-plan-e2e-nfd27/sub-nginx-bp2pf state: UpgradePending (csv nginxvxs8b-stable): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"install-plan-e2e-nfd27", Name:"install-ltzzp", UID:"5c57b75d-4981-44d0-ae1c-711b63ee6624", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"4819", FieldPath:""}
  waiting 1.400314311s for subscription install-plan-e2e-nfd27/sub-nginx-bp2pf to have installplan ref: has ref &v1.ObjectReference{Kind:"InstallPlan", Namespace:"install-plan-e2e-nfd27", Name:"install-ltzzp", UID:"5c57b75d-4981-44d0-ae1c-711b63ee6624", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"4819", FieldPath:""}
  STEP: Attempt to get InstallPlan @ 12/01/23 15:58:18.233
  waiting 198.94321ms for installplan install-plan-e2e-nfd27/install-ltzzp to be phases [Failed Complete], in phase Installing
  waiting 3.999229475s for installplan install-plan-e2e-nfd27/install-ltzzp to be phases [Failed Complete], in phase Complete
  STEP: Expect correct RBAC resources to be resolved and created @ 12/01/23 15:58:22.431
  Found unexpected step registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterServiceVersion", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"RoleBinding", Namespace:""}: name has prefix: true kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterServiceVersion", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterRole", Namespace:""}: name has prefix: true kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterServiceVersion", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterRoleBinding", Namespace:""}: name has prefix: true kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterServiceVersion", Namespace:""}, expected registry.ResourceKey{Name:"ins262hx.cluster.com", Kind:"CustomResourceDefinition", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterServiceVersion", Namespace:""}, expected registry.ResourceKey{Name:"nginx-sah9fvt", Kind:"ServiceAccount", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterServiceVersion", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"Role", Namespace:""}: name has prefix: true kinds match false
  STEP: This operator was installed into a global operator group, so the roles should have been lifted to clusterroles @ 12/01/23 15:58:22.432
  Found unexpected step registry.ResourceKey{Name:"ins262hx.cluster.com", Kind:"CustomResourceDefinition", Namespace:""}, expected registry.ResourceKey{Name:"nginx-sah9fvt", Kind:"ServiceAccount", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"ins262hx.cluster.com", Kind:"CustomResourceDefinition", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"Role", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"ins262hx.cluster.com", Kind:"CustomResourceDefinition", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"RoleBinding", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"ins262hx.cluster.com", Kind:"CustomResourceDefinition", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterRole", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"ins262hx.cluster.com", Kind:"CustomResourceDefinition", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterRoleBinding", Namespace:""}: name has prefix: false kinds match false
  STEP: This operator was installed into a global operator group, so the roles should have been lifted to clusterroles @ 12/01/23 15:58:22.432
  Found unexpected step registry.ResourceKey{Name:"nginx-sah9fvt", Kind:"ServiceAccount", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterRole", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginx-sah9fvt", Kind:"ServiceAccount", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterRoleBinding", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginx-sah9fvt", Kind:"ServiceAccount", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"Role", Namespace:""}: name has prefix: false kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginx-sah9fvt", Kind:"ServiceAccount", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"RoleBinding", Namespace:""}: name has prefix: false kinds match false
  STEP: This operator was installed into a global operator group, so the roles should have been lifted to clusterroles @ 12/01/23 15:58:22.432
  Found unexpected step registry.ResourceKey{Name:"nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy", Kind:"Role", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"RoleBinding", Namespace:""}: name has prefix: true kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy", Kind:"Role", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterRole", Namespace:""}: name has prefix: true kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy", Kind:"Role", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterRoleBinding", Namespace:""}: name has prefix: true kinds match false
  STEP: This operator was installed into a global operator group, so the roles should have been lifted to clusterroles @ 12/01/23 15:58:22.432
  Found unexpected step registry.ResourceKey{Name:"nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy", Kind:"RoleBinding", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterRole", Namespace:""}: name has prefix: true kinds match false
  Found unexpected step registry.ResourceKey{Name:"nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy", Kind:"RoleBinding", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterRoleBinding", Namespace:""}: name has prefix: true kinds match false
  STEP: This operator was installed into a global operator group, so the roles should have been lifted to clusterroles @ 12/01/23 15:58:22.535
  Found unexpected step registry.ResourceKey{Name:"nginxvxs8b-stable-6X2c5zjdqVzHMCd0XvoNPFuGw7nzI6mlyNeNEN", Kind:"ClusterRole", Namespace:""}, expected registry.ResourceKey{Name:"nginxvxs8b-stable", Kind:"ClusterRoleBinding", Namespace:""}: name has prefix: true kinds match false
  STEP: This operator was installed into a global operator group, so the roles should have been lifted to clusterroles @ 12/01/23 15:58:22.639
  STEP: This operator was installed into a global operator group, so the roles should have been lifted to clusterroles @ 12/01/23 15:58:22.639
  STEP: Should have removed every matching step @ 12/01/23 15:58:22.639
  deleting csv install-plan-e2e-nfd27/nginxvxs8b-stable
  STEP: Explicitly delete the CSV @ 12/01/23 15:58:22.64
  STEP: Looking for no error OR IsNotFound error @ 12/01/23 15:58:22.651
  STEP: eventually seeing all ClusterRoles, ClusterRoleBindings, and ServiceAccounts deleted @ 12/01/23 15:58:22.651
  found 2 ClusterRoleBindings: nginxvxs8b-stable-6X2c5zjdqVzHMCd0XvoNPFuGw7nzI6mlyNeNEN, nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy
  found 2 ClusterRoleBindings: nginxvxs8b-stable-6X2c5zjdqVzHMCd0XvoNPFuGw7nzI6mlyNeNEN, nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy
  found 2 ClusterRoleBindings: nginxvxs8b-stable-6X2c5zjdqVzHMCd0XvoNPFuGw7nzI6mlyNeNEN, nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy
  found no ClusterRoleBindings
  found 2 ClusterRoles: nginxvxs8b-stable-6X2c5zjdqVzHMCd0XvoNPFuGw7nzI6mlyNeNEN, nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy
  found no ClusterRoleBindings
  found 2 ClusterRoles: nginxvxs8b-stable-6X2c5zjdqVzHMCd0XvoNPFuGw7nzI6mlyNeNEN, nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy
  found no ClusterRoleBindings
  found 2 ClusterRoles: nginxvxs8b-stable-6X2c5zjdqVzHMCd0XvoNPFuGw7nzI6mlyNeNEN, nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy
  found no ClusterRoleBindings
  found 2 ClusterRoles: nginxvxs8b-stable-6X2c5zjdqVzHMCd0XvoNPFuGw7nzI6mlyNeNEN, nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy
  found no ClusterRoleBindings
  found 2 ClusterRoles: nginxvxs8b-stable-6X2c5zjdqVzHMCd0XvoNPFuGw7nzI6mlyNeNEN, nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy
  found no ClusterRoleBindings
  found 2 ClusterRoles: nginxvxs8b-stable-6X2c5zjdqVzHMCd0XvoNPFuGw7nzI6mlyNeNEN, nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy
  found no ClusterRoleBindings
  found 2 ClusterRoles: nginxvxs8b-stable-6X2c5zjdqVzHMCd0XvoNPFuGw7nzI6mlyNeNEN, nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy
  found 2 ClusterRoleBindings: nginxvxs8b-stable-6X2c5zjdqVzHMCd0XvoNPFuGw7nzI6mlyNeNEN, nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy
// ...
   found 2 ClusterRoleBindings: nginxvxs8b-stable-6X2c5zjdqVzHMCd0XvoNPFuGw7nzI6mlyNeNEN, nginxvxs8b-stable-nginx--5aAa1Tz0X96o93paaNjb9RfECGJOFaheZgi5dy
  Deleting config map nginx-catalogcjc5j-configmap...
  Deleting catalog source nginx-catalogcjc5j...
  waiting for the catalog source nginx-catalogcjc5j-jngn8 pod to be deleted...
  [FAILED] in [It] - /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go:2926 @ 12/01/23 16:08:23.761
  collecting the install-plan-e2e-nfd27 namespace artifacts as the 'creation with permissions' test case failed
  collecting logs in the ./artifacts/ artifacts directory
  tearing down the install-plan-e2e-nfd27 namespace
  resetting e2e kube client
  deleting install-plan-e2e-nfd27/install-plan-e2e-nfd27-operatorgroup
  deleting <global>/install-plan-e2e-nfd27
  garbage collecting CRDs
  << Timeline

  [FAILED] Timed out after 600.085s.
  Expected
      <bool>: false
  to be true
  In [It] at: /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go:2926 @ 12/01/23 16:08:23.761

  Full Stack Trace
    github.com/operator-framework/operator-lifecycle-manager/test/e2e.glob..func16.13()
    	/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/installplan_e2e_test.go:2926 +0x2026

@stevekuznetsov
Copy link
Member Author

the fact that the CRBs don't exist, then do ... makes me wonder if the delete actually worked

@stevekuznetsov
Copy link
Member Author

I don't see any mention of finalizers in the codebase so I honestly can't tell how they expected this cleanup to work ... likely a fix here is to add a finalizer to the CSV to ensure we clean up when it's deleted ... ?

@stevekuznetsov
Copy link
Member Author

This is the code that does this cleanup but it doesn't seem like there's a finalizer in play at all - so we're really not guaranteed to have cleanup happen and it's not surprising that we flake out. Looks like the fix here is just to add one.

@stevekuznetsov
Copy link
Member Author

stevekuznetsov commented Dec 1, 2023 •
edited
Loading

Unfortunately this might be a tricky one to implement since the queueinformer library doesn't actually queue keys for events, it calls handlers in an edge-driven way. At the minimum we won't be putting the logic for handling deletions in the ToSyncerWithDelete() call ... since the object won't be deleted until we remove the finalizer. We will need to handle it in the main sync loop and key off of !csv.ObjectMeta.DeletionTimestamp.IsZero()

@tmshort tmshort self-assigned this Dec 7, 2023
@tmshort
Copy link
Contributor

tmshort commented Dec 7, 2023

As stated, the delete does not work; it seems that unless the CSV is in a (e.g.) success state, it will not be deleted, so that will have to be looked at. By waiting until the CSV is in a success state, and then deleting, the test passes.

tmshort added a commit to tmshort/operator-lifecycle-manager that referenced this issue Dec 7, 2023
@tmshort
Fix e2e Install Plan creation with permission
d2442e6 
Fix operator-framework#3108

Attempting to delete a CSV that is not in a terminal (e.g. Success) state
fails silently (i.e. the CSV is not deleted). So, wait until the CSV is
in the success state before deleting.

Add logging.

Signed-off-by: Todd Short <todd.short@me.com>
This was referenced Dec 7, 2023
Merged
Closed
tmshort added a commit to tmshort/operator-lifecycle-manager that referenced this issue Jan 2, 2024
@tmshort
Fix Create InstallPlan with Permissions test
1a29a18 
Fix operator-framework#3108
The Subscription needs to be deleted before deleting the CSV, otherwise
the Subscription will recreate the CSV, and subsequently, the CR/CRBs
are not deleted.

Update some test logging as well.

Signed-off-by: Todd Short <todd.short@me.com>
@tmshort tmshort mentioned this issue Jan 3, 2024
Closed
tmshort added a commit to tmshort/operator-lifecycle-manager that referenced this issue Jan 3, 2024
@tmshort
Fix Create InstallPlan with Permissions test
0fc56ce 
Fix operator-framework#3108
The Subscription needs to be deleted before deleting the CSV, otherwise
the Subscription will recreate the CSV, and subsequently, the CR/CRBs
are not deleted.

Update some test logging as well.

Signed-off-by: Todd Short <todd.short@me.com>
github-merge-queue bot pushed a commit that referenced this issue Jan 3, 2024
@tmshort
Fix Create InstallPlan with Permissions test (#3129)
8215c7f 
Fix #3108
The Subscription needs to be deleted before deleting the CSV, otherwise
the Subscription will recreate the CSV, and subsequently, the CR/CRBs
are not deleted.

Update some test logging as well.

Signed-off-by: Todd Short <todd.short@me.com>
@tmshort tmshort mentioned this issue Jan 11, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tmshort tmshort

Labels
kind/flake Categorizes issue or PR as related to a flaky test.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix e2e Install Plan creation with permission tmshort/operator-lifecycle-manager
2 participants
@stevekuznetsov @tmshort