Skip to content

Conversation

@porridge
Copy link
Collaborator

This should help us establish direction for investigation of ROX-8049, since right now it is not clear whether the cause for delay is in negative caching, in operator framework or in the helm operator code.

It adds some noise but it's not too bad:

Comparison (blank lines mine for clarity):

Before

2021-09-24T09:08:43.373+0200	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2021-09-24T09:08:43.373+0200	INFO	setup	skipping webhook setup, ENABLE_WEBHOOKS==false
2021-09-24T09:08:43.376+0200	INFO	controllers.Helm	Watching resource	{"group": "platform.stackrox.io", "version": "v1alpha1", "kind": "Central"}
2021-09-24T09:08:43.380+0200	INFO	controllers.Helm	Watching resource	{"group": "platform.stackrox.io", "version": "v1alpha1", "kind": "SecuredCluster"}
2021-09-24T09:08:43.380+0200	INFO	setup	starting manager
2021-09-24T09:08:43.380+0200	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2021-09-24T09:08:43.380+0200	INFO	controller-runtime.manager.controller.securedcluster-controller	Starting EventSource	{"source": "kind source: platform.stackrox.io/v1alpha1, Kind=SecuredCluster"}
2021-09-24T09:08:43.380+0200	INFO	controller-runtime.manager.controller.central-controller	Starting EventSource	{"source": "kind source: platform.stackrox.io/v1alpha1, Kind=Central"}
2021-09-24T09:08:43.485+0200	INFO	controller-runtime.manager.controller.securedcluster-controller	Starting EventSource	{"source": "kind source: /v1, Kind=Secret"}
2021-09-24T09:08:43.485+0200	INFO	controller-runtime.manager.controller.central-controller	Starting EventSource	{"source": "kind source: /v1, Kind=Secret"}
2021-09-24T09:08:43.586+0200	INFO	controller-runtime.manager.controller.central-controller	Starting Controller
2021-09-24T09:08:43.587+0200	INFO	controller-runtime.manager.controller.securedcluster-controller	Starting Controller
2021-09-24T09:08:43.587+0200	INFO	controller-runtime.manager.controller.securedcluster-controller	Starting workers	{"worker count": 1}
2021-09-24T09:08:43.587+0200	INFO	controller-runtime.manager.controller.central-controller	Starting workers	{"worker count": 1}

2021-09-24T09:09:23.577+0200	DEBUG	controllers.Helm	Starting install
I0924 09:09:24.636975   46578 request.go:655] Throttling request took 1.047072756s, request: GET:https://kubernetes.docker.internal:6443/apis/networking.k8s.io/v1?timeout=32s
W0924 09:09:24.932949   46578 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0924 09:09:24.935699   46578 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2021-09-24T09:09:25.057+0200	DEBUG	controllers.Helm	creating 22 resource(s)
W0924 09:09:25.077557   46578 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0924 09:09:25.077988   46578 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2021-09-24T09:09:25.260+0200	INFO	controllers.Helm	Release installed	{"central": "default/stackrox-central-services", "name": "stackrox-central-services", "version": 1}
2021-09-24T09:09:25.440+0200	DEBUG	controllers.Helm	preparing upgrade for stackrox-central-services
2021-09-24T09:09:26.755+0200	DEBUG	controllers.Helm	performing update for stackrox-central-services
2021-09-24T09:09:26.770+0200	DEBUG	controllers.Helm	dry run for stackrox-central-services
W0924 09:09:26.819614   46578 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0924 09:09:26.829351   46578 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0924 09:09:26.832278   46578 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0924 09:09:26.837478   46578 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2021-09-24T09:09:27.043+0200	INFO	controllers.Helm	Release reconciled	{"central": "default/stackrox-central-services", "name": "stackrox-central-services", "version": 1}

2021-09-24T09:10:01.130+0200	DEBUG	controllers.Helm	uninstall: Deleting stackrox-central-services
I0924 09:10:02.223889   46578 request.go:655] Throttling request took 1.041884775s, request: GET:https://kubernetes.docker.internal:6443/apis/authentication.k8s.io/v1?timeout=32s
2021-09-24T09:10:02.242+0200	DEBUG	controllers.Helm	Starting delete for "scanner-db" Service
2021-09-24T09:10:02.242+0200	DEBUG	controllers.Helm	Starting delete for "central" Service
2021-09-24T09:10:02.242+0200	DEBUG	controllers.Helm	Starting delete for "scanner" Service
2021-09-24T09:10:02.305+0200	DEBUG	controllers.Helm	Starting delete for "scanner-db" Deployment
2021-09-24T09:10:02.305+0200	DEBUG	controllers.Helm	Starting delete for "scanner" Deployment
2021-09-24T09:10:02.305+0200	DEBUG	controllers.Helm	Starting delete for "central" Deployment
2021-09-24T09:10:02.319+0200	DEBUG	controllers.Helm	Starting delete for "stackrox-central-psp" RoleBinding
2021-09-24T09:10:02.319+0200	DEBUG	controllers.Helm	Starting delete for "stackrox-central-diagnostics" RoleBinding
2021-09-24T09:10:02.319+0200	DEBUG	controllers.Helm	Starting delete for "stackrox-scanner-psp" RoleBinding
2021-09-24T09:10:02.331+0200	DEBUG	controllers.Helm	Starting delete for "stackrox-central-diagnostics" Role
2021-09-24T09:10:02.343+0200	DEBUG	controllers.Helm	Starting delete for "stackrox-default-central-psp" ClusterRole
2021-09-24T09:10:02.343+0200	DEBUG	controllers.Helm	Starting delete for "stackrox-default-scanner-psp" ClusterRole
2021-09-24T09:10:02.356+0200	DEBUG	controllers.Helm	Starting delete for "central-endpoints" ConfigMap
2021-09-24T09:10:02.356+0200	DEBUG	controllers.Helm	Starting delete for "scanner-config" ConfigMap
2021-09-24T09:10:02.356+0200	DEBUG	controllers.Helm	Starting delete for "central-config" ConfigMap
2021-09-24T09:10:02.371+0200	DEBUG	controllers.Helm	Starting delete for "central" ServiceAccount
2021-09-24T09:10:02.371+0200	DEBUG	controllers.Helm	Starting delete for "scanner" ServiceAccount
2021-09-24T09:10:02.383+0200	DEBUG	controllers.Helm	Starting delete for "stackrox-default-central" PodSecurityPolicy
2021-09-24T09:10:02.383+0200	DEBUG	controllers.Helm	Starting delete for "stackrox-default-scanner" PodSecurityPolicy
W0924 09:10:02.393961   46578 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0924 09:10:02.393987   46578 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2021-09-24T09:10:02.394+0200	DEBUG	controllers.Helm	Starting delete for "allow-ext-to-central" NetworkPolicy
2021-09-24T09:10:02.394+0200	DEBUG	controllers.Helm	Starting delete for "scanner" NetworkPolicy
2021-09-24T09:10:02.394+0200	DEBUG	controllers.Helm	Starting delete for "scanner-db" NetworkPolicy
2021-09-24T09:10:02.411+0200	DEBUG	controllers.Helm	purge requested for stackrox-central-services
2021-09-24T09:10:02.443+0200	INFO	controllers.Helm	Release uninstalled	{"central": "default/stackrox-central-services", "name": "stackrox-central-services", "version": 1}

After

2021-09-24T09:12:22.971+0200	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2021-09-24T09:12:22.971+0200	INFO	setup	skipping webhook setup, ENABLE_WEBHOOKS==false
2021-09-24T09:12:22.974+0200	INFO	controllers.Helm	Watching resource	{"group": "platform.stackrox.io", "version": "v1alpha1", "kind": "Central"}
2021-09-24T09:12:22.978+0200	INFO	controllers.Helm	Watching resource	{"group": "platform.stackrox.io", "version": "v1alpha1", "kind": "SecuredCluster"}
2021-09-24T09:12:22.978+0200	INFO	setup	starting manager
2021-09-24T09:12:22.979+0200	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2021-09-24T09:12:22.979+0200	INFO	controller-runtime.manager.controller.securedcluster-controller	Starting EventSource	{"source": "kind source: platform.stackrox.io/v1alpha1, Kind=SecuredCluster"}
2021-09-24T09:12:22.979+0200	INFO	controller-runtime.manager.controller.central-controller	Starting EventSource	{"source": "kind source: platform.stackrox.io/v1alpha1, Kind=Central"}
2021-09-24T09:12:23.083+0200	INFO	controller-runtime.manager.controller.securedcluster-controller	Starting EventSource	{"source": "kind source: /v1, Kind=Secret"}
2021-09-24T09:12:23.083+0200	INFO	controller-runtime.manager.controller.central-controller	Starting EventSource	{"source": "kind source: /v1, Kind=Secret"}
2021-09-24T09:12:23.183+0200	INFO	controller-runtime.manager.controller.securedcluster-controller	Starting Controller
2021-09-24T09:12:23.183+0200	INFO	controller-runtime.manager.controller.central-controller	Starting Controller
2021-09-24T09:12:23.183+0200	INFO	controller-runtime.manager.controller.securedcluster-controller	Starting workers	{"worker count": 1}
2021-09-24T09:12:23.183+0200	INFO	controller-runtime.manager.controller.central-controller	Starting workers	{"worker count": 1}

2021-09-24T09:12:33.643+0200	DEBUG	controllers.Helm	reconciling...	{"central": "default/stackrox-central-services"}
2021-09-24T09:12:35.069+0200	DEBUG	controllers.Helm	Starting install
I0924 09:12:36.124786   47834 request.go:655] Throttling request took 1.044414816s, request: GET:https://kubernetes.docker.internal:6443/apis/networking.k8s.io/v1?timeout=32s
W0924 09:12:36.354983   47834 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0924 09:12:36.358207   47834 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2021-09-24T09:12:36.440+0200	DEBUG	controllers.Helm	creating 22 resource(s)
W0924 09:12:36.454992   47834 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0924 09:12:36.454997   47834 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2021-09-24T09:12:36.611+0200	INFO	controllers.Helm	Release installed	{"central": "default/stackrox-central-services", "name": "stackrox-central-services", "version": 1}
2021-09-24T09:12:36.658+0200	DEBUG	controllers.Helm	reconciling...	{"central": "default/stackrox-central-services"}
2021-09-24T09:12:36.757+0200	DEBUG	controllers.Helm	preparing upgrade for stackrox-central-services
2021-09-24T09:12:38.053+0200	DEBUG	controllers.Helm	performing update for stackrox-central-services
2021-09-24T09:12:38.065+0200	DEBUG	controllers.Helm	dry run for stackrox-central-services
W0924 09:12:38.101994   47834 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0924 09:12:38.107936   47834 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0924 09:12:38.111774   47834 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0924 09:12:38.116127   47834 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2021-09-24T09:12:38.288+0200	INFO	controllers.Helm	Release reconciled	{"central": "default/stackrox-central-services", "name": "stackrox-central-services", "version": 1}

2021-09-24T09:18:09.237+0200	DEBUG	controllers.Helm	reconciling...	{"central": "default/stackrox-central-services"}
2021-09-24T09:18:09.330+0200	DEBUG	controllers.Helm	uninstall: Deleting stackrox-central-services
I0924 09:18:10.424882   47834 request.go:655] Throttling request took 1.047191126s, request: GET:https://kubernetes.docker.internal:6443/apis/node.k8s.io/v1beta1?timeout=32s
2021-09-24T09:18:10.440+0200	DEBUG	controllers.Helm	Starting delete for "scanner-db" Service
2021-09-24T09:18:10.440+0200	DEBUG	controllers.Helm	Starting delete for "central" Service
2021-09-24T09:18:10.440+0200	DEBUG	controllers.Helm	Starting delete for "scanner" Service
2021-09-24T09:18:10.480+0200	DEBUG	controllers.Helm	Starting delete for "scanner-db" Deployment
2021-09-24T09:18:10.480+0200	DEBUG	controllers.Helm	Starting delete for "scanner" Deployment
2021-09-24T09:18:10.480+0200	DEBUG	controllers.Helm	Starting delete for "central" Deployment
2021-09-24T09:18:10.499+0200	DEBUG	controllers.Helm	Starting delete for "stackrox-central-psp" RoleBinding
2021-09-24T09:18:10.499+0200	DEBUG	controllers.Helm	Starting delete for "stackrox-scanner-psp" RoleBinding
2021-09-24T09:18:10.499+0200	DEBUG	controllers.Helm	Starting delete for "stackrox-central-diagnostics" RoleBinding
2021-09-24T09:18:10.510+0200	DEBUG	controllers.Helm	Starting delete for "stackrox-central-diagnostics" Role
2021-09-24T09:18:10.518+0200	DEBUG	controllers.Helm	Starting delete for "stackrox-default-central-psp" ClusterRole
2021-09-24T09:18:10.518+0200	DEBUG	controllers.Helm	Starting delete for "stackrox-default-scanner-psp" ClusterRole
2021-09-24T09:18:10.528+0200	DEBUG	controllers.Helm	Starting delete for "scanner-config" ConfigMap
2021-09-24T09:18:10.528+0200	DEBUG	controllers.Helm	Starting delete for "central-config" ConfigMap
2021-09-24T09:18:10.528+0200	DEBUG	controllers.Helm	Starting delete for "central-endpoints" ConfigMap
2021-09-24T09:18:10.541+0200	DEBUG	controllers.Helm	Starting delete for "scanner" ServiceAccount
2021-09-24T09:18:10.541+0200	DEBUG	controllers.Helm	Starting delete for "central" ServiceAccount
2021-09-24T09:18:10.548+0200	DEBUG	controllers.Helm	Starting delete for "stackrox-default-scanner" PodSecurityPolicy
2021-09-24T09:18:10.548+0200	DEBUG	controllers.Helm	Starting delete for "stackrox-default-central" PodSecurityPolicy
W0924 09:18:10.556096   47834 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0924 09:18:10.557235   47834 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2021-09-24T09:18:10.557+0200	DEBUG	controllers.Helm	Starting delete for "scanner-db" NetworkPolicy
2021-09-24T09:18:10.557+0200	DEBUG	controllers.Helm	Starting delete for "allow-ext-to-central" NetworkPolicy
2021-09-24T09:18:10.557+0200	DEBUG	controllers.Helm	Starting delete for "scanner" NetworkPolicy
2021-09-24T09:18:10.567+0200	DEBUG	controllers.Helm	purge requested for stackrox-central-services
2021-09-24T09:18:10.608+0200	INFO	controllers.Helm	Release uninstalled	{"central": "default/stackrox-central-services", "name": "stackrox-central-services", "version": 1}
2021-09-24T09:18:10.672+0200	DEBUG	controllers.Helm	reconciling...	{"central": "default/stackrox-central-services"}
2021-09-24T09:18:10.675+0200	DEBUG	controllers.Helm	resource not found, nothing to do	{"central": "default/stackrox-central-services"}

@porridge porridge merged commit bbfc3e1 into main Sep 27, 2021
@porridge porridge deleted the porridge/ROX-8049-trace branch September 27, 2021 05:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants