Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Charm goes into blocked with "Not all resources found in cluster." #930

Open
orfeas-k opened this issue Jun 11, 2024 · 2 comments
Open

Charm goes into blocked with "Not all resources found in cluster." #930

orfeas-k opened this issue Jun 11, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@orfeas-k
Copy link
Contributor

orfeas-k commented Jun 11, 2024 •
edited
Loading

Bug Description

argo-controller/0*            blocked   idle   10.244.0.39                  [kubernetes:crds-cm-and-secrets] Not all resources found in cluster.  This may be transient if we haven't tried to de...

This probably relates to how kubernetesComponent works.

That being said, it resolves itself after a while (sth like 20 minutes later?)

To Reproduce

Deploy CKF from latest/edge bundle.yaml file.

Environment

AKS 1.28
juju 3.5.1 with agent 3.5.0

Relevant Log Output

$ jst (juju status)
Model     Controller     Cloud/Region    Version  SLA          Timestamp
kubeflow  kf-controller  aks/westeurope  3.5.0    unsupported  11:54:28+03:00

App                        Version                  Status   Scale  Charm                    Channel              Rev  Address       Exposed  Message
admission-webhook                                   active       1  admission-webhook        latest/edge          312  10.0.49.139   no       
argo-controller                                     waiting      1  argo-controller          latest/edge          489  10.0.96.114   no       installing agent
dex-auth                                            active       1  dex-auth                 latest/edge          474  10.0.120.239  no       
envoy                                               active       1  envoy                    latest/edge          207  10.0.83.231   no       
istio-ingressgateway                                active       1  istio-gateway            latest/edge          913  10.0.63.210   no       
istio-pilot                                         active       1  istio-pilot              latest/edge          872  10.0.57.131   no       
jupyter-controller                                  active       1  jupyter-controller       latest/edge          962  10.0.253.162  no       
jupyter-ui                                          active       1  jupyter-ui               latest/edge          884  10.0.79.228   no       
katib-controller                                    active       1  katib-controller         latest/edge/pr-185   553  10.0.209.225  no       
katib-db                   8.0.36-0ubuntu0.22.04.1  active       1  mysql-k8s                8.0/edge             146  10.0.94.241   no       
katib-db-manager                                    active       1  katib-db-manager         latest/edge/pr-185   517  10.0.80.0     no       
katib-ui                                            active       1  katib-ui                 latest/edge          518  10.0.46.198   no       
kfp-api                                             active       1  kfp-api                  latest/edge         1348  10.0.69.72    no       
kfp-db                     8.0.36-0ubuntu0.22.04.1  active       1  mysql-k8s                8.0/edge             146  10.0.69.59    no       
kfp-metadata-writer                                 active       1  kfp-metadata-writer      latest/edge          392  10.0.117.239  no       
kfp-persistence                                     active       1  kfp-persistence          latest/edge         1355  10.0.10.135   no       
kfp-profile-controller                              active       1  kfp-profile-controller   latest/edge         1313  10.0.15.8     no       
kfp-schedwf                                         active       1  kfp-schedwf              latest/edge         1367  10.0.112.16   no       
kfp-ui                                              active       1  kfp-ui                   latest/edge         1349  10.0.183.58   no       
kfp-viewer                                          active       1  kfp-viewer               latest/edge         1381  10.0.125.51   no       
kfp-viz                                             active       1  kfp-viz                  latest/edge         1300  10.0.83.53    no       
knative-eventing                                    active       1  knative-eventing         latest/edge          401  10.0.238.181  no       
knative-operator                                    active       1  knative-operator         latest/edge          376  10.0.251.51   no       
knative-serving                                     active       1  knative-serving          latest/edge          402  10.0.205.88   no       
kserve-controller                                   active       1  kserve-controller        latest/edge          543  10.0.45.208   no       
kubeflow-dashboard                                  active       1  kubeflow-dashboard       latest/edge          543  10.0.199.224  no       
kubeflow-profiles                                   active       1  kubeflow-profiles        latest/edge          385  10.0.91.22    no       
kubeflow-roles                                      active       1  kubeflow-roles           latest/edge          218  10.0.255.1    no       
kubeflow-volumes                                    active       1  kubeflow-volumes         latest/edge          307  10.0.70.51    no       
metacontroller-operator                             active       1  metacontroller-operator  latest/edge          280  10.0.32.176   no       
minio                      res:oci-image@1755999    active       1  minio                    latest/edge          316  10.0.150.254  no       
mlmd                                                active       1  mlmd                     latest/edge          184  10.0.63.139   no       
oidc-gatekeeper                                     active       1  oidc-gatekeeper          latest/edge          382  10.0.93.201   no       
pvcviewer-operator                                  active       1  pvcviewer-operator       latest/edge           95  10.0.251.221  no       
seldon-controller-manager                           active       1  seldon-core              latest/edge          691  10.0.129.45   no       
tensorboard-controller                              active       1  tensorboard-controller   latest/edge          288  10.0.60.4     no       
tensorboards-web-app                                active       1  tensorboards-web-app     latest/edge          276  10.0.242.228  no       
training-operator                                   active       1  training-operator        latest/edge          378  10.0.70.1     no       

Unit                          Workload  Agent  Address       Ports          Message
admission-webhook/0*          active    idle   10.244.0.38                  
argo-controller/0*            blocked   idle   10.244.0.39                  [kubernetes:crds-cm-and-secrets] Not all resources found in cluster.  This may be transient if we havent tried to de...
dex-auth/0*                   active    idle   10.244.0.40                  
envoy/0*                      active    idle   10.244.0.43                  
istio-ingressgateway/0*       active    idle   10.244.0.41                  
istio-pilot/0*                active    idle   10.244.0.42                  
jupyter-controller/0*         active    idle   10.244.0.44                  
jupyter-ui/0*                 active    idle   10.244.1.118                 
katib-controller/0*           active    idle   10.244.0.45                  
katib-db-manager/0*           active    idle   10.244.1.120                 
katib-db/0*                   active    idle   10.244.0.47                  Primary
katib-ui/0*                   active    idle   10.244.0.46                  
kfp-api/0*                    active    idle   10.244.1.121                 
kfp-db/0*                     active    idle   10.244.1.122                 Primary
kfp-metadata-writer/0*        active    idle   10.244.0.49                  
kfp-persistence/0*            active    idle   10.244.0.52                  
kfp-profile-controller/0*     active    idle   10.244.0.53                  
kfp-schedwf/0*                active    idle   10.244.0.54                  
kfp-ui/0*                     active    idle   10.244.1.123                 
kfp-viewer/0*                 active    idle   10.244.0.57                  
kfp-viz/0*                    active    idle   10.244.1.124                 
knative-eventing/0*           active    idle   10.244.0.48                  
knative-operator/0*           active    idle   10.244.0.60                  
knative-serving/0*            active    idle   10.244.0.50                  
kserve-controller/0*          active    idle   10.244.1.127                 
kubeflow-dashboard/0*         active    idle   10.244.0.58                  
kubeflow-profiles/0*          active    idle   10.244.1.129                 
kubeflow-roles/0*             active    idle   10.244.0.55                  
kubeflow-volumes/0*           active    idle   10.244.1.125                 
metacontroller-operator/0*    active    idle   10.244.0.56                  
minio/0*                      active    idle   10.244.0.59   9000-9001/TCP  
mlmd/0*                       active    idle   10.244.0.62                  
oidc-gatekeeper/0*            active    idle   10.244.1.126                 
pvcviewer-operator/0*         active    idle   10.244.1.130                 
seldon-controller-manager/0*  active    idle   10.244.0.61                  
tensorboard-controller/0*     active    idle   10.244.1.128                 
tensorboards-web-app/0*       active    idle   10.244.0.63                  
training-operator/0*          active    idle   10.244.0.64         

╰─$ kfl argo-controller-0  -c argo-controller
2024-06-11T08:04:24.989Z [pebble] HTTP API server listening on ":38813".
2024-06-11T08:04:24.989Z [pebble] Started daemon.
2024-06-11T08:04:55.240Z [pebble] GET /v1/notices?timeout=30s 30.000296088s 200
2024-06-11T08:05:25.241Z [pebble] GET /v1/notices?timeout=30s 30.000677203s 200
2024-06-11T08:05:55.241Z [pebble] GET /v1/notices?timeout=30s 30.000277459s 200
2024-06-11T08:06:25.242Z [pebble] GET /v1/notices?timeout=30s 30.000342723s 200
2024-06-11T08:06:55.243Z [pebble] GET /v1/notices?timeout=30s 30.000644026s 200
2024-06-11T08:07:25.244Z [pebble] GET /v1/notices?timeout=30s 30.000251745s 200
2024-06-11T08:07:55.245Z [pebble] GET /v1/notices?timeout=30s 30.000304423s 200
2024-06-11T08:08:16.378Z [pebble] GET /v1/plan?format=yaml 86.602µs 200
2024-06-11T08:08:16.380Z [pebble] POST /v1/layers 360.207µs 200
2024-06-11T08:08:16.472Z [pebble] POST /v1/services 90.928064ms 202
2024-06-11T08:08:16.571Z [pebble] GET /v1/notices?timeout=30s 21.325890557s 200
2024-06-11T08:08:16.571Z [pebble] Service "argo-controller" starting: workflow-controller --configmap argo-workflow-controller-configmap --executor-image gcr.io/ml-pipeline/argoexec:v3.4.16-license-compliance
2024-06-11T08:08:16.589Z [argo-controller] time="2024-06-11T08:08:16Z" level=info msg="index config" indexWorkflowSemaphoreKeys=true
2024-06-11T08:08:16.591Z [argo-controller] time="2024-06-11T08:08:16Z" level=info msg="cron config" cronSyncPeriod=10s
2024-06-11T08:08:16.591Z [argo-controller] time="2024-06-11T08:08:16Z" level=info msg="Memoization caches will be garbage-collected if they have not been hit after" gcAfterNotHitDuration=30s
2024-06-11T08:08:16.591Z [argo-controller] time="2024-06-11T08:08:16.591Z" level=info msg="not enabling pprof debug endpoints"
2024-06-11T08:08:16.623Z [argo-controller] time="2024-06-11T08:08:16.623Z" level=info msg="Configuration:\nartifactRepository:\n  archiveLogs: true\n  s3:\n    accessKeySecret:\n      key: accesskey\n      name: mlpipeline-minio-artifact\n    bucket: mlpipeline\n    endpoint: minio.kubeflow:9000\n    insecure: true\n    secretKeySecret:\n      key: secretkey\n      name: mlpipeline-minio-artifact\nexecutor:\n  imagePullPolicy: IfNotPresent\n  name: \"\"\n  resources: {}\ninitialDelay: 0s\nmetricsConfig: {}\nnodeEvents: {}\npodSpecLogStrategy: {}\nsso:\n  clientId:\n    key: \"\"\n  clientSecret:\n    key: \"\"\n  issuer: \"\"\n  redirectUrl: \"\"\n  sessionExpiry: 0s\ntelemetryConfig: {}\n"
2024-06-11T08:08:16.623Z [argo-controller] time="2024-06-11T08:08:16.623Z" level=info msg="Persistence configuration disabled"
2024-06-11T08:08:16.623Z [argo-controller] time="2024-06-11T08:08:16.623Z" level=info executorImage="gcr.io/ml-pipeline/argoexec:v3.4.16-license-compliance" executorImagePullPolicy=IfNotPresent managedNamespace=
2024-06-11T08:08:16.624Z [argo-controller] I0611 08:08:16.624311      16 leaderelection.go:248] attempting to acquire leader lease kubeflow/workflow-controller...
2024-06-11T08:08:16.624Z [argo-controller] time="2024-06-11T08:08:16.624Z" level=info msg="Starting dummy metrics server at localhost:9090/metrics"
2024-06-11T08:08:16.791Z [argo-controller] I0611 08:08:16.791838      16 leaderelection.go:258] successfully acquired lease kubeflow/workflow-controller
2024-06-11T08:08:16.792Z [argo-controller] time="2024-06-11T08:08:16.791Z" level=info msg="new leader" leader=argo-controller
2024-06-11T08:08:16.792Z [argo-controller] time="2024-06-11T08:08:16.792Z" level=info msg="Starting Workflow Controller" defaultRequeueTime=10s version=v3.4.16
2024-06-11T08:08:16.792Z [argo-controller] time="2024-06-11T08:08:16.792Z" level=info msg="Current Worker Numbers" podCleanup=4 workflowTtlWorkers=4 workflowWorkers=32
2024-06-11T08:08:16.792Z [argo-controller] time="2024-06-11T08:08:16.792Z" level=info msg="Successfully shutdown dummy metrics server at localhost:9090/metrics"
2024-06-11T08:08:16.792Z [argo-controller] time="2024-06-11T08:08:16.792Z" level=info msg="Starting prometheus metrics server at localhost:9090/metrics"
2024-06-11T08:08:16.792Z [argo-controller] time="2024-06-11T08:08:16.792Z" level=info msg="Watching task results" labelSelector="!workflows.argoproj.io/controller-instanceid,workflows.argoproj.io/workflow"
2024-06-11T08:08:16.792Z [argo-controller] time="2024-06-11T08:08:16.792Z" level=info msg=Plugins executorPlugins=false
2024-06-11T08:08:16.807Z [argo-controller] time="2024-06-11T08:08:16.807Z" level=info msg="Manager initialized successfully"
2024-06-11T08:08:16.841Z [argo-controller] time="2024-06-11T08:08:16.841Z" level=info msg="Received Workflow Controller config map kubeflow/argo-workflow-controller-configmap update"
2024-06-11T08:08:16.887Z [argo-controller] time="2024-06-11T08:08:16.887Z" level=info msg="Configuration:\nartifactRepository:\n  archiveLogs: true\n  s3:\n    accessKeySecret:\n      key: accesskey\n      name: mlpipeline-minio-artifact\n    bucket: mlpipeline\n    endpoint: minio.kubeflow:9000\n    insecure: true\n    secretKeySecret:\n      key: secretkey\n      name: mlpipeline-minio-artifact\nexecutor:\n  imagePullPolicy: IfNotPresent\n  name: \"\"\n  resources: {}\ninitialDelay: 0s\nmetricsConfig: {}\nnodeEvents: {}\npodSpecLogStrategy: {}\nsso:\n  clientId:\n    key: \"\"\n  clientSecret:\n    key: \"\"\n  issuer: \"\"\n  redirectUrl: \"\"\n  sessionExpiry: 0s\ntelemetryConfig: {}\n"
2024-06-11T08:08:16.887Z [argo-controller] time="2024-06-11T08:08:16.887Z" level=info msg="Persistence configuration disabled"
2024-06-11T08:08:16.887Z [argo-controller] time="2024-06-11T08:08:16.887Z" level=info executorImage="gcr.io/ml-pipeline/argoexec:v3.4.16-license-compliance" executorImagePullPolicy=IfNotPresent managedNamespace=
2024-06-11T08:08:17.005Z [argo-controller] time="2024-06-11T08:08:17.005Z" level=info msg="Performing periodic GC" periodicity=5m0s
2024-06-11T08:08:17.005Z [argo-controller] time="2024-06-11T08:08:17.005Z" level=info msg="Persistence disabled - so archived workflow GC disabled - you must restart the controller if you enable this"
2024-06-11T08:08:17.005Z [argo-controller] time="2024-06-11T08:08:17.005Z" level=info msg="Starting CronWorkflow controller"
2024-06-11T08:08:17.006Z [argo-controller] time="2024-06-11T08:08:17.005Z" level=info msg="Starting workflow garbage collector controller (retentionWorkers 4)"
2024-06-11T08:08:17.006Z [argo-controller] time="2024-06-11T08:08:17.006Z" level=info msg="Started workflow garbage collection"
2024-06-11T08:08:17.006Z [argo-controller] W0611 08:08:17.006122      16 shared_informer.go:401] The sharedIndexInformer has started, run more than once is not allowed
2024-06-11T08:08:17.636Z [pebble] GET /v1/changes/1/wait?timeout=4.000s 1.162624211s 200
2024-06-11T08:08:17.637Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A16.473056513Z&timeout=30s 1.065463412s 200
2024-06-11T08:08:17.660Z [pebble] GET /v1/services 156.603µs 200
2024-06-11T08:08:47.641Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.000299529s 200
2024-06-11T08:09:17.642Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.000222777s 200
2024-06-11T08:09:47.643Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.001175966s 200
2024-06-11T08:10:17.651Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.000888236s 200
2024-06-11T08:10:47.652Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.000410318s 200
2024-06-11T08:11:17.654Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.001256712s 200
2024-06-11T08:11:47.655Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.000528584s 200
2024-06-11T08:12:17.656Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.00049291s 200
2024-06-11T08:12:47.657Z [pebble] GET /v1/notices?after=2024-06-11T08%3A08%3A17.572753164Z&timeout=30s 30.000524557s 200




─$ juju debug-log -i argo-controller -l info
unit-argo-controller-0: 11:04:24 INFO juju.cmd running containerAgent [3.5.0 01a596920ff5a632babf5aacf485cc78fe1ed5df gc go1.21.9]
unit-argo-controller-0: 11:04:24 INFO juju.cmd.containeragent.unit start "unit"
unit-argo-controller-0: 11:04:24 INFO juju.worker.upgradesteps upgrade steps for 3.5.0 have already been run.
unit-argo-controller-0: 11:04:24 INFO juju.worker.probehttpserver starting http server on 127.0.0.1:65301
unit-argo-controller-0: 11:04:24 INFO juju.api connection established to "wss://10.0.230.216:17070/model/b11aee1c-fb16-4a93-85ec-d6a3785776ca/api"
unit-argo-controller-0: 11:04:24 INFO juju.worker.apicaller [b11aee] "unit-argo-controller-0" successfully connected to "10.0.230.216:17070"
unit-argo-controller-0: 11:04:24 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-argo-controller-0: 11:04:24 INFO juju.worker.logger logger worker started
unit-argo-controller-0: 11:04:24 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-argo-controller-0: 11:04:43 INFO unit.argo-controller/0.juju-log Running legacy hooks/install.
unit-argo-controller-0: 11:04:44 INFO unit.argo-controller/0.juju-log HTTP Request: GET https://10.0.0.1/api/v1/namespaces/kubeflow/services/argo-controller "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:04:44 INFO unit.argo-controller/0.juju-log HTTP Request: PATCH https://10.0.0.1/api/v1/namespaces/kubeflow/services/argo-controller "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:04:44 INFO unit.argo-controller/0.juju-log Kubernetes service 'argo-controller' patched successfully
unit-argo-controller-0: 11:04:44 INFO unit.argo-controller/0.juju-log Starting `execute_components` for event 'ArgoControllerOperator/on/install[1]'
unit-argo-controller-0: 11:04:44 INFO unit.argo-controller/0.juju-log Executing component: 'leadership-gate'
unit-argo-controller-0: 11:04:44 INFO unit.argo-controller/0.juju-log Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:04:44 INFO unit.argo-controller/0.juju-log Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:04:45 INFO unit.argo-controller/0.juju-log Execution for component 'relation:object_storage' complete.  Component now has status 'BlockedStatus('Expected data from exactly 1 related applications - got 0.')'
unit-argo-controller-0: 11:04:45 INFO unit.argo-controller/0.juju-log execute_components execution loop complete.
unit-argo-controller-0: 11:04:45 INFO unit.argo-controller/0.juju-log Status of all CharmReconciler Components:
unit-argo-controller-0: 11:04:45 INFO unit.argo-controller/0.juju-log Status: BlockedStatus('[relation:object_storage] Expected data from exactly 1 related applications - got 0.')
unit-argo-controller-0: 11:04:45 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus("[kubernetes:crds-cm-and-secrets] Execution pending - waiting on relation:object_storage (BlockedStatus('Expected data from exactly 1 related applications - got 0.')).")
unit-argo-controller-0: 11:04:45 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (MaintenanceStatus("Execution pending - waiting on relation:object_storage (BlockedStatus(\'Expected data from exactly 1 related applications - got 0.\')).")), relation:object_storage (BlockedStatus(\'Expected data from exactly 1 related applications - got 0.\')).')
unit-argo-controller-0: 11:04:45 INFO unit.argo-controller/0.juju-log Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:04:45 INFO unit.argo-controller/0.juju-log Status of unit set to: BlockedStatus('[relation:object_storage] Expected data from exactly 1 related applications - got 0.')
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Starting `execute_components` for event 'ArgoControllerOperator/on/leader_elected[11]'
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Executing component: 'leadership-gate'
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Execution for component 'relation:object_storage' complete.  Component now has status 'WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')'
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log execute_components execution loop complete.
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Status of all CharmReconciler Components:
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Status: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus("[kubernetes:crds-cm-and-secrets] Execution pending - waiting on relation:object_storage (WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')).")
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (MaintenanceStatus("Execution pending - waiting on relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).")), relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).')
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:04:47 INFO unit.argo-controller/0.juju-log Status of unit set to: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Starting `execute_components` for event 'ArgoControllerOperator/on/argo_controller_pebble_ready[16]'
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Executing component: 'leadership-gate'
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Execution for component 'relation:object_storage' complete.  Component now has status 'WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')'
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log execute_components execution loop complete.
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Status of all CharmReconciler Components:
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Status: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus("[kubernetes:crds-cm-and-secrets] Execution pending - waiting on relation:object_storage (WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')).")
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (MaintenanceStatus("Execution pending - waiting on relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).")), relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).')
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:04:48 INFO unit.argo-controller/0.juju-log Status of unit set to: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:04:49 INFO unit.argo-controller/0.juju-log Starting `execute_components` for event 'ArgoControllerOperator/on/config_changed[21]'
unit-argo-controller-0: 11:04:49 INFO unit.argo-controller/0.juju-log Executing component: 'leadership-gate'
unit-argo-controller-0: 11:04:49 INFO unit.argo-controller/0.juju-log Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:04:49 INFO unit.argo-controller/0.juju-log Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:04:49 INFO unit.argo-controller/0.juju-log Execution for component 'relation:object_storage' complete.  Component now has status 'WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')'
unit-argo-controller-0: 11:04:49 INFO unit.argo-controller/0.juju-log execute_components execution loop complete.
unit-argo-controller-0: 11:04:50 INFO unit.argo-controller/0.juju-log Status of all CharmReconciler Components:
unit-argo-controller-0: 11:04:50 INFO unit.argo-controller/0.juju-log Status: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:04:50 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus("[kubernetes:crds-cm-and-secrets] Execution pending - waiting on relation:object_storage (WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')).")
unit-argo-controller-0: 11:04:50 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (MaintenanceStatus("Execution pending - waiting on relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).")), relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).')
unit-argo-controller-0: 11:04:50 INFO unit.argo-controller/0.juju-log Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:04:50 INFO unit.argo-controller/0.juju-log Status of unit set to: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:04:51 INFO unit.argo-controller/0.juju-log Running legacy hooks/start.
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Starting `execute_components` for event 'ArgoControllerOperator/on/config_changed[31]'
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Executing component: 'leadership-gate'
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Execution for component 'relation:object_storage' complete.  Component now has status 'WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')'
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log execute_components execution loop complete.
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Status of all CharmReconciler Components:
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Status: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus("[kubernetes:crds-cm-and-secrets] Execution pending - waiting on relation:object_storage (WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')).")
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (MaintenanceStatus("Execution pending - waiting on relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).")), relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).')
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:04:54 INFO unit.argo-controller/0.juju-log Status of unit set to: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Starting `execute_components` for event 'ArgoControllerOperator/on/object_storage_relation_changed[41]'
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'leadership-gate'
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'relation:object_storage' complete.  Component now has status 'WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')'
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: execute_components execution loop complete.
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Status of all CharmReconciler Components:
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Status: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Status: MaintenanceStatus("[kubernetes:crds-cm-and-secrets] Execution pending - waiting on relation:object_storage (WaitingStatus('List of <ops.model.Relation object-storage:8> versions not found for apps: minio')).")
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (MaintenanceStatus("Execution pending - waiting on relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).")), relation:object_storage (WaitingStatus(\'List of <ops.model.Relation object-storage:8> versions not found for apps: minio\')).')
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:06:20 INFO unit.argo-controller/0.juju-log object-storage:8: Status of unit set to: WaitingStatus('[relation:object_storage] List of <ops.model.Relation object-storage:8> versions not found for apps: minio')
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Starting `execute_components` for event 'ArgoControllerOperator/on/object_storage_relation_changed[46]'
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'leadership-gate'
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'relation:object_storage' complete.  Component now has status 'BlockedStatus('Expected data from exactly 1 related applications - got 0.')'
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: execute_components execution loop complete.
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Status of all CharmReconciler Components:
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Status: BlockedStatus('[relation:object_storage] Expected data from exactly 1 related applications - got 0.')
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Status: MaintenanceStatus("[kubernetes:crds-cm-and-secrets] Execution pending - waiting on relation:object_storage (BlockedStatus('Expected data from exactly 1 related applications - got 0.')).")
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (MaintenanceStatus("Execution pending - waiting on relation:object_storage (BlockedStatus(\'Expected data from exactly 1 related applications - got 0.\')).")), relation:object_storage (BlockedStatus(\'Expected data from exactly 1 related applications - got 0.\')).')
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:06:31 INFO unit.argo-controller/0.juju-log object-storage:8: Status of unit set to: BlockedStatus('[relation:object_storage] Expected data from exactly 1 related applications - got 0.')
unit-argo-controller-0: 11:07:05 INFO unit.argo-controller/0.juju-log object-storage:8: Starting `execute_components` for event 'ArgoControllerOperator/on/object_storage_relation_changed[51]'
unit-argo-controller-0: 11:07:05 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'leadership-gate'
unit-argo-controller-0: 11:07:05 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:07:05 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:07:06 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'relation:object_storage' complete.  Component now has status 'BlockedStatus('Expected data from exactly 1 related applications - got 0.')'
unit-argo-controller-0: 11:07:06 INFO unit.argo-controller/0.juju-log object-storage:8: execute_components execution loop complete.
unit-argo-controller-0: 11:07:06 INFO unit.argo-controller/0.juju-log object-storage:8: Status of all CharmReconciler Components:
unit-argo-controller-0: 11:07:06 INFO unit.argo-controller/0.juju-log object-storage:8: Status: BlockedStatus('[relation:object_storage] Expected data from exactly 1 related applications - got 0.')
unit-argo-controller-0: 11:07:06 INFO unit.argo-controller/0.juju-log object-storage:8: Status: MaintenanceStatus("[kubernetes:crds-cm-and-secrets] Execution pending - waiting on relation:object_storage (BlockedStatus('Expected data from exactly 1 related applications - got 0.')).")
unit-argo-controller-0: 11:07:06 INFO unit.argo-controller/0.juju-log object-storage:8: Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (MaintenanceStatus("Execution pending - waiting on relation:object_storage (BlockedStatus(\'Expected data from exactly 1 related applications - got 0.\')).")), relation:object_storage (BlockedStatus(\'Expected data from exactly 1 related applications - got 0.\')).')
unit-argo-controller-0: 11:07:06 INFO unit.argo-controller/0.juju-log object-storage:8: Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:07:06 INFO unit.argo-controller/0.juju-log object-storage:8: Status of unit set to: BlockedStatus('[relation:object_storage] Expected data from exactly 1 related applications - got 0.')
unit-argo-controller-0: 11:08:06 INFO unit.argo-controller/0.juju-log object-storage:8: Starting `execute_components` for event 'ArgoControllerOperator/on/object_storage_relation_changed[56]'
unit-argo-controller-0: 11:08:06 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'leadership-gate'
unit-argo-controller-0: 11:08:06 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'leadership-gate' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:08:06 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'relation:object_storage'
unit-argo-controller-0: 11:08:06 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'relation:object_storage' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:08:06 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'kubernetes:crds-cm-and-secrets'
unit-argo-controller-0: 11:08:07 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:08 INFO unit.argo-controller/0.juju-log object-storage:8: Rendering manifests
unit-argo-controller-0: 11:08:08 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterworkflowtemplates.argoproj.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:08 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/compositecontrollers.metacontroller.k8s.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:08 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/controllerrevisions.metacontroller.k8s.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:09 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/cronworkflows.argoproj.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:09 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/decoratorcontrollers.metacontroller.k8s.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:09 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/scheduledworkflows.kubeflow.org?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:09 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/viewers.kubeflow.org?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:09 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/workflowartifactgctasks.argoproj.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:09 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/workfloweventbindings.argoproj.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:09 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/workflows.argoproj.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:10 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/workflowtaskresults.argoproj.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:10 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/workflowtasksets.argoproj.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:10 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions/workflowtemplates.argoproj.io?force=true&fieldManager=lightkube "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:10 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/api/v1/namespaces/kubeflow/secrets/mlpipeline-minio-artifact?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-argo-controller-0: 11:08:10 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: PATCH https://10.0.0.1/api/v1/namespaces/kubeflow/configmaps/argo-workflow-controller-configmap?force=true&fieldManager=lightkube "HTTP/1.1 201 Created"
unit-argo-controller-0: 11:08:10 INFO unit.argo-controller/0.juju-log object-storage:8: Reconcile completed successfully
unit-argo-controller-0: 11:08:12 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:12 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/configmaps?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:12 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:13 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/secrets?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:13 INFO unit.argo-controller/0.juju-log object-storage:8: Rendering manifests
unit-argo-controller-0: 11:08:13 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'kubernetes:crds-cm-and-secrets' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:08:15 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:15 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/configmaps?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:15 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:15 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/secrets?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:15 INFO unit.argo-controller/0.juju-log object-storage:8: Rendering manifests
unit-argo-controller-0: 11:08:16 INFO unit.argo-controller/0.juju-log object-storage:8: Executing component: 'container:argo-controller'
unit-argo-controller-0: 11:08:16 INFO unit.argo-controller/0.juju-log object-storage:8: PebbleServiceComponent.get_layer executing
unit-argo-controller-0: 11:08:17 INFO unit.argo-controller/0.juju-log object-storage:8: PebbleServiceComponent.get_layer executing
unit-argo-controller-0: 11:08:17 INFO unit.argo-controller/0.juju-log object-storage:8: Execution for component 'container:argo-controller' complete.  Component now has status 'ActiveStatus('')'
unit-argo-controller-0: 11:08:18 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:19 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/configmaps?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:19 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:19 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/secrets?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:19 INFO unit.argo-controller/0.juju-log object-storage:8: Rendering manifests
unit-argo-controller-0: 11:08:20 INFO unit.argo-controller/0.juju-log object-storage:8: execute_components execution loop complete.
unit-argo-controller-0: 11:08:21 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:21 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/configmaps?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:22 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:22 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/secrets?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:22 INFO unit.argo-controller/0.juju-log object-storage:8: Rendering manifests
unit-argo-controller-0: 11:08:23 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:23 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/configmaps?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:23 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:24 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/secrets?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:24 INFO unit.argo-controller/0.juju-log object-storage:8: Rendering manifests
unit-argo-controller-0: 11:08:25 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:25 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/configmaps?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:25 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/apis/apiextensions.k8s.io/v1/customresourcedefinitions?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:26 INFO unit.argo-controller/0.juju-log object-storage:8: HTTP Request: GET https://10.0.0.1/api/v1/secrets?labelSelector=app.kubernetes.io%2Finstance%3Dargo-controller-kubeflow%2Ckubernetes-resource-handler-scope%3Dcrds-cm-and-secrets "HTTP/1.1 200 OK"
unit-argo-controller-0: 11:08:26 INFO unit.argo-controller/0.juju-log object-storage:8: Rendering manifests
unit-argo-controller-0: 11:08:26 INFO unit.argo-controller/0.juju-log object-storage:8: Status of all CharmReconciler Components:
unit-argo-controller-0: 11:08:26 INFO unit.argo-controller/0.juju-log object-storage:8: Status: BlockedStatus("[kubernetes:crds-cm-and-secrets] Not all resources found in cluster.  This may be transient if we haven't tried to deploy them yet.")
unit-argo-controller-0: 11:08:26 INFO unit.argo-controller/0.juju-log object-storage:8: Status: MaintenanceStatus('[container:argo-controller] Execution pending - waiting on kubernetes:crds-cm-and-secrets (BlockedStatus("Not all resources found in cluster.  This may be transient if we haven\'t tried to deploy them yet.")).')
unit-argo-controller-0: 11:08:26 INFO unit.argo-controller/0.juju-log object-storage:8: Status: ActiveStatus('[leadership-gate] ')
unit-argo-controller-0: 11:08:26 INFO unit.argo-controller/0.juju-log object-storage:8: Status: ActiveStatus('[relation:object_storage] ')
unit-argo-controller-0: 11:08:26 INFO unit.argo-controller/0.juju-log object-storage:8: Status of unit set to: BlockedStatus("[kubernetes:crds-cm-and-secrets] Not all resources found in cluster.  This may be transient if we haven't tried to deploy them yet.")

Additional Context

No response
@orfeas-k orfeas-k added bug Something isn't working enhancement New feature or request labels Jun 11, 2024
@orfeas-k orfeas-k reopened this Jun 11, 2024
@orfeas-k orfeas-k removed the enhancement New feature or request label Jun 11, 2024
@orfeas-k
Copy link
Contributor Author

I deployed to a different cluster and observed the same behaviour with 2 other charms

kfp-profile-controller/0*     blocked      idle   10.244.0.21                 [kubernetes:secrets-and-compositecontroller] Not all resources found in cluster.  This may be transient if we haven't...
kfp-schedwf/0*                maintenance  idle   10.244.0.22                 [kfp-schedwf-pebble-service] Execution pending - waiting on kubernetes:auth-and-crds (BlockedStatus("Not all resource...

@orfeas-k orfeas-k transferred this issue from canonical/argo-operators Jun 11, 2024
@orfeas-k orfeas-k changed the title Charm goes into blocked with "Not al resources found in cluster." Charm goes into blocked with "Not all resources found in cluster." Jun 11, 2024
@syncronize-issues-to-jira Syncronize issues to Jira
Copy link

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5883.

This message was autogenerated

@orfeas-k orfeas-k reopened this Jun 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@orfeas-k