apimanager_status_reconciler.go: add debug logs

[ilargitxiki]

No description provided.

[openshift-ci]

Hi @ilargitxiki. Thanks for your PR.

I'm waiting for a 3scale member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[eguzki]

It's me testing from an account without permissions :)

[codeclimate]

Code Climate has analyzed commit 25aa184 and detected 0 issues on this pull request.

View more on Code Climate.

[eguzki]

/ok-to-test

[austincunningham]

Verification

• checkout this branch

• ran the operator locally

• Created an apimanger cr and s3 secret

• Check the logs for the updates to logging

• apimangerAvailableCondition is there

1.6817183590378149e+09  DEBUG   controllers.APIManager  Status apimanagerAvailableCondition     {"Status Reconciler": "3scale-test/apimanager-sample", "deploymentsAvailable": false, "defaultRoutesReady": false}

• we see defaultRouteReady: route not found

1.68171826334244e+09    DEBUG   controllers.APIManager  Status defaultRoutesReady: route not found      {"Status Reconciler": "3scale-test/apimanager-sample", "expectedRouteHost": "master.apps.aucunnin.d3ys.s1.devshift.org"}
1.6817182633424456e+09  DEBUG   controllers.APIManager  Status defaultRoutesReady: route not found      {"Status Reconciler": "3scale-test/apimanager-sample", "expectedRouteHost": "3scale.apps.aucunnin.d3ys.s1.devshift.org"}
1.6817182633424509e+09  DEBUG   controllers.APIManager  Status defaultRoutesReady: route not found      {"Status Reconciler": "3scale-test/apimanager-sample", "expectedRouteHost": "3scale-admin.apps.aucunnin.d3ys.s1.devshift.org"}

• we don't see defaultRouteReady: route not ready we can see "defaultRoutesReady": false to "defaultRoutesReady": true in the logs without ever hitting that log

1.6817185085388563e+09  DEBUG   controllers.APIManager  Status apimanagerAvailableCondition     {"Status Reconciler": "3scale-test/apimanager-sample", "deploymentsAvailable": false, "defaultRoutesReady": false}
1.681718508538874e+09   INFO    olm     Found deployments with status   {"stopped": [], "starting": ["apicast-production"], "ready": ["apicast-staging", "backend-cron", "backend-listener", "backend-redis", "backend-worker", "system-app", "system-memcache", "system-mysql", "system-redis", "system-sidekiq", "system-sphinx", "zync", "zync-database", "zync-que"]}
1.6817185085389156e+09  DEBUG   controllers.APIManager  Status  {"Status Reconciler": "3scale-test/apimanager-sample", "status is different": false}
1.6817185085389223e+09  DEBUG   controllers.APIManager  Status was not updated  {"Status Reconciler": "3scale-test/apimanager-sample"}
1.681718520803766e+09   INFO    controllers.APIManager  ReconcileAPIManager     {"apimanager": "3scale-test/apimanager-sample", "Operator version": "0.11.0", "3scale release": "2.14"}
1.681718520803782e+09   INFO    controllers.WebConsole  Reconciling ReconcileWebConsole {"webconsole": "3scale-test/zync-3scale-provider-lfcqx", "Operator version": "0.11.0"}
1.6817185209039953e+09  DEBUG   controllers.WebConsole  END     {"webconsole": "3scale-test/zync-3scale-provider-lfcqx"}
1.6817185209040525e+09  INFO    controllers.WebConsole  Reconciling ReconcileWebConsole {"webconsole": "3scale-test/zync-3scale-provider-697wn", "Operator version": "0.11.0"}
1.6817185210030448e+09  DEBUG   controllers.WebConsole  END     {"webconsole": "3scale-test/zync-3scale-provider-697wn"}
1.6817185210030916e+09  INFO    controllers.WebConsole  Reconciling ReconcileWebConsole {"webconsole": "3scale-test/zync-3scale-api-tcb7h", "Operator version": "0.11.0"}
1.6817185211016154e+09  DEBUG   controllers.WebConsole  END     {"webconsole": "3scale-test/zync-3scale-api-tcb7h"}
1.6817185211016762e+09  INFO    controllers.WebConsole  Reconciling ReconcileWebConsole {"webconsole": "3scale-test/zync-3scale-provider-lfcqx", "Operator version": "0.11.0"}
1.6817185211058075e+09  DEBUG   controllers.APIManager  START   {"Status Reconciler": "3scale-test/apimanager-sample"}
1.6817185211066296e+09  DEBUG   controllers.APIManager  Status apimanagerAvailableCondition     {"Status Reconciler": "3scale-test/apimanager-sample", "deploymentsAvailable": false, "defaultRoutesReady": true}

Guessing the route comes up before an full reconcile so we will never hit this .

[austincunningham]

Not merged My review stands it was #805 that merged

[eguzki]

we don't see defaultRouteReady: route not ready we can see "defaultRoutesReady": false to "defaultRoutesReady": true in the logs without ever hitting that log

If no issues are found, "defaultRouteReady: route not ready" should not be seen. That's when the route has not been accepted by Openshift. A common case of not being accepted is when the domain name has already been used.I hit that state when I re-used the same domain name in two different namespaces when deploying 3scale. The route created for backend was in "Not Ready" state and that log line showed up. It is easy to reproduce, just create a route with, let's say 3scale.apps.aucunnin.d3ys.s1.devshift.org before creating the APIManager CR.

[austincunningham]

Ok so I created a second apimanager CR in the default namespace with the same wildcarddomain
didn't see defaultRouteReady: route not ready in the logs and the install fail with crashing pods .

[eguzki]

Ok so I created a second apimanager CR in the default namespace with the same wildcarddomain didn't see defaultRouteReady: route not ready in the logs and the install fail with crashing pods .

Crashing pods? That should be something else, out of the scope of this PR

[austincunningham]

Ok created a dummy route to match the 3scale-admin, then deployed 3scale operator , see the debug error defaultRouteReady: route not ready in the operator logs, see the 3scale admin route gets rejected by openshift in the ui and gets removed.

• verified