-
Notifications
You must be signed in to change notification settings - Fork 215
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
reference ContainerCluster is not ready #578
Comments
Hi @travisrandolph-bestbuy, the updating is normal behavior for config connector and can't be changed; all of our resources attempt to reconcile roughly every 10 minutes. As for the warnings, can you provide some more detail? Is that the entirety of the warning message? |
Also, can you check if your resources have label leaser on, you'll need to turn if off to avoid the API calls for releasing the leases. |
@mbzomowski We are not doing anything to manage conflict. Below is an example of the update in the logs.
|
Hello! As a quick clarification of what @mbzomowski was saying: although it is true that there will be an API GET every 10 minutes to ensure that the resource is up to date, it is not necessarily true that the update will occur every 10 minutes: only in the case where there was a change that needed to be applied. So update requests every 10 minutes implies that there is something up with the resource: either it is not updating properly, or perhaps some erroneous diff detection on the part of Config Connector. @travisrandolph-bestbuy can you send us a redacted version of the resource so we can see if we can reproduce locally? The output you sent looks to be logs from the controller, which can be helpful for diagnostics but doesn't help us spot issues like whether label leasing is on or reproducing locally. Redacted output from a |
@toumorokoshi Here's the output.
|
Thanks for the info! Based on this annotation:
there is no conflict prevention on, and most likely there is some field that is being detected as being modified over and over again. Let me reach to our current bug rotation to follow-up. |
Hi @travisrandolph-bestbuy , sorry for the delayed response. I was trying to create a GKE cluster with the configuration you provided (except for On the other hand, I noticed that if there is any update to the cluster, there should be a corresponding log for the update operation. Could you check the GKE operation logs associated with the cluster to get more information about what has been updated and whether the changes in the operations match the yaml? An example query to search for GKE operations in Logs Explorer would be:
|
@maqiuyujoyce Here is the update I'm seeing regularly.
|
Hi @travisrandolph-bestbuy, I think this issue is probably being caused by Officially, the field only recognizes all-caps values as per the field's description in the docs. I can file a bug to allow for all-lower-case values as well to avoid this issue in the future, but we probably won't be able to prioritize this improvement for a while if the all-caps values do work as intended. If using |
@jcanseco Updating the release channel to use upper case resolved some of our clusters. One of the logs looks like it was trying to update the maintenance window. After looking into it we were leaving the Zulu time specification off the end of the start and end times. Once I updated my configs to '2019-09-02T15:00:00Z', it stopped trying to update. After these changes most of our clusters are fine now. The last problem seems to be with 'masterAuthorizedNetworksConfig'. I've added the mismatch in the second code box below. We can't add the boolean, so I'm not sure we can do anything to fix that.
|
Hey @travisrandolph-bestbuy, I'm glad to hear that most of your clusters are no longer seeing the issue. Re: Would you be willing to share the |
@jcanseco After pushing both the prior changes to all clusters our resources are no longer updating every 10 minutes. I'm closing this one out! |
@travisrandolph-bestbuy, that's great to hear! Thanks for keeping us updated. |
Hi,
I need help understanding ContainerCluster. It seems like it's updating itself every 10th minute. This leads to our ContainerNodePool's sends out warnings:
reference ContainerCluster {{REDACTED}} is not ready
Additional Diagnostic Information
Kubernetes Cluster Version
Client Version: v1.19.2
Server Version: v1.20.10-gke.1600
Config Connector Version
1.60.0
Config Connector Mode
namespaced
The text was updated successfully, but these errors were encountered: