Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After upgrading to 0.17, an exception error occurs when modifying clickhouseinstallation #842

Closed
czhfe opened this issue Dec 22, 2021 · 13 comments
Labels
possible bug Something isn't working

Comments

@czhfe
Copy link

czhfe commented Dec 22, 2021

After upgrading to 0.17 an exception error occurs when modifying clickhouseinstallation, which prevents modification and deletion of clickhouseinstallation. the error is as follows:

The ClickHouseInstallation "clickhouse" is invalid: status.errors: Invalid value: "null": status.errors in body must be of type array: "null"

@Slach
Copy link
Collaborator

Slach commented Dec 22, 2021

How exactly do you upgrade to 0.17?
Did you upgrade crds or just update image in deployment?

Could you share the results for following command?

kubectl get deploy --all-namespaces | grep clickhouse-operator

@czhfe
Copy link
Author

czhfe commented Dec 22, 2021

How exactly do you upgrade to 0.17? Did you upgrade or just update image in deployment?crds

Could you share the results for following command?

kubectl get deploy --all-namespaces | grep clickhouse-operator

Upgrade operator first, then upgrade crd.

image

@Slach
Copy link
Collaborator

Slach commented Dec 22, 2021

Could you upgrade CRDs first
your deployment not look like after standard installation way
it should be clickhouse-operator instead of clickhouse-operator-altinity-clickhouse-operator?
Did you use some Helm chart or something else?

Could you remove this deployment (it safe)

kubecetl delete deploy -n kube-system clickhouse-operator-altinity-clickhouse-operator

after it and just apply

kubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml

and check results

kubectl get deploy clickhouse-operator -n kube-system 

@czhfe
Copy link
Author

czhfe commented Dec 22, 2021

Could you upgrade CRDs first your deployment not look like after it should be clickhouse-operator instead of ? Did you use some Helm chart or something else?standard installation way``clickhouse-operator-altinity-clickhouse-operator

Could you remove this deployment (it safe)

kubecetl delete deploy -n kube-system clickhouse-operator-altinity-clickhouse-operator

after it and just apply

kubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml

and check results

kubectl get deploy clickhouse-operator -n kube-system 

I wrote the helm chart deployment myself

@Slach
Copy link
Collaborator

Slach commented Dec 22, 2021

@czhfe ok. try update crds
in your helm chart
and re-install clickhouse-operator

@czhfe
Copy link
Author

czhfe commented Dec 22, 2021

@czhfe ok. try update crds in your helm chart and re-install clickhouse-operator

Now I do not understand why the operator and crd updated here will error, there is no solution

@Slach
Copy link
Collaborator

Slach commented Dec 22, 2021

It solution cause new CRDs define status as subresources.

Compare your current CRDs with
https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml

@czhfe
Copy link
Author

czhfe commented Dec 22, 2021

It solution cause new CRDs define status as subresources.

Compare your current CRDs with https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml

I upgraded from 0.13.5 to 0.17

@alex-zaitsev alex-zaitsev added the possible bug Something isn't working label Dec 24, 2021
@czhfe
Copy link
Author

czhfe commented Dec 29, 2021

Upgrade from clickhouse-operator 0.13.5 to 0.17.0. perform the following steps.

  1. Upgrade crd first
  2. Upgrade operator and other resources

After the upgrade, modify ClickHouseInstallation CR encountered the following error

image

@Slach
Copy link
Collaborator

Slach commented Dec 29, 2021

Do you have status in your edited resource? If yes, please remove this sections and try to save

Could you share the following command results?

kubectl get chi -n ch1 clickhouse -o yaml

@czhfe
Copy link
Author

czhfe commented Dec 30, 2021

Do you have in your edited resource? If yes, please remove this sections and try to savestatus

Could you share the following command results?

kubectl get chi -n ch1 clickhouse -o yaml

Yes, I have tested this before by deleting the status section and trying to save it later, but the result is still the same.

@czhfe
Copy link
Author

czhfe commented Dec 30, 2021

Problem recurrence:

The /spec/templating/policy in the clickhouseinstallations.clickhouse.altinity.com CRD prior to version 0.16.0 did not have its default value explicitly defined, resulting in that default value being "".
The clickhouseinstallations.clickhouse.altinity.com CRD was upgraded in version 0.16.0 of the clickhouse operator (this can be seen in the release notes in RELEASES).
In version 0.16.0 and later /spec/templating/policy was changed to be configurable as auto or manual, with the default value being manual. clickhouseinstallations.clickhouse.altinity.com before 0.16.0
CR before 0.16.0 did not explicitly define that the /spec/templating/policy field would be added automatically, so clickhouse operators and CRD versions after 0.16.0 cannot automatically update the /spec/templating/policy field, which would block
clickhouse operator to update the CR for clickhouseinstallations.clickhouse.altinity.com, which causes an error

@alex-zaitsev
Copy link
Member

Should be fixed in earlier released, but additional rules were added at https://github.com/Altinity/clickhouse-operator/releases/tag/release-0.22.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
possible bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants