Dangerous upgrade from 1.6.0 to 1.7.0 if the procedure is not followed

[mtparet]

Bug Report

What did you do?

I did not read the upgrade procedure (my fault) and I started an helm upgrade from 1.6.0 to 1.7.0 without first applying new crd ressources.

What did you expect to see?

Upgrade that failed.

What did you see instead? Under which circumstances?

All elastic ressources (elasticsearch, beats, kibana) were destroyed from the cluster.

Environment

• ECK version:

1.7.0

• Kubernetes information:

  insert any information about your Kubernetes environment that could help us:

  • Cloud: EKS 1.21

• Logs:

I did not find relevants logs 

[pebrc]

How exactly did you do the upgrade? If I simply run

helm upgrade elastic-operator elastic/eck-operator -n elastic-system without following the upgrade instructions I get an error as in:

Error: UPGRADE FAILED: cannot patch "agents.agent.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "agents.agent.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema && cannot patch "apmservers.apm.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "apmservers.apm.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema && cannot patch "elasticmapsservers.maps.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "elasticmapsservers.maps.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema && cannot patch "elasticsearches.elasticsearch.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "elasticsearches.elasticsearch.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema && cannot patch "enterprisesearches.enterprisesearch.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "enterprisesearches.enterprisesearch.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema && cannot patch "kibanas.kibana.k8s.elastic.co" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "kibanas.kibana.k8s.elastic.co" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema

Which should prompt users to investigate further and hopefully discover our upgrade instructions. (I tried this on OCP 4.8 which is also based on K8s 1.21 but will take a look at EKS as well)

[mtparet]

Hum good question, I was using https://github.com/k3s-io/helm-controller. The log could indicate there was a delete action on release... :/

$ kubectl logs -f helm-install-elastic-operator-jzwnf -n elastic-system 
CHART="${CHART//%\{KUBERNETES_API\}%/${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}}"
set +v -x
+ cp /var/run/secrets/kubernetes.io/serviceaccount/ca.crt /usr/local/share/ca-certificates/
+ update-ca-certificates
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
+ [[ '' != \t\r\u\e ]]
+ export HELM_HOST=127.0.0.1:44134
+ HELM_HOST=127.0.0.1:44134
+ helm_v2 init --skip-refresh --client-only --stable-repo-url https://charts.helm.sh/stable/
+ tiller --listen=127.0.0.1:44134 --storage=secret
Creating /root/.helm 
Creating /root/.helm/repository 
Creating /root/.helm/repository/cache 
Creating /root/.helm/repository/local 
Creating /root/.helm/plugins 
Creating /root/.helm/starters 
Creating /root/.helm/cache/archive 
Creating /root/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://charts.helm.sh/stable/ 
Adding local repo with URL: http://127.0.0.1:8879/charts 
$HELM_HOME has been configured at /root/.helm.
Not installing Tiller due to 'client-only' flag having been set
++ jq -r '.Releases | length'
++ helm_v2 ls --all '^elastic-operator$' --output json
[main] 2021/08/04 12:19:18 Starting Tiller v2.17.0 (tls=false)
[main] 2021/08/04 12:19:18 GRPC listening on 127.0.0.1:44134
[main] 2021/08/04 12:19:18 Probes listening on :44135
[main] 2021/08/04 12:19:18 Storage driver is Secret
[main] 2021/08/04 12:19:18 Max history per release is 0
[storage] 2021/08/04 12:19:18 listing all releases with filter
+ V2_CHART_EXISTS=
+ [[ '' == \1 ]]
+ [[ '' == \v\2 ]]
+ [[ -n '' ]]
+ shopt -s nullglob
+ helm_content_decode
+ set -e
+ ENC_CHART_PATH=/chart/elastic-operator.tgz.base64
+ CHART_PATH=/elastic-operator.tgz
+ [[ ! -f /chart/elastic-operator.tgz.base64 ]]
+ return
+ [[ install != \d\e\l\e\t\e ]]
+ helm_repo_init
+ grep -q -e 'https\?://'
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
+ [[ eck-operator == stable/* ]]
+ [[ -n https://helm.elastic.co ]]
+ helm_v3 repo add elastic-operator https://helm.elastic.co
"elastic-operator" has been added to your repositories
+ helm_v3 repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "elastic-operator" chart repository
Update Complete. ⎈Happy Helming!⎈
+ helm_update install --namespace elastic-system --repo https://helm.elastic.co --version 1.7.0
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
++ tr '[:upper:]' '[:lower:]'
++ jq -r '"\(.[0].app_version),\(.[0].status)"'
++ helm_v3 ls --all -f '^elastic-operator$' --namespace elastic-system --output json
+ LINE=1.7.0,failed
+ IFS=,
+ read -r INSTALLED_VERSION STATUS _
+ VALUES=
+ [[ install = \d\e\l\e\t\e ]]
+ [[ 1.7.0 =~ ^(|null)$ ]]
+ [[ failed =~ ^(pending-install|pending-upgrade|pending-rollback)$ ]]
+ [[ failed == \d\e\p\l\o\y\e\d ]]
+ [[ failed =~ ^(deleted|failed|null|unknown)$ ]]
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
+ helm_v3 uninstall elastic-operator --namespace elastic-system
release "elastic-operator" uninstalled
+ echo Deleted
+ helm_v3 install --namespace elastic-system --repo https://helm.elastic.co --version 1.7.0 elastic-operator eck-operator
Deleted
NAME: elastic-operator
LAST DEPLOYED: Wed Aug  4 12:19:23 2021
NAMESPACE: elastic-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Inspect the operator logs by running the following command:
   kubectl logs -n elastic-system sts/elastic-operator
+ exit

[mtparet]

I opened an issue in the helm-controller repository k3s-io/helm-controller#110.

[pebrc]

I am closing this as the problem seems to be with helm-controller. Let's reopen if that changes.