Question: Support for 2.4

[arve0]

Hi! 👋
Thanks for the operator. Any plans for supporting 2.4 of OpenSearch? Did not find any issues, discussions or roadmap about it. I see that maximum supported version is 2.3.0 in the README.

As I understand the code in ResolveImage, the operator will not stop you from defining say version 2.4.1.

[idanl21]

Hey @arve0, you are right that the operator will not stop you from installing version 2.4.1 although, we do not recommend to create a cluster in version 2.4.1 with the operator cause we didnt did the right tests and research, in a case of major API and behaviors changes you can get some bugs from it.

[swoehrl-mw]

Just to expand on what @idanl21 wrote:
You can install newer versions of OpenSearch 2.x, there is no technical block for it. And in most cases it should just work as the important APIs should stay compatible but as Idan stated there is always a chance for behaviour changes or bugs. So feel free to try it on a test cluster.

We will always try to test new OpenSearch versions with the operator and update the compatibility matrix, but that will lag behind.

[arve0]

Thanks. I’ll test an install and upgrade.

Are there automated tests in this repo? Or is the testing by you done manually?

[swoehrl-mw]

Are there automated tests in this repo? Or is the testing by you done manually?

There is an open PR (#343) to automate such tests so that in the future we just need to update the opensearch version and the tests will do the rest. Until then it's a manual process.

[turesheim]

So we tested 2.4.1 in our TEST cluster by patching the OpenSearch resource. All worker nodes and the dashboard nodes restarted as expected and came up and running 2.4.1 instead of 2.3.0. However none of the master nodes restarted and manually restarting these did not result in them coming up with the expected version.
After recreating the entire cluster it is running nicely with 2.4.1 and we have so far not found any more issues.

[swoehrl-mw]

Hi @turesheim.

After recreating the entire cluster it is running nicely with 2.4.1 and we have so far not found any more issues.

Good to know, thanks for sharing your experience.

However none of the master nodes restarted and manually restarting these did not result in them coming up with the expected version.

Could you try to recreate the problem and open a new issue with more details and a reproducible setup? That way we can try to locate the root cause and see if we have a bug in the operator upgrade logic.

[arve0]

OK, so another update.

What happens seems to be this:

1. Kibana deployment is updated to 2.5.0
2. Kibana rollout fails, as pod never goes ready
3. Kibana logs this info and error message:

{"type":"log","@timestamp":"2023-02-01T14:06:16Z","tags":["info","savedobjects-service"],"pid":1,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
{"type":"log","@timestamp":"2023-02-01T14:06:16Z","tags":["error","savedobjects-service"],"pid":1,"message":"This version of OpenSearch Dashboards (v2.5.0) is incompatible with the following OpenSearch nodes in your cluster: v2.3.0 @ opensearch-nodes-0/10.131.0.92:9200 (10.131.0.92), v2.3.0 @ opensearch-nodes-1/10.129.2.47:9200 (10.129.2.47), v2.3.0 @ opensearch-nodes-2/10.128.2.50:9200 (10.128.2.50), v2.3.0 @ opensearch-masters-2/10.128.2.68:9200 (10.128.2.68), v2.3.0 @ opensearch-masters-1/10.131.0.89:9200 (10.131.0.89), v2.3.0 @ opensearch-masters-0/10.129.2.42:9200 (10.129.2.42)"}

Last 10 messages from operator is:

1.6752605400263982e+09	DEBUG	resource is in sync	{"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opensearch","namespace":"logs-domstol-no"}, "namespace": "logs-domstol-no", "name": "opensearch", "reconcileID": "1ad52dc7-efe4-41db-8eba-a47caa8b1e83", "reconciler": "cluster", "name": "opensearch-nodes", "namespace": "logs-domstol-no", "apiVersion": "v1", "kind": "Service"}
1.6752605400265512e+09	INFO	The existing statefulset VolumeClaimTemplate disk size is: 5Gi	{"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opensearch","namespace":"logs-domstol-no"}, "namespace": "logs-domstol-no", "name": "opensearch", "reconcileID": "1ad52dc7-efe4-41db-8eba-a47caa8b1e83"}
1.6752605400267541e+09	INFO	The cluster definition nodePool disk size is: 5Gi	{"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opensearch","namespace":"logs-domstol-no"}, "namespace": "logs-domstol-no", "name": "opensearch", "reconcileID": "1ad52dc7-efe4-41db-8eba-a47caa8b1e83"}
1.6752605400267613e+09	INFO	The existing disk size 5Gi is same as passed in disk size 5Gi	{"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opensearch","namespace":"logs-domstol-no"}, "namespace": "logs-domstol-no", "name": "opensearch", "reconcileID": "1ad52dc7-efe4-41db-8eba-a47caa8b1e83"}
1.6752605400355477e+09	DEBUG	resource is in sync	{"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opensearch","namespace":"logs-domstol-no"}, "namespace": "logs-domstol-no", "name": "opensearch", "reconcileID": "1ad52dc7-efe4-41db-8eba-a47caa8b1e83", "reconciler": "cluster", "name": "opensearch-nodes", "namespace": "logs-domstol-no", "apiVersion": "apps/v1", "kind": "StatefulSet"}
1.6752605400364783e+09	DEBUG	resource is in sync	{"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opensearch","namespace":"logs-domstol-no"}, "namespace": "logs-domstol-no", "name": "opensearch", "reconcileID": "1ad52dc7-efe4-41db-8eba-a47caa8b1e83", "reconciler": "dashboards", "name": "opensearch-dashboards-config", "namespace": "logs-domstol-no", "apiVersion": "v1", "kind": "ConfigMap"}
1.6752605400395093e+09	DEBUG	resource diff	{"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opensearch","namespace":"logs-domstol-no"}, "namespace": "logs-domstol-no", "name": "opensearch", "reconcileID": "1ad52dc7-efe4-41db-8eba-a47caa8b1e83", "reconciler": "dashboards", "name": "opensearch-dashboards", "namespace": "logs-domstol-no", "apiVersion": "apps/v1", "kind": "Deployment", "patch": "{\"spec\":{\"strategy\":{\"$retainKeys\":[\"type\"],\"rollingUpdate\":null}}}"}
1.675260540039928e+09	DEBUG	updating resource	{"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opensearch","namespace":"logs-domstol-no"}, "namespace": "logs-domstol-no", "name": "opensearch", "reconcileID": "1ad52dc7-efe4-41db-8eba-a47caa8b1e83", "reconciler": "dashboards", "name": "opensearch-dashboards", "namespace": "logs-domstol-no", "apiVersion": "apps/v1", "kind": "Deployment"}
1.675260540048887e+09	DEBUG	resource updated	{"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opensearch","namespace":"logs-domstol-no"}, "namespace": "logs-domstol-no", "name": "opensearch", "reconcileID": "1ad52dc7-efe4-41db-8eba-a47caa8b1e83", "reconciler": "dashboards", "name": "opensearch-dashboards", "namespace": "logs-domstol-no", "apiVersion": "apps/v1", "kind": "Deployment"}
1.6752605400502403e+09	DEBUG	resource is in sync	{"controller": "opensearchcluster", "controllerGroup": "opensearch.opster.io", "controllerKind": "OpenSearchCluster", "OpenSearchCluster": {"name":"opensearch","namespace":"logs-domstol-no"}, "namespace": "logs-domstol-no", "name": "opensearch", "reconcileID": "1ad52dc7-efe4-41db-8eba-a47caa8b1e83", "reconciler": "dashboards", "name": "opensearch-dashboards", "namespace": "logs-domstol-no", "apiVersion": "v1", "kind": "Service"}

Cluster definition is:

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  creationTimestamp: "2022-12-13T14:35:02Z"
  finalizers:
  - Opster
  generation: 14
  labels:
    kustomize.toolkit.fluxcd.io/name: ldn-opensearch
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: opensearch
  namespace: logs-domstol-no
  resourceVersion: "420334773"
  uid: a7940d57-fc27-432c-a10d-2a1b8111c09d
spec:
  bootstrap:
    nodeSelector:
      node-role.kubernetes.io/infra: ""
    resources: {}
    tolerations:
    - effect: NoSchedule
      key: infra
      operator: Equal
      value: reserved
  confMgmt:
    smartScaler: true
  dashboards:
    additionalConfig:
      opensearch_security.multitenancy.enabled: "true"
    enable: true
    opensearchCredentialsSecret:
      name: opensearch-admin-password
    replicas: 1
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 500m
        memory: 1Gi
    tolerations:
    - effect: NoSchedule
      key: infra
      operator: Equal
      value: reserved
    version: 2.5.0
  general:
    additionalConfig:
      cluster.info.update.interval: 1m
      cluster.routing.allocation.disk.watermark.flood_stage: 1gb
      cluster.routing.allocation.disk.watermark.high: 5gb
      cluster.routing.allocation.disk.watermark.low: 15gb
    drainDataNodes: true
    httpPort: 9200
    serviceName: opensearch
    vendor: opensearch
    version: 2.5.0
  initHelper: {}
  nodePools:
  - component: masters
    diskSize: 5Gi
    jvm: -Dhttp.proxyHost=webproxy.at.domstol.no -Dhttp.proxyPort=8088 -Dhttps.proxyHost=webproxy.at.domstol.no
      -Dhttps.proxyPort=8088 -Dhttp.nonProxyHosts=*.domstol.no|localhost|172.30.*|10.128.*|10.129.*|10.130.*|10.131.*
    nodeSelector:
      node-role.kubernetes.io/infra: ""
    persistence:
      pvc:
        accessModes:
        - ReadWriteOnce
        storageClass: domstol-nfs-storage
    replicas: 3
    resources:
      limits:
        cpu: 500m
        memory: 2Gi
      requests:
        cpu: 100m
        memory: 2Gi
    roles:
    - cluster_manager
    tolerations:
    - effect: NoSchedule
      key: infra
      operator: Equal
      value: reserved
  - component: nodes
    diskSize: 5Gi
    jvm: -Xmx3G -Xms3G -Dhttp.proxyHost=webproxy.at.domstol.no -Dhttp.proxyPort=8088
      -Dhttps.proxyHost=webproxy.at.domstol.no -Dhttps.proxyPort=8088 -Dhttp.nonProxyHosts=*.domstol.no|localhost|172.30.*|10.128.*|10.129.*|10.130.*|10.131.*
    nodeSelector:
      node-role.kubernetes.io/infra: ""
    persistence:
      pvc:
        accessModes:
        - ReadWriteOnce
        storageClass: domstol-nfs-storage
    replicas: 3
    resources:
      limits:
        memory: 5Gi
      requests:
        cpu: 100m
        memory: 4Gi
    roles:
    - data
    - ingest
    tolerations:
    - effect: NoSchedule
      key: infra
      operator: Equal
      value: reserved
  security:
    config:
      adminCredentialsSecret:
        name: opensearch-admin-password
      adminSecret: {}
      securityConfigSecret:
        name: securityconfig-secret
    tls:
      http:
        caSecret: {}
        generate: true
        secret: {}
      transport:
        caSecret: {}
        generate: true
        secret: {}
status:
  componentsStatus:
  - {}
  - {}
  - component: Upgrader
    description: nodes
    status: Upgrading
  initialized: true
  phase: RUNNING
  version: 2.3.0

[swoehrl-mw]

Closing this issue. Opensearch 2.4 is supported by operator version 2.2.

@arve0 Dashboards will only become ready once the underlying opensearch cluster is also upgraded. If you have problems with upgrading please comment in #420 so we do not mix topics in issues.