tracking: Test demo upgrades on nightly versions for 26.7.0

[Techassi]

Part of stackabletech/issues#847

This is testing that upgrades from the stable release to the nightly release of the operators
and products do not negatively impact the products.

Note

Record the results (issues, anomalies, or success) during the process in a comment on this issue.
Eg:

:green_circle: **airflow-scheduled-job**

The CRD had been updated and I needed to change the following in the manifest:
...

The following emojis can be used to indicate the status:

• hourglass: In progress / waiting
• red_circle: In progress with errors
• orange_circle: Completed with minor issues or anomalies
• green_circle: Completed without issues or errors

List of demos

Replace the items in the lists below with the applicable Pull Requests (if any). Also put your
GitHub handle next to the demo to indicate a particular demo was picked up for testing.

• airflow-scheduled-job @adwk67
• argo-cd-git-ops
• data-lakehouse-iceberg-trino-spark
• end-to-end-security @Techassi
• hbase-hdfs-load-cycling-data @adwk67
• jupyterhub-keycloak @Techassi
• jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data @sweb
• logging @Techassi
• nifi-kafka-druid-earthquake-data
• nifi-kafka-druid-water-level-data
• signal-processing @Techassi
• spark-k8s-anomaly-detection-taxi-data @NickLarsenNZ
• trino-iceberg @sbernauer
• trino-taxi-data @sbernauer

List of stacks

Similar to the demos above we need to test all stacks that don't have a demo.
Replace the items in the lists below with the applicable Pull Requests (if any). Also put your
GitHub handle next to the stack to indicate a particular stack was picked up for testing.

Tip

Some of the stacks have a tutorial to follow.

• monitoring
• observability @NickLarsenNZ
• openldap
• tutorial-openldap

Stable to Nightly Upgrade Testing Instructions

These instructions are for deploying and completing the stable demo, and then
upgrading operators, CRDs, and products to the nightly versions well as upgrading
the operators and CRDS.

Tip

Be sure to select the stable docs version on https://docs.stackable.tech/home/stable/demos/.

# For demo testing, , install demo (stable operators) for the stable release (26.3).
stackablectl demo install <DEMO_NAME>

# For stack testing, install stack via
stackablectl stack install <STACK_NAME>

# --- IMPORTANT ---
# Run through the stable demo/stack instructions (refer to the list above).

# Install nightly version of operators
stackablectl release upgrade dev

# Bump every stacklet to the version that used by the nightly demo.
# Typically, this is the latest supported version.
kubectl patch hbaseclusters/hbase --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"x.x.x"}]' # changed

# Check that the operator logs look good (no unexpected errors etc.)

# Run through the (still) nightly demo/stack instructions again.

HDFS rolling upgrade instructions

For demos that use HDFS, also test a rolling upgrade.
Also see https://docs.stackable.tech/home/nightly/hdfs/usage-guide/upgrading/

# In some cases (eg. in the end-to-end-security demo), HDFS is kerberized.
# This requires these commands to be run first.
# Pick "nn/..."
klist -kt /stackable/kerberos/keytab
kinit -kt /stackable/kerberos/keytab <PRINCIPAL_FROM_ABOVE>

# These commands need to be executed in a superuser environment (eg. the namenode pod)
hdfs dfsadmin -rollingUpgrade prepare
hdfs dfsadmin -rollingUpgrade query  # repeat until "Proceed with rolling upgrade"

# Now patch the productVersion in your regular shell
kubectl patch hdfs/<CLUSTER_NAME> --type=merge \
  --patch '{"spec": {"image": {"productVersion": "<NEW_VERSION>"}}}'

# Finalize the upgrade in the namenode pod
# Needs another kinit because the container was restarted
hdfs dfsadmin -rollingUpgrade finalize

kubectl patch hdfs/<CLUSTER_NAME> --subresource=status --type=merge \
  --patch '{"status": {"deployedProductVersion": "<NEW_VERSION>"}}'

[Techassi]

🟢 logging

• 🟢 Install stable
• 🟢 Upgrade to nightly

[Techassi]

🟢 signal-processing (paused for now, seems to be a deeper issue which seems to go back to at least 25.11)

• 🟢 Install stable
  Initially, the create-nifi-ingestion-job failed because it timed out when trying to connect to NiFi. NiFi seems to fail to connect to the Postgres database: JDBC driver class 'org.postgresql.Driver' not found. No JDBC driver classes found in the provided resources. @adwk67 discovered the issue and it is fixed by:
  • fix(signal-processing): fetch flow definition from release-26.3 #427
  • fix: Processing URL and namespace replacement #428
• 🟢 Upgrade to nightly

[sbernauer]

⌛ trino-iceberg

• 🟢 Install stable
• 🟢 Upgrade to nightly
  • Obviously had to address the CRD change from feat: Implement generic database connection hive-operator#674

[sbernauer]

⌛ trino-taxi-data

• 🟢 Install stable
• 🔴 Upgrade to nightly
  • 🟢 Obviously had to address the CRD change from feat: Implement generic database connection hive-operator#674
  • 🔴 And same for Superset: feat: Add generic database connection mechanism superset-operator#722. We found a bug related to SECRET_KEY rotation, see https://stackable-workspace.slack.com/archives/C0AV49795HQ/p1783672917029649 = > Fixed by fix: Correctly migrate SECRET_KEY from 26.3 setups superset-operator#754

[NickLarsenNZ]

⏳ spark-k8s-anomaly-detection-taxi-data

• 🟢 Baseline: Fine
• 🟠 Upgrade: Currently a problems:
  • 🔴 spark-k8s-operator due to now-namespaced sparkapptemplate. Revert "fix: make application templates namespaced (#694)" spark-k8s-operator#719
  • 🟢 hive CRD breaking change (Missing field metadataDatabase).
  • 🟠 superset CRD breking change (Missing field metadataDatabase, also missing credentialSecretName, and a new SECRET_KEY is being generated instead of using the established one: 26.3 -> 0.0.0-dev SECRET_KEY rotation breaks stacklet superset-operator#753)

Details

Hive operator nightly:

2026-07-10T07:28:11.361462Z  INFO reconciling object{object.ref=HiveCluster.v1alpha1.hive.stackable.tech/hive-iceberg.default object.reason=related object updated: StatefulSet.v1.apps/hive-iceberg-metastore-default.default}: stackable_hive_operator::controller: Starting reconcile
2026-07-10T07:28:11.361657Z ERROR stackable_operator::logging::controller: Failed to reconcile object controller.name="hivecluster.hive.stackable.tech" error=reconciler for object HiveCluster.v1alpha1.hive.stackable.tech/hive-iceberg.default failed error.sources=[HiveCluster object is invalid, Missing field metadataDatabase]

Spark operator nightly:

Error: failed to run webhook server

Caused by:
    0: failed to update certificate
    1: conversion webhook error
    2: failed to patch CRD "sparkapptemplates.spark.stackable.tech"
    3: ApiError: CustomResourceDefinition.apiextensions.k8s.io "sparkapptemplates.spark.stackable.tech" is invalid: spec.scope: Invalid value: "Namespaced": field is immutable: Invalid (Status { status: Some(Failure), code: 422, message: "CustomResourceDefinition.apiextensions.k8s.io \"sparkapptemplates.spark.stackable.tech\" is invalid: spec.scope: Invalid value: \"Namespaced\": field is immutable", metadata: Some(ListMeta { continue_: None, remaining_item_count: None, resource_version: None, self_link: None, shard_info: None }), reason: "Invalid", details: Some(StatusDetails { name: "sparkapptemplates.spark.stackable.tech", group: "apiextensions.k8s.io", kind: "CustomResourceDefinition", uid: "", causes: [StatusCause { reason: "FieldValueInvalid", message: "Invalid value: \"Namespaced\": field is immutable", field: "spec.scope" }], retry_after_seconds: 0 }) })
    4: CustomResourceDefinition.apiextensions.k8s.io "sparkapptemplates.spark.stackable.tech" is invalid: spec.scope: Invalid value: "Namespaced": field is immutable: Invalid
stream closed: EOF for stackable-operators/spark-k8s-operator-deployment-575544b9c4-rr487 (spark-k8s-operator)

[adwk67]

🟢 airflow-scheduled-job

• 🟢 Baseline: Fine
• 🟢 Upgrade: Fine, other than:
  • (🟠spark-k8s-operator due to now-namespaced sparkapptemplate)
  • had to patch the airflow cluster to add the missing and required-in-nightly fields credentialsSecretName and metadataDatabase

[Techassi]

🟢 jupyterhub-keycloak

• 🟢 Install stable
• 🟢 Upgrade to nightly
  • notebook only completed successfully after I deleted old data from MinIO

[Techassi]

⌛ end-to-end-security

• 🟢 Install stable
• 🟠 Upgrade to nightly (rolling upgrade worked after kinit, superset broken)