chore(tracking): Test demo upgrades on nightly versions for 25.11.0

[Techassi]

Part of stackabletech/issues#759

This is testing that upgrades from the stable release to the nightly release of the operators and products do not negatively impact the products.

Note

Record the results (issues, anomalies, or success) during the process in a comment on this issue.
Eg:

:green_circle: **airflow-scheduled-job**

The CRD had been updated and I needed to change the following in the manifest:
...

The following emojis can be used to indicate the status:

• hourglass: In progress / waiting
• red_circle: In progress with errors
• orange_circle: Completed with minor issues or anomalies
• green_circle: Completed without issues or errors

List of demos

Replace the items in the lists below with the applicable Pull Requests (if any). Also put your GitHub handle next to the demo to indicate a particular demo was picked up for testing.

• airflow-scheduled-job @adwk67
• data-lakehouse-iceberg-trino-spark @labrenbe
• end-to-end-security @Techassi
• hbase-hdfs-load-cycling-data @adwk67
• jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data @adwk67
• jupyterhub-keycloak @adwk67
• logging @Techassi
• nifi-kafka-druid-earthquake-data @razvan
• nifi-kafka-druid-water-level-data @razvan
• signal-processing @Techassi
• spark-k8s-anomaly-detection-taxi-data @adwk67
• trino-iceberg @labrenbe
• trino-taxi-data @labrenbe

List of stacks

Unused stack upgrade testing is not needed. Demos already test product upgrades.
Therefore, the unused stack tests will be only done from scratch. See: #327

Stable to Nightly Upgrade Testing Instructions

These instructions are for deploying and completing the stable demo, and then upgrading operators, CRDs, and products to the nightly versions well as upgrading the operators and CRDS.

Tip

Be sure to select the stable docs version on https://docs.stackable.tech/home/stable/demos/.

# Make sure stackablectl 1.2.0 is installed
stackablectl --version

# Install demo (stable operators) for the stable release (25.7).
stackablectl demo install <DEMO_NAME>

# --- IMPORTANT ---
# Run through the stable demo instructions (refer to the list above).

# Get a list of installed operators
stackablectl operator installed --output=plain

# Uninstall operators for the stable release (25.7)
stackablectl release uninstall 25.7

# Update CRDs to nightly version (on main)
# Repeat this for every operator used by the demo (use the list from the earlier step before deleting the operators)
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/main/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/...-operator/main/deploy/helm/...-operator/crds/crds.yaml
# N.B. for the secret-operator this should not be necessary as the latest version will be deployed by the operator

# Install nightly version of operators (use the list from the earlier step before deleting the operators)
stackablectl operator install commons ...

# Optionally update the product versions in the CRDs (to the latest non-experimental version for the new release), e.g.:
kubectl patch hbaseclusters/hbase --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"x.x.x"}]' # changed

[adwk67]

🟢 jupyterhub-keycloak

• 🟢 Install stable from scratch
• 🟢 Upgrade to nightly

[Techassi]

🟢 logging

• 🟢 Install stable from scratch
• 🟢 Upgrade to nightly
  • fix: Force apply CRDs secret-operator#653 fixes the CRD conflicts.

[adwk67]

🟢 airflow-scheduled-job

• 🟢 Install stable from scratch
• 🟢 Upgrade to nightly
  • the secret operator CRDs have to be deleted to avoid errors when rolling out the nightly operator. This is expected when moving from 25.7 to 25.11 as the spec.conversion.strategy values are different (None and Webhook). Should be ok thereafter.
  • fix: Force apply CRDs secret-operator#653 fixes the CRD conflicts.
  • during the upgrade several scheduled jobs fail: this is expected and they can all be successfully re-started in the UI

[Techassi]

🟢 signal-processing

• 🟢 Install stable from scratch
• 🟢 Upgrade to nightly
  • fix: Force apply CRDs secret-operator#653 fixes the CRD conflicts.

[razvan]

Update

Retry with the this version of the secret operator stackabletech/secret-operator#653

🟢 nifi-kafka-druid-earthquake-data

• 🟢 25.7.0 install
• 🟢 upgrade to dev release

---

Was:

🔴 nifi-kafka-druid-earthquake-data

• 🟠 25.7.0 install from scratch
  • superset-druid-connection generates a job that writes to the the Superset Postgres DB. The job pod writes to the DB successfully but fails to return 0. Therefore another pod is created that fails because it tries to create duplicate keys in the DB.
• 🔴 upgrade operators to dev
  • after uninstalling the 25.7.0 release, need to manually delete secretclasses and truststores CRDs
    • ⚠️ while doing this I also deleted all product secret classes created by the demos which caused the problems below
  • after installing the dev release, Druid pods are left in Pending with errors related to the secret volumes.
  • Warning FailedBinding 24m (x6 over 24m) ephemeral_volume ephemeral volume tls-mount: PVC default/druid-broker-default-0-tls-mount was not created for pod default/druid-broker-defaul t-0 (pod is not owner)

[Techassi]

🟢 end-to-end-security

• The two bugs found where fixed

WAS

🔴 end-to-end-security

• 🟢 Install stable from scratch
  • When logging into Keycloak as the administrator, the following message is displayed in a banner at the top: "You are logged in as a temporary admin user. To harden security, create a permanent admin account and delete the temporary one." Is this something we want to/can address?
• 🔴 Upgrade to nightly
  • All pods are running and "healthy".
  • Trino queries fail with: SQL Error [84148225]: Query failed (#20251031_152856_00001_zgwra): Error accessing metadata file for table customer_analytics.customer
  • The Hive metastore reports the following error: ERROR [Metastore-Handler-Pool: Thread-223] server.TThreadPoolServer: Thrift Error occurred during processing of message. org.apache.thrift.transport.TTransportException: org.apache.thrift.transport.TTransportException: Socket is closed by peer.
  • @sbernauer figured out the fix: the STSs for HDFS need to be deleted which is detailed below.
  • 🔴 The HDFS upgrade is broken because of chore: ensure metrics are correctly exposed hdfs-operator#721 See chore: ensure metrics are correctly exposed hdfs-operator#721 for details => Created fix: Revert/remove headless suffix from headless service hdfs-operator#726
  • 🔴 The stackablectl release upgrade dev command is broken because of a 404. See https://stackable-workspace.slack.com/archives/C0985QFRVGC/p1762172924746249 for details => Created fix: Don't crash during release upgrade for SDP 25.11 stackable-cockpit#418

Details

Trino queries failed with

Caused by: java.io.IOException: Get status for file hdfs://hdfs/lakehouse/customer-analytics/customer-816e46755ef34461b9fdf33b0aec9593/metadata/00001-2cced164-2831-4ee0-81de-c5ae143d7985.metadata.json failed: Failed to query OPA backend
Caused by: java.nio.channels.UnresolvedAddressException

HDFS config still is

hadoop.security.authorization.opa.policy.url=http://opa.default.svc.cluster.local:8081/v1/data/hdfs/allow

The hdfs-operator logs

StatefulSet.apps "hdfs-journalnode-default" is invalid: spec: Forbidden: updates to statefulset spec for fields other than XXX are forbidden

Solution:

kubectl delete sts hdfs-namenode-default hdfs-journalnode-default hdfs-datanode-default

Afterwards I got

Caused by: java.io.IOException: Get status for file hdfs://hdfs/lakehouse/customer-analytics/customer-816e46755ef34461b9fdf33b0aec9593/metadata/00001-2cced164-2831-4ee0-81de-c5ae143d7985.metadata.json failed: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error

That's IMHO unrelated, but worth it's own research

[adwk67]

🟢 hbase-hdfs-load-cycling-data

• 🟢 Install stable from scratch
• 🟢 Upgrade to nightly
  • Everything works after the upgrade: hbase components (apart from the restserver) receive new listeners and the master/regionserver pods are restarted with the 0.0.0-dev tag in place of 25.7.0 (expected behaviour)
  • The HDFS pods are not restarted and continue to run with 25.7.0 tags. Also expected, as the listeners are stable across the restart.
  • Also: the secret-operator CRDs had to be deleted manually before upgrading the operators.
  • fix: Force apply CRDs secret-operator#653 fixes the CRD conflicts.

[adwk67]

🟢 jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data

• 🟢 Install stable from scratch
• 🟢 Upgrade to nightly
  • The SecretClass and TrustStore CRDs need to be deleted.
  • fix: Force apply CRDs secret-operator#653 fixes the CRD conflicts.

[adwk67]

🟢 spark-k8s-anomaly-detection-taxi-data

• 🟢 Install stable from scratch
• 🟢 Upgrade to nightly: secrets are deleted along with the secret-op CRDs and hive/trino pods stuck in pending
  • fix: Force apply CRDs secret-operator#653 fixes the CRD conflicts.

Secret-op logs this out after the bump to nightly, not sure if it is a problem:
(Update: as discussed in the daily 03.11, this is to be expected)

2025-10-31T17:26:27.631335Z  INFO stackable_secret_operator: Starting Stackable Operator for managing credentials built_info.pkg_version="0.0.0-dev" built_info.git_version="6a7fb74" built_info.target="x86_64-unknown-linux-gnu" built_info.built_time_utc="Fri, 31 Oct 2025 14:21:51 +0000" built_info.rustc_version="rustc 1.89.0 (29483883e 2025-08-04)"
2025-10-31T17:26:27.631577Z  INFO stackable_operator::utils::cluster_info: Fetching Kubernetes cluster domain from the local kubelet node_name=stackable-data-platform-worker2
2025-10-31T17:26:27.637911Z  INFO stackable_operator::utils::cluster_info: Using Kubernetes cluster domain from the kubelet config cluster_domain=cluster.local
2025-10-31T17:26:27.723523Z  INFO stackable_webhook::servers::conversion: run conversion webhook server
2025-10-31T17:26:27.723573Z  INFO stackable_webhook::maintainer: reconciling custom resource definitions k8s.crd.names=["secretclasses.secrets.stackable.tech", "truststores.secrets.stackable.tech"]
2025-10-31T17:26:27.831158Z  INFO stackable_secret_operator: applying default secretclass
2025-10-31T17:26:28.750505Z ERROR stackable_operator::logging::controller: Failed to reconcile object controller.name="truststore.secrets.stackable.tech" error=event queue error error.sources=[watch stream failed: ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }), ApiError: storage is (re)initializing: TooManyRequests (ErrorResponse { status: "Failure", message: "storage is (re)initializing", reason: "TooManyRequests", code: 429 }), storage is (re)initializing: TooManyRequests]
2025-10-31T17:26:29.551802Z ERROR stackable_operator::logging::controller: Failed to reconcile object controller.name="truststore.secrets.stackable.tech" error=event queue error error.sources=[error returned by apiserver during watch: too old resource version: 7783 (7792): Expired, too old resource version: 7783 (7792): Expired]
2025-10-31T17:26:31.153242Z ERROR stackable_operator::logging::controller: Failed to reconcile object controller.name="truststore.secrets.stackable.tech" error=event queue error error.sources=[error returned by apiserver during watch: too old resource version: 7783 (7792): Expired, too old resource version: 7783 (7792): Expired]

[labrenbe]

🟢 trino-iceberg

• 🟢 Install stable from scratch
• 🟢 Upgrade to nightly