Fix the data-lakehouse-iceberg-trino-spark demo

[NickLarsenNZ]

This came out of the release upgrade testing: #59 (comment)

Spark

No visualization information available for the streaming job run. but that seems to only be because the jobs completed. The active job show a visualisation, as does the completed water_level job.

Job list	Missing visualization

You can see exceptions on the right, and the jobs didn't seem to run (0 ms duration).

Postgres

There was also some warnings from the postgres helm install:

# Postgres gave the following warnings:
# WARNING: There are "resources" sections in the chart not set. Using "resourcesPreset" is not recommended for production. For production installations, please set the following values according to your workload needs:
#     - primary.resources
#     - readReplicas.resources

[mbaeuerle]

This might not be related but the URL to the bike status endpoint is not reachable any longer: https://ip-api.mobidata-bw.de/v1/NVBW/gbfs/v2/free_bike_status.json
Used in the shared bike status: https://github.com/stackabletech/demos/blob/main/demos/data-lakehouse-iceberg-trino-spark/LakehouseKafkaIngest.xml#L1388

[NickLarsenNZ]

This might not be related but the URL to the bike status endpoint is not reachable any longer: https://ip-api.mobidata-bw.de/v1/NVBW/gbfs/v2/free_bike_status.json Used in the shared bike status: https://github.com/stackabletech/demos/blob/main/demos/data-lakehouse-iceberg-trino-spark/LakehouseKafkaIngest.xml#L1388

Thanks for the heads-up @mbaeuerle. I'll take a look into this in the lead up to the SDP 24.11 release.

[xeniape]

@NickLarsenNZ I would argue the postgres warning was related to the postgres upgrade process and not the demo itself. At least I didn't see any errors or warnings in the running postgres instances while solving this issue. Do you have more information on what was impacted by that and does that need addressing as part of this issue?

Other than that I submitted a PR for the fix of the endpoints fetching the data. Kafka topics are created as a result and the spark streaming queries are running fine now. #115

[NickLarsenNZ]

Thanks for the endpoint fix @xeniape.

[NickLarsenNZ]

Marked as closed because:

• fix: Update Remote URLs in NiFi for fetching data #115 resolves the Spark issue.
• The Postgres issue seemed to come from upgrade testing.