Adapt spark k8s versions for release 24.7

[maltesander]

Description

• switch 3.5.1 to java 17
• removed versions 3.4.1 and 3.5.0

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Tasks

Beta Give feedback

1. Changes are OpenShift compatible
2. All added packages (via microdnf or otherwise) have a comment on why they are added
3. Things not downloaded from Red Hat repositories should be mirrored in the Stackable repository and downloaded from there
4. All packages should have (if available) signatures/hashes verified
5. Add an entry to the CHANGELOG.md file
6. Integration tests ran successfully

TIP: Running integration tests with a new product image

The image can be built and uploaded to the kind cluster with the following commands:

bake --product <product> --image-version <stackable-image-version>
kind load docker-image <image-tagged-with-the-major-version> --name=<name-of-your-test-cluster>

See the output of bake to retrieve the image tag for <image-tagged-with-the-major-version>.

[Maleware]

The PR looks great.

Sadly I can't get a green test for this on arm64:

--- FAIL: kuttl (306.81s)
    --- FAIL: kuttl/harness (0.00s)
        --- FAIL: kuttl/harness/iceberg_openshift-false_spark-3.5.1,docker.stackable.tech_stackable-experimental_spark-k8s_3.5.1-stackable0.0.0-dev (306.79s)
FAIL
ERROR:root:kuttl failed

I will comment later again and approve if we get it fixed

[razvan]

Sadly I can't get a green test for this on arm64:

There was a bug in the test template that is fixed here: stackabletech/spark-k8s-operator@70b5e68