Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Merged by Bors] - Add support for Spark 3.4.0 #243

Closed
wants to merge 14 commits into from
Closed

[Merged by Bors] - Add support for Spark 3.4.0 #243

wants to merge 14 commits into from

Conversation

sbernauer
Copy link
Member

@sbernauer sbernauer commented May 22, 2023 •
edited

Description

For #242

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

Edit tasklist title
Beta Give feedback Tasklist Author, more options

Delete tasklist

Delete tasklist block?
Are you sure? All relationships in this tasklist will be removed.
  1. Changes are OpenShift compatible
    Options
  2. CRD changes approved
    Options
  3. Helm chart can be installed and deployed operator works
    Options
  4. Integration tests passed (for non trivial changes)
    Options

Reviewer

Edit tasklist title
Beta Give feedback Tasklist Reviewer, more options

Delete tasklist

Delete tasklist block?
Are you sure? All relationships in this tasklist will be removed.
  1. Code contains useful comments
    Options
  2. (Integration-)Test cases added
    Options
  3. Documentation added or updated
    Options
  4. Changelog updated
    Options
  5. Cargo.toml only contains references to git tags (not specific commits or branches)
    Options

Acceptance

Edit tasklist title
Beta Give feedback Tasklist Acceptance, more options

Delete tasklist

Delete tasklist block?
Are you sure? All relationships in this tasklist will be removed.
  1. Feature Tracker has been updated
    Options
  2. Proper release label has been added
    Options

Once the review is done, comment bors r+ (or bors merge) to merge. Further information

@sbernauer
Fix Spark driver not getting imagePullPolicy set
b7bec6b 
Reported in https://stackable-workspace.slack.com/archives/C037Q9DPF4P/p1682326403038869
@sbernauer
fix tests
e128f3e
@sbernauer
Revert "Fix Spark driver not getting imagePullPolicy set"
0c14413 
This reverts commit b7bec6b.

The commit succeed in telling spark to set the pull policy on the driver main container. Vector container is still wrong.

Everything is a mess.
Everything is optional (event the image and the Spark version!!!).
Wrong image pull secrets and pull policy everywhere.
No ResolvedProductImage (and if used totally messed up).
Needs complete overhaul!
@sbernauer
Rename spark examples
80e9462
@sbernauer
WIP: Switch images to sandbox
3b38d1f
@sbernauer
Remove comment
548b15f
@sbernauer
Fix tests to support test multiple Spark versions
d4aa0b6
@sbernauer
Remove uneeded file
bbc1df1
@sbernauer sbernauer self-assigned this May 23, 2023
@maltesander maltesander self-requested a review May 23, 2023 08:51
maltesander
maltesander reviewed May 23, 2023
View reviewed changes
Copy link
Member

@maltesander maltesander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. Will test a little now.

CHANGELOG.md Outdated Show resolved Hide resolved
CHANGELOG.md Outdated Show resolved Hide resolved
tests/templates/kuttl/logging/05-deploy-automatic-log-config-spark-app.yaml.j2 Outdated Show resolved Hide resolved
sbernauer and others added 4 commits May 23, 2023 12:05
@sbernauer @maltesander
Update CHANGELOG.md
da65f42 
Co-authored-by: Malte Sander <contact@maltesander.com>
@sbernauer @maltesander
Update CHANGELOG.md
44c56c3 
Co-authored-by: Malte Sander <contact@maltesander.com>
@sbernauer
Switch back from sandbox to stackable
413b9ba
@sbernauer
revert pull policy always in tests
a27fa04
maltesander
maltesander requested changes May 23, 2023
View reviewed changes
Copy link
Member

@maltesander maltesander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are many references to 3.3.0-stackable0.3.0 instead e.g. 3.4.0-stackable0.0.0-dev left over.

@sbernauer
Copy link
Member Author

Started test https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/spark-k8s-operator-it-custom/86/

@sbernauer
Copy link
Member Author

Bumped stackable versions.
Test passed as well 🚀

maltesander
maltesander approved these changes May 23, 2023
View reviewed changes
Copy link
Member

@maltesander maltesander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@sbernauer
Copy link
Member Author

bors r+

bors bot pushed a commit that referenced this pull request May 23, 2023
@sbernauer
Add support for Spark 3.4.0 (#243)
f4441de 
# Description

For #242
@bors
Copy link

bors bot commented May 23, 2023

Pull request successfully merged into main.

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

@bors bors bot changed the title Add support for Spark 3.4.0 [Merged by Bors] - Add support for Spark 3.4.0 May 23, 2023
@bors bors bot closed this May 23, 2023
@bors bors bot deleted the spark-3.4.0 branch May 23, 2023 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maltesander maltesander maltesander approved these changes

Assignees

@sbernauer sbernauer

Labels
None yet
Projects
Stackable Engineering
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@sbernauer @maltesander