Skip to content

[Merged by Bors] - Use HDFS discovery configmap for deep storage #262

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 12 commits into from
Closed

[Merged by Bors] - Use HDFS discovery configmap for deep storage #262

wants to merge 12 commits into from

Conversation

maltesander
Copy link
Member

@maltesander maltesander commented Jul 5, 2022 •
edited
Loading

Description

  • HDFS deep storage now configured via discover config map instead of url to namenode
  • Added HDFS deep storage test
  • Increased install-druid cluster timeouts in tests
  • Adapted docs and examples

This is breaking due to a CRD change. Needs discussion?

closes #188

Review Checklist

  • Code contains useful comments
  • (Integration-)Test cases added (or not applicable)
  • Documentation added (or not applicable)
  • Changelog updated (or not applicable)
  • Cargo.toml only contains references to git tags (not specific commits or branches)
  • Helm chart can be installed and deployed operator works (or not applicable)

Once the review is done, comment bors r+ (or bors merge) to merge. Further information

@maltesander maltesander requested a review from a team July 5, 2022 10:54
@maltesander maltesander added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jul 5, 2022
sbernauer
sbernauer reviewed Jul 5, 2022
View reviewed changes
Copy link
Member

@sbernauer sbernauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a full review as i currently did not find the time to review properly, sorry

@maltesander maltesander requested a review from a team July 5, 2022 13:37
@sbernauer sbernauer assigned sbernauer and unassigned fhennig Jul 6, 2022
@maltesander maltesander requested a review from sbernauer July 6, 2022 10:19
sbernauer
sbernauer reviewed Jul 6, 2022
View reviewed changes
Copy link
Member

@sbernauer sbernauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actual change LGTM. I will play around a little with the different CRD versions and see how that works

@sbernauer
Copy link
Member

sbernauer commented Jul 6, 2022 •
edited
Loading

Current problem.
Maybe me kubectl apply is wrong as i have no idea what I'm doing ^^

sbernauer@sbernauer:~/stackabletech/druid-operator$ helm ls
NAME            NAMESPACE       REVISION        UPDATED                                         STATUS          CHART                   APP VERSION
druid-operator  default         1               2022-07-06 14:04:19.422411088 +0200 CEST        deployed        druid-operator-0.6.0    0.6.0      
sbernauer@sbernauer:~/stackabletech/druid-operator$ k get crd druidclusters.druid.stackable.tech -o yaml | grep alpha
  - name: v1alpha1
  - v1alpha1
sbernauer@sbernauer:~/stackabletech/druid-operator$ cargo r crd | kubectl apply -f -
    Finished dev [unoptimized + debuginfo] target(s) in 0.07s
     Running `target/debug/stackable-druid-operator crd`
Warning: resource customresourcedefinitions/druidclusters.druid.stackable.tech is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
The CustomResourceDefinition "druidclusters.druid.stackable.tech" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions

Installing from scratch (when no crd is deployed previously) works fine

@sbernauer
Copy link
Member

Integration test run: https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/druid-operator-it-custom/10/console

@maltesander
Copy link
Member Author

bors merge

bors bot pushed a commit that referenced this pull request Jul 6, 2022
@maltesander @sbernauer
Use HDFS discovery configmap for deep storage (#262)
e335933 
# Description

- HDFS deep storage now configured via discover config map instead of url to namenode
- Added HDFS deep storage test
- Increased install-druid cluster timeouts in tests
- Adapted docs and examples

This is breaking due to a CRD change. Needs discussion?

closes #188



Co-authored-by: Malte Sander <malte.sander.it@gmail.com>
Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de>
@bors
Copy link
Contributor

bors bot commented Jul 6, 2022

Pull request successfully merged into main.

Build succeeded:

@bors bors bot changed the title Use HDFS discovery configmap for deep storage [Merged by Bors] - Use HDFS discovery configmap for deep storage Jul 6, 2022
@bors bors bot closed this Jul 6, 2022
@bors bors bot deleted the feature/hdfs_discovery_config_map branch July 6, 2022 14:47
@adwk67 adwk67 mentioned this pull request Sep 5, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sbernauer sbernauer sbernauer approved these changes

Assignees

@sbernauer sbernauer

Labels
release-note Denotes a PR that will be considered when it comes time to generate release notes.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Use HDFS ConfigMap to get a reference to the HDFS end-point for deep storage
3 participants
@maltesander @sbernauer @fhennig