Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tiered Storage config #205

Merged
merged 13 commits into from
Jan 17, 2024
Merged

Tiered Storage config #205

merged 13 commits into from
Jan 17, 2024

Conversation

pellicano
Copy link
Contributor

@pellicano pellicano commented Jan 18, 2022 •
edited

Fixes #72

Motivation

Comply with Tiered Storage configuration.
https://pulsar.apache.org/docs/en/concepts-tiered-storage/
https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/

Modifications

Add a modified Helm Chart version from https://github.com/kafkaesque-io/pulsar-helm-chart.
Other reference Helm Chart: https://github.com/streamnative/charts

Verifying this change

  • Make sure that the change passes the CI checks.

@ckdarby
Copy link

ckdarby commented Jan 18, 2022

Hey @michaeljmarshall! @lhotari & @cdbartholomew mentioned that we should ping you when we made this contribution 👍

Let us know if there is anything missing or what we can do to move this forward!

michaeljmarshall
michaeljmarshall requested changes Jan 19, 2022
View reviewed changes
Copy link
Member

@michaeljmarshall michaeljmarshall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pellicano - thanks for your contribution! I left some minor comments that we should address and then it'll be ready to merge.

charts/pulsar/templates/broker-configmap.yaml Outdated Show resolved Hide resolved
charts/pulsar/templates/broker-statefulset.yaml Show resolved Hide resolved
charts/pulsar/values.yaml Outdated Show resolved Hide resolved
@pellicano
GitHub PR apache#205 changes (1st round)
3c86160 
Remove <= 2.6.0 configs.
Add missing GCS secret volumeMount.
Update GCS example name.
@pellicano
Cleanup comments
8489c3f
@pellicano
Copy link
Contributor Author

Thanks for the comments @michaeljmarshall!
Let me know if there's anything else.

michaeljmarshall
michaeljmarshall approved these changes Jan 20, 2022
View reviewed changes
Copy link
Member

@michaeljmarshall michaeljmarshall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for addressing my feedback, @pellicano.

lhotari
lhotari approved these changes Jan 20, 2022
View reviewed changes
Copy link
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work @pellicano . Thanks for contributing!

@lhotari
Copy link
Member

lhotari commented Jan 20, 2022

@sijie @codelipenghui @eolivelli @addisonj This PR would be a valuable addition to Apache Pulsar Helm chart. Please review.

@lhotari lhotari requested a review from addisonj January 20, 2022 05:57
@sijie
Copy link
Member

sijie commented Jan 20, 2022

@MarvinCai @tuteng can you review this change?

sijie
sijie previously requested changes Jan 20, 2022
View reviewed changes
charts/pulsar/values.yaml Outdated Show resolved Hide resolved
charts/pulsar/values.yaml Outdated Show resolved Hide resolved
@pellicano
GitHub PR apache#205 changes (2nd round)
3216222 
Moved storageOffload under broker section.
Fixed some typos.
Added AWS S3 IRSA annotation comment.
@pellicano
Merge branch 'master' of https://github.com/pellicano/pulsar-helm-chart
bdd67e5
@pellicano
Copy link
Contributor Author

Changes applied as suggested by @sijie. Thank you.
Let me know if there are any other changes.

michaeljmarshall
michaeljmarshall requested changes Jan 21, 2022
View reviewed changes
Copy link
Member

@michaeljmarshall michaeljmarshall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tuteng raises an important security concern, I'm switching my review to request changes until we move the keys to secrets.

charts/pulsar/templates/broker-configmap.yaml Outdated Show resolved Hide resolved
@pellicano
GitHub PR apache#205 changes (3rd round)
6c55955 
Moved AWS and Azure credentials into K8S secrets using same StreamNative Helm Chart approach.
@pellicano
Copy link
Contributor Author

pellicano commented Jan 24, 2022 •
edited

Addressed security concern related to AWS and Azure credentials.
Let me know if there's anything else.

michaeljmarshall
michaeljmarshall approved these changes Jan 24, 2022
View reviewed changes
Copy link
Member

@michaeljmarshall michaeljmarshall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@michaeljmarshall
Copy link
Member

@sijie - PTAL

@pellicano
Copy link
Contributor Author

@sijie if you may take a look, I appreciate. Thank you.

@pellicano
Copy link
Contributor Author

Any news here, @michaeljmarshall?
@sijie if you may please take a look. We appreciate. Thank you.

@ckdarby
Copy link

ckdarby commented Apr 8, 2022

@sijie Can you assign over to someone else to take a look?

@michaeljmarshall
Copy link
Member

@pellicano @ckdarby - are you able to rebase and then take a look at the failing status checks? Once that happens, I think we can merge this. Thanks!

@benlongo
Copy link

If needed I can open a new PR with this rebased.

Merge remote-tracking branch 'upstream/master'
af72886 
# Conflicts:
#	charts/pulsar/Chart.yaml
@michaeljmarshall
Copy link
Member

@pellicano - looks like some of the tests failed with this error:

Error: template: pulsar/templates/broker-statefulset.yaml:260:50: executing "pulsar/templates/broker-statefulset.yaml" at <eq .Values.broker.storageOffload.driver "aws-s3">: error calling eq: incompatible types for comparison

I assume this fails because it's comparing an unset value to a string value.

@lhotari lhotari dismissed sijie’s stale review January 17, 2024 10:44

outdated review

@lhotari lhotari merged commit cfa156f into apache:master Jan 17, 2024
26 checks passed
@lhotari lhotari mentioned this pull request Jan 17, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lhotari lhotari lhotari approved these changes

@tuteng tuteng tuteng approved these changes

@michaeljmarshall michaeljmarshall michaeljmarshall approved these changes

@codelipenghui codelipenghui Awaiting requested review from codelipenghui

@addisonj addisonj Awaiting requested review from addisonj

@MarvinCai MarvinCai Awaiting requested review from MarvinCai

@sijie sijie Awaiting requested review from sijie

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Please add support for Tiered Storage in the Helm Chart
7 participants
@pellicano @ckdarby @lhotari @sijie @michaeljmarshall @benlongo @tuteng