Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Merged by Bors] - Adjust NiFi repository sizes depending on configured PVC sizes #371

Closed
wants to merge 8 commits into from

Conversation

nightkr
Copy link
Member

@nightkr nightkr commented Nov 7, 2022

Description

Fixes #354

Currently this reserves a constant 90% of the PVC size. In practice this headroom "should" probably be scaled based on the complexity on the NiFi graph, but that's not really a thing we can predict easily.

This has been running over the weekend with a test cluster and it has cleaned up and stayed working over that period, with all volumes sitting around their respective fill points.

Review Checklist

  • Code contains useful comments
  • CRD change approved (or not applicable)
  • (Integration-)Test cases added (or not applicable)
  • Documentation added (or not applicable)
  • Changelog updated (or not applicable)
  • Cargo.toml only contains references to git tags (not specific commits or branches)
  • Helm chart can be installed and deployed operator works (or not applicable)

Once the review is done, comment bors r+ (or bors merge) to merge. Further information

@nightkr nightkr marked this pull request as ready for review November 7, 2022 11:26
@nightkr
Copy link
Member Author

nightkr commented Nov 7, 2022

We could do a fixed safety margin for provenance and flowfiles, but NiFi itself only accepts percentages for the content archive. Is it ok to just use percentage offsets for everything? @lfrancke

@nightkr nightkr requested a review from a team November 7, 2022 11:29
@nightkr nightkr self-assigned this Nov 7, 2022
@lfrancke
Copy link
Member

lfrancke commented Nov 7, 2022

I'm fine wtih that

@sbernauer sbernauer mentioned this pull request Nov 7, 2022
32 tasks
Copy link
Member

@sbernauer sbernauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, thanks for adding it to operator-rs and bumping operator-rs!
Feel free to merge after removing patch in Cargo.toml

rust/operator-binary/src/storage_quantity.rs Outdated Show resolved Hide resolved
bors bot pushed a commit to stackabletech/operator-rs that referenced this pull request Nov 8, 2022
## Description

This is used by stackabletech/nifi-operator#371.



Co-authored-by: Teo Klestrup Röijezon <teo.roijezon@stackable.de>
@nightkr
Copy link
Member Author

nightkr commented Nov 8, 2022

bors r+

bors bot pushed a commit that referenced this pull request Nov 8, 2022
# Description

Fixes #354

Currently this reserves a constant 90% of the PVC size. In practice this headroom "should" probably be scaled based on the complexity on the NiFi graph, but that's not really a thing we can predict easily.

This has been running over the weekend with a test cluster and it has cleaned up and stayed working over that period, with all volumes sitting around their respective fill points.
@bors
Copy link
Contributor

bors bot commented Nov 8, 2022

Pull request successfully merged into main.

Build succeeded:

@bors bors bot changed the title Adjust NiFi repository sizes depending on configured PVC sizes [Merged by Bors] - Adjust NiFi repository sizes depending on configured PVC sizes Nov 8, 2022
@bors bors bot closed this Nov 8, 2022
@bors bors bot deleted the feature/dynamic-repo-size branch November 8, 2022 09:08
Comment on lines -234 to -235
default:
capacity: null
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is the accidental breaking change

bors bot pushed a commit that referenced this pull request Nov 10, 2022
# Description

This was introduced in #371, when I fragmentified the cluster resources without adding moving the `#[serde(default)]` attributes into `#[fragment_attrs]` blocks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make repository-sizes dynamic based on pvc sizes
3 participants