ceph: auto grow OSDs size on PVCs #8078
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
travisn
requested changes
Jun 8, 2021
subhamkrai
requested changes
Jun 9, 2021
parth-gr
force-pushed
the
osd-size
branch
6 times, most recently
from
0a12d91
to
b61fb7c
Compare
BlaineEXE
reviewed
Jun 15, 2021
Comment on lines
609
to
616
| For ex: If you need to increase the size of OSD by 30% | ||
| ```console | ||
| Enter the growth rate percentage of OSD: | ||
| 30 | ||
| ``` |
There was a problem hiding this comment.
Why does the script take dynamic input instead of just using flags or arguments which allows someone to use it in an automatic situation? This means that Prometheus won't be able to take actions automatically.
There was a problem hiding this comment.
The script takes the input only for the first time when we start it(to know the growth percentage or count ), and after it automatically triggers actions based on the Prometheus alert, in a time interval of 5 mins.
travisn
requested changes
Jun 15, 2021
| .\increase-size-of-osd.sh size | ||
| ``` | ||
|
|
||
| After running the script it will ask for the `growth rate percentage`. |
There was a problem hiding this comment.
Instead of prompting for the arguments after starting the script, we should read the arguments to the script. For example:
./increase-size-of-osd.sh count --max 12 --growth-rate 3
OR
./increase-size-of-osd.sh size --max 2Ti --growth-rate 10%
leseb
requested changes
Jun 21, 2021
tests/scripts/auto-grow-storage.sh
Outdated
| i=0 | ||
| alertmanagerroute=$(oc get route -n openshift-monitoring | grep alertmanager-main | awk '{ print $2 }') | ||
| curl -sk -H "Authorization: Bearer $(oc sa get-token prometheus-k8s -n openshift-monitoring)" https://${alertmanagerroute}/api/v1/alerts | jq -r '.' >./tt.txt | ||
| export total_alerts=$(cat ./tt.txt | jq '.data | length') |
There was a problem hiding this comment.
Suggested change
| export total_alerts=$(cat ./tt.txt | jq '.data | length') | |
| export total_alerts=$(jq '.data | length' < tt.txt ) |
cat's sole purpose is to concatenate files :)
parth-gr
force-pushed
the
osd-size
branch
3 times, most recently
from
6793874
to
a4ef97c
Compare
parth-gr
force-pushed
the
osd-size
branch
3 times, most recently
from
0253088
to
ed1c35a
Compare
parth-gr
force-pushed
the
osd-size
branch
2 times, most recently
from
60b4670
to
a195977
Compare
parth-gr
force-pushed
the
osd-size
branch
2 times, most recently
from
6653a9e
to
c2f3613
Compare
travisn
requested changes
Aug 19, 2021
travisn
approved these changes
Aug 20, 2021
leseb
requested changes
Aug 31, 2021
When an OSD reaches OSD_NEARFULL state, we have to manually increase the PVC volume claim or manually increase the count of OSDs in the device set Added a script auto-grow-storage.sh which will i)automatically increase claim volume ii)automatically add number of OSDs Closes: rook#6101 Signed-off-by: parth-gr <paarora@redhat.com>
leseb
approved these changes
Sep 1, 2021
sp98
approved these changes
Sep 1, 2021
mergify bot
added a commit
that referenced
this pull request
Sep 1, 2021
ceph: auto grow OSDs size on PVCs (backport #8078)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When an OSDs reach OSD_NEARFULL state, we have to manually either increase the volume claim for provisioning more storage or add more OSDs to the cluster to keep the cluster Healthy.
If you need to automatically increase the provisioned volume(scale OSDs vertically).
Added a script auto-grow-storage.sh which will automatically increase claim volume, depending on the percent growth rate mentioned.
If you need to automatically increase the number of OSDs(scale OSDs Horizontally).
Added a script auto-grow-storage.sh which will automatically increase the OSDs of a device set, depending on the Count you mentioned.
Closes: #6101
Signed-off-by: parth-gr paarora@redhat.com
Tested on different scenarios:
For scaling vertically:
i) When the increment size of PVC becomes more than the max size.
ii) When max Size is labeled with the different units (Mi,Gi,Ti).
For scaling Horizontally:
i) when the increment count becomes more than the disk limit.
Description of your changes:
Which issue is resolved by this Pull Request:
Resolves #
Checklist:
make codegen) has been run to update object specifications, if necessary.