Pb 3707 large resource count support #1345
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lalat-das
requested review from
siva-portworx,
diptiranjanpx,
prashanthpx,
sgajawada-px and
px-kesavan
|
Can one of the admins verify this patch? |
lalat-das
force-pushed
the
pb-3707-largeResourceCountSupport
branch
from
7ed0ed9
to
bd618ee
Compare
lalat-das
force-pushed
the
pb-3707-largeResourceCountSupport
branch
3 times, most recently
from
9100ec9
to
a9e5ccc
Compare
px-kesavan
approved these changes
Mar 29, 2023
lalat-das
force-pushed
the
pb-3707-largeResourceCountSupport
branch
from
a9e5ccc
to
f860e56
Compare
lalat-das
force-pushed
the
pb-3707-largeResourceCountSupport
branch
from
bf0caff
to
ef5b6eb
Compare
lalat-das
force-pushed
the
pb-3707-largeResourceCountSupport
branch
from
0a462a6
to
d29b7f9
Compare
Added two fields to applicationbackup application restore CR spec for Signed-off-by: Lalatendu Das <ldas@purestorage.com>
- if the application backup cr size increase more than threshhold then strip the resource info - update the large resource flag to true - update the resource count so that mongo can refer it in px-backup side Signed-off-by: Lalatendu Das <ldas@purestorage.com>
- if the application restore cr size increase more than threshhold then strip the resource info - update the large resource enable flag to true - update the resource count so that mongo can refer it in px-backup side - Update the timestamp of restore CR when more thann 3000 resources are applied, so that it won't be considered a stale CR and get cleaned up. - Used a temporary list to gather up all the resources, so that timestamp update of CR shouldn't overwhem the GRPC/etcd channel Signed-off-by: Lalatendu Das <ldas@purestorage.com>
lalat-das
force-pushed
the
pb-3707-largeResourceCountSupport
branch
2 times, most recently
from
e909c8a
to
2396bbd
Compare
|
@siva-portworx Can you please verify the delete of go-routine logic that we discussed in the morning |
siva-portworx
approved these changes
Apr 3, 2023
- added a go routine to update the restore CR - In the Restore Resourcei path sending signal on channel to go-routine in every 15 mins - altered the caller of DeleteResources() to pass a nil channel Signed-off-by: Lalatendu Das <ldas@purestorage.com>
lalat-das
force-pushed
the
pb-3707-largeResourceCountSupport
branch
from
2396bbd
to
010705f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this? feature
What this PR does / why we need it: This adds support for taking backup which involves large count of resources. the pb-3696 explains in detail the root cause and its solution.
Does this PR change a user-facing CRD or CLI?: nope
Is a release note needed?: yes
Does this change need to be cherry-picked to a release branch?: no
Unit test detail
Note: the stork changes are vendored to px-backup and some changes in px-backup is done to exercise these unit tests
Backup of 10K resources with single namespace
Size of backup is around 1.7MB
backup succeed
The file shows resource.json 10000 resource
**Multi name space with 35000 resources. CR size almost 7MB
Restore Path Verification is done for below test cases