This repository has been archived by the owner on May 3, 2022. It is now read-only.
installation controller: reinstall deleted objects #205
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
juliogreff
force-pushed
the
jgreff/it-reinstall
branch
from
cdf9c1d
to
749661a
Compare
One of the biggest complaints we get from users not being able to finish their rollouts is that their incumbents are unhealthy, and one of the biggest causes for that is when Deployments disappear[1] from the target clusters. This happens because the installation controller does not even try to replace objects after its first pass is completed, and CanOverride is set to false. Now, the installation controller will do a full run of the installer on every sync, reinstalling any objects that are gone, whatever their kind. Additionally, we'll start to listen to Service and Deployment events, so that we can sync the InstallationTarget whenever someone messes with these objects in the target clusters. We do this only for Deployments and Services because they're the only ones that can cause shipper to hang, as it depends on them in the capacity and traffic controller, respectively. [1] Users swear they haven't deleted them. We have yet to find another cause for this mistery.
juliogreff
force-pushed
the
jgreff/it-reinstall
branch
2 times, most recently
from
7ed9c74
to
2076bd6
Compare
osdrv
approved these changes
Oct 7, 2019
parhamdoustdar
approved these changes
Oct 10, 2019
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
One of the biggest complaints we get from users not being able to finish
their rollouts is that their incumbents are unhealthy, and one of the
biggest causes for that is when Deployments disappear[1] from the target
clusters. This happens because the installation controller does not even
try to replace objects after its first pass is completed, and
CanOverride is set to false.
Now, the installation controller will do a full run of the installer on
every sync, reinstalling any objects that are gone, whatever their kind.
Additionally, we'll start to listen to Service and Deployment events, so
that we can sync the InstallationTarget whenever someone messes with
these objects in the target clusters. We do this only for Deployments
and Services because they're the only ones that can cause shipper to
hang, as it depends on them in the capacity and traffic controller,
respectively.
This PR closes #110
[1] Users swear they haven't deleted them. We have yet to find another
cause for this mistery.