Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redeployment of an application takes more time #1357

Closed
praveenkumar opened this issue Sep 8, 2017 · 5 comments
Closed

Redeployment of an application takes more time #1357

praveenkumar opened this issue Sep 8, 2017 · 5 comments

Comments

@praveenkumar
Copy link
Contributor

praveenkumar commented Sep 8, 2017

When using CentOS ISO with minishift, redeploy an application takes a lot of time (around 10 minutes) and event logs shows following warnings/errors.
Refer: https://issues.jboss.org/browse/CDK-166

E0906 18:56:30.970930    3220 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/secret/94d1f756-9333-11e7-a3b3-52540022febb-deployer-token-63kch\" (\"94d1f756-9333-11e7-a3b3-52540022febb\")" failed. No retries permitted until 2017-09-06 18:57:02.970907748 +0000 UTC (durationBeforeRetry 32s). 
Error: UnmountVolume.TearDown failed for volume "kubernetes.io/secret/94d1f756-9333-11e7-a3b3-52540022febb-deployer-token-63kch" (volume.spec.Name: "deployer-token-63kch") pod "94d1f756-9333-11e7-a3b3-52540022febb" (UID: "94d1f756-9333-11e7-a3b3-52540022febb") 
with: remove /var/lib/minishift/openshift.local.volumes/pods/94d1f756-9333-11e7-a3b3-52540022febb/volumes/kubernetes.io~secret/deployer-token-63kch: device or resource busy

This seems like a Kubernetes issue : kubernetes/kubernetes#51835

@LalatenduMohanty
Copy link
Member

LalatenduMohanty commented Sep 8, 2017

The root cause of the issue is that Kubernetes tries to delete the POD and associated kubernetes.io~secret but it can not delete it (hence the error device or resource busy). It only happens when oc cluster up access the config/data/volume directories through symbolic links.
Here are the symbolic links which Minishift uses as we use the persistent storage partition.

hostConfigDirectory  = "/var/lib/minishift/openshift.local.config"
hostDataDirectory    = "/var/lib/minishift/hostdata"
hostVolumesDirectory = "/var/lib/minishift/openshift.local.volumes"
hostPvDirectory      = "/var/lib/minishift/openshift.local.pv"

Workaround:

Provide following flags to minishift start as

--host-config-dir /mnt/sda1/var/lib/minishift/openshift.local.config \
--host-data-dir /mnt/sda1/var/lib/minishift/hostdata \
--host-pv-dir /mnt/sda1/var/lib/minishift/openshift.local.pv \
--host-volumes-dir /mnt/sda1/var/lib/minishift/openshift.local.volumes

@LalatenduMohanty LalatenduMohanty changed the title Redeployment of a application take more time than expected (device or resource busy) Redeployment of an application take more time Sep 8, 2017
@LalatenduMohanty LalatenduMohanty changed the title Redeployment of an application take more time Redeployment of an application takes more time Sep 8, 2017
@stale
Copy link

stale bot commented Nov 7, 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the status/stale label Nov 7, 2017
@coolbrg
Copy link
Contributor

coolbrg commented Nov 8, 2017

still to do

@stale stale bot removed the status/stale label Nov 8, 2017
@stale
Copy link

stale bot commented Jan 7, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the status/stale label Jan 7, 2018
@stale stale bot removed the status/stale label Jan 9, 2018
@LalatenduMohanty LalatenduMohanty added this to the v1.13.0 milestone Jan 12, 2018
@praveenkumar
Copy link
Contributor Author

Since it resolved downstream because of the new version of Openshift. I am closing this now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants