"minishift openshift restart" right after "minishift start" make router and docker registry unusable

[agajdosi]

General information

• Minishift version: current master
• OS: Linux - CentOS (on centos-ci)
• Hypervisor: KVM

Steps to reproduce

1. minishift start
2. minishift openshift restart
3. eval $(minishift oc-env)
4. oc get pods -n default

Expected

Pods docker-registry and router should be running.

Actual

On some system right after the minishift start the docker-registry and router pods' status is Running, however they are not READY:

NAME                            READY     STATUS    RESTARTS   AGE       IP               NODE
docker-registry-1-4np67         0/1       Running   0          15s       172.17.0.5       localhost
docker-registry-1-deploy        1/1       Running   0          21s       172.17.0.4       localhost
persistent-volume-setup-ndlx2   1/1       Running   0          22s       172.17.0.3       localhost
router-1-c9fmh                  0/1       Running   0          15s       192.168.122.40   localhost
router-1-deploy                 1/1       Running   0          21s       172.17.0.2       localhost

When OpenShift container is restarted in this situation the docker-registry and router containers can't handle it and fails. The failure is however quiet and thus will cause problems later.

Fix?

Docker-registry and router pods should be in Running state and READY after startup. Minishift should check this as part of Minishift start.

Also shouldn't we first prepare the local host node for shutdown, restart and then mark the node ready for deployments? (https://docs.openshift.com/container-platform/3.3/admin_guide/manage_nodes.html#rebooting-nodes)

oc adm drain localhost
restart origin container (minishift openshift restart)
oc adm uncordon localhost
wait until registry and router are ready

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[anjannath]

Still to do..

…

[LalatenduMohanty]

Docker-registry and router pods should be in Running state and READY after startup

@agajdosi How to check if the pods are ready? I guess what you are saying is that "running" state of these containers are enough to say these are ready, right?

[agajdosi]

@LalatenduMohanty TBH I do not fully understand the concept behind the fact oc provides information for STATUS (running, stopped, error...) and READY (0 or 1) for the pods. But it seems that after M start some pods are running, but sometimes are not ready, which causes troubles.

You can try with:

oc get pods -n default --as system:admin

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.