Starting minishift with metrics fails to resolve kubernetes master

[lulf]

When starting minishift with metrics, the metrics deployer pod complains that it cannot resolves the kubernetes master host. When I run 'oc cluster up --metrics', it works as expected.

Environment:
Fedora 25
minishift 1.0.0-beta.5, KVM driver

$ minishift ssh cat /etc/resolv.conf 
nameserver 192.168.124.1

Steps:

1. minishift start --metrics
2. oc login -u system:admin
3. oc logs metrics-deployer-pod -n openshift-infra

The end of the log:

PREFLIGHT CHECK FAILED
========================
validate_master_accessible: 
unable to access master url https://kubernetes.default.svc:443
See the error from 'curl https://kubernetes.default.svc:443' below     for details:
curl: (7) Failed connect to kubernetes.default.svc:443; Connection     refused

Deployment has been aborted prior to starting, as these failures     often indicate fatal problems.
Please evaluate any error messages above and determine how they     can be addressed.
To ignore this validation failure and continue, specify     IGNORE_PREFLIGHT=true.

PREFLIGHT CHECK FAILEPREFLIGHT CHECK FAILED

[hferentschik]

I just provisioned OpenShift with --metrics and it worked fine.

...
-- Installing registry ... OK
-- Installing router ... OK
-- Installing metrics ... OK
...

Also I can access Hawkular under https://metrics-openshift-infra.192.168.99.100.nip.io/hawkular/metrics

@lulf, so you are saying this is repeatable?

[praveenkumar]

I also tried it with centos iso on OSx and it worked as @hferentschik told. It takes some time because it pull different images for metrics and then start but once everything in place it does work.

[lulf]

Yes this is repeatable for me at least. One thing to note is that I had the same issue with 'oc cluster up --metrics' in the past, but resolved it by adding the firewalld zones for docker as described in step 3. in https://github.com/openshift/origin/blob/master/docs/cluster_up_down.md#linux . This somehow allowed the containers to resolve the kubernetes DNS. I'm suspecting something similar is happening inside the minishift vm.

[praveenkumar]

@lulf Finally I did test on the Fedora-25 machine and it did work as expected. I started it with 4096MB ram becaue metrics again run couple of containers. Also once everything deployed it takes couple of mins to metrics be available.

$ cat /etc/redhat-release 
Fedora release 25 (Twenty Five)

$ getenforce 
Enforcing

$ ./minishift start --metrics true --memory 4096
Starting local OpenShift cluster using 'kvm' hypervisor...
Downloading ISO 'https://github.com/minishift/minishift-b2d-iso/releases/download/v1.0.2/minishift-b2d.iso'
 40.00 MB / 40.00 MB [=================================================================================================================================================================================] 100.00% 0s
 22.37 MB / 22.37 MB [=================================================================================================================================================================================] 100.00% 0s
Provisioning OpenShift via '/home/prkumar/.minishift/cache/oc/v1.4.1/oc [cluster up --use-existing-config --host-config-dir /var/lib/minishift/openshift.local.config --host-data-dir /var/lib/minishift/hostdata --host-volumes-dir /var/lib/minishift/openshift.local.volumes --metrics true --routing-suffix 192.168.42.19.nip.io]'
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v1.4.1 image ... 
   Pulling image openshift/origin:v1.4.1
   Pulled 0/3 layers, 3% complete
   Pulled 1/3 layers, 36% complete
   Pulled 2/3 layers, 77% complete
   Pulled 3/3 layers, 100% complete
   Extracting
   Image pull complete
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... OK
-- Checking type of volume mount ... 
   Using Docker shared volumes for OpenShift volumes
-- Creating host directories ... OK
-- Finding server IP ... 
   Using 192.168.42.19 as the server IP
-- Starting OpenShift container ... 
   Creating initial OpenShift configuration
   Starting OpenShift using container 'origin'
   Waiting for API server to start listening
   OpenShift server started
-- Adding default OAuthClient redirect URIs ... OK
-- Installing registry ... OK
-- Installing router ... OK
-- Installing metrics ... OK
-- Importing image streams ... OK
-- Importing templates ... OK
-- Login to server ... OK
-- Creating initial project "myproject" ... OK
-- Removing temporary directory ... OK
-- Server Information ... 
   OpenShift server started.
   The server is accessible via web console at:
       https://192.168.42.19:8443

   The metrics service is available at:
       https://metrics-openshift-infra.192.168.42.19.nip.io

   You are logged in as:
       User:     developer
       Password: developer

   To login as administrator:
       oc login -u system:admin

$ curl -Ik https://metrics-openshift-infra.192.168.42.19.nip.io
HTTP/1.1 200 OK
Last-Modified: Sat, 19 Nov 2016 15:24:37 GMT
X-Powered-By: Undertow/1
Server: WildFly/10
Content-Length: 996
Content-Type: text/html
Accept-Ranges: bytes
Date: Mon, 27 Mar 2017 11:02:08 GMT
Set-Cookie: f5489604a2df0d7b0e8c7b7b0a17e360=f3b775371be11e639d54ae1e7df8cf7a; path=/; HttpOnly; Secure
Cache-control: private

[lulf]

I just tried this instead:

minishift start --metrics --openshift-version v1.5.0-rc.0

And this time the metrics deployer succeeds.

[praveenkumar]

@lulf Finally it worked for you but I still surprise why it is not working with v1.4.1 :(

[jorgemoralespou]

There was a bug on those images and a new version was pushed to dockerhub. Nothing related to minishift. I just tried this instead: minishift start --metrics --openshift-version v1.5.0-rc.0 And this time the metrics deployer succeeds

[lulf]

@jorgemoralespou Do you have a link to the issue?

[jorgemoralespou]

openshift/origin#13533

[coolbrg]

minishift start --metrics --openshift-version v1.5.0-rc.0

The latest release will now require the minimum version of openshift to be 1.5.0-rc.0. PR is added here #628.

[praveenkumar]

I am closing this issue since @lulf issue is resolved and also the issue was with images, nothing from minishift side.