openshift-web-console pod fails to start due to Error: User "system:serviceaccount:openshift-web-console:webconsole" cannot get clusterservicebrokers.servicecatalog.k8s.io at the cluster scope: User "system:serviceaccount:openshift-web-console:webconsole" cannot get clusterservicebrokers.servicecatalog.k8s.io at the cluster scope

[lucastheisen]

I am using openshift-ansible to provision my cluster. It seems to be happy up until it attempts to Verify that the web console is running where it fails:

fatal: [openshift1.pd.labs.mitre.org]: FAILED! => {
    "attempts": 120,
    "changed": false,
    "cmd": [
        "curl",
        "-k",
        "https://webconsole.openshift-web-console.svc/healthz"
    ],
    "delta": "0:00:01.015458",
    "end": "2018-01-18 17:40:48.117393",
    "invocation": {
        "module_args": {
            "_raw_params": "curl -k https://webconsole.openshift-web-console.svc/healthz",
            "_uses_shell": false,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": false
        }
    },
    "msg": "non-zero return code",
    "rc": 7,
    "start": "2018-01-18 17:40:47.101935",
    "stderr": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to webconsole.openshift-web-console.svc:443; Connection refused",
    "stderr_lines": [
        "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current",
        "                                 Dload  Upload   Total   Spent    Left  Speed",
        "",
        "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to webconsole.openshift-web-console.svc:443; Connection refused"
    ],
    "stdout": "",
    "stdout_lines": []
}

However, when i check in the web console from a browser using my public hostname, the app is live and I can log in as an administrator, switch to the openshift-web-console project, and check its pods. I see one pod for webconsole that fails to start with the following in its log:

W0118 23:10:54.087746       1 start.go:92] Warning: config.loggingPublicURL: Invalid value: "": required to view aggregated container logs in the console, web console start will continue.
W0118 23:10:54.123610       1 start.go:92] Warning: config.metricsPublicURL: Invalid value: "": required to view cluster metrics in the console, web console start will continue.
Error: User "system:serviceaccount:openshift-web-console:webconsole" cannot get clusterservicebrokers.servicecatalog.k8s.io at the cluster scope: User "system:serviceaccount:openshift-web-console:webconsole" cannot get clusterservicebrokers.servicecatalog.k8s.io at the cluster scope
Usage:
  origin-web-console [flags]
Flags:
      --alsologtostderr                                log to standard error as well as files
      --audit-log-format string                        Format of saved audits. "legacy" indicates 1-line text format for each event. "json" indicates structured json format. Requires the 'AdvancedAuditing' feature gate. Known formats are legacy,json. (default "json")
      --audit-log-maxage int                           The maximum number of days to retain old audit log files based on the timestamp encoded in their filename.
      --audit-log-maxbackup int                        The maximum number of old audit log files to retain.
      --audit-log-maxsize int                          The maximum size in megabytes of the audit log file before it gets rotated.
      --audit-log-path string                          If set, all requests coming to the apiserver will be logged to this file.  '-' means standard out.
      --audit-policy-file string                       Path to the file that defines the audit policy configuration. Requires the 'AdvancedAuditing' feature gate. With AdvancedAuditing, a profile is required to enable auditing.
      --audit-webhook-batch-buffer-size int            The size of the buffer to store events before batching and sending to the webhook. Only used in batch mode. (default 10000)
      --audit-webhook-batch-initial-backoff duration   The amount of time to wait before retrying the first failed requests. Only used in batch mode. (default 10s)
      --audit-webhook-batch-max-size int               The maximum size of a batch sent to the webhook. Only used in batch mode. (default 400)
      --audit-webhook-batch-max-wait duration          The amount of time to wait before force sending the batch that hadn't reached the max size. Only used in batch mode. (default 30s)
      --audit-webhook-batch-throttle-burst int         Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode. (default 15)
      --audit-webhook-batch-throttle-qps float32       Maximum average number of requests per second. Only used in batch mode. (default 10)
      --audit-webhook-config-file string               Path to a kubeconfig formatted file that defines the audit webhook configuration. Requires the 'AdvancedAuditing' feature gate.
      --audit-webhook-mode string                      Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the webhook to buffer and send events asynchronously. Known modes are batch,blocking. (default "batch")
      --config string                                  filename containing the WebConsoleConfig
      --contention-profiling                           Enable lock contention profiling, if profiling is enabled
      --enable-swagger-ui                              Enables swagger ui on the apiserver at /swagger-ui
      --log-flush-frequency duration                   Maximum number of seconds between log flushes (default 5s)
      --log_backtrace_at traceLocation                 when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                                 If non-empty, write log files in this directory
      --logtostderr                                    log to standard error instead of files (default true)
      --profiling                                      Enable profiling via web interface host:port/debug/pprof/ (default true)
      --stderrthreshold severity                       logs at or above this threshold go to stderr (default 2)
  -v, --v Level                                        log level for V logs
      --vmodule moduleSpec                             comma-separated list of pattern=N settings for file-filtered logging
F0118 23:10:55.152343 1 console.go:35] User "system:serviceaccount:openshift-web-console:webconsole" cannot get clusterservicebrokers.servicecatalog.k8s.io at the cluster scope: User "system:serviceaccount:openshift-web-console:webconsole" cannot get clusterservicebrokers.servicecatalog.k8s.io at the cluster scope

Version

ltheisen@openshift1.pd.labs.mitre.org(Cent7) ~$ oc version
oc v3.7.0+7ed6862
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

Steps To Reproduce

1. I just ran the openshift-ansible with my config and this happens
   This is completely uncustomized, but this is the deployment config that is getting used:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: '1'
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"apps/v1beta1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"openshift-web-console","webconsole":"true"},"name":"webconsole","namespace":"openshift-web-console"},"spec":{"replicas":1,"strategy":{"type":"Recreate"},"template":{"metadata":{"labels":{"webconsole":"true"},"name":"webconsole"},"spec":{"containers":[{"command":["/usr/bin/origin-web-console","--audit-log-path=-","-v=0","--config=/var/webconsole-config/webconsole-config.yaml"],"image":"docker.io/openshift/origin-web-console:latest","imagePullPolicy":"IfNotPresent","livenessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS"}},"name":"webconsole","ports":[{"containerPort":8443}],"readinessProbe":{"httpGet":{"path":"/healthz","port":8443,"scheme":"HTTPS"}},"volumeMounts":[{"mountPath":"/var/serving-cert","name":"serving-cert"},{"mountPath":"/var/webconsole-config","name":"webconsole-config"}]}],"nodeSelector":{"region":"infra"},"serviceAccountName":"webconsole","volumes":[{"name":"serving-cert","secret":{"defaultMode":400,"secretName":"webconsole-serving-cert"}},{"configMap":{"defaultMode":440,"name":"webconsole-config"},"name":"webconsole-config"}]}}}}
  creationTimestamp: '2018-01-18T20:44:27Z'
  generation: 1
  labels:
    app: openshift-web-console
    webconsole: 'true'
  name: webconsole
  namespace: openshift-web-console
  resourceVersion: '28163'
  selfLink: /apis/apps/v1beta1/namespaces/openshift-web-console/deployments/webconsole
  uid: 604b2497-fc90-11e7-8cfe-001a4a1d019b
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      webconsole: 'true'
  strategy:
    type: Recreate
  template:
    metadata:
      creationTimestamp: null
      labels:
        webconsole: 'true'
      name: webconsole
    spec:
      containers:
        - command:
            - /usr/bin/origin-web-console
            - '--audit-log-path=-'
            - '-v=0'
            - '--config=/var/webconsole-config/webconsole-config.yaml'
          image: 'docker.io/openshift/origin-web-console:latest'
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /
              port: 8443
              scheme: HTTPS
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          name: webconsole
          ports:
            - containerPort: 8443
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 8443
              scheme: HTTPS
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /var/serving-cert
              name: serving-cert
            - mountPath: /var/webconsole-config
              name: webconsole-config
      dnsPolicy: ClusterFirst
      nodeSelector:
        region: infra
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: webconsole
      serviceAccountName: webconsole
      terminationGracePeriodSeconds: 30
      volumes:
        - name: serving-cert
          secret:
            defaultMode: 400
            secretName: webconsole-serving-cert
        - configMap:
            defaultMode: 440
            name: webconsole-config
          name: webconsole-config
status:
  conditions:
    - lastTransitionTime: '2018-01-18T20:44:27Z'
      lastUpdateTime: '2018-01-18T20:44:27Z'
      message: Deployment does not have minimum availability.
      reason: MinimumReplicasUnavailable
      status: 'False'
      type: Available
    - lastTransitionTime: '2018-01-18T20:54:29Z'
      lastUpdateTime: '2018-01-18T20:54:29Z'
      message: ReplicaSet "webconsole-2959644228" has timed out progressing.
      reason: ProgressDeadlineExceeded
      status: 'False'
      type: Progressing
  observedGeneration: 1
  replicas: 1
  unavailableReplicas: 1
  updatedReplicas: 1

Current Result

Error: User "system:serviceaccount:openshift-web-console:webconsole" cannot get clusterservicebrokers.servicecatalog.k8s.io at the cluster scope: User "system:serviceaccount:openshift-web-console:webconsole" cannot get clusterservicebrokers.servicecatalog.k8s.io at the cluster scope

Expected Result

The pod starts successfully

Additional Information

root@openshift1.pd.labs.mitre.org(Cent7) ~$ oc adm diagnostics
[Note] Determining if client configuration exists for client/cluster diagnostics
Info:  Successfully read a client config file at '/root/.kube/config'
Info:  Using context for cluster-admin access: 'default/openshift1-pd-labs-mitre-org:8443/system:admin'
[Note] Performing systemd discovery

[Note] Running diagnostic: ConfigContexts[default/openshift1-pd-labs-mitre-org:8443/system:admin]
       Description: Validate client config context is complete and has connectivity

Info:  The current client config context is 'default/openshift1-pd-labs-mitre-org:8443/system:admin':
       The server URL is 'https://openshift1.pd.labs.mitre.org:8443'
       The user authentication is 'system:admin/openshift1-pd-labs-mitre-org:8443'
       The current project is 'default'
       Successfully requested project list; has access to project(s):
         [default kube-public kube-system logging management-infra openshift openshift-infra openshift-node openshift-web-console]

[Note] Running diagnostic: DiagnosticPod
       Description: Create a pod to run diagnostics from the application standpoint

Info:  Output from the diagnostic pod (image openshift/origin-deployer:v3.7.0):
       [Note] Running diagnostic: PodCheckAuth
              Description: Check that service account credentials authenticate as expected

       Info:  Service account token successfully authenticated to master
       Info:  Service account token was authenticated by the integrated registry.
       [Note] Running diagnostic: PodCheckDns
              Description: Check that DNS within a pod works as expected

       [Note] Summary of diagnostics execution (version v3.7.0+7ed6862):
       [Note] Completed with no errors or warnings seen.

[Note] Running diagnostic: NetworkCheck
       Description: Create a pod on all schedulable nodes and run network diagnostics from the application standpoint

root@openshift1.pd.labs.mitre.org(Cent7) ~$ oc get all -o json -n openshift-web-console
{
    "apiVersion": "v1",
    "items": [
        {
            "apiVersion": "v1",
            "kind": "Pod",
            "metadata": {
                "annotations": {
                    "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"openshift-web-console\",\"name\":\"webconsole-2959644228\",\"uid\":\"604c9589-fc90-11e7-8cfe-001a4a1d019b\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"27060\"}}\n",
                    "openshift.io/scc": "restricted"
                },
                "creationTimestamp": "2018-01-18T20:44:29Z",
                "generateName": "webconsole-2959644228-",
                "labels": {
                    "pod-template-hash": "2959644228",
                    "webconsole": "true"
                },
                "name": "webconsole-2959644228-h9psl",
                "namespace": "openshift-web-console",
                "ownerReferences": [
                    {
                        "apiVersion": "extensions/v1beta1",
                        "blockOwnerDeletion": true,
                        "controller": true,
                        "kind": "ReplicaSet",
                        "name": "webconsole-2959644228",
                        "uid": "604c9589-fc90-11e7-8cfe-001a4a1d019b"
                    }
                ],
                "resourceVersion": "45015",
                "selfLink": "/api/v1/namespaces/openshift-web-console/pods/webconsole-2959644228-h9psl",
                "uid": "60f189a0-fc90-11e7-8cfe-001a4a1d019b"
            },
            "spec": {
                "containers": [
                    {
                        "command": [
                            "/usr/bin/origin-web-console",
                            "--audit-log-path=-",
                            "-v=0",
                            "--config=/var/webconsole-config/webconsole-config.yaml"
                        ],
                        "image": "docker.io/openshift/origin-web-console:latest",
                        "imagePullPolicy": "IfNotPresent",
                        "livenessProbe": {
                            "failureThreshold": 3,
                            "httpGet": {
                                "path": "/",
                                "port": 8443,
                                "scheme": "HTTPS"
                            },
                            "periodSeconds": 10,
                            "successThreshold": 1,
                            "timeoutSeconds": 1
                        },
                        "name": "webconsole",
                        "ports": [
                            {
                                "containerPort": 8443,
                                "protocol": "TCP"
                            }
                        ],
                        "readinessProbe": {
                            "failureThreshold": 3,
                            "httpGet": {
                                "path": "/healthz",
                                "port": 8443,
                                "scheme": "HTTPS"
                            },
                            "periodSeconds": 10,
                            "successThreshold": 1,
                            "timeoutSeconds": 1
                        },
                        "resources": {},
                        "securityContext": {
                            "capabilities": {
                                "drop": [
                                    "KILL",
                                    "MKNOD",
                                    "SETGID",
                                    "SETUID"
                                ]
                            },
                            "privileged": false,
                            "runAsUser": 1000080000,
                            "seLinuxOptions": {
                                "level": "s0:c9,c4"
                            }
                        },
                        "terminationMessagePath": "/dev/termination-log",
                        "terminationMessagePolicy": "File",
                        "volumeMounts": [
                            {
                                "mountPath": "/var/serving-cert",
                                "name": "serving-cert"
                            },
                            {
                                "mountPath": "/var/webconsole-config",
                                "name": "webconsole-config"
                            },
                            {
                                "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
                                "name": "webconsole-token-8mb77",
                                "readOnly": true
                            }
                        ]
                    }
                ],
                "dnsPolicy": "ClusterFirst",
                "imagePullSecrets": [
                    {
                        "name": "webconsole-dockercfg-gv92r"
                    }
                ],
                "nodeName": "openshift4.pd.labs.mitre.org",
                "nodeSelector": {
                    "region": "infra"
                },
                "restartPolicy": "Always",
                "schedulerName": "default-scheduler",
                "securityContext": {
                    "fsGroup": 1000080000,
                    "seLinuxOptions": {
                        "level": "s0:c9,c4"
                    }
                },
                "serviceAccount": "webconsole",
                "serviceAccountName": "webconsole",
                "terminationGracePeriodSeconds": 30,
                "volumes": [
                    {
                        "name": "serving-cert",
                        "secret": {
                            "defaultMode": 400,
                            "secretName": "webconsole-serving-cert"
                        }
                    },
                    {
                        "configMap": {
                            "defaultMode": 440,
                            "name": "webconsole-config"
                        },
                        "name": "webconsole-config"
                    },
                    {
                        "name": "webconsole-token-8mb77",
                        "secret": {
                            "defaultMode": 420,
                            "secretName": "webconsole-token-8mb77"
                        }
                    }
                ]
            },
            "status": {
                "conditions": [
                    {
                        "lastProbeTime": null,
                        "lastTransitionTime": "2018-01-18T20:44:29Z",
                        "status": "True",
                        "type": "Initialized"
                    },
                    {
                        "lastProbeTime": null,
                        "lastTransitionTime": "2018-01-18T20:44:29Z",
                        "message": "containers with unready status: [webconsole]",
                        "reason": "ContainersNotReady",
                        "status": "False",
                        "type": "Ready"
                    },
                    {
                        "lastProbeTime": null,
                        "lastTransitionTime": "2018-01-18T20:44:29Z",
                        "status": "True",
                        "type": "PodScheduled"
                    }
                ],
                "containerStatuses": [
                    {
                        "containerID": "docker://5a6ce6469b2d1813b74aa6043f4304791d6a6e5276c4f1374e7dd991d0fe6b61",
                        "image": "docker.io/openshift/origin-web-console:latest",
                        "imageID": "docker-pullable://docker.io/openshift/origin-web-console@sha256:6802f6c63043467ca54a4736ebdeb4e3940405e9fc8ffbc2c7cbc5b426c81c3d",
                        "lastState": {
                            "terminated": {
                                "containerID": "docker://5a6ce6469b2d1813b74aa6043f4304791d6a6e5276c4f1374e7dd991d0fe6b61",
                                "exitCode": 255,
                                "finishedAt": "2018-01-18T23:36:43Z",
                                "reason": "Error",
                                "startedAt": "2018-01-18T23:36:43Z"
                            }
                        },
                        "name": "webconsole",
                        "ready": false,
                        "restartCount": 37,
                        "state": {
                            "waiting": {
                                "message": "Back-off 5m0s restarting failed container=webconsole pod=webconsole-2959644228-h9psl_openshift-web-console(60f189a0-fc90-11e7-8cfe-001a4a1d019b)",
                                "reason": "CrashLoopBackOff"
                            }
                        }
                    }
                ],
                "hostIP": "128.29.35.60",
                "phase": "Running",
                "podIP": "10.131.0.9",
                "qosClass": "BestEffort",
                "startTime": "2018-01-18T20:44:29Z"
            }
        },
        {
            "apiVersion": "v1",
            "kind": "Service",
            "metadata": {
                "annotations": {
                    "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{\"service.alpha.openshift.io/serving-cert-secret-name\":\"webconsole-serving-cert\"},\"labels\":{\"app\":\"openshift-web-console\"},\"name\":\"webconsole\",\"namespace\":\"openshift-web-console\"},\"spec\":{\"ports\":[{\"name\":\"https\",\"port\":443,\"targetPort\":8443}],\"selector\":{\"webconsole\":\"true\"}}}\n",
                    "service.alpha.openshift.io/serving-cert-secret-name": "webconsole-serving-cert",
                    "service.alpha.openshift.io/serving-cert-signed-by": "openshift-service-serving-signer@1516228937"
                },
                "creationTimestamp": "2018-01-18T20:44:28Z",
                "labels": {
                    "app": "openshift-web-console"
                },
                "name": "webconsole",
                "namespace": "openshift-web-console",
                "resourceVersion": "27077",
                "selfLink": "/api/v1/namespaces/openshift-web-console/services/webconsole",
                "uid": "605b6a37-fc90-11e7-8cfe-001a4a1d019b"
            },
            "spec": {
                "clusterIP": "172.30.123.247",
                "ports": [
                    {
                        "name": "https",
                        "port": 443,
                        "protocol": "TCP",
                        "targetPort": 8443
                    }
                ],
                "selector": {
                    "webconsole": "true"
                },
                "sessionAffinity": "None",
                "type": "ClusterIP"
            },
            "status": {
                "loadBalancer": {}
            }
        },
        {
            "apiVersion": "extensions/v1beta1",
            "kind": "Deployment",
            "metadata": {
                "annotations": {
                    "deployment.kubernetes.io/revision": "1",
                    "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"apps/v1beta1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"openshift-web-console\",\"webconsole\":\"true\"},\"name\":\"webconsole\",\"namespace\":\"openshift-web-console\"},\"spec\":{\"replicas\":1,\"strategy\":{\"type\":\"Recreate\"},\"template\":{\"metadata\":{\"labels\":{\"webconsole\":\"true\"},\"name\":\"webconsole\"},\"spec\":{\"containers\":[{\"command\":[\"/usr/bin/origin-web-console\",\"--audit-log-path=-\",\"-v=0\",\"--config=/var/webconsole-config/webconsole-config.yaml\"],\"image\":\"docker.io/openshift/origin-web-console:latest\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"httpGet\":{\"path\":\"/\",\"port\":8443,\"scheme\":\"HTTPS\"}},\"name\":\"webconsole\",\"ports\":[{\"containerPort\":8443}],\"readinessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8443,\"scheme\":\"HTTPS\"}},\"volumeMounts\":[{\"mountPath\":\"/var/serving-cert\",\"name\":\"serving-cert\"},{\"mountPath\":\"/var/webconsole-config\",\"name\":\"webconsole-config\"}]}],\"nodeSelector\":{\"region\":\"infra\"},\"serviceAccountName\":\"webconsole\",\"volumes\":[{\"name\":\"serving-cert\",\"secret\":{\"defaultMode\":400,\"secretName\":\"webconsole-serving-cert\"}},{\"configMap\":{\"defaultMode\":440,\"name\":\"webconsole-config\"},\"name\":\"webconsole-config\"}]}}}}\n"
                },
                "creationTimestamp": "2018-01-18T20:44:27Z",
                "generation": 1,
                "labels": {
                    "app": "openshift-web-console",
                    "webconsole": "true"
                },
                "name": "webconsole",
                "namespace": "openshift-web-console",
                "resourceVersion": "28163",
                "selfLink": "/apis/extensions/v1beta1/namespaces/openshift-web-console/deployments/webconsole",
                "uid": "604b2497-fc90-11e7-8cfe-001a4a1d019b"
            },
            "spec": {
                "progressDeadlineSeconds": 600,
                "replicas": 1,
                "revisionHistoryLimit": 2,
                "selector": {
                    "matchLabels": {
                        "webconsole": "true"
                    }
                },
                "strategy": {
                    "type": "Recreate"
                },
                "template": {
                    "metadata": {
                        "creationTimestamp": null,
                        "labels": {
                            "webconsole": "true"
                        },
                        "name": "webconsole"
                    },
                    "spec": {
                        "containers": [
                            {
                                "command": [
                                    "/usr/bin/origin-web-console",
                                    "--audit-log-path=-",
                                    "-v=0",
                                    "--config=/var/webconsole-config/webconsole-config.yaml"
                                ],
                                "image": "docker.io/openshift/origin-web-console:latest",
                                "imagePullPolicy": "IfNotPresent",
                                "livenessProbe": {
                                    "failureThreshold": 3,
                                    "httpGet": {
                                        "path": "/",
                                        "port": 8443,
                                        "scheme": "HTTPS"
                                    },
                                    "periodSeconds": 10,
                                    "successThreshold": 1,
                                    "timeoutSeconds": 1
                                },
                                "name": "webconsole",
                                "ports": [
                                    {
                                        "containerPort": 8443,
                                        "protocol": "TCP"
                                    }
                                ],
                                "readinessProbe": {
                                    "failureThreshold": 3,
                                    "httpGet": {
                                        "path": "/healthz",
                                        "port": 8443,
                                        "scheme": "HTTPS"
                                    },
                                    "periodSeconds": 10,
                                    "successThreshold": 1,
                                    "timeoutSeconds": 1
                                },
                                "resources": {},
                                "terminationMessagePath": "/dev/termination-log",
                                "terminationMessagePolicy": "File",
                                "volumeMounts": [
                                    {
                                        "mountPath": "/var/serving-cert",
                                        "name": "serving-cert"
                                    },
                                    {
                                        "mountPath": "/var/webconsole-config",
                                        "name": "webconsole-config"
                                    }
                                ]
                            }
                        ],
                        "dnsPolicy": "ClusterFirst",
                        "nodeSelector": {
                            "region": "infra"
                        },
                        "restartPolicy": "Always",
                        "schedulerName": "default-scheduler",
                        "securityContext": {},
                        "serviceAccount": "webconsole",
                        "serviceAccountName": "webconsole",
                        "terminationGracePeriodSeconds": 30,
                        "volumes": [
                            {
                                "name": "serving-cert",
                                "secret": {
                                    "defaultMode": 400,
                                    "secretName": "webconsole-serving-cert"
                                }
                            },
                            {
                                "configMap": {
                                    "defaultMode": 440,
                                    "name": "webconsole-config"
                                },
                                "name": "webconsole-config"
                            }
                        ]
                    }
                }
            },
            "status": {
                "conditions": [
                    {
                        "lastTransitionTime": "2018-01-18T20:44:27Z",
                        "lastUpdateTime": "2018-01-18T20:44:27Z",
                        "message": "Deployment does not have minimum availability.",
                        "reason": "MinimumReplicasUnavailable",
                        "status": "False",
                        "type": "Available"
                    },
                    {
                        "lastTransitionTime": "2018-01-18T20:54:29Z",
                        "lastUpdateTime": "2018-01-18T20:54:29Z",
                        "message": "ReplicaSet \"webconsole-2959644228\" has timed out progressing.",
                        "reason": "ProgressDeadlineExceeded",
                        "status": "False",
                        "type": "Progressing"
                    }
                ],
                "observedGeneration": 1,
                "replicas": 1,
                "unavailableReplicas": 1,
                "updatedReplicas": 1
            }
        },
        {
            "apiVersion": "extensions/v1beta1",
            "kind": "ReplicaSet",
            "metadata": {
                "annotations": {
                    "deployment.kubernetes.io/desired-replicas": "1",
                    "deployment.kubernetes.io/max-replicas": "1",
                    "deployment.kubernetes.io/revision": "1"
                },
                "creationTimestamp": "2018-01-18T20:44:27Z",
                "generation": 1,
                "labels": {
                    "pod-template-hash": "2959644228",
                    "webconsole": "true"
                },
                "name": "webconsole-2959644228",
                "namespace": "openshift-web-console",
                "ownerReferences": [
                    {
                        "apiVersion": "extensions/v1beta1",
                        "blockOwnerDeletion": true,
                        "controller": true,
                        "kind": "Deployment",
                        "name": "webconsole",
                        "uid": "604b2497-fc90-11e7-8cfe-001a4a1d019b"
                    }
                ],
                "resourceVersion": "27084",
                "selfLink": "/apis/extensions/v1beta1/namespaces/openshift-web-console/replicasets/webconsole-2959644228",
                "uid": "604c9589-fc90-11e7-8cfe-001a4a1d019b"
            },
            "spec": {
                "replicas": 1,
                "selector": {
                    "matchLabels": {
                        "pod-template-hash": "2959644228",
                        "webconsole": "true"
                    }
                },
                "template": {
                    "metadata": {
                        "creationTimestamp": null,
                        "labels": {
                            "pod-template-hash": "2959644228",
                            "webconsole": "true"
                        },
                        "name": "webconsole"
                    },
                    "spec": {
                        "containers": [
                            {
                                "command": [
                                    "/usr/bin/origin-web-console",
                                    "--audit-log-path=-",
                                    "-v=0",
                                    "--config=/var/webconsole-config/webconsole-config.yaml"
                                ],
                                "image": "docker.io/openshift/origin-web-console:latest",
                                "imagePullPolicy": "IfNotPresent",
                                "livenessProbe": {
                                    "failureThreshold": 3,
                                    "httpGet": {
                                        "path": "/",
                                        "port": 8443,
                                        "scheme": "HTTPS"
                                    },
                                    "periodSeconds": 10,
                                    "successThreshold": 1,
                                    "timeoutSeconds": 1
                                },
                                "name": "webconsole",
                                "ports": [
                                    {
                                        "containerPort": 8443,
                                        "protocol": "TCP"
                                    }
                                ],
                                "readinessProbe": {
                                    "failureThreshold": 3,
                                    "httpGet": {
                                        "path": "/healthz",
                                        "port": 8443,
                                        "scheme": "HTTPS"
                                    },
                                    "periodSeconds": 10,
                                    "successThreshold": 1,
                                    "timeoutSeconds": 1
                                },
                                "resources": {},
                                "terminationMessagePath": "/dev/termination-log",
                                "terminationMessagePolicy": "File",
                                "volumeMounts": [
                                    {
                                        "mountPath": "/var/serving-cert",
                                        "name": "serving-cert"
                                    },
                                    {
                                        "mountPath": "/var/webconsole-config",
                                        "name": "webconsole-config"
                                    }
                                ]
                            }
                        ],
                        "dnsPolicy": "ClusterFirst",
                        "nodeSelector": {
                            "region": "infra"
                        },
                        "restartPolicy": "Always",
                        "schedulerName": "default-scheduler",
                        "securityContext": {},
                        "serviceAccount": "webconsole",
                        "serviceAccountName": "webconsole",
                        "terminationGracePeriodSeconds": 30,
                        "volumes": [
                            {
                                "name": "serving-cert",
                                "secret": {
                                    "defaultMode": 400,
                                    "secretName": "webconsole-serving-cert"
                                }
                            },
                            {
                                "configMap": {
                                    "defaultMode": 440,
                                    "name": "webconsole-config"
                                },
                                "name": "webconsole-config"
                            }
                        ]
                    }
                }
            },
            "status": {
                "fullyLabeledReplicas": 1,
                "observedGeneration": 1,
                "replicas": 1
            }
        }
    ],
    "kind": "List",
    "metadata": {
        "resourceVersion": "",
        "selfLink": ""
    }
}

[visit https://docs.openshift.org/latest/welcome/index.html]

[aizuddin85]

are you able to query oc get scc as system:admin user?

[lucastheisen]

@aizuddin85 , from the master node it works just fine:

openshift1.pd.labs.mitre.org | SUCCESS | rc=0 >>
NAME               PRIV      CAPS      SELINUX     RUNASUSER          FSGROUP     SUPGROUP    PRIORITY   READONLYROOTFS   VOLUMES
anyuid             false     []        MustRunAs   RunAsAny           RunAsAny    RunAsAny    10         false            [configMap downwardAPI emptyDir persistentVolumeClaim projected secret]
hostaccess         false     []        MustRunAs   MustRunAsRange     MustRunAs   RunAsAny    <none>     false            [configMap downwardAPI emptyDir hostPath persistentVolumeClaim projected secret]
hostmount-anyuid   false     []        MustRunAs   RunAsAny           RunAsAny    RunAsAny    <none>     false            [configMap downwardAPI emptyDir hostPath nfs persistentVolumeClaim projected secret]
hostnetwork        false     []        MustRunAs   MustRunAsRange     MustRunAs   MustRunAs   <none>     false            [configMap downwardAPI emptyDir persistentVolumeClaim projected secret]
nonroot            false     []        MustRunAs   MustRunAsNonRoot   RunAsAny    RunAsAny    <none>     false            [configMap downwardAPI emptyDir persistentVolumeClaim projected secret]
privileged         true      [*]       RunAsAny    RunAsAny           RunAsAny    RunAsAny    <none>     false            [*]
restricted         false     []        MustRunAs   MustRunAsRange     MustRunAs   RunAsAny    <none>     false            [configMap downwardAPI emptyDir persistentVolumeClaim projected secret]

From any other node it fails (which i believe is expected)...

[lucastheisen]

Just pulled the latest openshift-ansible today... Seems resolved. Perhaps it was resolved by this commit

[missnebun]

I have the same problem. I got the latest openshift-ansible. I run the installer few times. Same problem. Not sure what is wrong ...

[missnebun]

[root@nyvm3103 ~]# curl -k https://webconsole.openshift-web-console.svc/healthz
curl: (7) Failed connect to webconsole.openshift-web-console.svc:443; Connection refused

[missnebun]

I have 3 Masters 3 Infra and 3 DataNodes with openshift_master_cluster_method=native.
Looks like the web console is trying to run in master

    "nodeSelector": {
        "node-role.kubernetes.io/master": "true"
    },

Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message

---

8m 2m 26 default-scheduler Warning FailedScheduling 0/6 nodes are available: 6 MatchNodeSelector.

Not sure variable I have to set for the web_console to run in to infra nodes ?

[missnebun]

Found the problem ...
Looks like @spadgett changed openshift_web_console_nodeselector variable to
openshift_web_console_nodeselector: {"node-role.kubernetes.io/master":"true"} . But if you have dedicated masters and Scheduling is Disabled then the web-console never comes online.
The sad part is that there is no documentation about this variable anywhere. https://docs.openshift.org/latest/install_config/install/advanced_install.html#configuring-host-variables .

[spadgett]

@missnebun It's not documented because you're running pre-release code. Can you give us some more detail on what you're doing so we can help? Are you installing 3.9?

[missnebun]

@spadgett Not at all. I installed origin 3.7. The mistake I made was to clone https://github.com/openshift/openshift-ansible repo and use that for the install. I spend entire weekend to get this ready for a POC today. More than 10 hours looking at the ansible playbooks and after 89 install runes finally worked ... Just a default variable :( Also the documentation needs to reflect:
ansible-playbook -i hosts openshift-ansible/playbooks/prerequisites.yml -vv
ansible-playbook -i hosts openshift-ansible/playbooks/deploy_cluster.yml -vv

There is no more ansible-playbook [-i /path/to/inventory]
~/openshift-ansible/playbooks/byo/config.yml

[spadgett]

@missnebun You shouldn't be using current master to install 3.7. The master branch is the prerelease installer for OpenShift 3.9. You want to switch to the release-3.7 branch to install 3.7. This is why the doc doesn't match what you see.

[missnebun]

@spadgett I find out in the hard way everything ... the cool part is that I have ansible-wmware-snapshot role done ... and few others cool tricks :) so not a total disaster :)) My wife said thank you do the 2 days spa :))) All good !!!

[jffree]

@missnebun Have you solved this problem? I have the same question.

[missnebun]

Yes ... you need to clone the 3.7 branch instead of master.

[jffree]

@missnebun Can you share your ansible inventory or any other deploy doc to me? Thank you!