apb/install.yaml does not exist anymore

[cmoulliard]

Issue

The script https://github.com/openshift/ansible-service-broker/blob/master/scripts/run_latest_build.sh refers to a file which does not exist anymore
AND which is needed to install OAB on kubernetes

#!/bin/bash

echo "========================================================================"
echo "                RUN LATEST BUILD IS NO LONGER SUPPORTED"
echo "========================================================================"
echo ""
echo "To install the broker, please use our apb/install.yaml. For example:"
echo "   curl https://raw.githubusercontent.com/openshift/ansible-service-broker/master/apb/install.yaml | kubectl create -f -"

[jmrodri]

@cmoulliard What version of kubernetes were you working with? That run_latest_build.sh script is dead it needs to get removed, we've moved to using an operator to deploy the broker. We'll have to retest the kubernetes install support.

[cmoulliard]

I'm using kubernetes 1.14 but I could install if needed a previous version such 1.12 or 1.13
Can you point to the doc about how I can install OAB operator please ? Do we have also to install the kubernetes service api broker as an operator too ? @jmrodri

[cmoulliard]

I'm doing a new test using docker image of docker.io/ansibleplaybookbundle/origin-ansible-service-broker:v4.0" and even if the serviceinstance, servicebinding and secret are well created and status ready, I don't see the deployment of the postgresql db

NAME                                                  CLASS                                   PLAN   STATUS   AGE
serviceinstance.servicecatalog.k8s.io/postgresql-db   ClusterServiceClass/dh-postgresql-apb   dev    Ready    11h

NAME                                                 SERVICE-INSTANCE   SECRET-NAME     STATUS   AGE
servicebinding.servicecatalog.k8s.io/postgresql-db   postgresql-db      postgresql-db   Ready    11h

NAME                         TYPE                                  DATA   AGE
secret/default-token-hgm8n   kubernetes.io/service-account-token   3      19h
secret/postgresql-db         Opaque                                6      11h

Service Instance detail

kc describe serviceinstance.servicecatalog.k8s.io/postgresql-db -n demo
Name:         postgresql-db
Namespace:    demo
Labels:       app.kubernetes.io/component=Service
              app.kubernetes.io/managed-by=component-operator
              app.kubernetes.io/name=fruit-backend-sb
              app.openshift.io/runtime=spring-boot
              app.openshift.io/version=
Annotations:  <none>
API Version:  servicecatalog.k8s.io/v1beta1
Kind:         ServiceInstance
Metadata:
  Creation Timestamp:  2019-04-16T17:34:21Z
  Finalizers:
    kubernetes-incubator/service-catalog
  Generation:        1
  Resource Version:  168
  Self Link:         /apis/servicecatalog.k8s.io/v1beta1/namespaces/demo/serviceinstances/postgresql-db
  UID:               decb37c8-606d-11e9-acec-a2789ba40b23
Spec:
  Cluster Service Class External Name:  dh-postgresql-apb
  Cluster Service Class Ref:
    Name:                              1dda1477cace09730bd8ed7a6505607e
  Cluster Service Plan External Name:  dev
  Cluster Service Plan Ref:
    Name:       7f4a5e35e4af2beb70076e72fab0b7ff
  External ID:  dec9dd42-606d-11e9-acec-a2789ba40b23
  Parameters:
    Postgresql Database:  my_data
    Postgresql Password:  secret
    Postgresql User:      luke
    Postgresql Version:   9.6
  Update Requests:        0
  User Info:
    Groups:
      system:masters
      system:authenticated
    UID:
    Username:  kubernetes-admin
Status:
  Async Op In Progress:  false
  Conditions:
    Last Transition Time:  2019-04-16T17:43:10Z
    Message:               The instance was provisioned successfully
    Reason:                ProvisionedSuccessfully
    Status:                True
    Type:                  Ready
  Deprovision Status:      Required
  External Properties:
    Cluster Service Plan External ID:    7f4a5e35e4af2beb70076e72fab0b7ff
    Cluster Service Plan External Name:  dev
    Parameter Checksum:                  f5626b04b46be0949a4f84c4940b3f79c8e58d5138d15c1595600e1dfa54679e
    Parameters:
      Postgresql Database:  my_data
      Postgresql Password:  secret
      Postgresql User:      luke
      Postgresql Version:   9.6
    User Info:
      Groups:
        system:masters
        system:authenticated
      UID:
      Username:                   kubernetes-admin
  Observed Generation:            1
  Orphan Mitigation In Progress:  false
  Provision Status:               Provisioned
  Reconciled Generation:          1
Events:                           <none>

[cmoulliard]

Sometimes I see this error

Stack of OAB -> https://gist.github.com/cmoulliard/177a07b2a67819c7a5744d57a0e0fded

[root@cloud component-operator-demo]# kc describe serviceinstance.servicecatalog.k8s.io/postgresql-db -n demo
Name:         postgresql-db
Namespace:    demo
Labels:       app.kubernetes.io/component=Service
              app.kubernetes.io/managed-by=component-operator
              app.kubernetes.io/name=fruit-backend-sb
              app.openshift.io/runtime=spring-boot
              app.openshift.io/version=
Annotations:  <none>
API Version:  servicecatalog.k8s.io/v1beta1
Kind:         ServiceInstance
Metadata:
  Creation Timestamp:  2019-04-17T07:30:57Z
  Finalizers:
    kubernetes-incubator/service-catalog
  Generation:        1
  Resource Version:  245
  Self Link:         /apis/servicecatalog.k8s.io/v1beta1/namespaces/demo/serviceinstances/postgresql-db
  UID:               bde8835a-60e2-11e9-8e92-bad91775db08
Spec:
  Cluster Service Class External Name:  dh-postgresql-apb
  Cluster Service Class Ref:
    Name:                              1dda1477cace09730bd8ed7a6505607e
  Cluster Service Plan External Name:  dev
  Cluster Service Plan Ref:
    Name:       7f4a5e35e4af2beb70076e72fab0b7ff
  External ID:  bde88266-60e2-11e9-8e92-bad91775db08
  Parameters:
    Postgresql Database:  my_data
    Postgresql Password:  secret
    Postgresql User:      luke
    Postgresql Version:   9.6
  Update Requests:        0
  User Info:
    Groups:
      system:masters
      system:authenticated
    UID:
    Username:  kubernetes-admin
Status:
  Async Op In Progress:  false
  Conditions:
    Last Transition Time:         2019-04-17T07:30:58Z
    Message:                      The instance references a broker "automation-broker" which has no OSB client created
    Reason:                       ReferencesNonexistentBroker
    Status:                       False
    Type:                         Ready
  Deprovision Status:             NotRequired
  Observed Generation:            1
  Orphan Mitigation In Progress:  false
  Provision Status:
  Reconciled Generation:          0
Events:
  Type     Reason                       Age                 From                                Message
  ----     ------                       ----                ----                                -------
  Warning  ReferencesNonexistentBroker  15s (x15 over 98s)  service-catalog-controller-manager  The instance references a broker "automation-broker" which has no OSB client created

Log of Kubernetes Service Catalog got it too

I0417 08:26:19.930339       1 controller_instance.go:1884] ServiceInstance "demo/postgresql-db" v245: Updating status
I0417 08:26:19.930503       1 request.go:942] Request Body: {"kind":"ServiceInstance","apiVersion":"servicecatalog.k8s.io/v1beta1","metadata":{"name":"postgresql-db","namespace":"demo","selfLink":"/apis/servicecatalog.k8s.io/v1beta1/namespaces/demo/serviceinstances/postgresql-db","uid":"bde8835a-60e2-11e9-8e92-bad91775db08","resourceVersion":"245","generation":1,"creationTimestamp":"2019-04-17T07:30:57Z","labels":{"app.kubernetes.io/component":"Service","app.kubernetes.io/managed-by":"component-operator","app.kubernetes.io/name":"fruit-backend-sb","app.openshift.io/runtime":"spring-boot","app.openshift.io/version":""},"finalizers":["kubernetes-incubator/service-catalog"]},"spec":{"clusterServiceClassExternalName":"dh-postgresql-apb","clusterServicePlanExternalName":"dev","clusterServiceClassRef":{"name":"1dda1477cace09730bd8ed7a6505607e"},"clusterServicePlanRef":{"name":"7f4a5e35e4af2beb70076e72fab0b7ff"},"parameters":{"postgresql_database":"my_data","postgresql_password":"secret","postgresql_user":"luke","postgresql_version":"9.6"},"externalID":"bde88266-60e2-11e9-8e92-bad91775db08","userInfo":{"username":"kubernetes-admin","uid":"","groups":["system:masters","system:authenticated"]},"updateRequests":0},"status":{"conditions":[{"type":"Ready","status":"False","lastTransitionTime":"2019-04-17T07:30:58Z","reason":"ReferencesNonexistentBroker","message":"The instance references a broker \"automation-broker\" which has no OSB client created"}],"asyncOpInProgress":false,"orphanMitigationInProgress":false,"reconciledGeneration":0,"observedGeneration":1,"provisionStatus":"","deprovisionStatus":"NotRequired"}}
I0417 08:26:19.930588       1 round_trippers.go:419] curl -k -v -XPUT  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: service-catalog/v0.1.42 (linux/amd64) kubernetes/49e5cc4/service-catalog-controller-manager" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjYXRhbG9nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InNlcnZpY2UtY2F0YWxvZy1jb250cm9sbGVyLW1hbmFnZXItdG9rZW4tdm0yY2siLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoic2VydmljZS1jYXRhbG9nLWNvbnRyb2xsZXItbWFuYWdlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjFhMzU2YmQ2LTYwMjUtMTFlOS1iYjViLTA4MDAyN2E5NWZhNSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjYXRhbG9nOnNlcnZpY2UtY2F0YWxvZy1jb250cm9sbGVyLW1hbmFnZXIifQ.kOqqfcZBZ0bZUSWaQCkC98rcMLGeCcvDkgj8biLkyBFy839yqmlnBAtS603MiyRGPjrWN19IawiqkEzt0005dRIC4wGUj27m5V-AA94fAp58ILhcYjVddmDULkLaJ_vd4Km7K4wk-PJojCTADnAlASJBJiOdJfvFIJ7njXzcfms6w-RJMTQhFUnhp1oskDOOsBKNdkpjCwe4Of65W2wsTL9mZO5sYwcOjDgAvH_4HB-D--HN0RkXSg8ReVZkIyRTCOmpQwbZrV32kMIX-OWPD3YPh5Mx5em9PwkEAGgP25r9WbhmEwyoK466G_KmUV-ScY1KbHPQcvk-9zEvXpNByg" 'https://10.96.0.1:443/apis/servicecatalog.k8s.io/v1beta1/namespaces/demo/serviceinstances/postgresql-db/status'
I0417 08:26:19.935161       1 round_trippers.go:438] PUT https://10.96.0.1:443/apis/servicecatalog.k8s.io/v1beta1/namespaces/demo/serviceinstances/postgresql-db/status 200 OK in 4 milliseconds
I0417 08:26:19.935179       1 round_trippers.go:444] Response Headers:
I0417 08:26:19.935183       1 round_trippers.go:447]     Content-Type: application/json
I0417 08:26:19.935186       1 round_trippers.go:447]     Date: Wed, 17 Apr 2019 08:26:19 GMT
I0417 08:26:19.935189       1 round_trippers.go:447]     Content-Length: 1578
I0417 08:26:19.935267       1 request.go:942] Response Body: {"kind":"ServiceInstance","apiVersion":"servicecatalog.k8s.io/v1beta1","metadata":{"name":"postgresql-db","namespace":"demo","selfLink":"/apis/servicecatalog.k8s.io/v1beta1/namespaces/demo/serviceinstances/postgresql-db/status","uid":"bde8835a-60e2-11e9-8e92-bad91775db08","resourceVersion":"245","generation":1,"creationTimestamp":"2019-04-17T07:30:57Z","labels":{"app.kubernetes.io/component":"Service","app.kubernetes.io/managed-by":"component-operator","app.kubernetes.io/name":"fruit-backend-sb","app.openshift.io/runtime":"spring-boot","app.openshift.io/version":""},"finalizers":["kubernetes-incubator/service-catalog"]},"spec":{"clusterServiceClassExternalName":"dh-postgresql-apb","clusterServicePlanExternalName":"dev","clusterServiceClassRef":{"name":"1dda1477cace09730bd8ed7a6505607e"},"clusterServicePlanRef":{"name":"7f4a5e35e4af2beb70076e72fab0b7ff"},"parameters":{"postgresql_database":"my_data","postgresql_password":"secret","postgresql_user":"luke","postgresql_version":"9.6"},"externalID":"bde88266-60e2-11e9-8e92-bad91775db08","userInfo":{"username":"kubernetes-admin","uid":"","groups":["system:masters","system:authenticated"]},"updateRequests":0},"status":{"conditions":[{"type":"Ready","status":"False","lastTransitionTime":"2019-04-17T07:30:58Z","reason":"ReferencesNonexistentBroker","message":"The instance references a broker \"automation-broker\" which has no OSB client created"}],"asyncOpInProgress":false,"orphanMitigationInProgress":false,"reconciledGeneration":0,"observedGeneration":1,"provisionStatus":"","deprovisionStatus":"NotRequired"}}
I0417 08:26:19.935428       1 controller.go:391] Error syncing ServiceInstance demo/postgresql-db (retry: 14/15): The instance references a broker "automation-broker" which has no OSB client created
I0417 08:26:19.935458       1 event.go:221] Event(v1.ObjectReference{Kind:"ServiceInstance", Namespace:"demo", Name:"postgresql-db", UID:"bde8835a-60e2-11e9-8e92-bad91775db08", APIVersion:"servicecatalog.k8s.io/v1beta1", ResourceVersion:"245", FieldPath:""}): type: 'Warning' reason: 'ReferencesNonexistentBroker' The instance references a broker "automation-broker" which has no OSB client created
I0417 08:26:19.935604       1 request.go:942] Request Body: {"count":207,"lastTimestamp":"2019-04-17T08:26:19Z","message":"The instance references a broker \"automation-broker\" which has no OSB client created"}

What means this message The instance references a broker "automation-broker" which has no OSB client created ?

[cmoulliard]

Is the OSB Client issue reported by the k8s service catalog been related that the service registered for the broker is automation-broker

kc describe clusterServiceClass/1dda1477cace09730bd8ed7a6505607e
Name:         1dda1477cace09730bd8ed7a6505607e
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  servicecatalog.k8s.io/v1beta1
Kind:         ClusterServiceClass
Metadata:
  Creation Timestamp:  2019-04-16T09:03:30Z
  Owner References:
    API Version:           servicecatalog.k8s.io/v1beta1
    Block Owner Deletion:  false
    Controller:            true
    Kind:                  ClusterServiceBroker
    Name:                  automation-broker
    UID:                   f2205a2d-6025-11e9-a396-2a118e2290a7
  Resource Version:        15
  Self Link:               /apis/servicecatalog.k8s.io/v1beta1/clusterserviceclasses/1dda1477cace09730bd8ed7a6505607e
  UID:                     815907f0-6026-11e9-a396-2a118e2290a7
Spec:
  Bindable:                     true
  Binding Retrievable:          false
  Cluster Service Broker Name:  automation-broker

and the kubernetes service is Broker

kc get svc -n automation-broker
NAME     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
broker   ClusterIP   10.108.254.144   <none>        1337/TCP,1338/TCP   22h

[jmrodri]

@cmoulliard the message The instance references a broker "automation-broker" which has no OSB client created is coming from Service Catalog. The error means that it can't find the connection (client used to talk to the broker) for this particular broker. But it seems based on the information you've given me that at some point it did think there was a broker there in order to get a response regarding the dh-postresql-apb.

[jmrodri]

Do you happen to have the information about the service brokers? kubectl get clusterservicebrokers and kubectl get servicebrokers.

[cmoulliard]

kubectl get clusterservicebrokers
NAME                URL                                                                 STATUS                 AGE
automation-broker   https://broker.automation-broker.svc:1338/ansible-service-broker/   ErrorFetchingCatalog   1d

kc describe clusterservicebroker/automation-broker
Name:         automation-broker
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  servicecatalog.k8s.io/v1beta1
Kind:         ClusterServiceBroker
Metadata:
  Creation Timestamp:  2019-04-16T08:59:30Z
  Finalizers:
    kubernetes-incubator/service-catalog
  Generation:        2
  Resource Version:  577
  Self Link:         /apis/servicecatalog.k8s.io/v1beta1/clusterservicebrokers/automation-broker
  UID:               f2205a2d-6025-11e9-a396-2a118e2290a7
Spec:
  Auth Info:
    Bearer:
      Secret Ref:
        Name:       automation-broker-client-token-zwhts
        Namespace:  automation-broker
  Ca Bundle:        LS0tLS1CRUdJ...S0tLQ==
  Relist Behavior:  Duration
  Relist Requests:  0
  URL:              https://broker.automation-broker.svc:1338/ansible-service-broker/
Status:
  Conditions:
    Last Transition Time:       2019-04-17T17:47:09Z
    Message:                    Error fetching catalog.Error getting broker catalog: Status: 404; ErrorMessage: <nil>; Description: <nil>; ResponseError: <nil>
    Reason:                     ErrorFetchingCatalog
    Status:                     False
    Type:                       Ready
  Last Catalog Retrieval Time:  2019-04-16T17:44:02Z
  Operation Start Time:         2019-04-17T17:47:09Z
  Reconciled Generation:        2
Events:
  Type     Reason                Age                   From                                Message
  ----     ------                ----                  ----                                -------
  Warning  ErrorFetchingCatalog  3m32s (x43 over 69m)  service-catalog-controller-manager  Error getting broker catalog: Status: 404; ErrorMessage: <nil>; Description: <nil>; ResponseError: <nil>

and

[root@cloud component-operator-demo]# kubectl get servicebrokers
No resources found.

[jmrodri]

@cmoulliard how did you ultimately deploy the broker and service catalog?

[cmoulliard]

Here is the script and steps that I'm using to install OAB and ServiceCatalog - https://github.com/snowdrop/openshift-infra/blob/f96140c2da563f7baa4f572b7eeb1b2dff870d68/kubernetes/create-k8s-cluster.sh#L157-L238

[cmoulliard]

Good news. If I setup a Kubernetes Cluster - v1.13, then that works

[root@cloud ~]# kubectl get serviceinstance,servicebinding,secrets,deployment,pods -n demo
NAME                                                  CLASS                                   PLAN   STATUS   AGE
serviceinstance.servicecatalog.k8s.io/postgresql-db   ClusterServiceClass/dh-postgresql-apb   dev    Ready    5m17s

NAME                                                 SERVICE-INSTANCE   SECRET-NAME     STATUS   AGE
servicebinding.servicecatalog.k8s.io/postgresql-db   postgresql-db      postgresql-db   Ready    5m17s

NAME                         TYPE                                  DATA   AGE
secret/default-token-57f7z   kubernetes.io/service-account-token   3      5m30s
secret/postgresql-db         Opaque                                6      111s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.extensions/fruit-backend-sb     1/1     1            1           5m19s
deployment.extensions/postgresql-9.6-dev   1/1     1            1           3m11s

NAME                                     READY   STATUS    RESTARTS   AGE
pod/fruit-backend-sb-546759956c-ddnbx    1/1     Running   0          5m7s
pod/postgresql-9.6-dev-8fd476488-dw5q9   1/1     Running   0          3m11s

[jmrodri]

So this seems to only affect kubenetes v1.14. That's good to know.

[cmoulliard]

Hmhmh. Not sure what was the issue but after creating again a new Centos7 VM where this script was used to install OAB, ServiceCatalog on k8s 1.14.1, that works now

NAME                                      READY   STATUS    RESTARTS   AGE
pod/fruit-backend-sb-59f45cc4d-m7tfq      1/1     Running   0          3m31s
pod/postgresql-9.6-dev-645dbfb467-zfn6z   1/1     Running   0          2m49s
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/fruit-backend-sb   ClusterIP   10.105.103.80    <none>        8080/TCP   3m42s
service/postgresql         ClusterIP   10.104.252.157   <none>        5432/TCP   2m50s
NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/fruit-backend-sb     1/1     1            1           3m42s
deployment.apps/postgresql-9.6-dev   1/1     1            1           2m49s
NAME                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/fruit-backend-sb-59f45cc4d      1         1         1       3m31s
replicaset.apps/fruit-backend-sb-d49bcf88       0         0         0       3m42s
replicaset.apps/postgresql-9.6-dev-645dbfb467   1         1         1       2m49s
NAME                                          RUNTIME       VERSION          SERVICE         INJECTED   AGE     STATUS
component.component.k8s.io/fruit-backend-sb   spring-boot   1.5.15.RELEASE   postgresql-db   Secret     3m53s   CreatingService
NAME                                                  CLASS                                   PLAN   STATUS   AGE
serviceinstance.servicecatalog.k8s.io/postgresql-db   ClusterServiceClass/dh-postgresql-apb   dev    Ready    3m41s
NAME                                                 SERVICE-INSTANCE   SECRET-NAME     STATUS   AGE
servicebinding.servicecatalog.k8s.io/postgresql-db   postgresql-db      postgresql-db   Ready    3m41s
NAME                         TYPE                                  DATA   AGE
secret/default-token-xs8rz   kubernetes.io/service-account-token   3      3m53s
secret/postgresql-db         Opaque                                6      2m15s

[jmrodri]

I used the bottom portion of your script to start a service-catalog and broker on 1.14. Initially the service-catalog had the ErrorFetchingCatalog. I believe the broker wasn't completely up yet. A couple minutes later I saw the clusterservicebroker was in Ready state. Let's leave this issue open for now since the original issue of the script is still valid. We need instructions for k8s with the broker and operator world.

Since things seem to be working, I will focus on just documenting the installation case.