ASB returning zero bytes to Service Catalog (ResponseError: unexpected end of JSON input)

[fortinj66]

Bug:

What happened:

in Openshift Origin 3.7.1 ASB returns no data (0 bytes) to the Service catalog when the Service Catalog
tries to update

What you expected to happen:
Service catalog to be updated successfully

From Service Catalog

0417 19:21:23.072083       1 controller_broker.go:173] ClusterServiceBroker "ansible-service-broker": Processing
I0417 19:21:23.075448       1 controller_broker.go:197] ClusterServiceBroker "ansible-service-broker": Creating client, URL: https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker 
I0417 19:21:23.075678       1 controller_broker.go:209] ClusterServiceBroker "ansible-service-broker": Processing adding/update event
W0417 19:21:23.103203       1 controller_broker.go:216] ClusterServiceBroker "ansible-service-broker": Error getting broker catalog: Status: 200; ErrorMessage: <nil>; Description: <nil>; ResponseError: unexpected end of JSON input
I0417 19:21:23.103248       1 controller_broker.go:683] ClusterServiceBroker "ansible-service-broker": Updating ready condition to False
I0417 19:21:23.103716       1 event.go:218] Event(v1.ObjectReference{Kind:"ClusterServiceBroker", Namespace:"", Name:"ansible-service-broker", UID:"d9f04ebe-3d04-11e8-8185-0a580a800003", APIVersion:"servicecatalog.k8s.io", ResourceVersion:"1204474", FieldPath:""}): type: 'Warning' reason: 'ErrorFetchingCatalog' Error getting broker catalog: Status: 200; ErrorMessage: <nil>; Description: <nil>; ResponseError: unexpected end of JSON input

From ASB

[2018-04-17T19:23:59.622Z] [INFO] AnsibleBroker::Catalog
10.130.2.1 - - [17/Apr/2018:19:23:59 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 0

I've has the same issue with two separate Openshift 3.7.1 clusters...

Service Catalog and ASB were deployed with the ansible-playbook

[djzager]

@fortinj66 Could you provide more of the broker logs? It isn't clear to me if this is on initial startup or after the broker has been running for sometime.

Some more questions if you are able:

1. Was the broker previously returning data on the /v2/catalog request and it is not now?
2. Does this persist indefinitely or does the broker start returning data on subsequent /v2/catalog requests?

[fortinj66]

Hi David,

I've attached the debug logs for the current pod...

1. I have not been able to get any 'proper' results since the installation...
2. It seems to persist indefinitely... I don't think it is credentials as it is returning 200 rather than 403...

asb.log

I also uploaded the Service Controller log:

controller.log

[fortinj66]

Here is the results running from inside the container (Note the content length):

sh-4.2$ curl -v -X GET -H "Authorization: Bearer xxx" -k https://localhost:1338/ansible-service-broker/v2/catalog
* About to connect() to localhost port 1338 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 1338 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=asb.openshift-ansible-service-broker.svc
*       start date: Apr 17 18:59:34 2018 GMT
*       expire date: Apr 16 18:59:35 2020 GMT
*       common name: asb.openshift-ansible-service-broker.svc
*       issuer: CN=openshift-service-serving-signer@1523394353
> GET /ansible-service-broker/v2/catalog HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:1338
> Accept: */*
> Authorization: Bearer xxx
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Tue, 17 Apr 2018 21:11:16 GMT
< Content-Length: 0
<
* Connection #0 to host localhost left intact

[fortinj66]

I believe I have found the culprit...

I tried adding the APBs individually and the process worked; data appeared when I queried the endpoint. I started adding groups of 10 APBs via the white_list parameter and added more after each success...

This worked until I added the group containing mssql-apb. After this group was added the endpoint returned 0 bytes. When I removed mssql-apb via black_list it worked...

I then tried just mssql-apb and the endpoint failed with zero bytes.

My final working config for the registries looks like this:

registry:
  - type: dockerhub
    name: dh
    url:  
    org:  ansibleplaybookbundle
    tag:  
    black_list:
        - ".*mssql-apb$"    
    white_list:
        - ".*-apb$"    

So there is something about mssql-apb that Ansible Service Broker doesn't like...

[djzager]

@fortinj66 Thank you for all the details. I'm interested in finding the root cause (ie. how mssql-apb could be causing this). Out of curiosity, what broker image are you using? If you are using a broker image tagged v3.7 or release-1.0, you may want to be grabbing release-1.0 apbs. We talk about that in our Compatibility section of our README and I'm working on a blog post to document our broker image's compatibility with different k8s/openshift and apb image versions.

Having said that, I'm curious if @jmrodri or @shawn-hurley have any thoughts on a particular APB causing the broker to return 0 bytes (I believe they would be curious the broker image/version also).

[shawn-hurley]

IIRC the 3.7 broker has pretty poor behavior around not understanding a spec.

I think the below config should fix the problem. What I think is happening is that your config is pulling the latest images and I don't know if it is a safe assumption that those images would work on 3.7.

registry:
  - type: dockerhub
    name: dh
    url:  
    org:  ansibleplaybookbundle
    tag:  v3.7 #should be pulling the specific 3.7 version of specs.
    black_list:
        - ".*mssql-apb$"    
    white_list:
        - ".*-apb$"    

/cc @dymurray @jmontleon

[fortinj66]

The broker image is:
image: 'ansibleplaybookbundle/origin-ansible-service-broker:v3.7'

@shawn-hurley I'll add the image tag and see what happens

[fortinj66]

@shawn-hurley Interestingly enough that did fix the issue. It also reduced the number of APBs installed to 17 from 38...

So i'm wondering if the ansible-playbook should be setting "tag:' to the ASB version instead of latest...

[shawn-hurley]

So i'm wondering if the ansible-playbook should be setting "tag:' to the ASB version instead of latest...

Are you referring to openshift-ansible playbook?

@fabianvf I think this is the line that should be changed: https://github.com/openshift/openshift-ansible/blob/release-3.7/roles/ansible_service_broker/vars/default_images.yml#L16, does that make sense?

edit: adding line from openshift ansible

[rthallisey]

Closing in favor of: openshift/openshift-ansible#8089