Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add troubleshooting documentation to the broker #479

Merged
merged 2 commits into from Oct 9, 2017
Merged

Add troubleshooting documentation to the broker #479

merged 2 commits into from Oct 9, 2017

Conversation

djzager
Copy link
Member

@djzager djzager commented Oct 5, 2017

Describe what this PR does and why we need it:

Documentation for troubles users/developers may see.

@openshift-ci-robot openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 5, 2017
@djzager djzager added needs-review and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 5, 2017
jmrodri
jmrodri suggested changes Oct 6, 2017
View reviewed changes
docs/troubleshooting.md Outdated

The purpose of this document is to provide troubleshooting steps for different
scenarios. Where possible sections and sub-section should be created to
classify different types of troubles.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where possible, sections and sub-sections ...

docs/troubleshooting.md Outdated
scenarios. Where possible sections and sub-section should be created to
classify different types of troubles.

## Errors related to Service Catalog communicating with the Broker
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Service Catalog and Broker communication issues

docs/troubleshooting.md Outdated
## Errors related to Service Catalog communicating with the Broker

### Certificate Authority

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a little more intro to the sections. Something like:

Sometimes the service-catalog is unable to communicate with the broker because of an unknown certificate authority.

Looking at the output below, we see the broker is running but there is a certificate signed by unknown authority preventing the service-catalog from connecting.

docs/troubleshooting.md Outdated
--------- -------- ----- ---- ------------- -------- ------ -------
6s 6s 1 service-catalog-controller-manager Warning ErrorFetchingCatalog Error getting broker catalog for broker "ansible-service-broker": Get https://asb.ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
6s 2s 9 service-catalog-controller-manager Warning ErrorFetchingCatalog Error getting broker catalog for broker "ansible-service-broker": Get https://asb.ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog: x509: certificate signed by unknown authority
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found the entire output confusing. I looked over it 3 times trying to find the error and finally had to scroll to the right to see the certificate error.

Consider doing the following:

  1. add more to the intro paragraph: The Status field of the description output will show the certificate signed by unknown authority error.

  2. then trim down the output to show the important piece we are focusing on.

$ oc describe servicebroker
Name:           ansible-service-broker
Namespace:
Labels:         <none>
Annotations:    openshift.io/generated-by=OpenShiftNewApp
API Version:    servicecatalog.k8s.io/v1alpha1
Kind:           ServiceBroker
...
Status:
  Conditions:
    Last Transition Time:       2017-10-05T17:22:01Z
    Message:                    Error fetching catalog. Error getting broker catalog for broker "ansible-service-broker": Get https://asb.ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog: x509: certificate signed by unknown authority
    Reason:                     ErrorFetchingCatalog
    Status:                     False
    Type:                       Ready
  Operation Start Time:         2017-10-05T17:22:02Z
  Reconciled Generation:        0
...

docs/troubleshooting.md Outdated

#### Resolution: Provide caBundle to service-catalog

We can get the caBundle with the following command:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to provide the service-catalog with the caBundle so that it can validate the certificate signing chain. We can get the caBundle with the following command:

docs/troubleshooting.md Outdated
38m 38m 1 service-catalog-controller-manager Warning ErrorFetchingCatalog Error getting broker catalog for broker "ansible-service-broker": Get https://asb.ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
38m 10m 107 service-catalog-controller-manager Warning ErrorFetchingCatalog Error getting broker catalog for broker "ansible-service-broker": Get https://asb.ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog: x509: certificate signed by unknown authority
10m 17s 34 service-catalog-controller-manager Warning ErrorFetchingCatalog Error getting broker catalog for broker "ansible-service-broker": Status: 401; ErrorMessage: <nil>; Description: invalid credentials, corrupt header; ResponseError: <nil>
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like with the previous error, I would only show the relevant pieces of the log:

$ oc describe servicebroker
Name:           ansible-service-broker
...
Spec:
  Auth Info:
    Bearer:
      Secret Ref:
        Kind:           Secret
        Name:           ansibleservicebroker-client
        Namespace:      ansible-service-broker
  Ca Bundle:            LS0t...
  ...
Status:
  Conditions:
    Last Transition Time:       2017-10-05T17:22:01Z
    Message:                    Error fetching catalog. Error getting broker catalog for broker "ansible-service-broker": Status: 401; ErrorMessage: <nil>; Description: invalid credentials, corrupt header; ResponseError: <nil>
    Reason:                     ErrorFetchingCatalog
    Status:                     False
    Type:                       Ready
  Operation Start Time:         2017-10-05T17:22:02Z
  Reconciled Generation:        0
Events:
  FirstSeen     LastSeen        Count   From                                    SubObjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                                    -------------   --------        ------                  -------
  38m           38m             1       service-catalog-controller-manager                      Warning         ErrorFetchingCatalog    Error getting broker catalog for broker "ansible-service-broker": Get https://asb.ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  38m           10m             107     service-catalog-controller-manager                      Warning         ErrorFetchingCatalog    Error getting broker catalog for broker "ansible-service-broker": Get https://asb.ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog: x509: certificate signed by unknown authority
  10m           17s             34      service-catalog-controller-manager                      Warning         ErrorFetchingCatalog    Error getting broker catalog for broker "ansible-service-broker": Status: 401; ErrorMessage: <nil>; Description: invalid credentials, corrupt header; ResponseError: <nil>

docs/troubleshooting.md Outdated

What you may notice in the output of `oc describe servicebroker` is that the service-catalog is being configured to use token based authentication
to communicate with the Ansible Service Broker. However, because of [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1498992), the broker
is configured to use basic auth by default. All that we need to do is 1) update the broker's configmap and 2) redeploy the broker.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Spec: section shows that the service-catalog is configured to use token based authentication to communicate with the broker. However, because of this bug, the broker is configured to use basic auth by default.

All that we need to do is 1) update the broker's configmap and 2) redeploy the broker.

docs/troubleshooting.md Outdated
$ oc edit configmap broker-config
```

Modifying the configuration like what you see below:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disable the basic auth by setting the enabled: entry to false like the example below:

docs/troubleshooting.md Outdated
to communicate with the Ansible Service Broker. However, because of [this bug](https://bugzilla.redhat.com/show_bug.cgi?id=1498992), the broker
is configured to use basic auth by default. All that we need to do is 1) update the broker's configmap and 2) redeploy the broker.

##### Update Broker's ConfigMap
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might be able to make this a plain bullet

  • Update Broker's ConfigMap

docs/troubleshooting.md Outdated
- enabled: true
```

##### Redeploy the Broker
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then you could also make this bullet:

  • Redeploy the Broker

@jmrodri
Copy link
Contributor

jmrodri commented Oct 6, 2017

@djzager great information, just a few tweaks to hopefully make it easier to consume.

@openshift-ci-robot openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 6, 2017
@jmrodri jmrodri merged commit 09cfe59 into openshift:master Oct 9, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jmrodri jmrodri jmrodri approved these changes

Assignees
No one assigned
Labels
needs-review size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@djzager @jmrodri @openshift-ci-robot