Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ClusterServiceBroker "ansible-service-broker" - Status: 400 -> Description: not found; ResponseError #813

Closed
cmoulliard opened this issue Mar 3, 2018 · 5 comments
Closed

ClusterServiceBroker "ansible-service-broker" - Status: 400 -> Description: not found; ResponseError #813

cmoulliard opened this issue Mar 3, 2018 · 5 comments
Assignees
@shawn-hurley
Labels
3.11 | release-1.3 Kubernetes 1.11 | Openshift 3.11 | Broker release-1.3

Comments

@cmoulliard
Copy link

cmoulliard commented Mar 3, 2018 •
edited

Info

This ticket contains most probably 2 errors as :

  • ServiceInstance is not created correctly Status: 400 -> Description: not found; ResponseError
  • The log of the POD reports errors about Tagged image (e.g v3.7) are missing for the ansibleplaybookbundle

What happened

Then, this is error will appear within the Openshift "overview" console of the project for a provisioned service - MySQL (APB)

apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceInstance
metadata:
  creationTimestamp: '2018-03-02T21:49:07Z'
  finalizers:
    - kubernetes-incubator/service-catalog
  generateName: dh-mysql-apb-
  generation: 1
  name: dh-mysql-apb-hmzfx
  namespace: project1
  resourceVersion: '26464'
  selfLink: >-
    /apis/servicecatalog.k8s.io/v1beta1/namespaces/project1/serviceinstances/dh-mysql-apb-hmzfx
  uid: 888ed617-1e63-11e8-aaea-0a580a8001b4
spec:
  clusterServiceClassExternalName: dh-mysql-apb
  clusterServiceClassRef:
    name: ddd528762894b277001df310a126d5ad
  clusterServicePlanExternalName: dev
  clusterServicePlanRef:
    name: 583f053f9ba165125a16cf9aff768017
  externalID: c4faddf1-5df0-4473-b3f4-45d03ed2a48c
  parametersFrom:
    - secretKeyRef:
        key: parameters
        name: dh-mysql-apb-parameters0xsul
  updateRequests: 0
  userInfo:
    extra:
      scopes.authorization.openshift.io:
        - 'user:full'
    groups:
      - 'system:authenticated:oauth'
      - 'system:authenticated'
    uid: ''
    username: admin
status:
  asyncOpInProgress: false
  conditions:
    - lastTransitionTime: '2018-03-02T21:49:07Z'
      message: >-
        Error provisioning ServiceInstance of ClusterServiceClass (K8S:
        "ddd528762894b277001df310a126d5ad" ExternalName: "dh-mysql-apb") at
        ClusterServiceBroker "ansible-service-broker": Status: 400;
        ErrorMessage: <nil>; Description: not found; ResponseError: <nil>
      reason: ProvisionCallFailed
      status: 'False'
      type: Ready
    - lastTransitionTime: '2018-03-02T21:49:07Z'
      message: >-
        Error provisioning ServiceInstance of ClusterServiceClass (K8S:
        "ddd528762894b277001df310a126d5ad" ExternalName: "dh-mysql-apb") at
        ClusterServiceBroker "ansible-service-broker": Status: 400;
        ErrorMessage: <nil>; Description: not found; ResponseError: <nil>
      reason: ClusterServiceBrokerReturnedFailure
      status: 'True'
      type: Failed
  deprovisionStatus: Required
  orphanMitigationInProgress: false
  reconciledGeneration: 1

** How to reproduce **

SSH access to a Fedora or CentOS vm machine is required

Inventory used on dedicated Hetzner machine

# Create an OSEv3 group that contains the masters and nodes groups
[OSEv3:children]
masters
nodes
etcd

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
ansible_user=root

# For Atomic VM
containerized=false

openshift_release=v3.7
openshift_image_tag=v3.7.0
openshift_pkg_version="-3.7.0"
openshift_deployment_type=origin
openshift_enable_service_catalog=true

openshift_hostname=46.4.81.220
openshift_master_cluster_public_hostname=46.4.81.220
openshift_master_default_subdomain=46.4.81.220.nip.io
openshift_disable_check=memory_availability,disk_availability,docker_image_availability
openshift_master_unsupported_embedded_etcd=true

public_ip_address=46.4.81.220

ansible_python_interpreter = /usr/bin/python3.6

ansible_service_broker_registry_whitelist=['.*-apb$']
ansible_service_broker_image_tag=v3.7
ansible_service_broker_registry_tag=v3.7
ansible_service_broker_remove=false
ansible_service_broker_install=true

# enable htpasswd auth
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
# htpasswd -nb admin admin
openshift_master_htpasswd_users={'admin': '$apr1$DloeoaY3$nqbN9fQBkyXgbj58buqEM.'}

# host group for masters
[masters]
46.4.81.220 openshift_public_hostname=46.4.81.220

# host group for etcd, should run on a node that is not schedulable
[etcd]
46.4.81.220

# host group for worker nodes, we list master node here so that
# openshift-sdn gets installed. We mark the master node as not
# schedulable.
[nodes]
46.4.81.220 openshift_node_labels="{'region':'infra','zone':'default'}" openshift_public_hostname=46.4.81.220 openshift_schedulable=true

Steps to install ASB using ansible

git clone -b release-3.7 git@github.com:openshift/openshift-ansible.git
ansible-playbook -i inventory/hetzner_host openshift-ansible/playbooks/byo/config.yml
ansible-playbook -i inventory/hetzner_host openshift-ansible/playbooks/byo/openshift-cluster/service-catalog.yml

A this stage, no errors are rerported as APB/APB_etcd PODS are running and we can browse the catalog. Next, browse the catalog from a project, select MySQL (APB) and create a service

Then, this is error will appear within the "overview" console of the project

Error provisioning ServiceInstance of ClusterServiceClass (K8S: "ddd528762894b277001df310a126d5ad" ExternalName: "dh-mysql-apb") at ClusterServiceBroker "ansible-service-broker": Status: 400; ErrorMessage: <nil>; Description: not found; ResponseError: <nil>

If we open the log of APB pods, then we will see such docker image errors

Using config file mounted to /etc/ansible-service-broker/config.yaml
--
  | ============================================================
  | ==           Starting Ansible Service Broker...           ==
  | ============================================================
  | [2018-03-02T21:48:02.848Z] [NOTICE] Initializing clients...
  | [2018-03-02T21:48:02.848Z] [INFO] == ETCD CX ==
  | [2018-03-02T21:48:02.848Z] [INFO] EtcdHost: asb-etcd.openshift-ansible-service-broker.svc
  | [2018-03-02T21:48:02.848Z] [INFO] EtcdPort: 2379
  | [2018-03-02T21:48:02.848Z] [INFO] Endpoints: [https://asb-etcd.openshift-ansible-service-broker.svc:2379 ]
  | [2018-03-02T21:48:02.858Z] [INFO] Etcd Version [Server: 3.3.1, Cluster: 3.3.0]
  | [2018-03-02T21:48:02.865Z] [INFO] Kubernetes version: v1.7.6+a08f5eeb62
  | [2018-03-02T21:48:02.865Z] [INFO] == REGISTRY CX ==
  | [2018-03-02T21:48:02.865Z] [INFO] Name: dh
  | [2018-03-02T21:48:02.865Z] [INFO] Type: dockerhub
  | [2018-03-02T21:48:02.865Z] [INFO] Url:
  | [2018-03-02T21:48:02.865Z] [INFO] == REGISTRY CX ==
  | [2018-03-02T21:48:02.865Z] [INFO] Name: localregistry
  | [2018-03-02T21:48:02.865Z] [INFO] Type: local_openshift
  | [2018-03-02T21:48:02.865Z] [INFO] Url:
  | [2018-03-02T21:48:02.865Z] [INFO] No cluster credential overrides provided, using k8s InClusterConfig
  | [2018-03-02T21:48:02.865Z] [INFO] Listening for provision messages
  | [2018-03-02T21:48:02.865Z] [INFO] Listening for deprovision messages
  | [2018-03-02T21:48:02.865Z] [INFO] Listening for provision messages
  | [2018-03-02T21:48:02.967Z] [INFO] Initiating Recovery Process
  | [2018-03-02T21:48:02.968Z] [INFO] No jobs to recover
  | [2018-03-02T21:48:02.968Z] [NOTICE]
  | [2018-03-02T21:48:02.968Z] [INFO] Broker configured to bootstrap on startup
  | [2018-03-02T21:48:02.968Z] [INFO] Attempting bootstrap...
  | [2018-03-02T21:48:02.968Z] [INFO] AnsibleBroker::Bootstrap
  | [2018-03-02T21:48:05.831Z] [ERROR] Image 'docker.io/ansibleplaybookbundle/iscsi-demo-target-apb:v3.7' may not exist in registry.
  | [2018-03-02T21:48:05.831Z] [ERROR] {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"v3.7"}}]}
  |  
  | [2018-03-02T21:48:06.055Z] [ERROR] Image 'docker.io/ansibleplaybookbundle/proxy-config-apb:v3.7' may not exist in registry.
  | [2018-03-02T21:48:06.056Z] [ERROR] {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"v3.7"}}]}
  |  
  | [2018-03-02T21:48:06.862Z] [ERROR] Image 'docker.io/ansibleplaybookbundle/virtualmachines-apb:v3.7' may not exist in registry.
  | [2018-03-02T21:48:06.862Z] [ERROR] {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"v3.7"}}]}
  |  
  | [2018-03-02T21:48:07.097Z] [ERROR] Image 'docker.io/ansibleplaybookbundle/photo-album-demo-api-apb:v3.7' may not exist in registry.
  | [2018-03-02T21:48:07.097Z] [ERROR] {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"v3.7"}}]}
  |  
  | [2018-03-02T21:48:07.325Z] [ERROR] Image 'docker.io/ansibleplaybookbundle/kibana-apb:v3.7' may not exist in registry.
  | [2018-03-02T21:48:07.325Z] [ERROR] {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"v3.7"}}]}
  |  
  | [2018-03-02T21:48:08.007Z] [ERROR] Image 'docker.io/ansibleplaybookbundle/awx-apb:v3.7' may not exist in registry.
  | [2018-03-02T21:48:08.007Z] [ERROR] {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"v3.7"}}]}
  |  
  | [2018-03-02T21:48:09.081Z] [ERROR] Image 'docker.io/ansibleplaybookbundle/keycloak-apb:v3.7' may not exist in registry.
  | [2018-03-02T21:48:09.081Z] [ERROR] {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"v3.7"}}]}
  |  
  | [2018-03-02T21:48:09.306Z] [ERROR] Image 'docker.io/ansibleplaybookbundle/prometheus-apb:v3.7' may not exist in registry.
  | [2018-03-02T21:48:09.306Z] [ERROR] {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"v3.7"}}]}
  |  
  | [2018-03-02T21:48:09.857Z] [ERROR] Image 'docker.io/ansibleplaybookbundle/photo-album-demo-ext-api-apb:v3.7' may not exist in registry.
  | [2018-03-02T21:48:09.857Z] [ERROR] {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"v3.7"}}]}
  |  
  | [2018-03-02T21:48:10.084Z] [ERROR] Image 'docker.io/ansibleplaybookbundle/photo-album-demo-app-apb:v3.7' may not exist in registry.
  | [2018-03-02T21:48:10.084Z] [ERROR] {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"v3.7"}}]}
  |  
  | [2018-03-02T21:48:11.114Z] [ERROR] Image 'docker.io/ansibleplaybookbundle/mongodb-apb:v3.7' may not exist in registry.
  | [2018-03-02T21:48:11.114Z] [ERROR] {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"v3.7"}}]}
  |  
  | [2018-03-02T21:48:11.741Z] [ERROR] Image 'docker.io/ansibleplaybookbundle/s2i-apb:v3.7' may not exist in registry.
  | [2018-03-02T21:48:11.741Z] [ERROR] {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"v3.7"}}]}
  |  
  | [2018-03-02T21:48:13.081Z] [ERROR] Image 'docker.io/ansibleplaybookbundle/es-apb:v3.7' may not exist in registry.
  | [2018-03-02T21:48:13.081Z] [ERROR] {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"v3.7"}}]}
  |  
  | [2018-03-02T21:48:13.67Z] [ERROR] Image 'docker.io/ansibleplaybookbundle/kubevirt-apb:v3.7' may not exist in registry.
  | [2018-03-02T21:48:13.67Z] [ERROR] {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"v3.7"}}]}
  |  
  | [2018-03-02T21:48:16.045Z] [INFO] Validating specs...
  | [2018-03-02T21:48:16.045Z] [NOTICE] All specs passed validation!
  | !!!!!!filter mode is none[2018-03-02T21:48:16.105Z] [INFO] Validating specs...
  | [2018-03-02T21:48:16.105Z] [NOTICE] All specs passed validation!
  | [2018-03-02T21:48:16.114Z] [NOTICE] Broker successfully bootstrapped on startup
  | [2018-03-02T21:48:16.177Z] [NOTICE] Listening on https://0.0.0.0:1338
  | [2018-03-02T21:48:16.177Z] [NOTICE] Ansible Service Broker Starting
  | [2018-03-02T21:48:27.998Z] [ERROR] Could not find a service instance in dao - 100: Key not found (/service_instance) [40]
  | 10.128.0.1 - - [02/Mar/2018:21:48:27 +0000] "DELETE /ansible-service-broker/v2/service_instances/cfd44c39-32d4-47a4-9aef-2980ae006355?accepts_incomplete=true&plan_id=583f053f9ba165125a16cf9aff768017&service_id=ddd528762894b277001df310a126d5ad HTTP/1.1" 410 3
  | 10.128.0.1 - - [02/Mar/2018:21:49:07 +0000] "PUT /ansible-service-broker/v2/service_instances/c4faddf1-5df0-4473-b3f4-45d03ed2a48c?accepts_incomplete=true HTTP/1.1" 400 33
  | 10.128.0.1 - - [02/Mar/2018:21:49:07 +0000] "PUT /ansible-service-broker/v2/service_instances/c4faddf1-5df0-4473-b3f4-45d03ed2a48c?accepts_incomplete=true HTTP/1.1" 400 33
  | 10.128.0.1 - - [02/Mar/2018:21:49:07 +0000] "PUT /ansible-service-broker/v2/service_instances/c4faddf1-5df0-4473-b3f4-45d03ed2a48c?accepts_incomplete=true HTTP/1.1" 400 33
  | 10.128.0.1 - - [02/Mar/2018:21:49:08 +0000] "PUT /ansible-service-broker/v2/service_instances/c4faddf1-5df0-4473-b3f4-45d03ed2a48c?accepts_incomplete=true HTTP/1.1" 400 33
  | [2018-03-02T21:52:02.852Z] [INFO] AnsibleBroker::Catalog
  | 10.128.0.1 - - [02/Mar/2018:21:52:02 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 74264
@cmoulliard cmoulliard changed the title Tagged image (e.g v3.7) are missing for the ansibleplaybookbundle ClusterServiceBroker "ansible-service-broker" - Status: 400 -> Description: not found; ResponseError Mar 3, 2018
@shawn-hurley
Copy link
Contributor

Some images do not have a 3.7 tag because they were made after the 3.7 release. IMO we may want to change the logs from Error to Warning or Info for the manifest not found.

I assume that other APBs showed up in the catalog?

@eriknelson @jmrodri thoughts on the log message?

@cmoulliard
Copy link
Author

I assume that other APBs showed up in the catalog?

Yes. The ServiceInstance is not created correctly Status: 400 -> Description: not found; ResponseError is blocking by the way

@rthallisey rthallisey added 3.10 | release-1.2 Kubernetes 1.10 | Openshift 3.10 | Broker release-1.2 bug labels Mar 19, 2018
@gouyang gouyang mentioned this issue Mar 28, 2018
Closed
@gouyang
Copy link

gouyang commented Mar 28, 2018

I have the same issue when provision any service from the web console, version is 3.9.

# oc version
oc v3.9.0-alpha.4+9ab7a71
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server 
openshift v3.9.0-alpha.4+9ab7a71
kubernetes v1.9.1+a0ce1bc657

@debianmaster
Copy link

same here

@djzager djzager added 3.11 | release-1.3 Kubernetes 1.11 | Openshift 3.11 | Broker release-1.3 and removed 3.10 | release-1.2 Kubernetes 1.10 | Openshift 3.10 | Broker release-1.2 labels May 30, 2018
@shawn-hurley
Copy link
Contributor

I am closing this issue as I can not reproduce on 3.10 release. If it can be reproduced this can be reopened with a exact reproducer steps so that I can track down what is going wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shawn-hurley shawn-hurley

Labels
3.11 | release-1.3 Kubernetes 1.11 | Openshift 3.11 | Broker release-1.3
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@cmoulliard @djzager @gouyang @debianmaster @shawn-hurley @rthallisey