Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ElasticSearch fails to deploy with swagger_type error #6

Closed
leifmadsen opened this issue Apr 30, 2018 · 5 comments
Closed

ElasticSearch fails to deploy with swagger_type error #6

leifmadsen opened this issue Apr 30, 2018 · 5 comments

Comments

@leifmadsen
Copy link

Attempting to deploy the ElasticSearch APB on OpenShift Origin 3.9 and running into the following error:

TASK [provision-es-apb : Create ImageStream] ***********************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to patch object: Evaluating spec_tags: unable to get swagger_type for from in __compare_obj_list() for item {'from': {'kind': 'DockerImage', 'name': 'registry.centos.org/rhsyseng/elasticsearch:6.2.1'}, 'name': '6.2.1'} in model tag_reference"}

I've validated that the broker_config ConfigMap is setup to have openshift with sandbox_role: admin and rolled out a new DeploymentConfig for ASB. (I know it works because I had an issue with unable to create a RoleBinding, but after the change that works fine. :))

Here are the logs for the ASB pod, and then the logs from the ES pod during deployment.

ASB logs starting the deployment

$ oc logs po/asb-2-bll7p
Using config file mounted to /etc/ansible-service-broker/config.yaml
============================================================
==           Starting Ansible Service Broker...           ==
============================================================
2018/04/30 15:17:00 Unable to get log.logfile from config
[2018-04-30T15:17:00.418Z] [NOTICE] - Initializing clients...
[2018-04-30T15:17:00.419Z] [INFO] - == ETCD CX ==
[2018-04-30T15:17:00.419Z] [INFO] - EtcdHost: asb-etcd.openshift-ansible-service-broker.svc
[2018-04-30T15:17:00.419Z] [INFO] - EtcdPort: 2379
[2018-04-30T15:17:00.419Z] [INFO] - Endpoints: [https://asb-etcd.openshift-ansible-service-broker.svc:2379]
[2018-04-30T15:17:00.434Z] [INFO] - Etcd Version [Server: 3.3.4, Cluster: 3.3.0]
[2018-04-30T15:17:00.441Z] [INFO] - OpenShift version: v3.9.0+d0f9aed-12
[2018-04-30T15:17:00.452Z] [INFO] - Kubernetes version: v1.9.1+a0ce1bc657
[2018-04-30T15:17:00.454Z] [INFO] - == REGISTRY CX == 
[2018-04-30T15:17:00.454Z] [INFO] - Name: dh
[2018-04-30T15:17:00.454Z] [INFO] - Type: dockerhub
[2018-04-30T15:17:00.454Z] [INFO] - Url: 
[2018-04-30T15:17:00.454Z] [INFO] - == REGISTRY CX == 
[2018-04-30T15:17:00.454Z] [INFO] - Name: localregistry
[2018-04-30T15:17:00.454Z] [INFO] - Type: local_openshift
[2018-04-30T15:17:00.454Z] [INFO] - Url: 
[2018-04-30T15:17:00.454Z] [INFO] - Initiating Recovery Process
[2018-04-30T15:17:00.454Z] [INFO] - Listening for provision messages
[2018-04-30T15:17:00.454Z] [INFO] - Listening for deprovision messages
[2018-04-30T15:17:00.454Z] [INFO] - Listening for update messages
[2018-04-30T15:17:00.454Z] [INFO] - Listening for binding messages
[2018-04-30T15:17:00.454Z] [INFO] - Listening for binding messages
[2018-04-30T15:17:00.456Z] [INFO] - Recovery complete
[2018-04-30T15:17:00.456Z] [NOTICE] - recover called
[2018-04-30T15:17:00.456Z] [INFO] - Broker configured to bootstrap on startup
[2018-04-30T15:17:00.456Z] [INFO] - Attempting bootstrap...
[2018-04-30T15:17:00.456Z] [INFO] - AnsibleBroker::Bootstrap
[2018-04-30T15:17:06.73Z] [INFO] - No runtime label found. Set runtime=1. Will use 'exec' to gather bind credentials
[2018-04-30T15:17:08.413Z] [INFO] - Didn't find encoded Spec label. Assuming image is not APB and skiping
[2018-04-30T15:17:12.647Z] [INFO] - Didn't find encoded Spec label. Assuming image is not APB and skiping
[2018-04-30T15:17:14.255Z] [INFO] - No runtime label found. Set runtime=1. Will use 'exec' to gather bind credentials
[2018-04-30T15:17:14.255Z] [INFO] - Validating specs...
[2018-04-30T15:17:14.255Z] [NOTICE] - All specs passed validation!
[2018-04-30T15:17:14.37Z] [INFO] - Validating specs...
[2018-04-30T15:17:14.37Z] [NOTICE] - All specs passed validation!
[2018-04-30T15:17:15.796Z] [NOTICE] - Broker successfully bootstrapped on startup
[2018-04-30T15:17:16.054Z] [NOTICE] - Listening on https://[::]:1338
[2018-04-30T15:17:16.054Z] [NOTICE] - Ansible Service Broker Starting
10.128.0.1 - - [30/Apr/2018:15:18:25 +0000] "GET /ansible-service-broker/v2/service_instances/152ef77e-2e6f-4665-a1aa-54af2a0bc414/last_operation?operation=85e01e16-e926-4913-b8fd-5622e113379c&plan_id=11d395a0f8377ed0f81fb6148720a6fd&service_id=aff6d7bb9c7f57c9ce8b742228e4caa3 HTTP/1.1" 200 124
[2018-04-30T15:18:25.612Z] [INFO] - ASYNC provisioning in progress
[2018-04-30T15:18:25.613Z] [NOTICE] - ============================================================
[2018-04-30T15:18:25.613Z] [NOTICE] -                        PROVISIONING                         
[2018-04-30T15:18:25.613Z] [NOTICE] - ============================================================
[2018-04-30T15:18:25.613Z] [NOTICE] - Spec.ID: aff6d7bb9c7f57c9ce8b742228e4caa3
[2018-04-30T15:18:25.613Z] [NOTICE] - Spec.Name: dh-es-apb
[2018-04-30T15:18:25.613Z] [NOTICE] - Spec.Image: docker.io/ansibleplaybookbundle/es-apb:latest
[2018-04-30T15:18:25.613Z] [NOTICE] - Spec.Description: APB to deploy cluster-ready Elasticsearch
[2018-04-30T15:18:25.613Z] [NOTICE] - ============================================================
[2018-04-30T15:18:25.613Z] [INFO] - Checking if namespace sa-telemetry exists.
10.128.0.1 - - [30/Apr/2018:15:18:25 +0000] "PUT /ansible-service-broker/v2/service_instances/ba3605bd-d908-44bf-b5d4-3f75a95688f6?accepts_incomplete=true HTTP/1.1" 202 58
[2018-04-30T15:18:25.654Z] [NOTICE] - Creating RoleBinding apb-3bc430cb-62d3-4f74-b2ed-67f71d9a94cd
[2018-04-30T15:18:25.747Z] [INFO] - Provision requested for instance ba3605bd-d908-44bf-b5d4-3f75a95688f6, but job is already in progress
10.128.0.1 - - [30/Apr/2018:15:18:25 +0000] "PUT /ansible-service-broker/v2/service_instances/ba3605bd-d908-44bf-b5d4-3f75a95688f6?accepts_incomplete=true HTTP/1.1" 202 58
10.128.0.1 - - [30/Apr/2018:15:18:25 +0000] "GET /ansible-service-broker/v2/service_instances/ba3605bd-d908-44bf-b5d4-3f75a95688f6/last_operation?operation=e243b354-0a9c-4212-afd0-f1c8be168da9&plan_id=11d395a0f8377ed0f81fb6148720a6fd&service_id=aff6d7bb9c7f57c9ce8b742228e4caa3 HTTP/1.1" 200 29
[2018-04-30T15:18:25.903Z] [NOTICE] - Creating RoleBinding apb-3bc430cb-62d3-4f74-b2ed-67f71d9a94cd
[2018-04-30T15:18:25.936Z] [INFO] - Successfully created apb sandbox: [ apb-3bc430cb-62d3-4f74-b2ed-67f71d9a94cd ], with admin permissions in namespace dh-es-apb-prov-qphnl
[2018-04-30T15:18:25.936Z] [INFO] - Running post create sandbox fuctions if defined.
[2018-04-30T15:18:25.936Z] [INFO] - Proxy configuration present. Applying to APB before execution:
[2018-04-30T15:18:25.936Z] [INFO] - HTTP_PROXY=""
[2018-04-30T15:18:25.936Z] [INFO] - HTTPS_PROXY=""
[2018-04-30T15:18:25.936Z] [INFO] - NO_PROXY=""
[2018-04-30T15:18:25.936Z] [NOTICE] - Creating pod "apb-3bc430cb-62d3-4f74-b2ed-67f71d9a94cd" in the dh-es-apb-prov-qphnl namespace
[2018-04-30T15:18:26.031Z] [INFO] - Watch pod [ apb-3bc430cb-62d3-4f74-b2ed-67f71d9a94cd ] tick 1
10.128.0.1 - - [30/Apr/2018:15:18:26 +0000] "GET /ansible-service-broker/v2/service_instances/ba3605bd-d908-44bf-b5d4-3f75a95688f6/last_operation?operation=e243b354-0a9c-4212-afd0-f1c8be168da9&plan_id=11d395a0f8377ed0f81fb6148720a6fd&service_id=aff6d7bb9c7f57c9ce8b742228e4caa3 HTTP/1.1" 200 29
10.128.0.1 - - [30/Apr/2018:15:18:30 +0000] "GET /ansible-service-broker/v2/service_instances/ba3605bd-d908-44bf-b5d4-3f75a95688f6/last_operation?operation=e243b354-0a9c-4212-afd0-f1c8be168da9&plan_id=11d395a0f8377ed0f81fb6148720a6fd&service_id=aff6d7bb9c7f57c9ce8b742228e4caa3 HTTP/1.1" 200 29
[2018-04-30T15:18:31.033Z] [INFO] - Watch pod [ apb-3bc430cb-62d3-4f74-b2ed-67f71d9a94cd ] tick 2
[2018-04-30T15:18:36.037Z] [INFO] - Watch pod [ apb-3bc430cb-62d3-4f74-b2ed-67f71d9a94cd ] tick 3
[2018-04-30T15:18:36.057Z] [ERROR] - Provision or Update action failed - Pod [ apb-3bc430cb-62d3-4f74-b2ed-67f71d9a94cd ] failed with exit code [2]
[2018-04-30T15:18:36.057Z] [INFO] - Destroying APB sandbox...
[2018-04-30T15:18:36.063Z] [NOTICE] - Successfully deleted rolebinding apb-3bc430cb-62d3-4f74-b2ed-67f71d9a94cd, namespace dh-es-apb-prov-qphnl
[2018-04-30T15:18:36.069Z] [NOTICE] - Successfully deleted rolebinding apb-3bc430cb-62d3-4f74-b2ed-67f71d9a94cd, namespace sa-telemetry
[2018-04-30T15:18:36.075Z] [ERROR] - broker::Provision error occurred. Pod [ apb-3bc430cb-62d3-4f74-b2ed-67f71d9a94cd ] failed with exit code [2]
10.128.0.1 - - [30/Apr/2018:15:18:38 +0000] "GET /ansible-service-broker/v2/service_instances/ba3605bd-d908-44bf-b5d4-3f75a95688f6/last_operation?operation=e243b354-0a9c-4212-afd0-f1c8be168da9&plan_id=11d395a0f8377ed0f81fb6148720a6fd&service_id=aff6d7bb9c7f57c9ce8b742228e4caa3 HTTP/1.1" 200 122

ES deployment failure logs from pod

$ oc project dh-es-apb-prov-qphnl
Now using project "dh-es-apb-prov-qphnl" on server "https://openshift-master.home.61will.space:8443".
[centos@openshift-master ~]$ oc logs apb-3bc430cb-62d3-4f74-b2ed-67f71d9a94cd

PLAY [es-apb playbook to provision the application] ****************************

TASK [ansible.kubernetes-modules : Install latest openshift client] ************
skipping: [localhost]

TASK [ansibleplaybookbundle.asb-modules : debug] *******************************
skipping: [localhost]

TASK [provision-es-apb : [SET UP] tmp folder] **********************************
changed: [localhost]

TASK [provision-es-apb : Process ImageStream from template] ********************
changed: [localhost]

TASK [provision-es-apb : Create ImageStream] ***********************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to patch object: Evaluating spec_tags: unable to get swagger_type for from in __compare_obj_list() for item {'from': {'kind': 'DockerImage', 'name': 'registry.centos.org/rhsyseng/elasticsearch:6.2.1'}, 'name': '6.2.1'} in model tag_reference"}

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=2    unreachable=0    failed=1
@leifmadsen
Copy link
Author

So I think I identified an issue in this file: https://github.com/ansibleplaybookbundle/es-apb/blob/master/roles/provision-es-apb/templates/elasticsearch-is.yml.j2

If you look in the openshift_v1_image_stream code, then you see that spec_tags should be a list, and not a dictionary. Not sure if this is really a problem, but the tags setup under the spec of the ImageStream spec file seems to be the issue.

The error I'm seeing is the from line, since it seems the code can't process this setup. I haven't dug into the code to understand what it is trying to do, because I should already be in bed :)

@leifmadsen
Copy link
Author

Looks like it is trying to replace the spec_docker_image_repository usage as it specifies you should use the spec.tags.from setup. Maybe it's being done wrong? Not sure...

@leifmadsen
Copy link
Author

Well, I looked some more before bed, and I'm not sure the APB is really doing anything wrong, especially since I looked at the history, and this has pretty much been like this since it was created over 3 months ago.

I suspect something must have changed in the ansible-kubernetes-modules repository, perhaps in this module: https://github.com/ansible/ansible-kubernetes-modules/blob/master/library/openshift_v1_image_stream_tag.py

@abays
Copy link

abays commented May 1, 2018

Given the error:

unable to get swagger_type for from in __compare_obj_list() for item {'from': {'kind': 'DockerImage', 'name': 'registry.centos.org/rhsyseng/elasticsearch:6.2.1'}, 'name': '6.2.1'}

It seems to be complaining it can't find from in the swagger types for the loaded class. In this case, we believe the class is:

https://github.com/openshift/openshift-restclient-python/blob/master/openshift/client/models/v1_tag_reference.py

And so the swagger_types are defined at:

https://github.com/openshift/openshift-restclient-python/blob/2a798cd7f2a9d6d36c8d4a50f21f6f780ef860dd/openshift/client/models/v1_tag_reference.py#L33

As one can see, "from" is not a key in this swagger_type dictionary. However, "_from" is present in the dict. Is it possible that there is some sort of key translation that is missing, or that the template being used should be changed such that the item is defined as "_from" instead of "from"? Just thinking out loud.

@leifmadsen
Copy link
Author

Since I can't reproduce this now (I guess something new was published that fix this), I'm going to close it out for now. If I run into it again, I'll reopen the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@leifmadsen @abays